From 3dbb96d4e69fc5d65ffb0bddf951cba7bf96f862 Mon Sep 17 00:00:00 2001 From: "lei.chen" Date: Mon, 31 Aug 2026 14:25:13 +0000 Subject: [PATCH] merge-fix: drop Rongsong bookkeeping missed in merge (.kiro/ reports/ CHANGES.md) First git rm during merge was refused for worktree-modified files; only .wolf/ got removed. Clean up the remaining OpenWolf/spec bookkeeping and weekly reports so the merged branch carries source fixes only. --- .kiro/settings/rules/design-discovery-full.md | 93 -- .../settings/rules/design-discovery-light.md | 49 - .kiro/settings/rules/design-principles.md | 198 --- .kiro/settings/rules/design-review-gate.md | 50 - .kiro/settings/rules/design-review.md | 110 -- .kiro/settings/rules/design-synthesis.md | 29 - .kiro/settings/rules/ears-format.md | 49 - .kiro/settings/rules/gap-analysis.md | 144 -- .../rules/requirements-review-gate.md | 51 - .kiro/settings/rules/steering-principles.md | 90 -- .kiro/settings/rules/tasks-generation.md | 222 --- .../settings/rules/tasks-parallel-analysis.md | 41 - .kiro/settings/templates/specs/design.md | 331 ---- .kiro/settings/templates/specs/init.json | 24 - .../templates/specs/requirements-init.md | 9 - .../settings/templates/specs/requirements.md | 32 - .kiro/settings/templates/specs/research.md | 61 - .kiro/settings/templates/specs/tasks.md | 24 - .../steering-custom/api-standards.md | 69 - .../steering-custom/authentication.md | 67 - .../templates/steering-custom/database.md | 46 - .../templates/steering-custom/deployment.md | 54 - .../steering-custom/error-handling.md | 59 - .../templates/steering-custom/security.md | 55 - .../templates/steering-custom/testing.md | 47 - .kiro/settings/templates/steering/product.md | 18 - .../settings/templates/steering/structure.md | 41 - .kiro/settings/templates/steering/tech.md | 45 - .kiro/specs/.gitignore | 1 - .kiro/specs/admin-config-tool/brief.md | 613 -------- .kiro/specs/admin-config-tool/design.md | 547 ------- .kiro/specs/admin-config-tool/requirements.md | 185 --- .kiro/specs/admin-config-tool/research.md | 215 --- .kiro/specs/admin-config-tool/spec.json | 44 - .kiro/specs/admin-config-tool/tasks.md | 200 --- .kiro/specs/build-time-versioning/design.md | 241 --- .../build-time-versioning/requirements.md | 52 - .kiro/specs/build-time-versioning/research.md | 57 - .kiro/specs/build-time-versioning/spec.json | 22 - .kiro/specs/build-time-versioning/tasks.md | 132 -- .../clusterconfig-schema-split/design.md | 272 ---- .../requirements.md | 104 -- .../clusterconfig-schema-split/research.md | 153 -- .../clusterconfig-schema-split/spec.json | 22 - .../specs/clusterconfig-schema-split/tasks.md | 109 -- .../component-failover-adoption/design.md | 574 ------- .../requirements.md | 93 -- .../component-failover-adoption/research.md | 170 --- .../component-failover-adoption/spec.json | 38 - .../component-failover-adoption/tasks.md | 205 --- .kiro/specs/component-rejoin-resync/brief.md | 53 - .kiro/specs/component-rejoin-resync/spec.json | 24 - .kiro/specs/config-upgrade-snapshot/brief.md | 24 - .kiro/specs/config-upgrade-snapshot/design.md | 432 ------ .../config-upgrade-snapshot/requirements.md | 119 -- .../specs/config-upgrade-snapshot/research.md | 422 ------ .kiro/specs/config-upgrade-snapshot/spec.json | 31 - .kiro/specs/config-upgrade-snapshot/tasks.md | 197 --- .kiro/specs/controller-status/design.md | 313 ---- .kiro/specs/controller-status/requirements.md | 117 -- .kiro/specs/controller-status/research.md | 153 -- .kiro/specs/controller-status/spec.json | 42 - .kiro/specs/controller-status/tasks.md | 108 -- .../specs/dirty-content-fingerprint/design.md | 237 --- .../dirty-content-fingerprint/requirements.md | 56 - .../dirty-content-fingerprint/research.md | 79 - .../specs/dirty-content-fingerprint/spec.json | 22 - .../specs/dirty-content-fingerprint/tasks.md | 119 -- .kiro/specs/election-iomgr-refactor/design.md | 539 ------- .../election-iomgr-refactor/requirements.md | 206 --- .../specs/election-iomgr-refactor/research.md | 518 ------- .kiro/specs/election-iomgr-refactor/spec.json | 22 - .kiro/specs/election-iomgr-refactor/tasks.md | 189 --- .../eliminate-frontend-makefile/brief.md | 58 - .../eliminate-frontend-makefile/spec.json | 24 - .../specs/ha-commit-barrier-rollout/brief.md | 24 - .../specs/ha-commit-barrier-rollout/design.md | 457 ------ .../ha-commit-barrier-rollout/requirements.md | 92 -- .../ha-commit-barrier-rollout/research.md | 170 --- .../specs/ha-commit-barrier-rollout/spec.json | 24 - .../specs/ha-commit-barrier-rollout/tasks.md | 124 -- .../INVESTIGATION-failover-job-loss.md | 123 -- .kiro/specs/ha-commit-durability/brief.md | 73 - .kiro/specs/ha-commit-durability/design.md | 498 ------ .../ha-commit-durability/repro-851/ctrl-1.log | 164 -- .../ha-commit-durability/repro-851/ctrl-2.log | 99 -- .../ha-commit-durability/repro-851/ctrl-3.log | 213 --- .../repro-851/waldump-ctrl-1.txt | 57 - .../repro-851/waldump-ctrl-2.txt | 57 - .../repro-851/waldump-ctrl-3.txt | 57 - .../ha-commit-durability/requirements.md | 61 - .kiro/specs/ha-commit-durability/research.md | 220 --- .kiro/specs/ha-commit-durability/spec.json | 36 - .kiro/specs/ha-commit-durability/tasks.md | 176 --- .kiro/specs/ha-commit-engine/brief.md | 24 - .kiro/specs/ha-commit-engine/design.md | 334 ---- .kiro/specs/ha-commit-engine/requirements.md | 107 -- .kiro/specs/ha-commit-engine/research.md | 98 -- .kiro/specs/ha-commit-engine/spec.json | 24 - .kiro/specs/ha-commit-engine/tasks.md | 101 -- .kiro/specs/ha-config-bootstrap/brief.md | 1208 --------------- .kiro/specs/ha-config-bootstrap/design.md | 844 ----------- .../ha-config-bootstrap/election-trace.md | 358 ----- .../specs/ha-config-bootstrap/manual-tests.md | 388 ----- .../specs/ha-config-bootstrap/requirements.md | 151 -- .kiro/specs/ha-config-bootstrap/research.md | 794 ---------- .kiro/specs/ha-config-bootstrap/spec.json | 24 - .kiro/specs/ha-config-bootstrap/tasks.md | 293 ---- .../specs/ha-consistency-invariants/brief.md | 102 -- .../specs/ha-consistency-invariants/design.md | 425 ------ .../ha-consistency-invariants/requirements.md | 100 -- .../ha-consistency-invariants/research.md | 268 ---- .../specs/ha-consistency-invariants/spec.json | 24 - .kiro/specs/ha-convergence-recovery/brief.md | 24 - .kiro/specs/ha-convergence-recovery/design.md | 437 ------ .../ha-convergence-recovery/requirements.md | 125 -- .../specs/ha-convergence-recovery/research.md | 218 --- .kiro/specs/ha-convergence-recovery/spec.json | 24 - .kiro/specs/ha-convergence-recovery/tasks.md | 112 -- .kiro/specs/ha-elect-ctx-lifecycle/brief.md | 62 - .kiro/specs/ha-elect-ctx-lifecycle/design.md | 183 --- .../ha-elect-ctx-lifecycle/requirements.md | 55 - .../specs/ha-elect-ctx-lifecycle/research.md | 56 - .kiro/specs/ha-elect-ctx-lifecycle/spec.json | 32 - .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | 79 - .kiro/specs/ha-entity-snapshots/design.md | 559 ------- .../specs/ha-entity-snapshots/requirements.md | 64 - .kiro/specs/ha-entity-snapshots/research.md | 205 --- .kiro/specs/ha-entity-snapshots/spec.json | 33 - .kiro/specs/ha-entity-snapshots/tasks.md | 166 -- .../brief.md | 50 - .../design.md | 204 --- .../requirements.md | 55 - .../research.md | 64 - .../spec.json | 38 - .../tasks.md | 66 - .kiro/specs/ha-heartbeat-liveness/design.md | 382 ----- .../ha-heartbeat-liveness/manual-tests.md | 222 --- .../ha-heartbeat-liveness/requirements.md | 148 -- .kiro/specs/ha-heartbeat-liveness/research.md | 400 ----- .kiro/specs/ha-heartbeat-liveness/spec.json | 23 - .kiro/specs/ha-heartbeat-liveness/tasks.md | 103 -- .../specs/ha-peer-connection-reuse/design.md | 427 ------ .../ha-peer-connection-reuse/requirements.md | 75 - .../ha-peer-connection-reuse/research.md | 404 ----- .../specs/ha-peer-connection-reuse/spec.json | 23 - .kiro/specs/ha-peer-connection-reuse/tasks.md | 92 -- .kiro/specs/ha-prevote/design.md | 336 ---- .kiro/specs/ha-prevote/requirements.md | 211 --- .kiro/specs/ha-prevote/research.md | 366 ----- .kiro/specs/ha-prevote/spec.json | 38 - .kiro/specs/ha-prevote/tasks.md | 100 -- .kiro/specs/ha-repl-hardening/design.md | 468 ------ .../specs/ha-repl-hardening/raft-tla-audit.md | 91 -- .kiro/specs/ha-repl-hardening/requirements.md | 103 -- .kiro/specs/ha-repl-hardening/research.md | 588 ------- .kiro/specs/ha-repl-hardening/spec.json | 23 - .kiro/specs/ha-repl-hardening/tasks.md | 303 ---- .kiro/specs/ha-single-leader-role/brief.md | 25 - .kiro/specs/ha-single-leader-role/design.md | 338 ----- .../ha-single-leader-role/requirements.md | 56 - .kiro/specs/ha-single-leader-role/research.md | 193 --- .kiro/specs/ha-single-leader-role/spec.json | 24 - .kiro/specs/ha-single-leader-role/tasks.md | 95 -- .../specs/ha-term-safety/TLA-gap-analysis.md | 62 - .kiro/specs/ha-term-safety/design.md | 443 ------ .kiro/specs/ha-term-safety/requirements.md | 131 -- .kiro/specs/ha-term-safety/research.md | 122 -- .kiro/specs/ha-term-safety/spec.json | 34 - .kiro/specs/ha-term-safety/tasks.md | 229 --- .kiro/specs/ha-wal-replication/brief.md | 61 - .kiro/specs/ha-wal-replication/design.md | 114 -- .../specs/ha-wal-replication/requirements.md | 62 - .kiro/specs/ha-wal-replication/research.md | 106 -- .kiro/specs/ha-wal-replication/spec.json | 32 - .kiro/specs/ha-wal-replication/tasks.md | 128 -- .../job-array-single-wal-record/brief.md | 74 - .../job-array-single-wal-record/design.md | 486 ------ .../requirements.md | 125 -- .../job-array-single-wal-record/research.md | 172 --- .../job-array-single-wal-record/spec.json | 38 - .../job-array-single-wal-record/tasks.md | 245 --- .kiro/specs/libxce-failover/brief.md | 150 -- .kiro/specs/libxce-failover/design.md | 540 ------- .kiro/specs/libxce-failover/requirements.md | 179 --- .kiro/specs/libxce-failover/research.md | 248 --- .kiro/specs/libxce-failover/spec.json | 32 - .kiro/specs/libxce-failover/tasks.md | 118 -- .../specs/raft-adapter-conformance/design.md | 385 ----- .../raft-adapter-conformance/requirements.md | 120 -- .../raft-adapter-conformance/research.md | 231 --- .../specs/raft-adapter-conformance/spec.json | 34 - .kiro/specs/raft-adapter-conformance/tasks.md | 127 -- .kiro/specs/raft-apply-decoupling/brief.md | 86 -- .kiro/specs/raft-apply-decoupling/design.md | 449 ------ .../raft-apply-decoupling/facts-in-raft.md | 150 -- .../raft-apply-decoupling/requirements.md | 160 -- .kiro/specs/raft-apply-decoupling/research.md | 103 -- .kiro/specs/raft-apply-decoupling/spec.json | 22 - .kiro/specs/raft-apply-decoupling/tasks.md | 221 --- .kiro/specs/raft-commit-safety/brief.md | 31 - .kiro/specs/raft-commit-safety/design.md | 392 ----- .../specs/raft-commit-safety/requirements.md | 112 -- .kiro/specs/raft-commit-safety/research.md | 53 - .kiro/specs/raft-commit-safety/spec.json | 24 - .kiro/specs/raft-commit-safety/tasks.md | 165 -- .kiro/specs/raft-drive-async/brief.md | 92 -- .kiro/specs/raft-drive-async/design.md | 385 ----- .kiro/specs/raft-drive-async/requirements.md | 111 -- .kiro/specs/raft-drive-async/research.md | 100 -- .kiro/specs/raft-drive-async/spec.json | 32 - .kiro/specs/raft-drive-async/tasks.md | 163 -- .kiro/specs/raft-ha-core-safety/design.md | 392 ----- .../specs/raft-ha-core-safety/requirements.md | 102 -- .kiro/specs/raft-ha-core-safety/research.md | 990 ------------ .kiro/specs/raft-ha-core-safety/spec.json | 22 - .kiro/specs/raft-ha-core-safety/tasks.md | 227 --- .kiro/specs/raft-library/design.md | 662 -------- .kiro/specs/raft-library/requirements.md | 97 -- .kiro/specs/raft-library/research.md | 296 ---- .kiro/specs/raft-library/spec.json | 23 - .kiro/specs/raft-library/tasks.md | 185 --- .kiro/specs/raft-log-replication/brief.md | 41 - .kiro/specs/raft-log-replication/design.md | 414 ----- .../raft-log-replication/requirements.md | 143 -- .kiro/specs/raft-log-replication/research.md | 34 - .kiro/specs/raft-log-replication/spec.json | 24 - .kiro/specs/raft-log-replication/tasks.md | 205 --- .kiro/specs/raft-sim-conformance/brief.md | 26 - .kiro/specs/raft-sim-conformance/design.md | 594 -------- .../raft-sim-conformance/requirements.md | 163 -- .kiro/specs/raft-sim-conformance/research.md | 165 -- .kiro/specs/raft-sim-conformance/spec.json | 34 - .kiro/specs/raft-sim-conformance/tasks.md | 275 ---- .../requirements.md | 7 - .../resource-distribution-mapping/research.md | 30 - .../resource-distribution-mapping/spec.json | 22 - .../resource-value-verification/design.md | 360 ----- .../requirements.md | 73 - .../resource-value-verification/research.md | 102 -- .../resource-value-verification/spec.json | 22 - .../resource-value-verification/tasks.md | 138 -- .kiro/specs/resval-id-decoupling/design.md | 435 ------ .../resval-id-decoupling/requirements.md | 95 -- .kiro/specs/resval-id-decoupling/research.md | 303 ---- .kiro/specs/resval-id-decoupling/spec.json | 24 - .kiro/specs/resval-id-decoupling/tasks.md | 224 --- .kiro/specs/scheduler-allocation/brief.md | 210 --- .../scheduler-allocation/requirements.md | 619 -------- .kiro/specs/scheduler-allocation/spec.json | 27 - .../scheduler-config-version-tag/design.md | 283 ---- .../requirements.md | 55 - .../scheduler-config-version-tag/research.md | 73 - .../scheduler-config-version-tag/spec.json | 22 - .../scheduler-config-version-tag/tasks.md | 50 - .../scheduler-controller-msg-gap/design.md | 471 ------ .../requirements.md | 110 -- .../scheduler-controller-msg-gap/research.md | 304 ---- .../scheduler-controller-msg-gap/spec.json | 24 - .../scheduler-controller-msg-gap/tasks.md | 136 -- ....94ffc737-bfba-49ef-8dcb-f6e6a1bf0915.json | 36 - .kiro/specs/scheduler-core/brief.md | 168 -- .kiro/specs/scheduler-core/design.md | 794 ---------- .kiro/specs/scheduler-core/requirements.md | 1348 ----------------- .kiro/specs/scheduler-core/research.md | 98 -- .kiro/specs/scheduler-core/spec.json | 26 - .kiro/specs/scheduler-core/tasks.md | 431 ------ .../specs/scheduler-dispatch-commit/design.md | 344 ----- .../scheduler-dispatch-commit/requirements.md | 71 - .../scheduler-dispatch-commit/research.md | 76 - .../specs/scheduler-dispatch-commit/spec.json | 30 - .../specs/scheduler-dispatch-commit/tasks.md | 125 -- .kiro/specs/scheduler-framework/brief.md | 384 ----- .../specs/scheduler-framework/requirements.md | 886 ----------- .kiro/specs/scheduler-framework/spec.json | 25 - .kiro/specs/scheduler-preemption/brief.md | 183 --- .../scheduler-preemption/requirements.md | 111 -- .kiro/specs/scheduler-preemption/spec.json | 26 - .../scheduler-sim-decision-parity/brief.md | 44 - .../scheduler-sim-decision-parity/design.md | 273 ---- .../requirements.md | 50 - .../scheduler-sim-decision-parity/research.md | 62 - .../scheduler-sim-decision-parity/spec.json | 22 - .../scheduler-sim-decision-parity/tasks.md | 104 -- .../specs/scheduler-sim-diagnostics/brief.md | 58 - .../design.md | 212 --- .../requirements.md | 52 - .../research.md | 90 -- .../spec.json | 22 - .../tasks.md | 58 - .../scheduler-sim-repl-verbosity/design.md | 291 ---- .../requirements.md | 59 - .../scheduler-sim-repl-verbosity/research.md | 77 - .../scheduler-sim-repl-verbosity/spec.json | 22 - .../scheduler-sim-repl-verbosity/tasks.md | 69 - .../brief.md | 64 - .../design.md | 290 ---- .../requirements.md | 41 - .../research.md | 73 - .../spec.json | 23 - .../tasks.md | 95 -- .kiro/specs/worker-join-refactor/design.md | 450 ------ .../worker-join-refactor/requirements.md | 71 - .kiro/specs/worker-join-refactor/research.md | 346 ----- .kiro/specs/worker-join-refactor/spec.json | 22 - .kiro/specs/worker-join-refactor/tasks.md | 86 -- .kiro/specs/worker-rejoin-resync/brief.md | 53 - .kiro/specs/worker-rejoin-resync/spec.json | 24 - .kiro/specs/xce-test-cli-ux/requirements.md | 92 -- .kiro/specs/xce-test-cli-ux/spec.json | 22 - .kiro/specs/xce-test-e2e-primitives/design.md | 154 -- .../xce-test-e2e-primitives/requirements.md | 94 -- .../specs/xce-test-e2e-primitives/research.md | 21 - .kiro/specs/xce-test-e2e-primitives/spec.json | 27 - .kiro/specs/xce-test-e2e-primitives/tasks.md | 100 -- .kiro/specs/xce-test-rc-file/design.md | 204 --- .kiro/specs/xce-test-rc-file/requirements.md | 43 - .kiro/specs/xce-test-rc-file/research.md | 57 - .kiro/specs/xce-test-rc-file/spec.json | 22 - .kiro/specs/xce-test-rc-file/tasks.md | 64 - .kiro/steering/api-standards.md | 145 -- .kiro/steering/build-deployment.md | 163 -- .kiro/steering/config-design.md | 72 - .kiro/steering/error-handling.md | 100 -- .kiro/steering/ha-async-io.md | 105 -- .kiro/steering/persistence.md | 96 -- .kiro/steering/product.md | 70 - .kiro/steering/roadmap.md | 122 -- .../steering/scheduler-driver-equivalence.md | 36 - .kiro/steering/security.md | 71 - .kiro/steering/structure.md | 106 -- .kiro/steering/tech.md | 102 -- .kiro/steering/testing.md | 95 -- CHANGES.md | 374 ----- reports/aug03-aug09-2026.md | 86 -- reports/aug08-aug15-2026.md | 111 -- reports/aug16-aug22-2026.md | 103 -- reports/aug23-aug29-2026.md | 111 -- reports/jul06-jul11-2026.md | 191 --- reports/jul13-jul18-2026.md | 175 --- reports/jul19-jul26-2026.md | 225 --- reports/jul27-aug02-2026.md | 85 -- reports/jun29-jul04-2026.md | 48 - 343 files changed, 56290 deletions(-) delete mode 100644 .kiro/settings/rules/design-discovery-full.md delete mode 100644 .kiro/settings/rules/design-discovery-light.md delete mode 100644 .kiro/settings/rules/design-principles.md delete mode 100644 .kiro/settings/rules/design-review-gate.md delete mode 100644 .kiro/settings/rules/design-review.md delete mode 100644 .kiro/settings/rules/design-synthesis.md delete mode 100644 .kiro/settings/rules/ears-format.md delete mode 100644 .kiro/settings/rules/gap-analysis.md delete mode 100644 .kiro/settings/rules/requirements-review-gate.md delete mode 100644 .kiro/settings/rules/steering-principles.md delete mode 100644 .kiro/settings/rules/tasks-generation.md delete mode 100644 .kiro/settings/rules/tasks-parallel-analysis.md delete mode 100644 .kiro/settings/templates/specs/design.md delete mode 100644 .kiro/settings/templates/specs/init.json delete mode 100644 .kiro/settings/templates/specs/requirements-init.md delete mode 100644 .kiro/settings/templates/specs/requirements.md delete mode 100644 .kiro/settings/templates/specs/research.md delete mode 100644 .kiro/settings/templates/specs/tasks.md delete mode 100644 .kiro/settings/templates/steering-custom/api-standards.md delete mode 100644 .kiro/settings/templates/steering-custom/authentication.md delete mode 100644 .kiro/settings/templates/steering-custom/database.md delete mode 100644 .kiro/settings/templates/steering-custom/deployment.md delete mode 100644 .kiro/settings/templates/steering-custom/error-handling.md delete mode 100644 .kiro/settings/templates/steering-custom/security.md delete mode 100644 .kiro/settings/templates/steering-custom/testing.md delete mode 100644 .kiro/settings/templates/steering/product.md delete mode 100644 .kiro/settings/templates/steering/structure.md delete mode 100644 .kiro/settings/templates/steering/tech.md delete mode 100644 .kiro/specs/.gitignore delete mode 100644 .kiro/specs/admin-config-tool/brief.md delete mode 100644 .kiro/specs/admin-config-tool/design.md delete mode 100644 .kiro/specs/admin-config-tool/requirements.md delete mode 100644 .kiro/specs/admin-config-tool/research.md delete mode 100644 .kiro/specs/admin-config-tool/spec.json delete mode 100644 .kiro/specs/admin-config-tool/tasks.md delete mode 100644 .kiro/specs/build-time-versioning/design.md delete mode 100644 .kiro/specs/build-time-versioning/requirements.md delete mode 100644 .kiro/specs/build-time-versioning/research.md delete mode 100644 .kiro/specs/build-time-versioning/spec.json delete mode 100644 .kiro/specs/build-time-versioning/tasks.md delete mode 100644 .kiro/specs/clusterconfig-schema-split/design.md delete mode 100644 .kiro/specs/clusterconfig-schema-split/requirements.md delete mode 100644 .kiro/specs/clusterconfig-schema-split/research.md delete mode 100644 .kiro/specs/clusterconfig-schema-split/spec.json delete mode 100644 .kiro/specs/clusterconfig-schema-split/tasks.md delete mode 100644 .kiro/specs/component-failover-adoption/design.md delete mode 100644 .kiro/specs/component-failover-adoption/requirements.md delete mode 100644 .kiro/specs/component-failover-adoption/research.md delete mode 100644 .kiro/specs/component-failover-adoption/spec.json delete mode 100644 .kiro/specs/component-failover-adoption/tasks.md delete mode 100644 .kiro/specs/component-rejoin-resync/brief.md delete mode 100644 .kiro/specs/component-rejoin-resync/spec.json delete mode 100644 .kiro/specs/config-upgrade-snapshot/brief.md delete mode 100644 .kiro/specs/config-upgrade-snapshot/design.md delete mode 100644 .kiro/specs/config-upgrade-snapshot/requirements.md delete mode 100644 .kiro/specs/config-upgrade-snapshot/research.md delete mode 100644 .kiro/specs/config-upgrade-snapshot/spec.json delete mode 100644 .kiro/specs/config-upgrade-snapshot/tasks.md delete mode 100644 .kiro/specs/controller-status/design.md delete mode 100644 .kiro/specs/controller-status/requirements.md delete mode 100644 .kiro/specs/controller-status/research.md delete mode 100644 .kiro/specs/controller-status/spec.json delete mode 100644 .kiro/specs/controller-status/tasks.md delete mode 100644 .kiro/specs/dirty-content-fingerprint/design.md delete mode 100644 .kiro/specs/dirty-content-fingerprint/requirements.md delete mode 100644 .kiro/specs/dirty-content-fingerprint/research.md delete mode 100644 .kiro/specs/dirty-content-fingerprint/spec.json delete mode 100644 .kiro/specs/dirty-content-fingerprint/tasks.md delete mode 100644 .kiro/specs/election-iomgr-refactor/design.md delete mode 100644 .kiro/specs/election-iomgr-refactor/requirements.md delete mode 100644 .kiro/specs/election-iomgr-refactor/research.md delete mode 100644 .kiro/specs/election-iomgr-refactor/spec.json delete mode 100644 .kiro/specs/election-iomgr-refactor/tasks.md delete mode 100644 .kiro/specs/eliminate-frontend-makefile/brief.md delete mode 100644 .kiro/specs/eliminate-frontend-makefile/spec.json delete mode 100644 .kiro/specs/ha-commit-barrier-rollout/brief.md delete mode 100644 .kiro/specs/ha-commit-barrier-rollout/design.md delete mode 100644 .kiro/specs/ha-commit-barrier-rollout/requirements.md delete mode 100644 .kiro/specs/ha-commit-barrier-rollout/research.md delete mode 100644 .kiro/specs/ha-commit-barrier-rollout/spec.json delete mode 100644 .kiro/specs/ha-commit-barrier-rollout/tasks.md delete mode 100644 .kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md delete mode 100644 .kiro/specs/ha-commit-durability/brief.md delete mode 100644 .kiro/specs/ha-commit-durability/design.md delete mode 100644 .kiro/specs/ha-commit-durability/repro-851/ctrl-1.log delete mode 100644 .kiro/specs/ha-commit-durability/repro-851/ctrl-2.log delete mode 100644 .kiro/specs/ha-commit-durability/repro-851/ctrl-3.log delete mode 100644 .kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-1.txt delete mode 100644 .kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-2.txt delete mode 100644 .kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-3.txt delete mode 100644 .kiro/specs/ha-commit-durability/requirements.md delete mode 100644 .kiro/specs/ha-commit-durability/research.md delete mode 100644 .kiro/specs/ha-commit-durability/spec.json delete mode 100644 .kiro/specs/ha-commit-durability/tasks.md delete mode 100644 .kiro/specs/ha-commit-engine/brief.md delete mode 100644 .kiro/specs/ha-commit-engine/design.md delete mode 100644 .kiro/specs/ha-commit-engine/requirements.md delete mode 100644 .kiro/specs/ha-commit-engine/research.md delete mode 100644 .kiro/specs/ha-commit-engine/spec.json delete mode 100644 .kiro/specs/ha-commit-engine/tasks.md delete mode 100644 .kiro/specs/ha-config-bootstrap/brief.md delete mode 100644 .kiro/specs/ha-config-bootstrap/design.md delete mode 100644 .kiro/specs/ha-config-bootstrap/election-trace.md delete mode 100644 .kiro/specs/ha-config-bootstrap/manual-tests.md delete mode 100644 .kiro/specs/ha-config-bootstrap/requirements.md delete mode 100644 .kiro/specs/ha-config-bootstrap/research.md delete mode 100644 .kiro/specs/ha-config-bootstrap/spec.json delete mode 100644 .kiro/specs/ha-config-bootstrap/tasks.md delete mode 100644 .kiro/specs/ha-consistency-invariants/brief.md delete mode 100644 .kiro/specs/ha-consistency-invariants/design.md delete mode 100644 .kiro/specs/ha-consistency-invariants/requirements.md delete mode 100644 .kiro/specs/ha-consistency-invariants/research.md delete mode 100644 .kiro/specs/ha-consistency-invariants/spec.json delete mode 100644 .kiro/specs/ha-convergence-recovery/brief.md delete mode 100644 .kiro/specs/ha-convergence-recovery/design.md delete mode 100644 .kiro/specs/ha-convergence-recovery/requirements.md delete mode 100644 .kiro/specs/ha-convergence-recovery/research.md delete mode 100644 .kiro/specs/ha-convergence-recovery/spec.json delete mode 100644 .kiro/specs/ha-convergence-recovery/tasks.md delete mode 100644 .kiro/specs/ha-elect-ctx-lifecycle/brief.md delete mode 100644 .kiro/specs/ha-elect-ctx-lifecycle/design.md delete mode 100644 .kiro/specs/ha-elect-ctx-lifecycle/requirements.md delete mode 100644 .kiro/specs/ha-elect-ctx-lifecycle/research.md delete mode 100644 .kiro/specs/ha-elect-ctx-lifecycle/spec.json delete mode 100644 .kiro/specs/ha-elect-ctx-lifecycle/tasks.md delete mode 100644 .kiro/specs/ha-entity-snapshots/design.md delete mode 100644 .kiro/specs/ha-entity-snapshots/requirements.md delete mode 100644 .kiro/specs/ha-entity-snapshots/research.md delete mode 100644 .kiro/specs/ha-entity-snapshots/spec.json delete mode 100644 .kiro/specs/ha-entity-snapshots/tasks.md delete mode 100644 .kiro/specs/ha-heartbeat-committed-propagation/brief.md delete mode 100644 .kiro/specs/ha-heartbeat-committed-propagation/design.md delete mode 100644 .kiro/specs/ha-heartbeat-committed-propagation/requirements.md delete mode 100644 .kiro/specs/ha-heartbeat-committed-propagation/research.md delete mode 100644 .kiro/specs/ha-heartbeat-committed-propagation/spec.json delete mode 100644 .kiro/specs/ha-heartbeat-committed-propagation/tasks.md delete mode 100644 .kiro/specs/ha-heartbeat-liveness/design.md delete mode 100644 .kiro/specs/ha-heartbeat-liveness/manual-tests.md delete mode 100644 .kiro/specs/ha-heartbeat-liveness/requirements.md delete mode 100644 .kiro/specs/ha-heartbeat-liveness/research.md delete mode 100644 .kiro/specs/ha-heartbeat-liveness/spec.json delete mode 100644 .kiro/specs/ha-heartbeat-liveness/tasks.md delete mode 100644 .kiro/specs/ha-peer-connection-reuse/design.md delete mode 100644 .kiro/specs/ha-peer-connection-reuse/requirements.md delete mode 100644 .kiro/specs/ha-peer-connection-reuse/research.md delete mode 100644 .kiro/specs/ha-peer-connection-reuse/spec.json delete mode 100644 .kiro/specs/ha-peer-connection-reuse/tasks.md delete mode 100644 .kiro/specs/ha-prevote/design.md delete mode 100644 .kiro/specs/ha-prevote/requirements.md delete mode 100644 .kiro/specs/ha-prevote/research.md delete mode 100644 .kiro/specs/ha-prevote/spec.json delete mode 100644 .kiro/specs/ha-prevote/tasks.md delete mode 100644 .kiro/specs/ha-repl-hardening/design.md delete mode 100644 .kiro/specs/ha-repl-hardening/raft-tla-audit.md delete mode 100644 .kiro/specs/ha-repl-hardening/requirements.md delete mode 100644 .kiro/specs/ha-repl-hardening/research.md delete mode 100644 .kiro/specs/ha-repl-hardening/spec.json delete mode 100644 .kiro/specs/ha-repl-hardening/tasks.md delete mode 100644 .kiro/specs/ha-single-leader-role/brief.md delete mode 100644 .kiro/specs/ha-single-leader-role/design.md delete mode 100644 .kiro/specs/ha-single-leader-role/requirements.md delete mode 100644 .kiro/specs/ha-single-leader-role/research.md delete mode 100644 .kiro/specs/ha-single-leader-role/spec.json delete mode 100644 .kiro/specs/ha-single-leader-role/tasks.md delete mode 100644 .kiro/specs/ha-term-safety/TLA-gap-analysis.md delete mode 100644 .kiro/specs/ha-term-safety/design.md delete mode 100644 .kiro/specs/ha-term-safety/requirements.md delete mode 100644 .kiro/specs/ha-term-safety/research.md delete mode 100644 .kiro/specs/ha-term-safety/spec.json delete mode 100644 .kiro/specs/ha-term-safety/tasks.md delete mode 100644 .kiro/specs/ha-wal-replication/brief.md delete mode 100644 .kiro/specs/ha-wal-replication/design.md delete mode 100644 .kiro/specs/ha-wal-replication/requirements.md delete mode 100644 .kiro/specs/ha-wal-replication/research.md delete mode 100644 .kiro/specs/ha-wal-replication/spec.json delete mode 100644 .kiro/specs/ha-wal-replication/tasks.md delete mode 100644 .kiro/specs/job-array-single-wal-record/brief.md delete mode 100644 .kiro/specs/job-array-single-wal-record/design.md delete mode 100644 .kiro/specs/job-array-single-wal-record/requirements.md delete mode 100644 .kiro/specs/job-array-single-wal-record/research.md delete mode 100644 .kiro/specs/job-array-single-wal-record/spec.json delete mode 100644 .kiro/specs/job-array-single-wal-record/tasks.md delete mode 100644 .kiro/specs/libxce-failover/brief.md delete mode 100644 .kiro/specs/libxce-failover/design.md delete mode 100644 .kiro/specs/libxce-failover/requirements.md delete mode 100644 .kiro/specs/libxce-failover/research.md delete mode 100644 .kiro/specs/libxce-failover/spec.json delete mode 100644 .kiro/specs/libxce-failover/tasks.md delete mode 100644 .kiro/specs/raft-adapter-conformance/design.md delete mode 100644 .kiro/specs/raft-adapter-conformance/requirements.md delete mode 100644 .kiro/specs/raft-adapter-conformance/research.md delete mode 100644 .kiro/specs/raft-adapter-conformance/spec.json delete mode 100644 .kiro/specs/raft-adapter-conformance/tasks.md delete mode 100644 .kiro/specs/raft-apply-decoupling/brief.md delete mode 100644 .kiro/specs/raft-apply-decoupling/design.md delete mode 100644 .kiro/specs/raft-apply-decoupling/facts-in-raft.md delete mode 100644 .kiro/specs/raft-apply-decoupling/requirements.md delete mode 100644 .kiro/specs/raft-apply-decoupling/research.md delete mode 100644 .kiro/specs/raft-apply-decoupling/spec.json delete mode 100644 .kiro/specs/raft-apply-decoupling/tasks.md delete mode 100644 .kiro/specs/raft-commit-safety/brief.md delete mode 100644 .kiro/specs/raft-commit-safety/design.md delete mode 100644 .kiro/specs/raft-commit-safety/requirements.md delete mode 100644 .kiro/specs/raft-commit-safety/research.md delete mode 100644 .kiro/specs/raft-commit-safety/spec.json delete mode 100644 .kiro/specs/raft-commit-safety/tasks.md delete mode 100644 .kiro/specs/raft-drive-async/brief.md delete mode 100644 .kiro/specs/raft-drive-async/design.md delete mode 100644 .kiro/specs/raft-drive-async/requirements.md delete mode 100644 .kiro/specs/raft-drive-async/research.md delete mode 100644 .kiro/specs/raft-drive-async/spec.json delete mode 100644 .kiro/specs/raft-drive-async/tasks.md delete mode 100644 .kiro/specs/raft-ha-core-safety/design.md delete mode 100644 .kiro/specs/raft-ha-core-safety/requirements.md delete mode 100644 .kiro/specs/raft-ha-core-safety/research.md delete mode 100644 .kiro/specs/raft-ha-core-safety/spec.json delete mode 100644 .kiro/specs/raft-ha-core-safety/tasks.md delete mode 100644 .kiro/specs/raft-library/design.md delete mode 100644 .kiro/specs/raft-library/requirements.md delete mode 100644 .kiro/specs/raft-library/research.md delete mode 100644 .kiro/specs/raft-library/spec.json delete mode 100644 .kiro/specs/raft-library/tasks.md delete mode 100644 .kiro/specs/raft-log-replication/brief.md delete mode 100644 .kiro/specs/raft-log-replication/design.md delete mode 100644 .kiro/specs/raft-log-replication/requirements.md delete mode 100644 .kiro/specs/raft-log-replication/research.md delete mode 100644 .kiro/specs/raft-log-replication/spec.json delete mode 100644 .kiro/specs/raft-log-replication/tasks.md delete mode 100644 .kiro/specs/raft-sim-conformance/brief.md delete mode 100644 .kiro/specs/raft-sim-conformance/design.md delete mode 100644 .kiro/specs/raft-sim-conformance/requirements.md delete mode 100644 .kiro/specs/raft-sim-conformance/research.md delete mode 100644 .kiro/specs/raft-sim-conformance/spec.json delete mode 100644 .kiro/specs/raft-sim-conformance/tasks.md delete mode 100644 .kiro/specs/resource-distribution-mapping/requirements.md delete mode 100644 .kiro/specs/resource-distribution-mapping/research.md delete mode 100644 .kiro/specs/resource-distribution-mapping/spec.json delete mode 100644 .kiro/specs/resource-value-verification/design.md delete mode 100644 .kiro/specs/resource-value-verification/requirements.md delete mode 100644 .kiro/specs/resource-value-verification/research.md delete mode 100644 .kiro/specs/resource-value-verification/spec.json delete mode 100644 .kiro/specs/resource-value-verification/tasks.md delete mode 100644 .kiro/specs/resval-id-decoupling/design.md delete mode 100644 .kiro/specs/resval-id-decoupling/requirements.md delete mode 100644 .kiro/specs/resval-id-decoupling/research.md delete mode 100644 .kiro/specs/resval-id-decoupling/spec.json delete mode 100644 .kiro/specs/resval-id-decoupling/tasks.md delete mode 100644 .kiro/specs/scheduler-allocation/brief.md delete mode 100644 .kiro/specs/scheduler-allocation/requirements.md delete mode 100644 .kiro/specs/scheduler-allocation/spec.json delete mode 100644 .kiro/specs/scheduler-config-version-tag/design.md delete mode 100644 .kiro/specs/scheduler-config-version-tag/requirements.md delete mode 100644 .kiro/specs/scheduler-config-version-tag/research.md delete mode 100644 .kiro/specs/scheduler-config-version-tag/spec.json delete mode 100644 .kiro/specs/scheduler-config-version-tag/tasks.md delete mode 100644 .kiro/specs/scheduler-controller-msg-gap/design.md delete mode 100644 .kiro/specs/scheduler-controller-msg-gap/requirements.md delete mode 100644 .kiro/specs/scheduler-controller-msg-gap/research.md delete mode 100644 .kiro/specs/scheduler-controller-msg-gap/spec.json delete mode 100644 .kiro/specs/scheduler-controller-msg-gap/tasks.md delete mode 100644 .kiro/specs/scheduler-core/.solomd/session.94ffc737-bfba-49ef-8dcb-f6e6a1bf0915.json delete mode 100644 .kiro/specs/scheduler-core/brief.md delete mode 100644 .kiro/specs/scheduler-core/design.md delete mode 100644 .kiro/specs/scheduler-core/requirements.md delete mode 100644 .kiro/specs/scheduler-core/research.md delete mode 100644 .kiro/specs/scheduler-core/spec.json delete mode 100644 .kiro/specs/scheduler-core/tasks.md delete mode 100644 .kiro/specs/scheduler-dispatch-commit/design.md delete mode 100644 .kiro/specs/scheduler-dispatch-commit/requirements.md delete mode 100644 .kiro/specs/scheduler-dispatch-commit/research.md delete mode 100644 .kiro/specs/scheduler-dispatch-commit/spec.json delete mode 100644 .kiro/specs/scheduler-dispatch-commit/tasks.md delete mode 100644 .kiro/specs/scheduler-framework/brief.md delete mode 100644 .kiro/specs/scheduler-framework/requirements.md delete mode 100644 .kiro/specs/scheduler-framework/spec.json delete mode 100644 .kiro/specs/scheduler-preemption/brief.md delete mode 100644 .kiro/specs/scheduler-preemption/requirements.md delete mode 100644 .kiro/specs/scheduler-preemption/spec.json delete mode 100644 .kiro/specs/scheduler-sim-decision-parity/brief.md delete mode 100644 .kiro/specs/scheduler-sim-decision-parity/design.md delete mode 100644 .kiro/specs/scheduler-sim-decision-parity/requirements.md delete mode 100644 .kiro/specs/scheduler-sim-decision-parity/research.md delete mode 100644 .kiro/specs/scheduler-sim-decision-parity/spec.json delete mode 100644 .kiro/specs/scheduler-sim-decision-parity/tasks.md delete mode 100644 .kiro/specs/scheduler-sim-diagnostics/brief.md delete mode 100644 .kiro/specs/scheduler-sim-eager-reconciliation/design.md delete mode 100644 .kiro/specs/scheduler-sim-eager-reconciliation/requirements.md delete mode 100644 .kiro/specs/scheduler-sim-eager-reconciliation/research.md delete mode 100644 .kiro/specs/scheduler-sim-eager-reconciliation/spec.json delete mode 100644 .kiro/specs/scheduler-sim-eager-reconciliation/tasks.md delete mode 100644 .kiro/specs/scheduler-sim-repl-verbosity/design.md delete mode 100644 .kiro/specs/scheduler-sim-repl-verbosity/requirements.md delete mode 100644 .kiro/specs/scheduler-sim-repl-verbosity/research.md delete mode 100644 .kiro/specs/scheduler-sim-repl-verbosity/spec.json delete mode 100644 .kiro/specs/scheduler-sim-repl-verbosity/tasks.md delete mode 100644 .kiro/specs/scheduler-snapshot-job-allocation/brief.md delete mode 100644 .kiro/specs/scheduler-snapshot-job-allocation/design.md delete mode 100644 .kiro/specs/scheduler-snapshot-job-allocation/requirements.md delete mode 100644 .kiro/specs/scheduler-snapshot-job-allocation/research.md delete mode 100644 .kiro/specs/scheduler-snapshot-job-allocation/spec.json delete mode 100644 .kiro/specs/scheduler-snapshot-job-allocation/tasks.md delete mode 100644 .kiro/specs/worker-join-refactor/design.md delete mode 100644 .kiro/specs/worker-join-refactor/requirements.md delete mode 100644 .kiro/specs/worker-join-refactor/research.md delete mode 100644 .kiro/specs/worker-join-refactor/spec.json delete mode 100644 .kiro/specs/worker-join-refactor/tasks.md delete mode 100644 .kiro/specs/worker-rejoin-resync/brief.md delete mode 100644 .kiro/specs/worker-rejoin-resync/spec.json delete mode 100644 .kiro/specs/xce-test-cli-ux/requirements.md delete mode 100644 .kiro/specs/xce-test-cli-ux/spec.json delete mode 100644 .kiro/specs/xce-test-e2e-primitives/design.md delete mode 100644 .kiro/specs/xce-test-e2e-primitives/requirements.md delete mode 100644 .kiro/specs/xce-test-e2e-primitives/research.md delete mode 100644 .kiro/specs/xce-test-e2e-primitives/spec.json delete mode 100644 .kiro/specs/xce-test-e2e-primitives/tasks.md delete mode 100644 .kiro/specs/xce-test-rc-file/design.md delete mode 100644 .kiro/specs/xce-test-rc-file/requirements.md delete mode 100644 .kiro/specs/xce-test-rc-file/research.md delete mode 100644 .kiro/specs/xce-test-rc-file/spec.json delete mode 100644 .kiro/specs/xce-test-rc-file/tasks.md delete mode 100644 .kiro/steering/api-standards.md delete mode 100644 .kiro/steering/build-deployment.md delete mode 100644 .kiro/steering/config-design.md delete mode 100644 .kiro/steering/error-handling.md delete mode 100644 .kiro/steering/ha-async-io.md delete mode 100644 .kiro/steering/persistence.md delete mode 100644 .kiro/steering/product.md delete mode 100644 .kiro/steering/roadmap.md delete mode 100644 .kiro/steering/scheduler-driver-equivalence.md delete mode 100644 .kiro/steering/security.md delete mode 100644 .kiro/steering/structure.md delete mode 100644 .kiro/steering/tech.md delete mode 100644 .kiro/steering/testing.md delete mode 100644 CHANGES.md delete mode 100644 reports/aug03-aug09-2026.md delete mode 100644 reports/aug08-aug15-2026.md delete mode 100644 reports/aug16-aug22-2026.md delete mode 100644 reports/aug23-aug29-2026.md delete mode 100644 reports/jul06-jul11-2026.md delete mode 100644 reports/jul13-jul18-2026.md delete mode 100644 reports/jul19-jul26-2026.md delete mode 100644 reports/jul27-aug02-2026.md delete mode 100644 reports/jun29-jul04-2026.md diff --git a/.kiro/settings/rules/design-discovery-full.md b/.kiro/settings/rules/design-discovery-full.md deleted file mode 100644 index b80c2f26..00000000 --- a/.kiro/settings/rules/design-discovery-full.md +++ /dev/null @@ -1,93 +0,0 @@ -# Full Discovery Process for Technical Design - -## Objective -Conduct comprehensive research and analysis to ensure the technical design is based on complete, accurate, and up-to-date information. - -## Discovery Steps - -### 1. Requirements Analysis -**Map Requirements to Technical Needs** -- Extract all functional requirements from EARS format -- Identify non-functional requirements (performance, security, scalability) -- Determine technical constraints and dependencies -- List core technical challenges - -### 2. Existing Implementation Analysis -**Understand Current System** (if modifying/extending): -- Analyze codebase structure and architecture patterns -- Map reusable components, services, utilities -- Identify domain boundaries and data flows -- Document integration points and dependencies -- Determine approach: extend vs refactor vs wrap - -### 3. Technology Research -**Investigate Best Practices and Solutions**: -- **Use WebSearch** to find: - - Latest architectural patterns for similar problems - - Industry best practices for the technology stack - - Recent updates or changes in relevant technologies - - Common pitfalls and solutions - -- **Use WebFetch** to analyze: - - Official documentation for frameworks/libraries - - API references and usage examples - - Migration guides and breaking changes - - Performance benchmarks and comparisons - -### 4. External Dependencies Investigation -**For Each External Service/Library**: -- Search for official documentation and GitHub repositories -- Verify API signatures and authentication methods -- Check version compatibility with existing stack -- Investigate rate limits and usage constraints -- Find community resources and known issues -- Document security considerations -- Note any gaps requiring implementation investigation - -### 5. Architecture Pattern & Boundary Analysis -**Evaluate Architectural Options**: -- Compare relevant patterns (MVC, Clean, Hexagonal, Event-driven) -- Assess fit with existing architecture and steering principles -- Identify domain boundaries and ownership seams required to avoid team conflicts -- Consider scalability implications and operational concerns -- Evaluate maintainability and team expertise -- Document preferred pattern and rejected alternatives in `research.md` - -### 6. Risk Assessment -**Identify Technical Risks**: -- Performance bottlenecks and scaling limits -- Security vulnerabilities and attack vectors -- Integration complexity and coupling -- Technical debt creation vs resolution -- Knowledge gaps and training needs - -## Research Guidelines - -### When to Search -**Always search for**: -- External API documentation and updates -- Security best practices for authentication/authorization -- Performance optimization techniques for identified bottlenecks -- Latest versions and migration paths for dependencies - -**Search if uncertain about**: -- Architectural patterns for specific use cases -- Industry standards for data formats/protocols -- Compliance requirements (GDPR, HIPAA, etc.) -- Scalability approaches for expected load - -### Search Strategy -1. Start with official sources (documentation, GitHub) -2. Check recent blog posts and articles (last 6 months) -3. Review Stack Overflow for common issues -4. Investigate similar open-source implementations - -## Output Requirements -Capture all findings that impact design decisions in `research.md` using the shared template: -- Key insights affecting architecture, technology alignment, and contracts -- Constraints discovered during research -- Recommended approaches and selected architecture pattern with rationale -- Rejected alternatives and trade-offs (documented in the Design Decisions section) -- Updated domain boundaries that inform Components & Interface Contracts -- Risks and mitigation strategies -- Gaps requiring further investigation during implementation \ No newline at end of file diff --git a/.kiro/settings/rules/design-discovery-light.md b/.kiro/settings/rules/design-discovery-light.md deleted file mode 100644 index 7a00ae29..00000000 --- a/.kiro/settings/rules/design-discovery-light.md +++ /dev/null @@ -1,49 +0,0 @@ -# Light Discovery Process for Extensions - -## Objective -Quickly analyze existing system and integration requirements for feature extensions. - -## Focused Discovery Steps - -### 1. Extension Point Analysis -**Identify Integration Approach**: -- Locate existing extension points or interfaces -- Determine modification scope (files, components) -- Check for existing patterns to follow -- Identify backward compatibility requirements - -### 2. Dependency Check -**Verify Compatibility**: -- Check version compatibility of new dependencies -- Validate API contracts haven't changed -- Ensure no breaking changes in pipeline - -### 3. Quick Technology Verification -**For New Libraries Only**: -- Use WebSearch for official documentation -- Verify basic usage patterns -- Check for known compatibility issues -- Confirm licensing compatibility -- Record key findings in `research.md` (technology alignment section) - -### 4. Integration Risk Assessment -**Quick Risk Check**: -- Impact on existing functionality -- Performance implications -- Security considerations -- Testing requirements - -## When to Escalate to Full Discovery -Switch to full discovery if you find: -- Significant architectural changes needed -- Complex external service integrations -- Security-sensitive implementations -- Performance-critical components -- Unknown or poorly documented dependencies - -## Output Requirements -- Clear integration approach (note boundary impacts in `research.md`) -- List of files/components to modify -- New dependencies with versions -- Integration risks and mitigations -- Testing focus areas \ No newline at end of file diff --git a/.kiro/settings/rules/design-principles.md b/.kiro/settings/rules/design-principles.md deleted file mode 100644 index cb12e95f..00000000 --- a/.kiro/settings/rules/design-principles.md +++ /dev/null @@ -1,198 +0,0 @@ -# Technical Design Rules and Principles - -## Core Design Principles - -### 0. Boundary First -- **Boundary is mandatory; owner is optional** -- A design is not ready when it explains components but leaves responsibility seams ambiguous -- Define what the spec owns before elaborating how it works -- Explicitly record what is out of boundary -- Do not leak downstream-specific behavior or assumptions into upstream boundaries - -### 1. Type Safety is Mandatory -- **NEVER** use `any` type in TypeScript interfaces -- Define explicit types for all parameters and returns -- Use discriminated unions for error handling -- Specify generic constraints clearly - -### 2. Design vs Implementation -- **Focus on WHAT, not HOW** -- Define interfaces and contracts, not code -- Specify behavior through pre/post conditions -- Document architectural decisions, not algorithms - -### 3. Visual Communication -- **Simple features**: Basic component diagram or none -- **Medium complexity**: Architecture + data flow -- **High complexity**: Multiple diagrams (architecture, sequence, state) -- **Always pure Mermaid**: No styling, just structure - -### 4. Component Design Rules -- **Single Responsibility**: One clear purpose per component -- **Clear Boundaries**: Explicit domain ownership -- **Boundary Commitments First**: Before detailing components, state the responsibility boundary this design commits to -- **Dependency Direction**: Follow architectural layers -- **Interface Segregation**: Minimal, focused interfaces -- **Team-safe Interfaces**: Design boundaries that allow parallel implementation without merge conflicts -- **No Hidden Shared Ownership**: If two areas appear to co-own the same behavior or data, the design is incomplete -- **Research Traceability**: Record boundary decisions and rationale in `research.md` - -### 5. Data Modeling Standards -- **Domain First**: Start with business concepts -- **Consistency Boundaries**: Clear aggregate roots -- **Normalization**: Balance between performance and integrity -- **Evolution**: Plan for schema changes - -### 6. Error Handling Philosophy -- **Fail Fast**: Validate early and clearly -- **Graceful Degradation**: Partial functionality over complete failure -- **User Context**: Actionable error messages -- **Observability**: Comprehensive logging and monitoring - -### 7. Integration Patterns -- **Loose Coupling**: Minimize dependencies -- **Contract First**: Define interfaces before implementation -- **Versioning**: Plan for API evolution -- **Idempotency**: Design for retry safety -- **Contract Visibility**: Surface API and event contracts in design.md while linking extended details from `research.md` - -### 8. Dependency Direction -- **Define and enforce the dependency direction** in the architecture section of design.md (e.g., Types → Config → Repository → Service → Runtime → UI) -- Each layer imports only from layers to its left — never upward -- This constraint is not a suggestion; implementation and review should treat violations as errors -- When the File Structure Plan maps files to components, the dependency direction determines which imports are allowed - -## Documentation Standards - -### Language and Tone -- **Declarative**: "The system authenticates users" not "The system should authenticate" -- **Precise**: Specific technical terms over vague descriptions -- **Concise**: Essential information only -- **Formal**: Professional technical writing - -### Structure Requirements -- **Hierarchical**: Clear section organization -- **Traceable**: Requirements to components mapping -- **Complete**: All aspects covered for implementation -- **Consistent**: Uniform terminology throughout -- **Focused**: Keep design.md centered on architecture and contracts; move investigation logs and lengthy comparisons to `research.md` - -## Section Authoring Guidance - -### Global Ordering -- Default flow: Overview → Goals/Non-Goals → Boundary Commitments → Architecture → File Structure Plan → Components & Interfaces → Optional sections. -- Teams may swap Traceability earlier or place Data Models nearer Architecture when it improves clarity, but keep section headings intact. -- Within each section, follow **Summary → Scope → Decisions → Impacts/Risks** so reviewers can scan consistently. - -### Requirement IDs -- Reference requirements as `2.1, 2.3` without prefixes (no “Requirement 2.1”). -- All requirements MUST have numeric IDs. If a requirement lacks a numeric ID, stop and fix `requirements.md` before continuing. -- Use `N.M`-style numeric IDs where `N` is the top-level requirement number from requirements.md (for example, Requirement 1 → 1.1, 1.2; Requirement 2 → 2.1, 2.2). -- Every component, task, and traceability row must reference the same canonical numeric ID. - -### Technology Stack -- Include ONLY layers impacted by this feature (frontend, backend, data, messaging, infra). -- For each layer specify tool/library + version + the role it plays; push extended rationale, comparisons, or benchmarks to `research.md`. -- When extending an existing system, highlight deviations from the current stack and list new dependencies. - -### System Flows -- Add diagrams only when they clarify behavior: - - **Sequence** for multi-step interactions - - **Process/State** for branching rules or lifecycle - - **Data/Event** for pipelines or async patterns -- Always use pure Mermaid. If no complex flow exists, omit the entire section. - -### Requirements Traceability -- Use the standard table (`Requirement | Summary | Components | Interfaces | Flows`) to prove coverage. -- Collapse to bullet form only when a single requirement maps 1:1 to a component. -- Prefer the component summary table for simple mappings; reserve the full traceability table for complex or compliance-sensitive requirements. -- Re-run this mapping whenever requirements or components change to avoid drift. - -### Components & Interfaces Authoring -- Boundary Commitments should already make the ownership seam explicit before this section begins. -- Group components by domain/layer and provide one block per component. -- Begin with a summary table listing Component, Domain, Intent, Requirement coverage, key dependencies, and selected contracts. -- Table fields: Intent (one line), Requirements (`2.1, 2.3`), Owner/Reviewers (optional). -- Dependencies table must mark each entry as Inbound/Outbound/External and assign Criticality (`P0` blocking, `P1` high-risk, `P2` informational). -- Summaries of external dependency research stay here; detailed investigation (API signatures, rate limits, migration notes) belongs in `research.md`. -- design.md must remain a self-contained reviewer artifact. Reference `research.md` only for background, and restate any conclusions or decisions here. -- Contracts: tick only the relevant types (Service/API/Event/Batch/State). Unchecked types should not appear later in the component section. -- Service interfaces must declare method signatures, inputs/outputs, and error envelopes. API/Event/Batch contracts require schema tables or bullet lists covering trigger, payload, delivery, idempotency. -- Use **Integration & Migration Notes**, **Validation Hooks**, and **Open Questions / Risks** to document rollout strategy, observability, and unresolved decisions. -- Detail density rules: - - **Full block**: components introducing new boundaries (logic hooks, shared services, external integrations, data layers). - - **Summary-only**: presentational/UI components with no new boundaries (plus a short Implementation Note if needed). -- Implementation Notes must combine Integration / Validation / Risks into a single bulleted subsection to reduce repetition. -- Prefer lists or inline descriptors for short data (dependencies, contract selections). Use tables only when comparing multiple items. - -### Shared Interfaces & Props -- Define a base interface (e.g., `BaseUIPanelProps`) for recurring UI components and extend it per component to capture only the deltas. -- Hooks, utilities, and integration adapters that introduce new contracts should still include full TypeScript signatures. -- When reusing a base contract, reference it explicitly (e.g., “Extends `BaseUIPanelProps` with `onSubmitAnswer` callback”) instead of duplicating the code block. - -### Data Models -- Domain Model covers aggregates, entities, value objects, domain events, and invariants. Add Mermaid diagrams only when relationships are non-trivial. -- Logical Data Model should articulate structure, indexing, sharding, and storage-specific considerations (event store, KV/wide-column) relevant to the change. -- Data Contracts & Integration section documents API payloads, event schemas, and cross-service synchronization patterns when the feature crosses boundaries. -- Lengthy type definitions or vendor-specific option objects should be placed in the Supporting References section within design.md, linked from the relevant section. Investigation notes stay in `research.md`. -- Supporting References usage is optional; only create it when keeping the content in the main body would reduce readability. All decisions must still appear in the main sections so design.md stands alone. - -### Error/Testing/Security/Performance Sections -- Record only feature-specific decisions or deviations. Link or reference organization-wide standards (steering) for baseline practices instead of restating them. - -### Diagram & Text Deduplication -- Do not restate diagram content verbatim in prose. Use the text to highlight key decisions, trade-offs, or impacts that are not obvious from the visual. -- When a decision is fully captured in the diagram annotations, a short “Key Decisions” bullet is sufficient. - -### General Deduplication -- Avoid repeating the same information across Overview, Architecture, and Components. Reference earlier sections when context is identical. -- If a requirement/component relationship is captured in the summary table, do not rewrite it elsewhere unless extra nuance is added. - -## Diagram Guidelines - -### When to include a diagram -- **Architecture**: Use a structural diagram when 3+ components or external systems interact. -- **Sequence**: Draw a sequence diagram when calls/handshakes span multiple steps. -- **State / Flow**: Capture complex state machines or business flows in a dedicated diagram. -- **ER**: Provide an entity-relationship diagram for non-trivial data models. -- **Skip**: Minor one-component changes generally do not need diagrams. - -### Mermaid requirements -```mermaid -graph TB - Client --> ApiGateway - ApiGateway --> ServiceA - ApiGateway --> ServiceB - ServiceA --> Database -``` - -- **Plain Mermaid only** – avoid custom styling or unsupported syntax. -- **Node IDs** – alphanumeric plus underscores only (e.g., `Client`, `ServiceA`). Do not use `@`, `/`, or leading `-`. -- **Labels** – simple words. Do not embed parentheses `()`, square brackets `[]`, quotes `"`, or slashes `/`. - - ❌ `DnD[@dnd-kit/core]` → invalid ID (`@`). - - ❌ `UI[KanbanBoard(React)]` → invalid label (`()`). - - ✅ `DndKit[dnd-kit core]` → use plain text in labels, keep technology details in the accompanying description. - - ℹ️ Mermaid strict-mode will otherwise fail with errors like `Expecting 'SQE' ... got 'PS'`; remove punctuation from labels before rendering. -- **Edges** – show data or control flow direction. -- **Groups** – using Mermaid subgraphs to cluster related components is allowed; use it sparingly for clarity. - -## Quality Metrics -### Design Completeness Checklist -- All requirements addressed -- No implementation details leaked -- Clear component boundaries -- Explicit error handling -- Comprehensive test strategy -- Security considered -- Performance targets defined -- Migration path clear (if applicable) - -### Common Anti-patterns to Avoid -❌ Mixing design with implementation -❌ Vague interface definitions -❌ Missing error scenarios -❌ Ignored non-functional requirements -❌ Overcomplicated architectures -❌ Tight coupling between components -❌ Missing data consistency strategy -❌ Incomplete dependency analysis diff --git a/.kiro/settings/rules/design-review-gate.md b/.kiro/settings/rules/design-review-gate.md deleted file mode 100644 index 06bc5feb..00000000 --- a/.kiro/settings/rules/design-review-gate.md +++ /dev/null @@ -1,50 +0,0 @@ -# Design Review Gate - -Before writing `design.md`, review the draft design and repair local issues until the design passes or a true spec gap is discovered. - -## Requirements Coverage Review - -- Every numeric requirement ID from `requirements.md` must appear in the design traceability mapping and be backed by one or more concrete components, contracts, flows, data models, or operational decisions. -- Every requirement that introduces an external dependency, integration point, runtime prerequisite, migration concern, observability need, security constraint, or performance target must be reflected explicitly in `design.md`. -- If coverage is missing because the design draft is incomplete, repair the draft and review again. -- If coverage cannot be completed cleanly because requirements are ambiguous, contradictory, or underspecified, stop and return to the requirements phase instead of inventing design detail. - -## Architecture Readiness Review - -- Component boundaries must be explicit enough that implementation tasks can be assigned without guessing ownership. -- Interfaces, contracts, state transitions, and integration boundaries must be concrete enough for implementation and validation. -- Build-vs-adopt decisions that materially affect architecture must be captured in `design.md`, with deeper investigation left in `research.md` when present. -- Runtime prerequisites, migrations, rollout constraints, validation hooks, and failure modes must be surfaced when they materially affect implementation order or risk. - -## Boundary Readiness Review - -- The design must explicitly state what this spec owns. -- The design must explicitly state what is out of boundary. -- Allowed dependencies must be concrete enough that reviewers can detect boundary violations later. -- If data, behavior, or integration responsibility appears shared across multiple areas without a clear seam, stop and repair the design. -- If downstream assumptions are embedded in upstream components "for convenience," stop and repair the design. -- If the boundary cannot be explained in a few direct bullets, it is probably still too vague for task generation. -- If the design reveals multiple independent responsibility seams that could move separately, stop and split the spec or return to roadmap discovery instead of forcing them into one spec. - -## Executability Review - -- The design must be implementable as a sequence of bounded tasks without hidden prerequisites. -- Parallel-safe boundaries should be visible where the architecture intends concurrent implementation. -- Avoid speculative abstraction: remove components, adapters, or interfaces that exist only for hypothetical future scope. -- If a section is too vague for tasks to reference directly, rewrite it before finalizing the design. - -## Mechanical Checks - -Before applying judgment, verify these mechanically: -- **Requirements traceability**: Extract all numeric requirement IDs from `requirements.md`. Scan the design draft for each ID. Report any IDs not found in the design. -- **Boundary section populated**: `Boundary Commitments`, `Out of Boundary`, `Allowed Dependencies`, and `Revalidation Triggers` must not be empty or placeholder-only. -- **File Structure Plan populated**: The File Structure Plan section must contain concrete file paths (not just "TBD" or empty). Scan for placeholder text in that section. -- **Boundary ↔ file structure alignment**: The File Structure Plan must reflect the stated responsibility boundary. If files imply broader ownership than the boundary section claims, report a mismatch. -- **No orphan components**: Every component mentioned in the design must appear in the File Structure Plan with a file path. Scan for component names that have no corresponding file entry. - -## Review Loop - -- Run mechanical checks first, then judgment-based review. -- If issues are local to the draft, repair the draft and re-run the review gate. -- Keep the loop bounded: no more than 2 review-and-repair passes before escalating a real spec gap. -- Write `design.md` only after the review gate passes. diff --git a/.kiro/settings/rules/design-review.md b/.kiro/settings/rules/design-review.md deleted file mode 100644 index 3209d8df..00000000 --- a/.kiro/settings/rules/design-review.md +++ /dev/null @@ -1,110 +0,0 @@ -# Design Review Process - -## Objective -Conduct interactive quality review of technical design documents to ensure they are solid enough to proceed to implementation with acceptable risk. - -## Review Philosophy -- **Quality assurance, not perfection seeking** -- **Critical focus**: Limit to 3 most important concerns -- **Interactive dialogue**: Engage with designer, not one-way evaluation -- **Balanced assessment**: Recognize strengths and weaknesses -- **Clear decision**: Definitive GO/NO-GO with rationale - -## Scope & Non-Goals - -- Scope: Evaluate the quality of the design document against project context and standards to decide GO/NO-GO. -- Non-Goals: Do not perform implementation-level design, deep technology research, or finalize technology choices. Defer such items to the design phase iteration. - -## Core Review Criteria - -### 1. Existing Architecture Alignment (Critical) -- Integration with existing system boundaries and layers -- Consistency with established architectural patterns -- Proper dependency direction and coupling management -- Alignment with current module organization - -### 2. Design Consistency & Standards -- Adherence to project naming conventions and code standards -- Consistent error handling and logging strategies -- Uniform configuration and dependency management -- Alignment with established data modeling patterns - -### 3. Extensibility & Maintainability -- Design flexibility for future requirements -- Clear separation of concerns and single responsibility -- Testability and debugging considerations -- Appropriate complexity for requirements - -### 4. Type Safety & Interface Design -- Proper type definitions and interface contracts -- Avoidance of unsafe patterns (e.g., `any` in TypeScript) -- Clear API boundaries and data structures -- Input validation and error handling coverage - -## Review Process - -### Step 1: Analyze -Analyze design against all review criteria, focusing on critical issues impacting integration, maintainability, complexity, and requirements fulfillment. - -### Step 2: Identify Critical Issues (≤3) -For each issue: -``` -🔴 **Critical Issue [1-3]**: [Brief title] -**Concern**: [Specific problem] -**Impact**: [Why it matters] -**Suggestion**: [Concrete improvement] -**Traceability**: [Requirement ID/section from requirements.md] -**Evidence**: [Design doc section/heading] -``` - -### Step 3: Recognize Strengths -Acknowledge 1-2 strong aspects to maintain balanced feedback. - -### Step 4: Decide GO/NO-GO -- **GO**: No critical architectural misalignment, requirements addressed, clear implementation path, acceptable risks -- **NO-GO**: Fundamental conflicts, critical gaps, high failure risk, disproportionate complexity - -## Traceability & Evidence - -- Link each critical issue to the relevant requirement(s) from `requirements.md` (ID or section). -- Cite evidence locations in the design document (section/heading, diagram, or artifact) to support the assessment. -- When applicable, reference constraints from steering context to justify the issue. - -## Output Format - -### Design Review Summary -2-3 sentences on overall quality and readiness. - -### Critical Issues (≤3) -For each: Issue, Impact, Recommendation, Traceability (e.g., 1.1, 1.2), Evidence (design.md section). - -### Design Strengths -1-2 positive aspects. - -### Final Assessment -Decision (GO/NO-GO), Rationale (1-2 sentences), Next Steps. - -### Interactive Discussion -Engage on designer's perspective, alternatives, clarifications, and necessary changes. - -## Length & Focus - -- Summary: 2–3 sentences -- Each critical issue: 5–7 lines total (including Issue/Impact/Recommendation/Traceability/Evidence) -- Overall review: keep concise (~400 words guideline) - -## Review Guidelines - -1. **Critical Focus**: Only flag issues that significantly impact success -2. **Constructive Tone**: Provide solutions, not just criticism -3. **Interactive Approach**: Engage in dialogue rather than one-way evaluation -4. **Balanced Assessment**: Recognize both strengths and weaknesses -5. **Clear Decision**: Make definitive GO/NO-GO recommendation -6. **Actionable Feedback**: Ensure all suggestions are implementable - -## Final Checklist - -- **Critical Issues ≤ 3** and each includes Impact and Recommendation -- **Traceability**: Each issue references requirement ID/section -- **Evidence**: Each issue cites design doc location -- **Decision**: GO/NO-GO with clear rationale and next steps diff --git a/.kiro/settings/rules/design-synthesis.md b/.kiro/settings/rules/design-synthesis.md deleted file mode 100644 index 935e4d6f..00000000 --- a/.kiro/settings/rules/design-synthesis.md +++ /dev/null @@ -1,29 +0,0 @@ -# Design Synthesis - -After discovery and before writing the design document, apply these three lenses to the collected findings. This step requires the full picture — do not parallelize. - -## 1. Generalization - -Look across all requirements as a group. Identify cases where multiple requirements are variations of the same underlying problem. - -- If feature A is a special case of a more general capability X, design X with an interface that naturally supports A (and potentially B, C later) -- Keep the implementation scope to what the current requirements demand — generalize the interface, not the implementation -- Record identified generalizations in `research.md` under Design Decisions - -## 2. Build vs. Adopt - -For each major component in the emerging design, ask: is this problem already solved? - -- Search for established standards (RFCs, protocols), battle-tested libraries, or platform-native capabilities that address the requirement -- Prefer adopting existing solutions over building custom ones when they fit the requirements without significant adaptation -- If adopting: verify the solution is actively maintained, compatible with the project's stack (check steering), and meets non-functional requirements -- If building: document why existing solutions were rejected (capture in `research.md`) - -## 3. Simplification - -For each component and abstraction layer in the emerging design, ask: is this necessary? - -- Remove components that exist "just in case" or for hypothetical future requirements not in the current spec -- Flatten unnecessary abstraction layers — if an interface has only one implementation with no foreseeable second, it may not need the indirection -- Prefer fewer, cohesive components over many fine-grained ones -- The right design is the smallest one that satisfies all requirements and remains extensible at the interface level diff --git a/.kiro/settings/rules/ears-format.md b/.kiro/settings/rules/ears-format.md deleted file mode 100644 index cef8e7df..00000000 --- a/.kiro/settings/rules/ears-format.md +++ /dev/null @@ -1,49 +0,0 @@ -# EARS Format Guidelines - -## Overview -EARS (Easy Approach to Requirements Syntax) is the standard format for acceptance criteria in spec-driven development. - -EARS patterns describe the logical structure of a requirement (condition + subject + response) and are not tied to any particular natural language. -All acceptance criteria should be written in the target language configured for the specification (for example, `spec.json.language` / `en`). -Keep EARS trigger keywords and fixed phrases in English (`When`, `If`, `While`, `Where`, `The system shall`, `The [system] shall`) and localize only the variable parts (`[event]`, `[precondition]`, `[trigger]`, `[feature is included]`, `[response/action]`) into the target language. Do not interleave target-language text inside the trigger or fixed English phrases themselves. - -## Primary EARS Patterns - -### 1. Event-Driven Requirements -- **Pattern**: When [event], the [system] shall [response/action] -- **Use Case**: Responses to specific events or triggers -- **Example**: When user clicks checkout button, the Checkout Service shall validate cart contents - -### 2. State-Driven Requirements -- **Pattern**: While [precondition], the [system] shall [response/action] -- **Use Case**: Behavior dependent on system state or preconditions -- **Example**: While payment is processing, the Checkout Service shall display loading indicator - -### 3. Unwanted Behavior Requirements -- **Pattern**: If [trigger], the [system] shall [response/action] -- **Use Case**: System response to errors, failures, or undesired situations -- **Example**: If invalid credit card number is entered, then the website shall display error message - -### 4. Optional Feature Requirements -- **Pattern**: Where [feature is included], the [system] shall [response/action] -- **Use Case**: Requirements for optional or conditional features -- **Example**: Where the car has a sunroof, the car shall have a sunroof control panel - -### 5. Ubiquitous Requirements -- **Pattern**: The [system] shall [response/action] -- **Use Case**: Always-active requirements and fundamental system properties -- **Example**: The mobile phone shall have a mass of less than 100 grams - -## Combined Patterns -- While [precondition], when [event], the [system] shall [response/action] -- When [event] and [additional condition], the [system] shall [response/action] - -## Subject Selection Guidelines -- **Software Projects**: Use concrete system/service name (e.g., "Checkout Service", "User Auth Module") -- **Process/Workflow**: Use responsible team/role (e.g., "Support Team", "Review Process") -- **Non-Software**: Use appropriate subject (e.g., "Marketing Campaign", "Documentation") - -## Quality Criteria -- Requirements must be testable, verifiable, and describe a single behavior. -- Use objective language: "shall" for mandatory behavior, "should" for recommendations; avoid ambiguous terms. -- Follow EARS syntax: [condition], the [system] shall [response/action]. diff --git a/.kiro/settings/rules/gap-analysis.md b/.kiro/settings/rules/gap-analysis.md deleted file mode 100644 index c4e4bca8..00000000 --- a/.kiro/settings/rules/gap-analysis.md +++ /dev/null @@ -1,144 +0,0 @@ -# Gap Analysis Process - -## Objective -Analyze the gap between requirements and existing codebase to inform implementation strategy decisions. - -## Analysis Framework - -### 1. Current State Investigation - -- Scan for domain-related assets: - - Key files/modules and directory layout - - Reusable components/services/utilities - - Dominant architecture patterns and constraints - -- Extract conventions: - - Naming, layering, dependency direction - - Import/export patterns and dependency hotspots - - Testing placement and approach - -- Note integration surfaces: - - Data models/schemas, API clients, auth mechanisms - -### 2. Requirements Feasibility Analysis - -- From EARS requirements, list technical needs: - - Data models, APIs/services, UI/components - - Business rules/validation - - Non-functionals: security, performance, scalability, reliability - -- Identify gaps and constraints: - - Missing capabilities in current codebase - - Unknowns to be researched later (mark as "Research Needed") - - Constraints from existing architecture and patterns - -- Note complexity signals: - - Simple CRUD / algorithmic logic / workflows / external integrations - -### 3. Implementation Approach Options - -#### Option A: Extend Existing Components -**When to consider**: Feature fits naturally into existing structure - -- **Which files/modules to extend**: - - Identify specific files requiring changes - - Assess impact on existing functionality - - Evaluate backward compatibility concerns - -- **Compatibility assessment**: - - Check if extension respects existing interfaces - - Verify no breaking changes to consumers - - Assess test coverage impact - -- **Complexity and maintainability**: - - Evaluate cognitive load of additional functionality - - Check if single responsibility principle is maintained - - Assess if file size remains manageable - -**Trade-offs**: -- ✅ Minimal new files, faster initial development -- ✅ Leverages existing patterns and infrastructure -- ❌ Risk of bloating existing components -- ❌ May complicate existing logic - -#### Option B: Create New Components -**When to consider**: Feature has distinct responsibility or existing components are already complex - -- **Rationale for new creation**: - - Clear separation of concerns justifies new file - - Existing components are already complex - - Feature has distinct lifecycle or dependencies - -- **Integration points**: - - How new components connect to existing system - - APIs or interfaces exposed - - Dependencies on existing components - -- **Responsibility boundaries**: - - Clear definition of what new component owns - - Interfaces with existing components - - Data flow and control flow - -**Trade-offs**: -- ✅ Clean separation of concerns -- ✅ Easier to test in isolation -- ✅ Reduces complexity in existing components -- ❌ More files to navigate -- ❌ Requires careful interface design - -#### Option C: Hybrid Approach -**When to consider**: Complex features requiring both extension and new creation - -- **Combination strategy**: - - Which parts extend existing components - - Which parts warrant new components - - How they interact - -- **Phased implementation**: - - Initial phase: minimal viable changes - - Subsequent phases: refactoring or new components - - Migration strategy if needed - -- **Risk mitigation**: - - Incremental rollout approach - - Feature flags or configuration - - Rollback strategy - -**Trade-offs**: -- ✅ Balanced approach for complex features -- ✅ Allows iterative refinement -- ❌ More complex planning required -- ❌ Potential for inconsistency if not well-coordinated -### 4. Out-of-Scope for Gap Analysis - -- Defer deep research activities to the design phase. -- Record unknowns as concise "Research Needed" items only. - -### 5. Implementation Complexity & Risk - - - Effort: - - S (1–3 days): existing patterns, minimal deps, straightforward integration - - M (3–7 days): some new patterns/integrations, moderate complexity - - L (1–2 weeks): significant functionality, multiple integrations or workflows - - XL (2+ weeks): architectural changes, unfamiliar tech, broad impact - - Risk: - - High: unknown tech, complex integrations, architectural shifts, unclear perf/security path - - Medium: new patterns with guidance, manageable integrations, known perf solutions - - Low: extend established patterns, familiar tech, clear scope, minimal integration - -### Output Checklist - -- Requirement-to-Asset Map with gaps tagged (Missing / Unknown / Constraint) -- Options A/B/C with short rationale and trade-offs -- Effort (S/M/L/XL) and Risk (High/Medium/Low) with one-line justification each -- Recommendations for design phase: - - Preferred approach and key decisions - - Research items to carry forward - -## Principles - -- **Information over decisions**: Provide analysis and options, not final choices -- **Multiple viable options**: Offer credible alternatives when applicable -- **Explicit gaps and assumptions**: Flag unknowns and constraints clearly -- **Context-aware**: Align with existing patterns and architecture limits -- **Transparent effort and risk**: Justify labels succinctly diff --git a/.kiro/settings/rules/requirements-review-gate.md b/.kiro/settings/rules/requirements-review-gate.md deleted file mode 100644 index b5c61dcc..00000000 --- a/.kiro/settings/rules/requirements-review-gate.md +++ /dev/null @@ -1,51 +0,0 @@ -# Requirements Review Gate - -Before writing `requirements.md`, review the draft requirements and repair local issues until the draft passes or a true scope ambiguity is discovered. - -## Boundary Continuity - -Use boundary terminology consistently across phases without turning requirements into design: - -- **Discovery** identifies `Boundary Candidates` -- **Requirements** make inclusion, exclusion, and adjacent expectations explicit when scope could be misread -- **Design** turns those into `Boundary Commitments` -- **Tasks** use `_Boundary:_` to constrain executable work - -Requirements should clarify the feature boundary in user- or operator-observable terms, not in architecture ownership or implementation detail. - -## Scope and Coverage Review - -- The draft must cover the feature's core user journeys, major scope boundaries, primary error cases, and meaningful edge conditions that are visible to the user or operator. -- If the feature touches adjacent systems, specs, or workflows, the draft must make clear what this feature expects from them and what it does not own when that distinction affects user-visible behavior or operator expectations. -- Business/domain rules, compliance constraints, security/privacy expectations, and operational constraints that materially shape user-visible behavior must be reflected explicitly when they are in scope. -- If coverage is missing because the draft is incomplete, repair the draft and review again. -- If coverage cannot be completed cleanly because the project description or steering context is ambiguous, contradictory, or underspecified, stop and ask the user to clarify instead of guessing. - -## EARS and Testability Review - -- Every acceptance criterion must follow the EARS rules defined in `ears-format.md`. -- Every requirement must be testable, observable, and specific enough that later design and validation can verify it. -- Remove implementation details that belong in `design.md` rather than `requirements.md`. -- Requirement headings must use numeric IDs only; do not mix numeric and alphabetic labels. - -## Structure and Quality Review - -- Group related behaviors into coherent requirement areas without duplicating the same obligation across multiple sections. -- Make inclusion/exclusion boundaries explicit when the feature scope could otherwise be misread. -- Keep boundary statements lightweight and observable: describe feature responsibility and adjacent expectations without prescribing components, layers, or internal ownership. -- Ensure non-functional expectations remain user-observable or operator-observable; move technology choices and internal architecture detail out of requirements. -- Normalize vague language such as "fast", "robust", or "secure" into concrete user-visible expectations whenever the source material supports it. - -## Mechanical Checks - -Before applying judgment, verify these mechanically: -- **Numeric IDs present**: Every requirement heading has a numeric ID (1, 1.1, 2, etc.). Scan the draft for headings without IDs. -- **Acceptance criteria exist**: Every requirement has at least one EARS-format acceptance criterion. Scan for requirements with no "When/If/While/Where" acceptance statements. -- **No implementation language**: Scan for technology-specific terms (database names, framework names, API patterns) that belong in design, not requirements. Flag any found. - -## Review Loop - -- Run mechanical checks first, then judgment-based review. -- If issues are local to the draft, repair the draft and re-run the review gate. -- Keep the loop bounded: no more than 2 review-and-repair passes before escalating a real ambiguity back to the user. -- Write `requirements.md` only after the review gate passes. diff --git a/.kiro/settings/rules/steering-principles.md b/.kiro/settings/rules/steering-principles.md deleted file mode 100644 index b3ac5f5e..00000000 --- a/.kiro/settings/rules/steering-principles.md +++ /dev/null @@ -1,90 +0,0 @@ -# Steering Principles - -Steering files are **project memory**, not exhaustive specifications. - ---- - -## Content Granularity - -### Golden Rule -> "If new code follows existing patterns, steering shouldn't need updating." - -### ✅ Document -- Organizational patterns (feature-first, layered) -- Naming conventions (PascalCase rules) -- Import strategies (absolute vs relative) -- Architectural decisions (state management) -- Technology standards (key frameworks) - -### ❌ Avoid -- Complete file listings -- Every component description -- All dependencies -- Implementation details -- Agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`) -- Detailed documentation of `.kiro/` metadata directories (settings, automation) - -### Example Comparison - -**Bad** (Specification-like): -```markdown -- /components/Button.tsx - Primary button with variants -- /components/Input.tsx - Text input with validation -- /components/Modal.tsx - Modal dialog -... (50+ files) -``` - -**Good** (Project Memory): -```markdown -## UI Components (`/components/ui/`) -Reusable, design-system aligned primitives -- Named by function (Button, Input, Modal) -- Export component + TypeScript interface -- No business logic -``` - ---- - -## Security - -Never include: -- API keys, passwords, credentials -- Database URLs, internal IPs -- Secrets or sensitive data - ---- - -## Quality Standards - -- **Single domain**: One topic per file -- **Concrete examples**: Show patterns with code -- **Explain rationale**: Why decisions were made -- **Maintainable size**: 100-200 lines typical - ---- - -## Preservation (when updating) - -- Preserve user sections and custom examples -- Additive by default (add, don't replace) -- Add `updated_at` timestamp -- Note why changes were made - ---- - -## Notes - -- Templates are starting points, customize as needed -- Follow same granularity principles as core steering -- All steering files loaded as project memory -- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories -- Custom files equally important as core files - ---- - -## File-Specific Focus - -- **product.md**: Purpose, value, business context (not exhaustive features) -- **tech.md**: Key frameworks, standards, conventions (not all dependencies) -- **structure.md**: Organization patterns, naming rules (not directory trees) -- **Custom files**: Specialized patterns (API, testing, security, etc.) diff --git a/.kiro/settings/rules/tasks-generation.md b/.kiro/settings/rules/tasks-generation.md deleted file mode 100644 index 58060974..00000000 --- a/.kiro/settings/rules/tasks-generation.md +++ /dev/null @@ -1,222 +0,0 @@ -# Task Generation Rules - -## Core Principles - -### 1. Natural Language Descriptions -Focus on capabilities and outcomes, not code structure. - -**Describe**: -- What functionality to achieve -- Business logic and behavior -- Features and capabilities -- Domain language and concepts -- Data relationships and workflows - -**Avoid**: -- File paths and directory structure -- Function/method names and signatures -- Type definitions and interfaces -- Class names and API contracts -- Specific data structures - -**Rationale**: Implementation details (files, methods, types) are defined in design.md. Tasks describe the functional work to be done. - -### 2. Task Ordering Principle - -**Order implies dependency**: Task N implicitly depends on all tasks before it. This is the primary dependency mechanism. - -**Tasks must follow this phase order**: -1. **Foundation**: Environment setup, test infrastructure, shared utilities, database schema, configuration -2. **Core**: Primary feature implementation (parallel-capable tasks grouped here) -3. **Integration**: Wiring components together, cross-boundary connections -4. **Validation**: E2E tests, edge cases, regression checks - -**Rationale**: Foundation work unblocks everything else. Placing setup tasks early prevents downstream blocking. Core tasks can often run in parallel because foundation is already complete. - -### 3. Task Integration & Progression - -**Every task must**: -- Build on previous outputs (no orphaned code) -- Connect to the overall system (no hanging features) -- Progress incrementally (no big jumps in complexity) -- Respect architecture boundaries defined in design.md (Architecture Pattern & Boundary Map) -- Honor interface contracts documented in design.md -- Use major task summaries sparingly—omit detail bullets if the work is fully captured by child tasks. - -**End with integration tasks** to wire everything together. - -### 4. Dependency Declaration - -**Default**: Sequential ordering handles most dependencies (task N depends on tasks before it). - -**Explicit declaration required when**: -- A task depends on a specific task in a different major-task group (cross-boundary) -- The dependency is non-obvious from ordering alone -- A task can skip ahead of its position (declared via `(P)`) but still needs specific prior work - -**Format**: `_Depends: 1.2, 2.3_` — placed alongside `_Requirements:_` in task detail sections. - -**Do not over-annotate**: If a task simply depends on the task directly before it, ordering alone is sufficient. - -### 5. Boundary Scope - -**Each task should declare its component boundary** using design.md component/module names: -- `_Boundary: AuthService_` or `_Boundary: API Layer, UserRepository_` -- Helps validate parallel safety: tasks with non-overlapping boundaries are parallel candidates -- Helps agents understand scope: what to touch and what not to touch - -**When to use**: Required for tasks marked `(P)` to validate parallel safety. Omit for sequential tasks where scope is obvious from the description. - -**Boundary rule**: -- Each executable task should stay within a single responsibility boundary -- If work must cross boundaries, make it an explicit integration task rather than a normal implementation task -- Do not hide cross-boundary coordination inside a task that appears local - -### 6. Flexible Task Sizing - -**Guidelines**: -- **Major tasks**: As many sub-tasks as logically needed (group by cohesion) -- **Sub-tasks**: 1-3 hours each, 3-10 details per sub-task -- Balance between too granular and too broad - -**Don't force arbitrary numbers** - let logical grouping determine structure. - -### 7. Requirements Mapping - -**End each task detail section with**: -- `_Requirements: X.X, Y.Y_` listing **only numeric requirement IDs** (comma-separated). Never append descriptive text, parentheses, translations, or free-form labels. -- For cross-cutting requirements, list every relevant requirement ID. All requirements MUST have numeric IDs in requirements.md. If an ID is missing, stop and correct requirements.md before generating tasks. -- Reference components/interfaces from design.md when helpful (e.g., `_Contracts: AuthService API`) - -### 7.5 Observable Completion - -**Each executable task must include at least one detail bullet that describes the observable completed state**: -- Phrase it as a deliverable, runtime behavior, persisted state, UI state, endpoint behavior, test result, or integration outcome -- Avoid vague bullets like "implement support", "wire things up", or "handle logic" unless paired with a concrete observable result -- Prefer making one detail bullet clearly answer: "What will be true when this task is done?" -- Keep this within the existing task body; do not add extra bookkeeping fields - -### 8. Code-Only Focus - -**Include ONLY**: -- Coding tasks (implementation) -- Testing tasks (unit, integration, E2E) -- Technical setup tasks (infrastructure, configuration) - -**Exclude**: -- Deployment tasks -- Documentation tasks -- User testing -- Marketing/business activities - -## Task Plan Review Gate - -Before writing `tasks.md`, review the draft task plan and repair local issues until the plan passes or a true spec gap is discovered. - -### Coverage Review - -- Every requirement ID from `requirements.md` must appear in at least one task. -- Every design component, interface/contract, integration point, runtime prerequisite, and validation concern from `design.md` must be represented by at least one task. -- If coverage is missing because the task plan is incomplete, repair the draft tasks and review again. -- If coverage cannot be added cleanly because requirements or design are ambiguous, contradictory, or underspecified, stop and return to the requirements/design phase instead of papering over the gap in `tasks.md`. - -### Executability Review - -- Every sub-task must be executable as written, usually within 1-3 hours. -- Every sub-task must produce a verifiable deliverable (behavior, artifact, endpoint, UI state, config, migration, test, or integration result). -- Every executable sub-task must include at least one detail bullet that states the observable completion condition. -- Split tasks that combine multiple independently verifiable outcomes. -- Split tasks that combine multiple responsibility boundaries unless they are explicit integration tasks. -- If many tasks require broad `_Boundary:_` scopes or repeated cross-boundary coordination, stop and return to design or roadmap decomposition instead of forcing the spec through task generation. -- Merge or collapse tasks that are too small, bookkeeping-only, or not meaningful execution units. -- Make implicit prerequisites explicit as preceding tasks. -- Re-check `_Depends:_`, `_Boundary:_`, and `(P)` markers after edits so concurrency claims still match the design boundaries and dependency graph. - -### Review Loop - -- Run the review gate on the draft task plan before writing `tasks.md`. -- If issues are task-plan-local, repair the draft and re-run the review gate. -- Keep the loop bounded: no more than 2 review-and-repair passes before escalating a real spec gap. -- Write `tasks.md` only after the review gate passes. - -### Optional Test Coverage Tasks - -- When the design already guarantees functional coverage and rapid MVP delivery is prioritized, mark purely test-oriented follow-up work (e.g., baseline rendering/unit tests) as **optional** using the `- [ ]*` checkbox form. -- Only apply the optional marker when the sub-task directly references acceptance criteria from requirements.md in its detail bullets. -- Never mark implementation work or integration-critical verification as optional—reserve `*` for auxiliary/deferrable test coverage that can be revisited post-MVP. - -## Task Hierarchy Rules - -### Maximum 2 Levels -- **Level 1**: Major tasks (1, 2, 3, 4...) -- **Level 2**: Sub-tasks (1.1, 1.2, 2.1, 2.2...) -- **No deeper nesting** (no 1.1.1) -- If a major task would contain only a single actionable item, collapse the structure and promote the sub-task to the major level (e.g., replace `1.1` with `1.`). -- When a major task exists purely as a container, keep the checkbox description concise and avoid duplicating detailed bullets—reserve specifics for its sub-tasks. - -### Sequential Numbering -- Major tasks MUST increment: 1, 2, 3, 4, 5... -- Sub-tasks reset per major task: 1.1, 1.2, then 2.1, 2.2... -- Never repeat major task numbers - -### Parallel Analysis (default) -- Assume parallel analysis is enabled unless explicitly disabled (e.g. `--sequential` flag). -- `(P)` means: this task has no dependency on its immediately preceding peers and can run concurrently with them. -- Identify tasks that can run concurrently when **all** conditions hold: - - No data dependency on other pending tasks - - No shared file or resource contention - - No prerequisite review/approval from another task - - `_Boundary:_` annotations confirm non-overlapping component scopes -- Foundation-phase tasks (see Task Ordering Principle) are rarely `(P)` — they establish shared prerequisites. -- Core-phase tasks are the primary candidates for `(P)` since foundation is already complete. -- Validate that identified parallel tasks operate within separate boundaries defined in the Architecture Pattern & Boundary Map. -- Confirm API/event contracts from design.md do not overlap in ways that cause conflicts. -- `(P)` tasks with cross-boundary dependencies must declare `_Depends: X.X_` explicitly. -- Append `(P)` immediately after the task number for each parallel-capable task: - - Example: `- [ ] 2.1 (P) Build background worker` - - Apply to both major tasks and sub-tasks when appropriate. -- If sequential mode is requested, omit `(P)` markers entirely. -- Group parallel tasks logically (same parent when possible) and highlight any ordering caveats in detail bullets. -- Explicitly call out dependencies that prevent `(P)` even when tasks look similar. - -### Checkbox Format -```markdown -- [ ] 1. Foundation: environment and test infrastructure setup -- [ ] 1.1 Sub-task description - - Detail item 1 - - Detail item 2 - - Observable completion condition - - _Requirements: X.X_ - -- [ ] 2. Core feature A -- [ ] 2.1 (P) Sub-task description - - Detail items... - - Observable completion condition - - _Requirements: Y.Y_ - - _Boundary: AuthService_ - -- [ ] 2.2 (P) Sub-task description - - Detail items... - - Observable completion condition - - _Requirements: Z.Z_ - - _Boundary: UserRepository_ - -- [ ] 3. Integration and wiring -- [ ] 3.1 Sub-task description - - Detail items... - - Observable completion condition - - _Depends: 2.1, 2.2_ - - _Requirements: W.W_ -``` - -## Requirements Coverage - -**Mandatory Check**: -- ALL requirements from requirements.md MUST be covered -- Cross-reference every requirement ID with task mappings -- If gaps found: Return to requirements or design phase -- No requirement should be left without corresponding tasks - -Use `N.M`-style numeric requirement IDs where `N` is the top-level requirement number from requirements.md (for example, Requirement 1 → 1.1, 1.2; Requirement 2 → 2.1, 2.2), and `M` is a local index within that requirement group. - -Document any intentionally deferred requirements with rationale. diff --git a/.kiro/settings/rules/tasks-parallel-analysis.md b/.kiro/settings/rules/tasks-parallel-analysis.md deleted file mode 100644 index fb31a4a0..00000000 --- a/.kiro/settings/rules/tasks-parallel-analysis.md +++ /dev/null @@ -1,41 +0,0 @@ -# Parallel Task Analysis Rules - -## Purpose -Provide a consistent way to identify implementation tasks that can be safely executed in parallel while generating `tasks.md`. - -## Relationship to Task Ordering - -`(P)` means: this task has no dependency on its immediately preceding peers and can run concurrently with them. The Task Ordering Principle (see tasks-generation.md) ensures Foundation-phase tasks run first, making Core-phase tasks the primary `(P)` candidates. - -## When to Consider Tasks Parallel -Only mark a task as parallel-capable when **all** of the following are true: - -1. **No data dependency** on pending tasks. -2. **No conflicting files or shared mutable resources** are touched. -3. **No prerequisite review/approval** from another task is required beforehand. -4. **Foundation work complete**: Environment/setup work needed by this task is already satisfied by earlier Foundation-phase tasks. -5. **Non-overlapping boundaries**: `_Boundary:_` annotations confirm tasks operate on separate components. - -## Marking Convention -- Append `(P)` immediately after the numeric identifier for each qualifying task. - - Example: `- [ ] 2.1 (P) Build background worker for emails` -- Apply `(P)` to both major tasks and sub-tasks when appropriate. -- If sequential execution is requested (e.g. via `--sequential` flag), omit `(P)` markers entirely. -- Keep `(P)` **outside** of checkbox brackets to avoid confusion with completion state. - -## Grouping & Ordering Guidelines -- Group parallel tasks under the same parent whenever the work belongs to the same theme. -- List obvious prerequisites or caveats in the detail bullets (e.g., "Requires schema migration from 1.2"). -- When two tasks look similar but are not parallel-safe, call out the blocking dependency explicitly. -- Skip marking container-only major tasks (those without their own actionable detail bullets) with `(P)`—evaluate parallel execution at the sub-task level instead. - -## Quality Checklist -Before marking a task with `(P)`, ensure you have: - -- Verified that running this task concurrently will not create merge or deployment conflicts. -- Confirmed `_Boundary:_` annotations show non-overlapping component scopes. -- Captured any shared state expectations in the detail bullets. -- Confirmed that the implementation can be tested independently. -- Added `_Depends: X.X_` if this `(P)` task still requires specific prior work from a different major-task group. - -If any check fails, **do not** mark the task with `(P)` and explain the dependency in the task details. diff --git a/.kiro/settings/templates/specs/design.md b/.kiro/settings/templates/specs/design.md deleted file mode 100644 index 1d31e71a..00000000 --- a/.kiro/settings/templates/specs/design.md +++ /dev/null @@ -1,331 +0,0 @@ -# Design Document Template - ---- -**Purpose**: Provide sufficient detail to ensure implementation consistency across different implementers, preventing interpretation drift. - -**Approach**: -- Include essential sections that directly inform implementation decisions -- Omit optional sections unless critical to preventing implementation errors -- Match detail level to feature complexity -- Use diagrams and tables over lengthy prose - -**Warning**: Approaching 1000 lines indicates excessive feature complexity that may require design simplification or splitting into multiple specs. ---- - -> Sections may be reordered (e.g., surfacing Requirements Traceability earlier or moving Data Models nearer Architecture) when it improves clarity. Within each section, keep the flow **Summary → Scope → Decisions → Impacts/Risks** so reviewers can scan consistently. - -## Overview -2-3 paragraphs max -**Purpose**: This feature delivers [specific value] to [target users]. -**Users**: [Target user groups] will utilize this for [specific workflows]. -**Impact** (if applicable): Changes the current [system state] by [specific modifications]. - - -### Goals -- Primary objective 1 -- Primary objective 2 -- Success criteria - -### Non-Goals -- Explicitly excluded functionality -- Future considerations outside current scope -- Integration points deferred - -## Boundary Commitments - -State the responsibility boundary of this spec in concrete terms. Treat this as the anchor for architecture, tasks, and later validation. - -### This Spec Owns -- Capabilities and behaviors this spec is responsible for -- Data it owns or is authoritative for -- Interfaces or contracts it defines or stabilizes - -### Out of Boundary -- Related concerns this spec explicitly does NOT own -- Work deferred to another spec, existing subsystem, or later phase -- Changes this spec must not absorb as "just one more thing" - -### Allowed Dependencies -- Upstream systems/specs/components this design may depend on -- Shared infrastructure this design may use -- Dependency constraints that must not be violated - -### Revalidation Triggers -List the kinds of changes that should force dependent specs or consumers to re-check integration. - -- Contract shape changes -- Data ownership changes -- Dependency direction changes -- Startup/runtime prerequisite changes - -## Architecture - -> Reference detailed discovery notes in `research.md` only for background; keep design.md self-contained for reviewers by capturing all decisions and contracts here. -> Capture key decisions in text and let diagrams carry structural detail—avoid repeating the same information in prose. -> Supporting sections below should remain as light as possible unless they materially clarify the responsibility boundary, dependency rules, or integration seams. - -### Existing Architecture Analysis (if applicable) -When modifying existing systems: -- Current architecture patterns and constraints -- Existing domain boundaries to be respected -- Integration points that must be maintained -- Technical debt addressed or worked around - -### Architecture Pattern & Boundary Map -**RECOMMENDED**: Include Mermaid diagram showing the chosen architecture pattern and system boundaries (required for complex features, optional for simple additions) - -**Architecture Integration**: -- Selected pattern: [name and brief rationale] -- Domain/feature boundaries: [how responsibilities are separated to avoid conflicts] -- Existing patterns preserved: [list key patterns] -- New components rationale: [why each is needed] -- Steering compliance: [principles maintained] - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Frontend / CLI | | | | -| Backend / Services | | | | -| Data / Storage | | | | -| Messaging / Events | | | | -| Infrastructure / Runtime | | | | - -> Keep rationale concise here and, when more depth is required (trade-offs, benchmarks), add a short summary plus pointer to the Supporting References section and `research.md` for raw investigation notes. - -## File Structure Plan - -Map the directory structure and file responsibilities for this feature. This section directly drives task `_Boundary:_` annotations and implementation Task Briefs. Use the appropriate level of detail: - -- **Small features**: List individual files with responsibilities -- **Large features**: Describe directory-level structure + per-domain/module pattern, list only non-obvious files individually - -### Directory Structure -``` -src/ -├── domain-a/ # Domain A responsibility -│ ├── controller.ts # Endpoint handlers -│ ├── service.ts # Business logic -│ └── types.ts # Domain types -├── domain-b/ # Domain B (same pattern as domain-a) -└── shared/ - └── cross-cutting.ts # Non-obvious: why this exists -``` - -> For repeated structures, describe the pattern once (e.g., "domain-b follows same pattern as domain-a"). List individual files only when their responsibility isn't obvious from the path. - -### Modified Files -- `path/to/existing.ts` — What changes and why - -> Each file should have one clear responsibility. Group files that change together. For repeated structures, describe the pattern once rather than listing every file. -> Avoid duplicating what Components and Interfaces already describes — focus on the physical file layout that Components maps to. - -## System Flows - -Provide only the diagrams needed to explain non-trivial flows. Use pure Mermaid syntax. Common patterns: -- Sequence (multi-party interactions) -- Process / state (branching logic or lifecycle) -- Data / event flow (pipelines, async messaging) - -Skip this section entirely for simple CRUD changes. -> Describe flow-level decisions (e.g., gating conditions, retries) briefly after the diagram instead of restating each step. - -## Requirements Traceability - -Use this section for complex or compliance-sensitive features where requirements span multiple domains. Straightforward 1:1 mappings can rely on the Components summary table. - -Map each requirement ID (e.g., `2.1`) to the design elements that realize it. - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | | | | | -| 1.2 | | | | | - -> Omit this section only when a single component satisfies a single requirement without cross-cutting concerns. - -## Components and Interfaces - -Provide a quick reference before diving into per-component details. - -- Summaries can be a table or compact list. Example table: - | Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | - |-----------|--------------|--------|--------------|--------------------------|-----------| - | ExampleComponent | UI | Displays XYZ | 1, 2 | GameProvider (P0), MapPanel (P1) | Service, State | -- Only components introducing new boundaries (e.g., logic hooks, external integrations, persistence) require full detail blocks. Simple presentation components can rely on the summary row plus a short Implementation Note. - -Group detailed blocks by domain or architectural layer. For each detailed component, list requirement IDs as `2.1, 2.3` (omit “Requirement”). When multiple UI components share the same contract, reference a base interface/props definition instead of duplicating code blocks. - -### [Domain / Layer] - -#### [Component Name] - -| Field | Detail | -|-------|--------| -| Intent | 1-line description of the responsibility | -| Requirements | 2.1, 2.3 | -| Owner / Reviewers | (optional) | - -**Responsibilities & Constraints** -- Primary responsibility -- Domain boundary and transaction scope -- Data ownership / invariants - -**Dependencies** -- Inbound: Component/service name — purpose (Criticality) -- Outbound: Component/service name — purpose (Criticality) -- External: Service/library — purpose (Criticality) - -Summarize external dependency findings here; deeper investigation (API signatures, rate limits, migration notes) lives in `research.md`. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] ← check only the ones that apply. - -##### Service Interface -```typescript -interface [ComponentName]Service { - methodName(input: InputType): Result; -} -``` -- Preconditions: -- Postconditions: -- Invariants: - -##### API Contract -| Method | Endpoint | Request | Response | Errors | -|--------|----------|---------|----------|--------| -| POST | /api/resource | CreateRequest | Resource | 400, 409, 500 | - -##### Event Contract -- Published events: -- Subscribed events: -- Ordering / delivery guarantees: - -##### Batch / Job Contract -- Trigger: -- Input / validation: -- Output / destination: -- Idempotency & recovery: - -##### State Management -- State model: -- Persistence & consistency: -- Concurrency strategy: - -**Implementation Notes** -- Integration: -- Validation: -- Risks: - -## Data Models - -Focus on the portions of the data landscape that change with this feature. - -### Domain Model -- Aggregates and transactional boundaries -- Entities, value objects, domain events -- Business rules & invariants -- Optional Mermaid diagram for complex relationships - -### Logical Data Model - -**Structure Definition**: -- Entity relationships and cardinality -- Attributes and their types -- Natural keys and identifiers -- Referential integrity rules - -**Consistency & Integrity**: -- Transaction boundaries -- Cascading rules -- Temporal aspects (versioning, audit) - -### Physical Data Model -**When to include**: When implementation requires specific storage design decisions - -**For Relational Databases**: -- Table definitions with data types -- Primary/foreign keys and constraints -- Indexes and performance optimizations -- Partitioning strategy for scale - -**For Document Stores**: -- Collection structures -- Embedding vs referencing decisions -- Sharding key design -- Index definitions - -**For Event Stores**: -- Event schema definitions -- Stream aggregation strategies -- Snapshot policies -- Projection definitions - -**For Key-Value/Wide-Column Stores**: -- Key design patterns -- Column families or value structures -- TTL and compaction strategies - -### Data Contracts & Integration - -**API Data Transfer** -- Request/response schemas -- Validation rules -- Serialization format (JSON, Protobuf, etc.) - -**Event Schemas** -- Published event structures -- Schema versioning strategy -- Backward/forward compatibility rules - -**Cross-Service Data Management** -- Distributed transaction patterns (Saga, 2PC) -- Data synchronization strategies -- Eventual consistency handling - -Skip subsections that are not relevant to this feature. - -## Error Handling - -### Error Strategy -Concrete error handling patterns and recovery mechanisms for each error type. - -### Error Categories and Responses -**User Errors** (4xx): Invalid input → field-level validation; Unauthorized → auth guidance; Not found → navigation help -**System Errors** (5xx): Infrastructure failures → graceful degradation; Timeouts → circuit breakers; Exhaustion → rate limiting -**Business Logic Errors** (422): Rule violations → condition explanations; State conflicts → transition guidance - -**Process Flow Visualization** (when complex business logic exists): -Include Mermaid flowchart only for complex error scenarios with business workflows. - -### Monitoring -Error tracking, logging, and health monitoring implementation. - -## Testing Strategy - -### Default sections (adapt names/sections to fit the domain) -- Unit Tests: 3–5 items from core functions/modules (e.g., auth methods, subscription logic) -- Integration Tests: 3–5 cross-component flows (e.g., webhook handling, notifications) -- E2E/UI Tests (if applicable): 3–5 critical user paths (e.g., forms, dashboards) -- Performance/Load (if applicable): 3–4 items (e.g., concurrency, high-volume ops) - -## Optional Sections (include when relevant) - -### Security Considerations -_Use this section for features handling auth, sensitive data, external integrations, or user permissions. Capture only decisions unique to this feature; defer baseline controls to steering docs._ -- Threat modeling, security controls, compliance requirements -- Authentication and authorization patterns -- Data protection and privacy considerations - -### Performance & Scalability -_Use this section when performance targets, high load, or scaling concerns exist. Record only feature-specific targets or trade-offs and rely on steering documents for general practices._ -- Target metrics and measurement strategies -- Scaling approaches (horizontal/vertical) -- Caching strategies and optimization techniques - -### Migration Strategy -Include a Mermaid flowchart showing migration phases when schema/data movement is required. -- Phase breakdown, rollback triggers, validation checkpoints - -## Supporting References (Optional) -- Create this section only when keeping the information in the main body would hurt readability (e.g., very long TypeScript definitions, vendor option matrices, exhaustive schema tables). Keep decision-making context in the main sections so the design stays self-contained. -- Link to the supporting references from the main text instead of inlining large snippets. -- Background research notes and comparisons continue to live in `research.md`, but their conclusions must be summarized in the main design. diff --git a/.kiro/settings/templates/specs/init.json b/.kiro/settings/templates/specs/init.json deleted file mode 100644 index 410b1cd5..00000000 --- a/.kiro/settings/templates/specs/init.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "{{FEATURE_NAME}}", - "created_at": "{{TIMESTAMP}}", - "updated_at": "{{TIMESTAMP}}", - "language": "en", - "phase": "initialized", - "approvals": { - "requirements": { - "generated": false, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false -} - - diff --git a/.kiro/settings/templates/specs/requirements-init.md b/.kiro/settings/templates/specs/requirements-init.md deleted file mode 100644 index edde05de..00000000 --- a/.kiro/settings/templates/specs/requirements-init.md +++ /dev/null @@ -1,9 +0,0 @@ -# Requirements Document - -## Project Description (Input) -{{PROJECT_DESCRIPTION}} - -## Requirements - - - diff --git a/.kiro/settings/templates/specs/requirements.md b/.kiro/settings/templates/specs/requirements.md deleted file mode 100644 index 42424014..00000000 --- a/.kiro/settings/templates/specs/requirements.md +++ /dev/null @@ -1,32 +0,0 @@ -# Requirements Document - -## Introduction -{{INTRODUCTION}} - - -## Boundary Context (Optional) -- **In scope**: {{IN_SCOPE_BEHAVIORS}} -- **Out of scope**: {{OUT_OF_SCOPE_BEHAVIORS}} -- **Adjacent expectations**: {{ADJACENT_SYSTEM_OR_SPEC_EXPECTATIONS}} - -## Requirements - -### Requirement 1: {{REQUIREMENT_AREA_1}} - -**Objective:** As a {{ROLE}}, I want {{CAPABILITY}}, so that {{BENEFIT}} - -#### Acceptance Criteria -1. When [event], the [system] shall [response/action] -2. If [trigger], then the [system] shall [response/action] -3. While [precondition], the [system] shall [response/action] -4. Where [feature is included], the [system] shall [response/action] -5. The [system] shall [response/action] - -### Requirement 2: {{REQUIREMENT_AREA_2}} -**Objective:** As a {{ROLE}}, I want {{CAPABILITY}}, so that {{BENEFIT}} - -#### Acceptance Criteria -1. When [event], the [system] shall [response/action] -2. When [event] and [condition], the [system] shall [response/action] - - diff --git a/.kiro/settings/templates/specs/research.md b/.kiro/settings/templates/specs/research.md deleted file mode 100644 index b7c32906..00000000 --- a/.kiro/settings/templates/specs/research.md +++ /dev/null @@ -1,61 +0,0 @@ -# Research & Design Decisions Template - ---- -**Purpose**: Capture discovery findings, architectural investigations, and rationale that inform the technical design. - -**Usage**: -- Log research activities and outcomes during the discovery phase. -- Document design decision trade-offs that are too detailed for `design.md`. -- Provide references and evidence for future audits or reuse. ---- - -## Summary -- **Feature**: `` -- **Discovery Scope**: New Feature / Extension / Simple Addition / Complex Integration -- **Key Findings**: - - Finding 1 - - Finding 2 - - Finding 3 - -## Research Log -Document notable investigation steps and their outcomes. Group entries by topic for readability. - -### [Topic or Question] -- **Context**: What triggered this investigation? -- **Sources Consulted**: Links, documentation, API references, benchmarks -- **Findings**: Concise bullet points summarizing the insights -- **Implications**: How this affects architecture, contracts, or implementation - -_Repeat the subsection for each major topic._ - -## Architecture Pattern Evaluation -List candidate patterns or approaches that were considered. Use the table format where helpful. - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Hexagonal | Ports & adapters abstraction around core domain | Clear boundaries, testable core | Requires adapter layer build-out | Aligns with existing steering principle X | - -## Design Decisions -Record major decisions that influence `design.md`. Focus on choices with significant trade-offs. - -### Decision: `` -- **Context**: Problem or requirement driving the decision -- **Alternatives Considered**: - 1. Option A — short description - 2. Option B — short description -- **Selected Approach**: What was chosen and how it works -- **Rationale**: Why this approach fits the current project context -- **Trade-offs**: Benefits vs. compromises -- **Follow-up**: Items to verify during implementation or testing - -_Repeat the subsection for each decision._ - -## Risks & Mitigations -- Risk 1 — Proposed mitigation -- Risk 2 — Proposed mitigation -- Risk 3 — Proposed mitigation - -## References -Provide canonical links and citations (official docs, standards, ADRs, internal guidelines). -- [Title](https://example.com) — brief note on relevance -- ... diff --git a/.kiro/settings/templates/specs/tasks.md b/.kiro/settings/templates/specs/tasks.md deleted file mode 100644 index 98298741..00000000 --- a/.kiro/settings/templates/specs/tasks.md +++ /dev/null @@ -1,24 +0,0 @@ -# Implementation Plan - -## Task Format Template - -Use whichever pattern fits the work breakdown: - -### Major task only -- [ ] {{NUMBER}}. {{TASK_DESCRIPTION}}{{PARALLEL_MARK}} - - {{DETAIL_ITEM_1}} *(Include details only when needed. If the task stands alone, omit bullet items.)* - - _Requirements: {{REQUIREMENT_IDS}}_ - -### Major + Sub-task structure -- [ ] {{MAJOR_NUMBER}}. {{MAJOR_TASK_SUMMARY}} -- [ ] {{MAJOR_NUMBER}}.{{SUB_NUMBER}} {{SUB_TASK_DESCRIPTION}}{{SUB_PARALLEL_MARK}} - - {{DETAIL_ITEM_1}} - - {{DETAIL_ITEM_2}} - - {{OBSERVABLE_COMPLETION_ITEM}} *(At least one detail item should state the observable completion condition for this task.)* - - _Requirements: {{REQUIREMENT_IDS}}_ *(IDs only; do not add descriptions or parentheses.)* - - _Boundary: {{COMPONENT_NAMES}}_ *(Only for (P) tasks. Omit when scope is obvious.)* - - _Depends: {{TASK_IDS}}_ *(Only for non-obvious cross-boundary dependencies. Most tasks omit this.)* - -> **Parallel marker**: Append ` (P)` only to tasks that can be executed in parallel. Omit the marker when running in `--sequential` mode. -> -> **Optional test coverage**: When a sub-task is deferrable test work tied to acceptance criteria, mark the checkbox as `- [ ]*` and explain the referenced requirements in the detail bullets. diff --git a/.kiro/settings/templates/steering-custom/api-standards.md b/.kiro/settings/templates/steering-custom/api-standards.md deleted file mode 100644 index acf6d4e3..00000000 --- a/.kiro/settings/templates/steering-custom/api-standards.md +++ /dev/null @@ -1,69 +0,0 @@ -# API Standards - -[Purpose: consistent API patterns for naming, structure, auth, versioning, and errors] - -## Philosophy -- Prefer predictable, resource-oriented design -- Be explicit in contracts; minimize breaking changes -- Secure by default (auth first, least privilege) - -## Endpoint Pattern -``` -/{version}/{resource}[/{id}][/{sub-resource}] -``` -Examples: -- `/api/v1/users` -- `/api/v1/users/:id` -- `/api/v1/users/:id/posts` - -HTTP verbs: -- GET (read, safe, idempotent) -- POST (create) -- PUT/PATCH (update) -- DELETE (remove, idempotent) - -## Request/Response - -Request (typical): -```json -{ "data": { ... }, "metadata": { "requestId": "..." } } -``` - -Success: -```json -{ "data": { ... }, "meta": { "timestamp": "...", "version": "..." } } -``` - -Error: -```json -{ "error": { "code": "ERROR_CODE", "message": "...", "field": "optional" } } -``` -(See error-handling for rules.) - -## Status Codes (pattern) -- 2xx: Success (200 read, 201 create, 204 delete) -- 4xx: Client issues (400 validation, 401/403 auth, 404 missing) -- 5xx: Server issues (500 generic, 503 unavailable) -Choose the status that best reflects the outcome. - -## Authentication -- Credentials in standard location -``` -Authorization: Bearer {token} -``` -- Reject unauthenticated before business logic - -## Versioning -- Version via URL/header/media-type -- Breaking change → new version -- Non-breaking → same version -- Provide deprecation window and comms - -## Pagination/Filtering (if applicable) -- Pagination: `page`, `pageSize` or cursor-based -- Filtering: explicit query params -- Sorting: `sort=field:asc|desc` -Return pagination metadata in `meta`. - ---- -_Focus on patterns and decisions, not endpoint catalogs._ diff --git a/.kiro/settings/templates/steering-custom/authentication.md b/.kiro/settings/templates/steering-custom/authentication.md deleted file mode 100644 index e6afadfc..00000000 --- a/.kiro/settings/templates/steering-custom/authentication.md +++ /dev/null @@ -1,67 +0,0 @@ -# Authentication & Authorization Standards - -[Purpose: unify auth model, token/session lifecycle, permission checks, and security] - -## Philosophy -- Clear separation: authentication (who) vs authorization (what) -- Secure by default: least privilege, fail closed, short-lived tokens -- UX-aware: friction where risk is high, smooth otherwise - -## Authentication - -### Method (choose + rationale) -- Options: JWT, Session, OAuth2, hybrid -- Choice: [our method] because [reason] - -### Flow (high-level) -``` -1) User proves identity (credentials or provider) -2) Server verifies and issues token/session -3) Client sends token per request -4) Server verifies token and proceeds -``` - -### Token/Session Lifecycle -- Storage: httpOnly cookie or Authorization header -- Expiration: short-lived access, longer refresh (if used) -- Refresh: rotate tokens; respect revocation -- Revocation: blacklist/rotate on logout/compromise - -### Security Pattern -- Enforce TLS; never expose tokens to JS when avoidable -- Bind token to audience/issuer; include minimal claims -- Consider device binding and IP/risk checks for sensitive actions - -## Authorization - -### Permission Model -- Choose one: RBAC / ABAC / ownership-based / hybrid -- Define roles/attributes centrally; avoid hardcoding across codebase - -### Checks (where to enforce) -- Route/middleware: coarse-grained gate -- Domain/service: fine-grained decisions -- UI: conditional rendering (no security reliance) - -Example pattern: -```typescript -requirePermission('resource:action'); // route -if (!user.can('resource:action')) throw ForbiddenError(); // domain -``` - -### Ownership -- Pattern: owner OR privileged role can act -- Verify on entity boundary before mutation - -## Passwords & MFA -- Passwords: strong policy, hashed (bcrypt/argon2), never plaintext -- Reset: time-limited token, single-use, notify user -- MFA: step-up for risky operations (policy-driven) - -## API-to-API Auth -- Use API keys or OAuth client credentials -- Scope keys minimally; rotate and audit usage -- Rate limit by identity (user/key) - ---- -_Focus on patterns and decisions. No library-specific code._ diff --git a/.kiro/settings/templates/steering-custom/database.md b/.kiro/settings/templates/steering-custom/database.md deleted file mode 100644 index bfd7c592..00000000 --- a/.kiro/settings/templates/steering-custom/database.md +++ /dev/null @@ -1,46 +0,0 @@ -# Database Standards - -[Purpose: guide schema design, queries, migrations, and integrity] - -## Philosophy -- Model the domain first; optimize after correctness -- Prefer explicit constraints; let database enforce invariants -- Query only what you need; measure before optimizing - -## Naming & Types -- Tables: `snake_case`, plural (`users`, `order_items`) -- Columns: `snake_case` (`created_at`, `user_id`) -- FKs: `{table}_id` referencing `{table}.id` -- Types: timezone-aware timestamps; strong IDs; precise money types - -## Relationships -- 1:N: FK in child -- N:N: join table with compound key -- 1:1: FK + UNIQUE - -## Migrations -- Immutable migrations; always add rollback -- Small, focused steps; test on non-prod first -- Naming: `{seq}_{action}_{object}` (e.g., `002_add_email_index`) - -## Query Patterns -- ORM for simple CRUD and safety; raw SQL for complex/perf-critical -- Avoid N+1 (eager load/batching); paginate large sets -- Index FKs and frequently filtered/sorted columns - -## Connection & Transactions -- Use pooling (size/timeouts based on workload) -- One connection per unit of work; close/return promptly -- Wrap multi-step changes in transactions - -## Data Integrity -- Use NOT NULL/UNIQUE/CHECK/FK constraints -- Validate at DB when appropriate (defense in depth) -- Prefer generated columns for consistent derivations - -## Backup & Recovery -- Regular backups with retention; test restores -- Document RPO/RTO targets; monitor backup jobs - ---- -_Focus on patterns and decisions. No environment-specific settings._ diff --git a/.kiro/settings/templates/steering-custom/deployment.md b/.kiro/settings/templates/steering-custom/deployment.md deleted file mode 100644 index 0d97a0bd..00000000 --- a/.kiro/settings/templates/steering-custom/deployment.md +++ /dev/null @@ -1,54 +0,0 @@ -# Deployment Standards - -[Purpose: safe, repeatable releases with clear environment and pipeline patterns] - -## Philosophy -- Automate; test before deploy; verify after deploy -- Prefer incremental rollout with fast rollback -- Production changes must be observable and reversible - -## Environments -- Dev: fast iteration; debugging enabled -- Staging: mirrors prod; release validation -- Prod: hardened; monitored; least privilege - -## CI/CD Flow -``` -Code → Test → Build → Scan → Deploy (staged) → Verify -``` -Principles: -- Fail fast on tests/scans; block deploy -- Artifact builds are reproducible (lockfiles, pinned versions) -- Manual approval for prod; auditable trail - -## Deployment Strategies -- Rolling: gradual instance replacement -- Blue-Green: switch traffic between two pools -- Canary: small % users first, expand on health -Choose per risk profile; document default. - -## Zero-Downtime & Migrations -- Health checks gate traffic; graceful shutdown -- Backwards-compatible DB changes during rollout -- Separate migration step; test rollback paths - -## Rollback -- Keep previous version ready; automate revert -- Rollback faster than fix-forward; document triggers - -## Configuration & Secrets -- 12-factor config via env; never commit secrets -- Secret manager; rotate; least privilege; audit access -- Validate required env vars at startup - -## Health & Monitoring -- Endpoints: `/health`, `/health/live`, `/health/ready` -- Monitor latency, error rate, throughput, saturation -- Alerts on SLO breaches/spikes; tune to avoid fatigue - -## Incident Response & DR -- Standard playbook: detect → assess → mitigate → communicate → resolve → post-mortem -- Backups with retention; test restore; defined RPO/RTO - ---- -_Focus on rollout patterns and safeguards. No provider-specific steps._ diff --git a/.kiro/settings/templates/steering-custom/error-handling.md b/.kiro/settings/templates/steering-custom/error-handling.md deleted file mode 100644 index 2d534398..00000000 --- a/.kiro/settings/templates/steering-custom/error-handling.md +++ /dev/null @@ -1,59 +0,0 @@ -# Error Handling Standards - -[Purpose: unify how errors are classified, shaped, propagated, logged, and monitored] - -## Philosophy -- Fail fast where possible; degrade gracefully at system boundaries -- Consistent error shape across the stack (human + machine readable) -- Handle known errors close to source; surface unknowns to a global handler - -## Classification (decide handling by source) -- Client: Input/validation/user action issues → 4xx -- Server: System failures/unexpected exceptions → 5xx -- Business: Rule/state violations → 4xx (e.g., 409) -- External: 3rd-party/network failures → map to 5xx or 4xx with context - -## Error Shape (single canonical format) -```json -{ - "error": { - "code": "ERROR_CODE", - "message": "Human-readable message", - "requestId": "trace-id", - "timestamp": "ISO-8601" - } -} -``` -Principles: stable code enums, no secrets, include trace info. - -## Propagation (where to convert) -- API layer: Convert domain errors → HTTP status + canonical body -- Service layer: Throw typed business errors, avoid stringly-typed errors -- Data/external layer: Wrap provider errors with safe, actionable codes -- Unknown errors: Bubble to global handler → 500 + generic message - -Example pattern: -```typescript -try { return await useCase(); } -catch (e) { - if (e instanceof BusinessError) return respondMapped(e); - logError(e); return respondInternal(); -} -``` - -## Logging (context over noise) -Log: operation, userId (if available), code, message, stack, requestId, minimal context. -Do not log: passwords, tokens, secrets, full PII, full bodies with sensitive data. -Levels: ERROR (failures), WARN (recoverable/edge), INFO (key events), DEBUG (diagnostics). - -## Retry (only when safe) -Retry when: network/timeouts/transient 5xx AND operation is idempotent. -Do not retry: 4xx, business errors, non-idempotent flows. -Strategy: exponential backoff + jitter, capped attempts; require idempotency keys. - -## Monitoring & Health -Track: error rates by code/category, latency, saturation; alert on spikes/SLI breaches. -Expose health: `/health` (live), `/health/ready` (ready). Link errors to traces. - ---- -_Focus on patterns and decisions. No implementation details or exhaustive lists._ diff --git a/.kiro/settings/templates/steering-custom/security.md b/.kiro/settings/templates/steering-custom/security.md deleted file mode 100644 index aca172a3..00000000 --- a/.kiro/settings/templates/steering-custom/security.md +++ /dev/null @@ -1,55 +0,0 @@ -# Security Standards - -[Purpose: define security posture with patterns for validation, authz, secrets, and data] - -## Philosophy -- Defense in depth; least privilege; secure by default; fail closed -- Validate at boundaries; sanitize for context; never trust input -- Separate authentication (who) and authorization (what) - -## Input & Output -- Validate at API boundaries and UI forms; enforce types and constraints -- Sanitize/escape based on destination (HTML, SQL, shell, logs) -- Prefer allow-lists over block-lists; reject early with minimal detail - -## Authentication & Authorization -- Authentication: verify identity; issue short-lived tokens/sessions -- Authorization: check permissions before actions; deny by default -- Centralize policies; avoid duplicating checks across code - -Pattern: -```typescript -if (!user.hasPermission('resource:action')) throw ForbiddenError(); -``` - -## Secrets & Configuration -- Never commit secrets; store in secret manager or env -- Rotate regularly; audit access; scope minimal -- Validate required env vars at startup; fail fast on missing - -## Sensitive Data -- Minimize collection; mask/redact in logs; encrypt at rest and in transit -- Restrict access by role/need-to-know; track access to sensitive records - -## Session/Token Security -- httpOnly + secure cookies where possible; TLS everywhere -- Short expiration; rotate on refresh; revoke on logout/compromise -- Bind tokens to audience/issuer; include minimal claims - -## Logging (security-aware) -- Log auth attempts, permission denials, and sensitive operations -- Never log passwords, tokens, secrets, full PII; avoid full bodies -- Include requestId and context to correlate events - -## Headers & Transport -- Enforce TLS; HSTS -- Set security headers (CSP, X-Frame-Options, X-Content-Type-Options) -- Prefer modern crypto; disable weak protocols/ciphers - -## Vulnerability Posture -- Prefer secure libraries; keep dependencies updated -- Static/dynamic scans in CI; track and remediate -- Educate team on common classes; encode as patterns above - ---- -_Focus on patterns and principles. Link concrete configs to ops docs._ diff --git a/.kiro/settings/templates/steering-custom/testing.md b/.kiro/settings/templates/steering-custom/testing.md deleted file mode 100644 index f4e22fe9..00000000 --- a/.kiro/settings/templates/steering-custom/testing.md +++ /dev/null @@ -1,47 +0,0 @@ -# Testing Standards - -[Purpose: guide what to test, where tests live, and how to structure them] - -## Philosophy -- Test behavior, not implementation -- Prefer fast, reliable tests; minimize brittle mocks -- Cover critical paths deeply; breadth over 100% pursuit - -## Organization -Options: -- Co-located: `component.tsx` + `component.test.tsx` -- Separate: `/src/...` and `/tests/...` -Pick one as default; allow exceptions with rationale. - -Naming: -- Files: `*.test.*` or `*.spec.*` -- Suites: what is under test; Cases: expected behavior - -## Test Types -- Unit: single unit, mocked dependencies, very fast -- Integration: multiple units together, mock externals only -- E2E: full flows, minimal mocks, only for critical journeys - -## Structure (AAA) -```typescript -it('does X when Y', () => { - // Arrange - const input = setup(); - // Act - const result = act(input); - // Assert - expect(result).toEqual(expected); -}); -``` - -## Mocking & Data -- Mock externals (API/DB); never mock the system under test -- Use factories/fixtures; reset state between tests -- Keep test data minimal and intention-revealing - -## Coverage -- Target: [% overall]; higher for critical domains -- Enforce thresholds in CI; exceptions require review rationale - ---- -_Focus on patterns and decisions. Tool-specific config lives elsewhere._ diff --git a/.kiro/settings/templates/steering/product.md b/.kiro/settings/templates/steering/product.md deleted file mode 100644 index a0f43fa7..00000000 --- a/.kiro/settings/templates/steering/product.md +++ /dev/null @@ -1,18 +0,0 @@ -# Product Overview - -[Brief description of what this product does and who it serves] - -## Core Capabilities - -[3-5 key capabilities, not exhaustive features] - -## Target Use Cases - -[Primary scenarios this product addresses] - -## Value Proposition - -[What makes this product unique or valuable] - ---- -_Focus on patterns and purpose, not exhaustive feature lists_ diff --git a/.kiro/settings/templates/steering/structure.md b/.kiro/settings/templates/steering/structure.md deleted file mode 100644 index 01162476..00000000 --- a/.kiro/settings/templates/steering/structure.md +++ /dev/null @@ -1,41 +0,0 @@ -# Project Structure - -## Organization Philosophy - -[Describe approach: feature-first, layered, domain-driven, etc.] - -## Directory Patterns - -### [Pattern Name] -**Location**: `/path/` -**Purpose**: [What belongs here] -**Example**: [Brief example] - -### [Pattern Name] -**Location**: `/path/` -**Purpose**: [What belongs here] -**Example**: [Brief example] - -## Naming Conventions - -- **Files**: [Pattern, e.g., PascalCase, kebab-case] -- **Components**: [Pattern] -- **Functions**: [Pattern] - -## Import Organization - -```typescript -// Example import patterns -import { Something } from '@/path' // Absolute -import { Local } from './local' // Relative -``` - -**Path Aliases**: -- `@/`: [Maps to] - -## Code Organization Principles - -[Key architectural patterns and dependency rules] - ---- -_Document patterns, not file trees. New files following patterns shouldn't require updates_ diff --git a/.kiro/settings/templates/steering/tech.md b/.kiro/settings/templates/steering/tech.md deleted file mode 100644 index c7b25b40..00000000 --- a/.kiro/settings/templates/steering/tech.md +++ /dev/null @@ -1,45 +0,0 @@ -# Technology Stack - -## Architecture - -[High-level system design approach] - -## Core Technologies - -- **Language**: [e.g., TypeScript, Python] -- **Framework**: [e.g., React, Next.js, Django] -- **Runtime**: [e.g., Node.js 20+] - -## Key Libraries - -[Only major libraries that influence development patterns] - -## Development Standards - -### Type Safety -[e.g., TypeScript strict mode, no `any`] - -### Code Quality -[e.g., ESLint, Prettier rules] - -### Testing -[e.g., Jest, coverage requirements] - -## Development Environment - -### Required Tools -[Key tools and version requirements] - -### Common Commands -```bash -# Dev: [command] -# Build: [command] -# Test: [command] -``` - -## Key Technical Decisions - -[Important architectural choices and rationale] - ---- -_Document standards and patterns, not every dependency_ diff --git a/.kiro/specs/.gitignore b/.kiro/specs/.gitignore deleted file mode 100644 index 4c5f2067..00000000 --- a/.kiro/specs/.gitignore +++ /dev/null @@ -1 +0,0 @@ -.claude/ diff --git a/.kiro/specs/admin-config-tool/brief.md b/.kiro/specs/admin-config-tool/brief.md deleted file mode 100644 index 221f797b..00000000 --- a/.kiro/specs/admin-config-tool/brief.md +++ /dev/null @@ -1,613 +0,0 @@ -# Brief: admin-config-tool - -## Problem - -XCE operators have no tool to build and deploy cluster configuration. Currently: - -- **Config authoring**: YAML config files (`configs/hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`, `limits.yaml`, `xce.yaml`, `scaler.yaml`) exist as documentation examples with JSON schemas for validation (`configs/schema/*.schema.json`), but **no tool compiles them** into the `ClusterConfig` Cap'n Proto binary that the controller expects. -- **Config deployment**: `OP_upgradeConfig` exists on the controller side (`config_repl.c:1591` `do_upgrade_config_with_bytes()` — receives binary blob, verifies SHA-256, saves, pushes to standbys, awaits majority ACK, signals restart), but **no CLI sends it**. Today: manual scp + restart. - -End users use `xcecli` (Go/Cobra → REST → apiserver → CGO → controller) for non-admin operations. There is no admin-facing tool. - -**Who**: XCE cluster operators. **Pain**: manual config management, error-prone YAML→binary translation, no primary auto-discovery. - -## Current State - -| What | Where | Status | -|------|-------|--------| -| `ClusterConfig` capnp struct | `src/schema/controller.capnp:694-708` | 13 fields (@0-@12), wire-ready — down from 21 after `clusterconfig-schema-split` removed `port`/`candidates`/hash-size/`xstrings` | -| YAML config examples | `configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml` | Human-authored, NOT loaded by any code | -| JSON schemas | `configs/schema/*.schema.json` | JSON Schema 2020-12 | -| `OP_upgradeConfig` handler | `config_repl.c:1591` `do_upgrade_config_with_bytes()` | Receives binary, verifies, pushes, awaits majority | -| `UpgradeConfigRequest/Reply` | `controller.capnp:792-797` | Defined in schema | -| libxce upgrade API | `request.c:1046` has extractors only | No send function | -| Controller wire protocol | TCP+TLS, capnp-encoded `Request`/`Reply`, same CA as controller-to-controller | Exists, used by all C daemons | - -**Key gap**: YAML → ClusterConfig binary tool does not exist. Upload CLI does not exist. - -## Desired Outcome - -Single `xceadmin` Python CLI that operators use for cluster configuration management: - -``` -# Build: auto-discover YAML files + schemas from standard directory layout -xceadmin config build configs/ # default output: cluster.bin -xceadmin config build configs/ -o cluster.bin # explicit output -xceadmin config build configs/ -o - # stdout (for piping) - -# Push: zero-config dev (same as xce_test), single --controller for production -xceadmin config push cluster.bin # default: localhost:9991 -xceadmin config push --controller ctrl-1:9991 cluster.bin # production: any one controller -``` - -**Standard config directory layout** (auto-discovered): -``` -configs/ -├── hosts.yaml # optional — missing = empty hosts section -├── queues.yaml # optional -├── resources.yaml # optional -├── users.yaml # optional -├── limits.yaml # optional -├── xce.yaml # optional — missing = clusterName/heartbeat/etc. left unset -└── scaler.yaml # optional -``` -No `schema/` subdirectory — **RESOLVED (2026-07-21)**: the 7 JSON schemas are bundled with the `xceadmin` package itself (`xcecommon` `package_data`, synced from `src/configs/schema/` at build time), not supplied per `configs_dir`. This mirrors the `controller.capnp` bundling decision and for the same reason: the schemas are the formal spec of what `xcecommon.mapping`'s field-extraction code actually expects, so they need to stay version-locked to the `xceadmin` release doing the mapping, not left for an operator to drift out of sync. `src/configs/schema/*.schema.json` remains in the repo as the source of truth that gets synced — an operator's `configs_dir` never needs its own `schema/` folder, and one present there is not read. - -- **config build**: Scan directory → validate present YAML files against `xceadmin`'s bundled JSON schemas → map to `ClusterConfig` capnp struct → serialize to `cluster.bin` → output SHA-256. Missing files = empty section (not error). No schema directory to require — bundled, not operator-supplied. -- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. `configGeneration` is fully controller-owned: the primary assigns `new_gen = current_gen + 1` on receipt and returns it as `versionTag`; xceadmin never reads, sets, or overrides it — no `--generation` flag. -- **Shared package**: `xcecommon` Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools - -## Approach - -**Chosen: Python CLI with single dependency on `controller.capnp`.** - -``` -src/schema/controller.capnp ← single source of truth - │ - ├──▶ c-capnproto → libxce → controller (C daemon, existing) - │ - └──▶ pycapnp → xceadmin (Python, new) - │ - ├── config build: YAML → ClusterConfig → bytes → cluster.bin - └── config push: Request{OP_upgradeConfig, ...} → TLS → Reply -``` - -- **Language**: Python. YAML parsing trivial (`yaml`), JSON Schema validation (`jsonschema`), capnp encoding (`pycapnp`), CLI (`click` or `argparse`). -- **Protocol**: TCP+TLS directly to controller (NOT through apiserver). Zero-config dev experience — default `localhost:9991`, same as xce_test. Single `--controller` flag for production override. -- **Dependencies**: `controller.capnp` schema file only. Zero CGO, zero libxce. Pure Python, open-source ready. -- **Auto primary detection**: Connect to given controller → controller reply carries peer list + primary address (standby redirect, already in reply schema) → follow redirect to primary → send request. One controller is enough to bootstrap full cluster discovery. Same mechanism xce_test uses -- **No CGO**: Wire protocol is TLS connect → send one capnp `Request` → receive one capnp `Reply` → exit. ~150 lines of Python. No iomgr event loop, no vtask thread pool, no persistent connections — those are for daemons, not one-shot CLI tools. - -### Architecture Decisions (from 2026-07-06 discovery session) - -| Decision | Rationale | -|----------|-----------| -| **Python** over Go | Future open-source release; shared capnp infra with planned Python xcecli rewrite; `pip install` no compilation | -| **Pure Python wire** over CGO+libxce | Open-source friendly; no C compilation; wire protocol is one-shot send/receive (~150 lines) | -| **Direct controller** over apiserver | Admin path needs no RBAC middleware; apiserver may be restarting during config upgrade; controller SSL is same CA | -| **Single `controller.capnp` dependency** | Schema IS the protocol contract; versioned; no extra API surface | -| **Separate `xceadmin` binary** over extending xcecli | `xcecli` = user tool (HTTP → apiserver); `xceadmin` = operator tool (TLS → controller); different audiences, different protocols | -| **Auto primary detection** | Operator shouldn't need to know which controller is primary; connect to any one, discover peers + primary from reply | -| **Config build offline** | YAML→binary is pure data transform; no controller connection needed; cluster.bin can be pre-built, reviewed, version-controlled | - -### xcecli vs xceadmin - -| | xcecli | xceadmin | -|---|--------|----------| -| **Audience** | End users | Operators | -| **Protocol** | HTTP/HTTPS (REST, JSON) | TCP+TLS (capnp binary) | -| **Target** | apiserver | controller (primary) | -| **Auth** | JWT token (login → apiserver) | None — same zero-config TLS as xce_test | -| **Auth middleware** | Full: WhitelistURL→WhitelistIP→RateLimit→JWT→CORS→RBAC | None — direct TCP+TLS, same trust model as controller peers | -| **Use cases** | Submit jobs, list queues, check status | Push config, cluster admin, force snapshot | -| **Language** | Go/Cobra (current), Python (future) | Python | - -## Project Decomposition (from 2026-07-06 discovery) - -This discovery session (Path E — mixed decomposition) identified 3 workstreams: - -| # | Workstream | Type | Language | Dependencies | -|---|-----------|------|----------|--------------| -| 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only | -| 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) | -| 3 | **perf-tuning-extraction** | DIRECT work | doc | None | - -**Dependency flow:** -``` -admin-config-tool (standalone, ships first) - │ - │ config push triggers controller-side path - ▼ -config-upgrade-snapshot (controller: replicate → snapshot → restart → activate) - │ - │ blocked on - ▼ -entity-bearing-snapshots (upstream blocker, tracked in roadmap) -``` - -## Configuration Inclusion Rules - -Decided 2026-07-20. Governs `ClusterConfig`/`cluster.bin` field classification now and for any new field added in the future — apply both rules to every candidate field; excluding on either is sufficient. - -> Promoted to project-wide steering: `.kiro/steering/config-design.md` holds the general rule (reusable for worker/scheduler/apiserver shared-config decisions, not just `ClusterConfig`). This section keeps the `ClusterConfig`-specific field classification; see the steering doc for the rule statement and rationale. - -1. **Independence rule**: if a parameter can change without needing to stay consistent with the rest of the cluster's config (i.e., changing it on one node doesn't require every other node to match), keep it OUT of `cluster.bin`. Example: `port` — each controller's own listen port is local, other nodes don't need to know or match it via config push. -2. **Frequency rule**: if a parameter is expected to change often relative to the rest of cluster config, keep it OUT of `cluster.bin`. Example: `candidates` — adding/removing a standby is a routine operational action; requiring a config build+push for it would make membership changes as heavyweight as a schema change. -3. **Default (both rules fail → include)**: a parameter that must be identical across every controller instance AND changes rarely belongs IN `cluster.bin`. Example: `clusterName` — divergence between nodes is a correctness bug, and it's set once at bootstrap. - -**Correction (2026-07-20, after investigating `clusterconfig-schema-split`)**: the `heartbeatInterval`/`inactiveHeartbeatCount` fields actually govern controller→**worker** liveness (marking a worker unavailable), not the HA/Raft controller-to-controller election heartbeat as originally assumed here — that election heartbeat has no `ClusterConfig` field at all (100% hardcoded, and per a bootstrap-availability constraint it never can have one — see `.kiro/steering/config-design.md` Rule 0). The worker-liveness fields actually *pass* the independence rule (one controller's worker-timeout tolerance doesn't need to match another's) — they stay IN `cluster.bin` by deliberate operational-uniformity override, not because they fail the rules. Verdict unchanged; reasoning corrected. - -**Applying the rules** (reference table for future field additions): - -| Field | Independence rule | Frequency rule | Verdict | -|---|---|---|---| -| `clusterName` | fails (must match cluster-wide) | fails (rarely changes) | IN | -| heartbeat/inactive-count (worker liveness) | **passes** (per-controller worker-timeout tolerance, no cross-node dependency) | — | IN (operational-uniformity override — see correction above) | -| `port` | passes (per-node, no cross-node dependency) | — | OUT | -| `candidates` | passes (adding a standby doesn't require other nodes' config to change) | passes (routine op) | OUT | -| `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash size | passes (perf tuning, local effect only) | — | OUT | -| `xstrings` hash size | passes (perf tuning, local effect only) | — | OUT | -| `limits` | fails (quotas must be enforced identically on every controller) | fails (rarely changes) | IN | -| `fetchJobInterval` | passes (per-controller scheduler-loop tuning, no cross-node dependency) | — | IN (operational-uniformity override, bundled into `xce.yaml` with the other cluster-identity/liveness fields rather than split into a separate file) | -| `netioTimeout` | passes (per-controller network tuning, no cross-node dependency) | — | IN (same override, same file — see `fetchJobInterval` row) | -| `hproviders` (scaler providers) | fails (a queue's scaler reference must resolve identically after failover) | fails (rarely changes) | IN | -| `configGeneration` | N/A — not operator input; controller-assigned on every push | N/A | OUT (different reason: not a config value at all, it's protocol state) | - -When a new field is proposed for `ClusterConfig` in the future, run it through rules 1–2 before adding it to xceadmin's YAML→capnp mapping; document the verdict in this table. - -## Scope - -- **In**: - - `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration`, `port`, and the per-instance perf-tuning fields (manager hash-table sizes, `xstrings`) are left unset — not xceadmin's to set. - - `xceadmin config push`: single controller address (default `localhost:9991`), auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `configGeneration` assignment is entirely controller-side (primary assigns `current + 1`, returns it as `versionTag`) — no client probe, increment, or override flag. - - `xceadmin config push --dry-run`: (nice-to-have) validate controller will accept the config without committing it - - `xcecommon` shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types - - Build system integration: pip-based install (editable for dev, pinned for release), nix dev shell packages, xce-build.sh component target - - Skills integration (see Skill Integration section below): - - `xce-build`: add `xceadmin` component target (Python package build + test) - - `xce-deploy-ha`: replace manual `ha/data/cluster.bin` prerequisite with `xceadmin config build` step; add `--build-config` flag for on-the-fly YAML→binary generation - - `xce-test-ha`: (future) config push validation round in soak loop — deferred until `config-upgrade-snapshot` lands on controller side - - Validation: reject configs using reserved name `"lost_and_found"` for queue/host (defense-in-depth — controller enforces at runtime via FATAL, tool prevents at authoring time) -- **Out**: - - Controller-side config replication/snapshot/restart logic (owned by `config-upgrade-snapshot`) - - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`, upstream blocker) - - Performance tuning / per-instance parameter extraction (separate direct work item) — `port`, manager hash-table-size fields, `xstrings`; these stay local to each controller instance, never in `cluster.bin` - - xcecli Python rewrite (future, separate spec) - - xce-test-ha config push validation round (deferred until `config-upgrade-snapshot` lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective) - -## Skill Integration - -Three existing skills touch `cluster.bin` — the binary config blob that `xceadmin config build` produces. Currently `cluster.bin` is a hand-crafted opaque artifact with no visible provenance. `xceadmin` replaces that with a reproducible build step. - -### xce-build — add `xceadmin` component - -New component target for the Python package: - -``` -/xce-build xceadmin # pip install -e . (editable, for dev) -/xce-build xceadmin release # pip install . (pinned, for production) -``` - -Build path differs from C components: Python, not C. Runs inside nix dev shell (pycapnp needs capnproto C++ runtime from nix). Build output: `xceadmin` CLI entry point + `xcecommon` package importable. - -This is the **blocking** integration — other skills can't use xceadmin until this lands. - -### xce-deploy-ha — generate cluster.bin from YAML - -**Current flow:** -``` -ha/data/cluster.bin (pre-built, manual, opaque) → deploy-ha.sh → controller/ctrl-1/conf/ -``` - -**New flow:** -``` -configs/*.yaml → xceadmin config build → cluster.bin → deploy-ha.sh → ... -``` - -Changes to the skill: - -| What | Before | After | -|------|--------|-------| -| Prerequisites | `cluster.bin` must exist under `ha/data/` | `cluster.bin` built from `configs/*.yaml` via `xceadmin config build` | -| Config build step | None (manual) | `xceadmin config build configs/ -o ha/data/cluster.bin` (auto, or skip if `--use-existing-config`) | -| `--build-config` flag | N/A | New flag: force rebuild cluster.bin from YAML before deploy | - -`deploy-ha.sh` gains an optional precondition: if `cluster.bin` missing or `--build-config` passed, run `xceadmin config build`. Otherwise use existing `cluster.bin` as-is (backward compatible — operators can still hand-craft for custom configs). - -### xce-test-ha — config push validation (future) - -Deferred until `config-upgrade-snapshot` lands on controller side. When ready: - -- **New soak round type**: kill primary, push new config via `xceadmin config push`, verify controller accepted + standbys received -- **Coverage**: exercises `OP_upgradeConfig` → replication → majority ACK path under failover -- **Verdict**: config push failure = FAIL (config consistency is a data-safety property) - -Not scoped into admin-config-tool tasks — belongs to `config-upgrade-snapshot` spec. Listed here for cross-spec visibility. - -### Integration dependency order - -``` -xce-build (+xceadmin) ← THIS SPEC, blocking - │ - ▼ -xce-deploy-ha ← THIS SPEC, immediate after xce-build - │ - ▼ -xce-test-ha ← config-upgrade-snapshot spec (future) - (config push round) blocked on entity-bearing-snapshots -``` - -## Harness Enhancement - -The `xceadmin` tool is Python — first Python component in a C/Go project. This requires harness changes across the development toolchain. - -### xce-build.sh — new `xceadmin` component - -Add a new case to `src/tools/xce-build.sh`: - -```bash -xceadmin) - echo "=== Building xceadmin Python package ===" - pip install -e . --quiet - echo "=== Running xceadmin tests ===" - python -m pytest tests/ -v - ;; -``` - -Python path — no meson setup. `pip install -e .` for dev (editable), `pip install .` for release. Tests via `pytest`, not meson test. Runs inside nix dev shell (capnproto C++ runtime needed for pycapnp compile). The `release` profile uses `pip install .` without `-e`. - -### xce-build SKILL.md — add component + usage note - -- **Usage line**: `Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin` -- **Profile notes**: `xceadmin` ignores meson profiles — it always uses pip. The profile arg affects C components only; xceadmin builds regardless of profile. -- **Examples**: `/xce-build xceadmin` (editable), `/xce-build xceadmin release` (pinned) - -### xce-deploy-ha SKILL.md — config build step - -Changes to the skill document: - -- **Prerequisites**: replace "TLS certs and a `cluster.bin` must exist under `ha/data/`" with "TLS certs must exist under `ha/data/`. `cluster.bin` is built from `configs/*.yaml` via `xceadmin config build` (or use existing if `--use-existing-config`)." -- **New `--build-config` flag**: forces rebuild of `cluster.bin` from YAML before deploy. Without it, uses existing `cluster.bin` as-is (backward compatible). -- **New step** (between resolve paths and stop running): optional `xceadmin config build configs/ -o ha/data/cluster.bin` if `--build-config` passed or `cluster.bin` missing. -- **Troubleshooting**: add "`pycapnp` import error" → run `/xce-build xceadmin` first. - -### settings.local.json — Python permissions - -Add Bash permission allowlist entries for xceadmin development: - -```json -"Bash(pip install *)", -"Bash(python -m pytest *)", -"Bash(xceadmin config build *)", -"Bash(xceadmin config push *)", -"Bash(ruff format *)", -"Bash(ruff check *)" -``` - -### PostToolUse hooks — Python formatting + lint - -Add a ruff format hook for Python file edits (`settings.json`): - -```json -{ - "matcher": "Write|Edit", - "hooks": [ - { - "type": "command", - "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff format --quiet \"$FILE_PATH\" || true", - "timeout": 5 - } - ] -} -``` - -Add a ruff check hook for lint feedback: - -```json -{ - "matcher": "Write|Edit", - "hooks": [ - { - "type": "command", - "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff check --quiet \"$FILE_PATH\" || true", - "timeout": 5 - } - ] -} -``` - -### PreToolUse hook — YAML schema guard (nice-to-have) - -Validate config YAML files against JSON schemas before accepting edits. Prevents commits of invalid config: - -```json -{ - "matcher": "Write|Edit", - "hooks": [ - { - "type": "command", - "command": "[[ \"$FILE_PATH\" == configs/*.yaml ]] && xceadmin config validate \"$FILE_PATH\" || true", - "timeout": 5 - } - ] -} -``` - -This requires `xceadmin config validate` subcommand (nice-to-have: validates a single YAML file against its schema, no binary output). Deferred to a follow-up task after core `config build` lands. - -### `/xce-admin` skill (deferred) - -Thin skill wrapper around `xceadmin config build/push` for agent-driven workflows. Not needed for spec implementation — CLI is sufficient. Deferred until CLI is stable and operator workflows are validated. - -### Harness change summary - -| # | What | Where | Priority | Phase | -|---|------|-------|----------|-------| -| 1 | `xceadmin` case in build script | `src/tools/xce-build.sh` | **Blocking** | Implementation | -| 2 | Component list + pip note | `.claude/skills/xce-build/SKILL.md` | **Blocking** | Implementation | -| 3 | `--build-config` + config build step | `.claude/skills/xce-deploy-ha/SKILL.md` | **Blocking** | Implementation | -| 4 | pip/pytest/xceadmin permissions | `.claude/settings.local.json` | **Blocking** | Implementation | -| 5 | ruff format hook on `.py` edits | `.claude/settings.json` | Nice-to-have | Implementation | -| 6 | YAML schema guard on config edits | `.claude/settings.json` | Nice-to-have | Follow-up | -| 7 | `/xce-admin` skill wrapper | `.claude/skills/xce-admin/SKILL.md` | Deferred | Post-spec | - -## Boundary Candidates - -- **`xceadmin` CLI** — operator-facing commands, argument parsing, output formatting -- **`xcecommon` Python package** — shared capnp schema loading, wire protocol (TLS+capnp), config model types. Reusable by future Python xcecli. -- **`config-build` module** — YAML→ClusterConfig mapping, JSON Schema validation, serialization -- **`config-push` module** — primary auto-detection, TLS+capnp send/receive, result parsing - -## Upstream / Downstream - -- **Upstream**: `controller.capnp` schema (source of truth for ClusterConfig, Request/Reply, OP codes) -- **Downstream**: Controller `OP_upgradeConfig` handler (`config_repl.c` — already exists) -- **Consumed by**: - - `config-upgrade-snapshot` spec — xceadmin is the trigger; controller-side handles replication/snapshot/restart - - `xce-build` skill — adds `xceadmin` component target (blocking integration) - - `xce-deploy-ha` skill — `xceadmin config build` replaces manual `cluster.bin` prerequisite - - `xce-test-ha` skill — (future) config push validation round, deferred until config-upgrade-snapshot lands -- **Future consumer**: Python xcecli rewrite (imports `xcecommon` for capnp schema types) - -## Constraints - -- **Python ≥ 3.10** (match system Python, avoid bleeding-edge features) -- **pycapnp** requires capnproto C++ runtime (`libcapnp-dev`) — separate from project's c-capnproto (C bindings). Both capnp runtimes can coexist; binary format is standardized. -- **Zero-config TLS** — same dev experience as xce_test: default `localhost:9991`, no cert flags needed. Controller peer trust model (X.509 CN validation via `validate_host()` — tracked in roadmap hardening). -- **No dependency on a running apiserver** — admin tool connects directly to controller. Config deployment works even when apiserver is down or restarting. -- **Open-source readiness** — pure Python, `pip install` friendly, no compilation step, clear single-schema dependency. - -## Nix Environment - -`xceadmin` is the first Python component in the XCE project. The nix dev shell currently provides C/C++ tooling only — zero Python packages. All needed Python packages are available in nixpkgs. - -### Packages to add - -Current `src/nixSupport/xce.nix` `nativeBuildInputs` already includes `capnproto` (line 21) — the C++ runtime that pycapnp wraps. Need to add: - -| Package | Nix attr | Purpose | Size | -|---------|----------|---------|------| -| Python 3 | `python3` | Interpreter (≥ 3.10) | ~50MB | -| pycapnp | `python3Packages.pycapnp` | capnp schema → Python types | small | -| PyYAML | `python3Packages.pyyaml` | YAML config parsing | small | -| jsonschema | `python3Packages.jsonschema` | JSON Schema validation | small | -| click | `python3Packages.click` | CLI argument parsing | small | -| pytest | `python3Packages.pytest` | Test framework | small | -| ruff | `python3Packages.ruff` | Python formatter + linter | ~10MB | - -All confirmed available in nixpkgs (unstable): pycapnp 2.2.3, pyyaml 6.0.3, jsonschema 4.26.0, click 8.3.1, pytest 9.0.3, ruff 0.15.20. - -### Where to add - -Two options: - -**Option A: Add to `xce.nix`** (recommended). Every dev gets Python toolchain. Simpler — one flake, one shell. Python deps are ~60MB total, negligible vs current C toolchain. - -**Option B: Separate `xceadmin.nix`**. Lighter shell for C-only devs. More maintenance — two nix files, two shells, cross-component workflows need manual env switching. - -Recommend **Option A** — xceadmin is part of the standard toolchain (like xcecli), not an optional extra. All devs should be able to `xceadmin config build` without switching shells. - -### New `nativeBuildInputs` block (addition to xce.nix) - -```nix -# Python — for xceadmin (admin config tool) -python3 -python3Packages.pycapnp -python3Packages.pyyaml -python3Packages.jsonschema -python3Packages.click -python3Packages.pytest -python3Packages.ruff -``` - -### pycapnp build note - -`pycapnp` wraps the capnproto C++ library at compile time (`pip install` compiles a C extension). Nix's `python3Packages.pycapnp` handles this correctly — it links against nix-provided `capnproto`. No additional `libcapnp-dev` or system package needed. This is one of the advantages of using nix pycapnp over `pip install pycapnp`. - -### CI implications - -- CI already uses nix dev shell — adding packages to `xce.nix` makes them available in CI automatically -- `config build` golden-file tests need pycapnp for capnp encode (can't mock without schema) -- `config push` mock tests need pycapnp + TLS (Python `ssl` stdlib, no extra dep) - -## Testing Strategy - -### config build (offline — pure data transform) - -**Golden-file tests** (primary). Given a `configs/` directory, verify output matches expected: - -``` -tests/ -├── fixtures/ -│ ├── minimal/ # hosts.yaml + queues.yaml only -│ │ ├── configs/ -│ │ │ ├── hosts.yaml -│ │ │ └── queues.yaml # no schema/ — validated against xceadmin's bundled schemas -│ │ └── expected.bin # pre-built golden output -│ ├── full/ # all 7 YAML files -│ └── invalid/ # schema violations → expect error -├── test_config_build.py # pytest: build → compare binary, verify SHA-256 -└── test_schema_validation.py # pytest: each schema violation case -``` - -- Framework: `pytest` (standard Python, no deps beyond dev) -- Zero external dependencies — no controller, no network -- Pre-build golden files once; commit to repo -- Schema validation: test each error case (missing required field, wrong type, invalid enum, reserved name `lost_and_found`) - -### config push (online — needs controller) - -**Option A: Mock controller** (unit/integration). Python TLS server that speaks enough capnp wire protocol to accept `OP_upgradeConfig` and return canned `UpgradeConfigReply`. - -**Option B: Real controller** (E2E). Piggyback on existing `/xce-test-ha` cluster. After cluster is up, `xceadmin config push` a known config and verify controller accepts it. - -- Option A for dev loop + CI (fast, no cluster setup) -- Option B for acceptance (real SSL, real replication, real controller) - -Recommended: Option A for spec tasks. Option B deferred to xce-test-ha integration after config-upgrade-snapshot lands. - -### CI Integration - -- `config build` tests: run in any Python CI (no C deps needed — pycapnp only for encode, can mock) -- `config push` mock tests: need pycapnp + controller.capnp compiled for Python -- Target: 80%+ coverage (project standard) - -## Open Design Questions - -Issues identified during 2026-07-07 review — need resolution before or during requirements generation. - -### Critical (resolve before requirements) - -| # | Question | Why it matters | Candidates | -|---|----------|---------------|------------| -| 1 | **YAML→capnp mapping incomplete** | **RESOLVED — all 13 fields accounted for.** `ClusterConfig` is now 13 fields (@0-@12, post-`clusterconfig-schema-split`), covered by 7 YAML files: `resources`→`resources.yaml`, `queues`→`queues.yaml`, `ugroups`→`users.yaml`, `hgroups`+`hostConfigs`→`hosts.yaml`, `limits`→`limits.yaml` (6 fields, 5 files). `clusterName`/`heartbeatInterval`/`inactiveHeartbeatCount`/`fetchJobInterval`/`netioTimeout`→`xce.yaml` (5 fields, 1 file, Req 1.11). `hproviders`→`scaler.yaml` (1 field, 1 file, Req 1.12). `configGeneration` excluded, controller-assigned (1 field, row 2). 6+5+1+1 = 13. | Closed — no candidates remain | -| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: fully controller-owned. Controller auto-increments `new_gen = current_gen + 1` on every `OP_upgradeConfig` receipt (already implemented, `config_repl.c`) and returns it as `versionTag`. xceadmin never reads, sets, or overrides it — `config build` leaves the field unset, `config push` sends the binary as-is, no `--generation` flag. Same generation as current = no-op is a controller-internal detail, not something xceadmin needs to detect. | ✅ Decided: controller-only, no client involvement | -| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). **RESOLVED — excluded from configuration** (Req 1.10): standby add/remove changes independently and more frequently than the rest of cluster config, so it fails both Configuration Inclusion Rules below. xceadmin never sources it from YAML/CLI, `cluster.bin` leaves it unset. Open sub-question: how peer membership IS sourced/updated instead (today: CLI args to `deploy-ha.sh`) — not a `cluster.bin` field going forward | (a) keep as `deploy-ha.sh` CLI args (status quo, out of xceadmin scope), (b) separate `xceadmin peer add/remove` command (future), (c) controller-side dynamic membership (future, own spec) | -| 4 | **YAML key ≠ capnp field name** | YAML `groupname` → capnp `GroupConfig.name`, YAML `exclude_members` → capnp `exmembers`. Every field needs explicit mapping definition | Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible | -| 5 | **Nested struct defaults** | `QueueConf` has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults | Use controller's `get_default_conf()` values from `myconf.c`; document which fields are operator-facing vs system-defaulted | - -### Important (resolve during design) - -| # | Question | Why it matters | Candidates | -|---|----------|---------------|------------| -| 6 | **Wire framing format** | Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject | Document in design.md; mock controller test catches framing errors | -| 7 | **TLS cert discovery** | ✅ Decided: reuse server-side cert bundle. xceadmin loads `xce.crt`/`xce.key` from `ha/data/` by default (same files `deploy-ha.sh` copies for controller-to-controller TLS), overridable via `--cert-dir`. No separate CA flag, no client cert issuance — same trust model as `xce_test`. See Requirement 2.3/2.4. | -| 8 | **capnp schema compilation** | ✅ **RESOLVED (design phase)**: no `capnp compile -o python` codegen step exists — `pycapnp` loads `controller.capnp` via runtime dynamic reflection (`capnp.load()`, `DynamicStruct`), not static generated code. The only build-time step is a plain file sync (not compilation): `src/schema/{controller,c}.capnp` and `src/configs/schema/*.schema.json` are copied into `xcecommon`'s `package_data` by `xce-build.sh`'s `xceadmin)` case so both schema types survive `pip install .` (non-editable), resolved at runtime via `importlib.resources`. See `design.md` `xcecommon.schema`. | -| 9 | **Python package layout** | Where does code live? Affects pip install, import paths, xce-build.sh | (a) `src/xceadmin/` alongside C code, (b) `tools/xceadmin/` with deploy-ha.sh, (c) top-level `xceadmin/` | -| 10 | **existing cluster.bin provenance** | `ha/data/cluster.bin` is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it | Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs | - -## Configuration Lifecycle & Rollback (Future Roadmap) - -### Source of truth: Git-managed YAML - -YAML config files (`configs/*.yaml`) are the authoritative source of truth. Operators manage them in git — version control, code review, audit trail are all free. `cluster.bin` is a **derived artifact** (like a compiled binary), never committed to git. - -``` -git tag v1.2.0 → YAML at v1.2.0 → xceadmin config build → cluster.bin (deterministic) - │ - xceadmin config push - │ - ▼ - controller cluster -``` - -**Deterministic builds**: same YAML + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` is left unset by `xceadmin config build` and never touched by `xceadmin config push` — the primary controller alone assigns it on receipt (`current + 1`), so it plays no part in build determinism. Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries. - -### What git gives us (free) - -| Capability | How | -|-----------|-----| -| Version history | `git log configs/` | -| Diff between versions | `git diff v1.2.0 v1.3.0 -- configs/` | -| Rollback source | `git checkout v1.2.0 -- configs/ && xceadmin config build` | -| Audit trail | `git blame configs/queues.yaml` | -| Code review | PR against `configs/` branch | - -### What still needs building - -Git manages the **source**. These parts need implementation: - -| # | Capability | What it does | Owned by | -|---|-----------|-------------|----------| -| R1 | **Binary diff** | `xceadmin config diff cluster-v1.bin cluster-v2.bin` — human-readable diff of two built config binaries (decode both → compare field-by-field → pretty-print changes). Useful for pre-push review: "what will this config change actually do?" | deferred (2026-07-21) — no spec owner yet | -| R2 | **Dry-run push** | `xceadmin config push --dry-run cluster.bin` — validate controller will accept the config (SHA check, version check) without committing. Safety net before real push | admin-config-tool | -| R3 | **Config change snapshot** | After config upgrade, take an entity-bearing snapshot of the new configuration state. This anchors the config change in the snapshot chain — without it, rollback has no durable record of what the cluster looked like before/after the change. Controller-side: insert snapshot step into the `OP_upgradeConfig` pipeline (between step 5 "majority ACK" and step 6 "commit WAL_CONFIG_MARKER" in the current handler). | config-upgrade-snapshot | -| R4 | **Controller version history** | Controller tracks last N config versions (tag + SHA + timestamp). `xceadmin config history` queries it. Needed for: "what config is the cluster running?" and "what was running before the last push?" Depends on R3 (snapshot chain provides durable version records). | config-upgrade-snapshot | -| R5 | **Controller-side rollback** | `xceadmin config rollback` (or `xceadmin config push --rollback-to v1.2.0`) — push a previous config version. Controller-side concerns: (a) validate rollback target is safe (no running jobs reference deleted queues/hosts), (b) snapshot before rollback for forward-recovery, (c) bump configGeneration even on rollback (monotonic), (d) restart/activate the rolled-back config | config-upgrade-snapshot | -| R6 | **Rollback safety validation** | Before rolling back, verify: (a) no running jobs use queues being deleted, (b) no active workers on hosts being removed, (c) resource definitions haven't changed in ways that break in-flight allocations. Controller-side check, client-side initiated | config-upgrade-snapshot | -| R7 | **Config activation status** | `xceadmin config status` — show: current config tag, when applied, which nodes have it, any pending restart, snapshot status. Operator visibility into config rollout state | config-upgrade-snapshot | -| R8 | **Config promotion pipeline** | Dev → staging → production config promotion. Same YAML source, different `--peers`/`--cluster-name` flags at build time. Git branches map to environments | admin-config-tool (future) | - -### Controller-side pipeline: current vs target - -``` -CURRENT (config_repl.c — exists today): - xceadmin push → SHA verify → decode → gen++ → save → replicate → WAL marker → restart signal - ↑ - step 8 MISSING -TARGET (after config-upgrade-snapshot): - xceadmin push → SHA verify → decode → gen++ → save → replicate → [SNAPSHOT] → WAL marker → restart signal - ↑ - R3: entity-bearing - snapshot of new config -``` - -The snapshot insertion point matters: it goes AFTER replication (all standbys have the config) but BEFORE the WAL marker commit (so the snapshot is part of the committed state). This ensures a rollback target always has a consistent snapshot to restore from. - -### Rollback workflow (end-to-end vision) - -``` -# 1. Discover what went wrong -xceadmin config diff cluster-current.bin cluster-suspect.bin -# → "queues.normal.priority changed 10→1, queues.express removed" - -# 2. Find the last good version -xceadmin config history -# → v1.2.0 (active), v1.1.0 (previous), v1.0.0 (initial) - -# 3. Dry-run the rollback -xceadmin config build --git-ref v1.1.0 -o cluster-v1.1.0.bin -xceadmin config push --dry-run cluster-v1.1.0.bin -# → "OK: controller will accept this config (generation 3→4)" - -# 4. Execute rollback -xceadmin config push cluster-v1.1.0.bin -# → "Pushed (tag=v1.1.0, SHA=..., acked=3/3). Controller restarting." - -# 5. Verify -xceadmin config status -# → "v1.1.0 active on 3/3 nodes" -``` - -### Rollback safety: what can go wrong - -| Scenario | Risk | Mitigation | -|----------|------|------------| -| Rollback deletes a queue | Running jobs on that queue become unmanageable | R5: controller refuses rollback while jobs reference deleted queues | -| Rollback removes a host | Workers on that host lose config | R5: controller refuses rollback while active workers on removed hosts | -| Rollback changes resource defs | In-flight allocations invalid | Controller fences: drain before rollback, or mark resources "changing" | -| Rollback during network partition | Standbys miss the config push | Majority ACK required (existing OP_upgradeConfig behavior) | -| Rapid rollback→forward→rollback | configGeneration churn, snapshot chain fragmentation | Rate-limit config changes (1 per N seconds); snapshot GC | - -### What's in-scope for admin-config-tool vs deferred - -| Capability | This spec | Why | -|-----------|-----------|-----| -| YAML source in git | ✅ (documentation) | Free — already how configs/ works | -| Deterministic build | ✅ (core) | Same YAML → same binary (generation not set — controller-owned) | -| Binary diff (R1) | ❌ deferred (2026-07-21) | Descoped from admin-config-tool — pure Python/offline, no controller dependency, but no operator has asked for it yet; revisit as follow-up once `config push` sees real-world usage | -| Dry-run push (R2) | ✅ (nice-to-have) | Needs `ProbeConfigTag` RPC (already exists in controller) | -| Config change snapshot (R3) | ❌ deferred | Controller-side — insert entity-bearing snapshot into OP_upgradeConfig pipeline; blocked on entity-bearing-snapshots | -| Version history (R4) | ❌ deferred | Controller-side — depends on R3 snapshot chain | -| Rollback (R5) | ❌ deferred | Controller-side — needs snapshot chain + restart | -| Safety validation (R6) | ❌ deferred | Controller-side — needs job/worker state queries | -| Config status (R7) | ❌ deferred | Controller-side — needs cluster-wide state tracking | -| Promotion pipeline (R8) | ❌ deferred | Wraps git branches + xceadmin build — future enhancement | - -## Existing Spec Touchpoints - -- **Extends**: `config-upgrade-snapshot` (provides the client-side trigger that config-upgrade-snapshot's controller-side path responds to; R3 config-change-snapshot is the missing controller-side step — insert entity-bearing snapshot between replication and WAL marker commit in the `OP_upgradeConfig` pipeline) -- **Adjacent**: `ha-term-safety` (the WAL_CONFIG_MARKER this tool's upload commits anchors on the Raft log via ha-term-safety's term-index infrastructure) -- **Upstream blocker**: `entity-bearing-snapshots` — must land before R3 (config change snapshot) and R4 (version history) can be implemented -- **Blocks**: nothing — xceadmin ships independently; the current controller pipeline (SHA verify → decode → gen++ → save → replicate → WAL marker → restart) already works without the snapshot step -- **Future**: configuration lifecycle (R1–R8 above) — split across admin-config-tool (R2, R8) and config-upgrade-snapshot (R3–R7); R1 (binary diff) deferred, no spec owner yet diff --git a/.kiro/specs/admin-config-tool/design.md b/.kiro/specs/admin-config-tool/design.md deleted file mode 100644 index e69f73b4..00000000 --- a/.kiro/specs/admin-config-tool/design.md +++ /dev/null @@ -1,547 +0,0 @@ -# Technical Design: admin-config-tool - -## Overview - -`xceadmin` is a standalone Python 3.10+ CLI that compiles operator-authored YAML into the `ClusterConfig` Cap'n Proto binary the XCE controller expects (`config build`), and delivers that binary to a running controller cluster over the same TLS+Cap'n Proto wire protocol controllers already speak to each other (`config push`). It is the first Python component in an otherwise C/Go codebase and the first external (non-daemon) client of the controller's raw wire protocol. - -**Users**: XCE cluster operators managing cluster topology, queues, resources, users, limits, autoscaler providers, and cluster-identity settings through git-managed YAML rather than hand-crafted binaries or manual `scp`. - -**Impact**: Today `cluster.bin` is a hand-built, provenance-free artifact and `OP_upgradeConfig` has no external caller. This spec adds the missing YAML→binary compiler and the missing binary→controller uploader, closing both gaps client-side. - -### Goals -- Deterministic YAML → `cluster.bin` compilation covering all 13 `ClusterConfig` fields (9.1–9.3, 1.1–1.12). -- TLS+capnp delivery of `cluster.bin` to a controller, following redirects to the primary (2.1–2.7, 3.1–3.4). -- Fail-fast validation with actionable errors before anything reaches a running cluster (4.1–4.5, 7.1–7.6). -- A reusable `xcecommon` package so a future Python `xcecli` does not re-derive capnp/wire-protocol code (5.1–5.4). -- Toolchain integration: `xce-build`, `xce-deploy-ha`, nix dev shell (8.1–8.6). - -### Non-Goals -- Controller-side replication, snapshot, restart, rollback, version history (owned by `config-upgrade-snapshot`). -- Fixing the controller-side `OP_upgradeConfig` dispatch defect discovered during this design's discovery phase (see Risks) — that is a controller bug, not an xceadmin gap. -- `config diff` — descoped to roadmap follow-up during requirements review (2026-07-21). -- Peer/candidate membership management, per-instance perf tuning (`port`, hash-table sizes, `xstrings`) — physically absent from `ClusterConfig` after `clusterconfig-schema-split`. - -## Boundary Commitments - -### This Spec Owns -- The `xceadmin` CLI (`config build`, `config push`, `config push --dry-run`). -- The `xcecommon` package: capnp schema loading, YAML→capnp field mapping, JSON Schema + business-rule validation, TLS+framing wire client. Both schema types (`controller.capnp` and the 7 JSON schemas) are packaged as bundled copies (`package_data`), synced from `src/schema/` and `src/configs/schema/` at build time — not read from an operator-supplied path at runtime (2026-07-21 revision; superseded Requirement 1.6, see Requirement 5.5). -- Build/toolchain integration: `src/xceadmin/meson.build` (`meson compile`/`nix build .#xcecore[_debug]`), `xce-build.sh` delegating dispatch, nix dev-shell Python packages, `xce-deploy-ha` `--build-config` flag — see File Structure Plan Design Review correction / Amendment. -- Test fixtures and golden files under `src/xceadmin/tests/`. - -### Out of Boundary -- ~~**Controller-side `OP_upgradeConfig` dispatch**~~ — **RESOLVED 2026-07-25 (bug-1468).** `cluster.c`'s `myreq_handler` `OPCode_upgradeConfig` case had its primary/standby guard polarity inverted by an unrelated commit (`fd93997f`, 2026-05-31: `if (ha_is_primary(...)) reject` instead of the correct `if (!ha_is_primary(...)) reject`), which rejected the one role — primary — that `ha_dispatch_gate()` ever lets reach this switch case. Restored to the correct polarity; the primary now reaches `do_upgrade_config_with_bytes()` via the network as designed. Live-verified: `test_e2e_live_push.py::test_live_config_push_against_real_single_node_controller`, `test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path`. -- ~~**Redirect wiring for `OP_upgradeConfig`**~~ — **RESOLVED 2026-08-26 (bug-1700/1701).** `cluster.c`'s generic `ha_dispatch_gate()` rejection path now calls `fill_redirect_reply()` whenever the gate rejects with `-XCE_ERR_STANDBY` (`cluster.c:411-437`) — a fix at the gate itself, covering every opcode that hits that rejection, not an `OPCode_upgradeConfig`-specific patch. `RequestReply.redirectHost`/`redirectPort` are now populated when a standby rejects `config push`, and xceadmin's already-generic redirect-following (Allowed Dependencies) transparently retries against the primary. Live-verified: `test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds`. -- **`ProbeConfigTag` server-side handling** — `xce_probe_config_tag()` (`config_repl.c:1829`) has no server-side dispatch case anywhere in `cluster.c` and no production caller; it is dormant plumbing. xceadmin's dry-run client implements the wire contract but has nothing live to call. -- Entity-bearing snapshots, config rollback, version history, config activation status (owned by `config-upgrade-snapshot`, blocked upstream on `entity-bearing-snapshots`). -- TLS mutual authentication / hostname verification — none exists anywhere in the codebase (`SSL_VERIFY_NONE` is set explicitly for the controller's own outbound client context, `myapp.c:109`; no `X509_check_host` call exists). xceadmin matches this existing posture; it neither introduces nor removes an authentication guarantee. - -### Allowed Dependencies -- `src/schema/controller.capnp` and `src/schema/c.capnp` — schema source of truth, read-only. `xceadmin` packages a synced *copy* (see File Structure Plan / `xcecommon.schema`) rather than referencing this path directly at runtime — `src/schema/` itself stays owned by the C build, not by this spec. -- `src/configs/*.yaml` — the 7 example configs already authored (this session) as operator-facing documentation; xceadmin's test fixtures are separate, purpose-built golden inputs under `src/xceadmin/tests/fixtures/`, not these files. `src/configs/schema/*.schema.json` is different in kind from the YAML examples: it's the source of truth `xcecommon` packages a synced copy of (2026-07-21 revision — see `xcecommon.validation`), the same relationship `src/schema/controller.capnp` already has. -- `xce.crt`/`xce.key` under `ha/data/` (or `--cert-dir`) — same cert bundle the controller cluster already uses. -- Nix `python3Packages`: `pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` (net-new additions to `src/nixSupport/xce.nix`). -- `src/xceadmin/meson.build`'s `custom_target()` as the build-integration point, delegated to by `src/tools/xce-build.sh`'s case-statement dispatch pattern — see Design Review correction / Amendment below. - -### Revalidation Triggers -- `controller.capnp` `ClusterConfig` field renumbering/addition/removal (already happened once this session via `clusterconfig-schema-split`) — invalidates the field-mapping table in Data Models. -- ~~A fix to the `cluster.c`/`ha.c` dispatch-gate contradiction~~ — **TRIGGERED 2026-07-25 (bug-1468).** `config push` E2E now runs against a real cluster (`test_e2e_live_push.py`, `test_e2e_live_ha_push.py`), superseding the "mock controller only" framing in Testing Strategy. -- ~~`fill_redirect_reply()` extended to `OPCode_upgradeConfig`~~ — **TRIGGERED 2026-08-26 (bug-1700/1701).** Closed via a broader, gate-level fix (every opcode hitting the `-XCE_ERR_STANDBY` rejection gets a redirect, not an `OPCode_upgradeConfig`-specific patch) — see Boundary Commitments. Live auto-primary-discovery testing for config push now exists: `test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds`. -- A server-side handler added for `OPCode_probeConfigTag` — unlocks `config push --dry-run` E2E testing. - -## Architecture - -### Existing Architecture Analysis -XCE's daemon-to-daemon wire protocol is hand-rolled: `iomgr`/`vtaskmgr` (epoll loop + thread-pool dispatch) drive a raw-`SSL`-over-TCP channel framed as `uint64_t` size prefix (native/host byte order, **not** network byte order — confirmed at `netio.c:190-199`, `request.c:863-938`, `config_repl.c:1925-1968`, no byte-swap anywhere in the path) followed by that many bytes of a Cap'n Proto `Request`/`Reply` message. There is no Cap'n Proto RPC layer (no vat network) — every exchange is one framed message out, one framed message back. `src/lib/request.c` already contains a minimal one-shot raw-SSL helper pattern (`xce_probe_config_tag`-adjacent code, `request.c:863-938`) that this design mirrors from the Python side. - -No precedent exists for a non-daemon client of this protocol: `src/tools/` has no socket/capnp client, `xce_test` is a WAL/snapshot file-inspection CLI with no network code, and `xcecli` (Go) speaks HTTP/JSON to `apiserver`, never this raw protocol. `xceadmin` is the first. - -### Architecture Pattern & Boundary Map - -**Selected pattern**: layered CLI — thin `click` entry point over a reusable library (`xcecommon`), matching the project's existing `controllers → services → store/backend` inward-dependency convention (`structure.md`) adapted for a one-shot tool (no persistent store; "runtime" is a single TLS exchange or a single file write). - -```mermaid -graph TB - CLI[xceadmin cli] --> Build[xceadmin build] - CLI --> Push[xceadmin push] - Build --> Validation[xcecommon validation] - Build --> Mapping[xcecommon mapping] - Push --> Wire[xcecommon wire] - Push --> Discovery[xceadmin discovery] - Discovery --> Wire - Mapping --> Schema[xcecommon schema] - Wire --> Schema - Validation --> YamlFiles[YAML configs] - Mapping --> ClusterBin[cluster bin file] - Wire --> Controller[XCE controller] -``` - -**Architecture Integration**: -- Selected pattern: layered CLI over a shared library — the library (`xcecommon`) has zero CLI/`click` dependency so a future Python `xcecli` (brief.md Future Roadmap) can import it directly. -- Domain boundaries: `xcecommon` owns protocol/schema/mapping concerns; `xceadmin` owns orchestration and CLI ergonomics (exit codes, stderr/stdout conventions, flag parsing). Neither owns controller-side behavior. -- Existing patterns preserved: framing format, TLS cert bundle reuse, opcode/union shapes are consumed exactly as the C side defines them — no protocol reinterpretation. -- New components rationale: `xcecommon` did not exist (first Python in the repo); every module below is net-new. -- Steering compliance: Cap'n Proto for wire format (`structure.md` — "Cap'n Proto for wire formats"), component lives under `src/<component>/` (`structure.md` — "Organization Philosophy"), fail-fast input validation (`security.md` — "Validate at the API boundary"). - -**Dependency Direction**: `xcecommon.schema` → `{xcecommon.mapping, xcecommon.wire}` → `xcecommon.validation` (independent, no inbound dependency from schema/mapping/wire) → `{xceadmin.build, xceadmin.discovery}` → `xceadmin.push` → `xceadmin.cli`. Each layer imports only from layers to its left; `xcecommon.*` never imports anything from `xceadmin.*` — this is what keeps `xcecommon` importable by a future Python `xcecli` without dragging in `click` or CLI-specific exit-code handling. - -**Parallel-safe implementation boundaries**: within `xcecommon`, `validation.py` has no dependency on `schema.py`/`mapping.py`/`wire.py` and can be implemented independently of them; `mapping.py` and `wire.py` both depend only on `schema.py`, not on each other, so they can be implemented in parallel once `schema.py` is stable. Within `xceadmin`, `build.py` (depends on `xcecommon.{validation,mapping}`) and `discovery.py`+`push.py` (depend on `xcecommon.wire`) touch disjoint `xcecommon` modules and can be implemented in parallel once the underlying `xcecommon` layer is stable. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| CLI | `click` (nixpkgs `python3Packages.click`) | `config build`/`config push`/`config push --dry-run` subcommand groups | Grouped-command idiom: `@click.group()` → `@cli.group()` → `@config.command()` | -| Capnp | `pycapnp` ~2.2.x (nixpkgs) | Load `controller.capnp`, build/encode `ClusterConfig`, decode `Request`/`Reply` | Dynamic-reflection (`DynamicStruct`) API; `$C.mapname(...)` annotations are declared-and-inert, not a parse hazard (verified against `capnp.pyx` source) | -| YAML | `pyyaml` (nixpkgs) | Parse the 7 config files | — | -| Schema validation | `jsonschema` (nixpkgs), Draft 2020-12 | Validate each YAML against `xceadmin`'s bundled JSON schemas (`package_data`, same mechanism as `xcecommon.schema`'s `controller.capnp`) | Source of truth is `src/configs/schema/*.schema.json`; synced into the package at build time, never read from `configs_dir` at runtime (2026-07-21 revision — see `xcecommon.validation`) | -| Transport | Python stdlib `ssl` + `socket` | TLS-wrapped TCP to controller | `ssl.SSLContext` with `check_hostname=False`, `verify_mode=ssl.CERT_NONE` — matches the controller's own outbound posture (`myapp.c:109`), not a weakening | -| Test framework | `pytest` (nixpkgs) | Golden-file + mock-controller tests | Plain byte-equality fixture comparison (see Testing Strategy — `pytest-regressions` considered, not adopted, to avoid an unverified new nix package) | -| Packaging | `pyproject.toml` + `pip install -e .` (dev) / `pip install .` (release) | Editable vs. pinned install | Runs inside nix dev shell so `pycapnp`'s C-extension build links nix-provided `capnproto` | - -## File Structure Plan - -### Directory Structure -``` -src/xceadmin/ -├── pyproject.toml # package metadata; entry point xceadmin = xceadmin.cli:main -│ # Wired into meson via src/xceadmin/meson.build (2026-07-22 -│ # amendment — see Design Review correction below) -├── src/ -│ ├── xcecommon/ # shared library — no click/CLI dependency, reusable by future Python xcecli -│ │ ├── __init__.py -│ │ ├── errors.py # XceAdminError base + all subclasses (single CLI dispatch surface) -│ │ ├── schema_files/ # packaged copies, synced from src/ at build (not hand-edited) -│ │ │ ├── controller.capnp # <- src/schema/controller.capnp -│ │ │ ├── c.capnp # <- src/schema/c.capnp -│ │ │ └── json/ # <- src/configs/schema/*.schema.json (7 files) -│ │ ├── schema.py # capnp.load(controller.capnp) wrapper + typed accessors; resolves schema_files/ via importlib.resources -│ │ ├── mapping.py # declarative YAML-section -> capnp-field table + 2 mapper shapes (list, scalar) -│ │ ├── validation.py # JSON Schema validation (bundled schema_files/json/), reserved-name / dup / undefined-ref checks -│ │ └── wire.py # TLS connect, 8-byte native-order framing, generic send_request() -│ └── xceadmin/ -│ ├── __init__.py -│ ├── cli.py # click group: config build / config push / config push --dry-run -│ ├── build.py # config build orchestration: scan dir -> validate -> map -> serialize -> sha256 -│ ├── push.py # config push orchestration: connect -> discover primary -> send -> display -│ └── discovery.py # generic redirect-following (Req 3) — opcode-agnostic -└── tests/ - ├── fixtures/ - │ ├── minimal/configs/{hosts,queues}.yaml + expected.bin # no schema/ — validated against xceadmin's bundled schemas - │ ├── full/configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml + expected.bin - │ └── invalid/configs/... # one fixture dir per validation-error case (4.1, 4.2, 4.3, 4.4, 4.5) - ├── test_schema_bundling.py # default_schema_path()/default_schema_dir() resolve under the active install (5.5) — only meaningful under `xce-build xceadmin release`, see Testing Strategy - ├── test_mapping.py # pure unit tests: map_list_section/map_scalar_section against in-memory dicts, no file I/O (1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3) — gap closed during task generation, was in Testing Strategy's "Unit — mapping" bullet but missing from this file list - ├── test_config_build.py # golden-file build tests (1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 9.1, 9.2, 9.3) - ├── test_schema_validation.py # per-error-case validation tests (4.1, 4.2, 4.3, 4.4, 4.5) - ├── test_wire_framing.py # framing pack/unpack against known byte sequences (2.1) - └── test_config_push_mock.py # mock TLS server, opcode + redirect + error-reply cases (2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, 3.3, 3.4, 7.1, 7.2, 7.3) -``` - -`xcecommon` and `xceadmin` are separate top-level packages under one `src/xceadmin/src/` tree, installed together via one `pyproject.toml` for now (two console-ready packages, one distributable unit) — splitting into two distributions is deferred until a second consumer (the future Python `xcecli`) actually exists (Simplification: no speculative packaging split). - -### Modified Files -- `src/tools/xce-build.sh` — new `xceadmin)` case: sync `src/schema/{controller,c}.capnp` → `src/xceadmin/src/xcecommon/schema_files/` and `src/configs/schema/*.schema.json` → `src/xceadmin/src/xcecommon/schema_files/json/` (`cp`, source of truth stays `src/schema/` and `src/configs/schema/` respectively — 2026-07-21 revision extends the sync to JSON schemas, Requirement 5.5), then `pip install -e .`/`pip install .` by profile, then `pytest` (8.1, 8.2). The sync step runs on every build so a `pip install .` (non-editable, Req 8.2) always packages the current schemas as `package_data` — without it, `load_schema()`/`xcecommon.validation`'s default resolution would silently package stale or missing copies. **`pytest` runs after whichever install just happened, not before** — this ordering is what makes the `release` profile's test run (8.2) the only one that actually exercises `default_schema_path()`/`default_schema_dir()` resolving under a real non-editable `site-packages` layout; the default (dev/editable) profile's test run (8.1) never would, since editable installs keep resolving into the checked-out `schema_files/` copy either way. **Design Review correction (original, 2026-07-21)**: `xceadmin` is dispatched directly by this case statement and is deliberately **not** wired into `src/meson.build`/`src/meson.options`/a `src/xceadmin/meson.build` — `xcecli`'s `custom_target()` pattern fits a single static-binary output; `pip install -e .` mutates a venv/site-packages tree with no single trackable output artifact, so forcing it through meson's build graph would be a dead end. `xceadmin` sits orthogonal to the meson component option entirely, matching what `brief.md`'s harness sketch already specified before this design pass over-engineered a meson integration. - -**Amendment (2026-07-22, supersedes the correction above)**: the user reopened this decision — standard-onboarding consistency (README 3.3 `nix build .#xcecore[_debug]` and 3.4 `meson compile` working for xceadmin exactly like every other component) outweighs the multi-file-output concern. `src/xceadmin/meson.build` wraps the install+test cycle in a `custom_target()` whose output is a stamp file (`xceadmin-build.stamp`), not a tracked multi-file tree — the real work (out-of-tree `pip install .` into a build-dir-local venv, then `pytest`) runs in `src/xceadmin/meson-build.sh`, invoked with `build_always_stale: true` so it reruns unconditionally rather than relying on ninja's file-mtime cache (matching this project's existing "clean rebuild every invocation" convention, same as every `xce-build.sh` component's `rm -rf "$BUILD_DIR"`). `src/meson.build` gains `subdir('xceadmin')` inside the `component != 'xcecli'` block, so it builds under `component=all`/`component=xcecore` (where `xcecore`'s `mesonFlags` already sets `-Dcomponent=xcecore`) but not under a pure `xcecli` build. `src/tools/xce-build.sh`'s `xceadmin)` case now just delegates (`setup_build; meson compile -C "$BUILD_DIR" xceadmin`) instead of carrying its own separate pip/venv implementation — the one meaningful behavior change is that the editable-vs-pinned distinction (Req 8.1 vs. 8.2) collapses: every `xce-build.sh xceadmin` invocation now runs the pinned/non-editable install path, a strictly stronger test (it exercises real `package_data` resolution on every run, not just under a `release` profile). A purity constraint the original design's rejection didn't have to solve: several tests reach sibling directories via a fixed relative path (`src/configs/`, `src/schema/`, `ha/data/xce.crt`/`.key`) that assume the checked-out repo's exact directory nesting; `meson-build.sh` resolves this by staging a `repo-mirror/` tree in the build dir with symlinks recreating that exact layout, rather than touching the tests' deliberate fail-loud (not skip-on-missing-layout) assertions. Verified both paths end-to-end: `meson compile -C xce.build xceadmin` (165/165 passed, real `ha/data/xce.crt` symlinked from the live checkout) and `nix build --option sandbox false .#xcecore_debug` (128 passed / 37 skipped — the skips are the existing `skipif` on `ha/data/xce.crt` not present, correctly firing since nix's git-tracked-only flake source has no committed cert files; 0 failed). AppImage bundling (README 3.5) is explicitly deferred — out of scope for this amendment, tracked as roadmap future work, not implemented here. -- `src/nixSupport/xce.nix` — add `python3`, `python3Packages.{pycapnp,pyyaml,jsonschema,click,pytest,ruff}` to `nativeBuildInputs` (8.6). -- `.claude/skills/xce-build/SKILL.md` — document the `xceadmin` component and its pip-based (non-meson-profile) build path (8.1, 8.2). -- `src/tools/deploy-ha.sh` — **(gap closed during task generation — the actual runtime behavior for 8.3-8.5 belongs here, not just in documentation)** add a `--build-config` flag; before starting the cluster, if `--build-config` is passed or `ha/data/cluster.bin` is missing, run `xceadmin config build configs/ -o ha/data/cluster.bin` (8.3, 8.4); if `cluster.bin` already exists and `--build-config` is not passed, use it as-is, unchanged (8.5). -- `.claude/skills/xce-deploy-ha/SKILL.md` — document the `--build-config` flag and the new precondition step added to `deploy-ha.sh` above (8.3, 8.4, 8.5). -- `.claude/settings.local.json` / `.claude/settings.json` — permission allowlist and ruff format/check hooks for `*.py` (full JSON already drafted in `brief.md` Harness Enhancement section; applied verbatim here, not re-derived). - -## System Flows - -### Config Build -```mermaid -flowchart TD - Start[xceadmin config build dir] --> ScanFiles[scan 7 known yaml filenames] - ScanFiles --> PerFile{file present} - PerFile -- no --> EmptySection[treat section as empty] - PerFile -- yes --> Validate[validate against bundled jsonschema] - Validate -- fail --> ErrValidation[report file field constraint exit nonzero] - Validate -- pass --> BusinessRules[reserved name dup check undefined ref warn] - BusinessRules --> Map[map to ClusterConfig fields] - EmptySection --> Map - Map --> Encode[pycapnp encode] - Encode --> Write[write cluster bin and sha256 to stderr] -``` - -Gating conditions: a missing individual YAML file is not an error (1.5, empty section) — there is no "missing schema" error mode at all (1.6, superseded 2026-07-21: schemas are bundled with `xceadmin`, not read from `configs_dir`). Schema validation failures are hard errors (4.1); business-rule warnings (undefined references) do not stop the build (4.4). - -### Config Push (including auto-discovery) -```mermaid -sequenceDiagram - participant Op as Operator - participant X as xceadmin push - participant C1 as Controller node1 - participant C2 as Controller node primary - - Op->>X: config push cluster.bin - X->>C1: TLS connect, Request OP_upgradeConfig - C1-->>X: Reply - alt redirectHost set - X->>C2: TLS connect, resend Request - C2-->>X: Reply UpgradeConfigReply - end - X->>Op: versionTag ackedCount error, exit code -``` - -At most 3 redirect hops (3.2); each hop is a fresh TLS connection with the same 30s timeout (3.4, 7.6). `fill_redirect_reply()` now fires for a standby's `OP_upgradeConfig` rejection too (bug-1700/1701, see Boundary Commitments) — the `alt` branch above is live-verified against a real 3-node cluster (`test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds`), not just mock-controller tests. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|---|---|---|---|---| -| 1 | Config Build | `xceadmin.build`, `xcecommon.{validation,mapping,schema}` | `build_cluster_config()`, `map_section()` | Config Build | -| 2 | Config Push | `xceadmin.push`, `xcecommon.wire` | `send_request()`, `push_config()` | Config Push | -| 3 | Auto Primary Discovery | `xceadmin.discovery` | `follow_redirects()` | Config Push | -| 4 | Validation | `xcecommon.validation` | `validate_yaml()`, `check_business_rules()` | Config Build | -| 5 | xcecommon Shared Package | `xcecommon.{schema,wire,mapping,validation}` | package-level exports, `default_schema_path()`, `default_schema_dir()` | — | -| 6 | Dry-Run Push (Nice-to-Have) | `xceadmin.push`, `xcecommon.wire` | `probe_config()` | Config Push (dry-run variant, not diagrammed — mirrors push minus commit) | -| 7 | Error Handling & Resilience | `xceadmin.cli`, `xcecommon.wire` | `click.ClickException` mapping, socket timeout handling | Config Push | -| 8 | Skill & Harness Integration | build/nix/skill files (File Structure Plan — Modified Files) | — | — | -| 9 | Deterministic Builds | `xcecommon.mapping`, `xceadmin.build` | `build_cluster_config()` (no wall-clock/host/user inputs) | Config Build | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|---|---|---|---|---|---| -| `xcecommon.schema` | Library | Load `controller.capnp`, expose typed roots | 5.1, 5.5, 7.4 | `controller.capnp` (P0) | Service | -| `xcecommon.mapping` | Library | YAML section dict → capnp field values | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3 | `xcecommon.schema` (P0) | Service | -| `xcecommon.validation` | Library | JSON Schema + business-rule checks | 4.1, 4.2, 4.3, 4.4, 4.5, 5.5 | `xcecommon/schema_files/json/*.schema.json` (P0, bundled) | Service | -| `xcecommon.wire` | Library | TLS connect, framing, generic request/reply | 2.1, 2.3, 2.4, 5.2, 6.1, 7.1, 7.3, 7.6 | `xce.crt`/`xce.key` (P0), `xcecommon.schema` (P0) | Service | -| `xceadmin.build` | CLI orchestration | `config build` command | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 9.1, 9.2, 9.3 | `xcecommon.{validation,mapping}` (P0) | Service | -| `xceadmin.push` | CLI orchestration | `config push` / `--dry-run` | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6 | `xcecommon.wire` (P0), `xceadmin.discovery` (P0) | Service | -| `xceadmin.discovery` | CLI orchestration | Redirect-following | 3.1, 3.2, 3.3, 3.4 | `xcecommon.wire` (P0) | Service | -| `xceadmin.cli` | CLI entry | Argument parsing, exit codes | 1.2, 1.3, 2.2, 2.4, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 8.1, 8.2 | all `xceadmin.*` (P0) | Service | - -Only components introducing a new boundary get a full block below; `xceadmin.cli` is presentation-only (click wiring) and is covered by the summary row plus its Implementation Note. - -### Library - -#### xcecommon.schema - -| Field | Detail | -|-------|--------| -| Intent | Load and cache the compiled `controller.capnp` schema; expose the `ClusterConfig`, `Request`, `Reply`, `UpgradeConfigRequest/Reply`, `ProbeConfigTagRequest/Reply` types. | -| Requirements | 5.1, 5.5, 7.4 | - -**Responsibilities & Constraints** -- Single load point — `capnp.load(path, imports=[schema_dir])` called once per process; other modules import the returned module object, never re-load. -- Resolves `c.capnp`'s sibling-directory import via the `imports` argument, pointed at wherever `default_schema_path()` resolved the copy to. -- Read-only: never mutates or re-serializes the schema itself. -- **Default path resolution (Design Review correction, 2026-07-21 — a hosted-discussion follow-up caught that the first draft never specified this, which meant `pip install .` / Req 8.2 was unimplementable as designed)**: `default_schema_path()` uses `importlib.resources.files("xcecommon") / "schema_files" / "controller.capnp"` — this resolves correctly for both `pip install -e .` (Req 8.1, still points into the checked-out repo's `schema_files/` copy) and `pip install .` (Req 8.2, resolves into whatever `site-packages` copy `package_data` installed) identically, unlike a `__file__`-relative `../../../schema/` walk, which only survives editable installs. `--schema-path <path>` (CLI) / `XCEADMIN_SCHEMA_PATH` (env var) override the default for troubleshooting, per 7.4's "suggest checking the installation." - -**Dependencies** -- External: `pycapnp` (P0) — schema compilation and `DynamicStruct` message API. -- External: `xcecommon/schema_files/{controller,c}.capnp` (P0) — packaged copy, synced from `src/schema/` at build time (File Structure Plan → Modified Files); failure path is 7.4. - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```python -def default_schema_path() -> str: - """importlib.resources.files('xcecommon') / 'schema_files' / 'controller.capnp', - overridable via --schema-path / XCEADMIN_SCHEMA_PATH.""" - -def load_schema(schema_path: str | None = None) -> "capnp.Capnp": - """Load controller.capnp (default_schema_path() if schema_path is None); - raise SchemaNotFoundError if the resolved path or its c.capnp import is missing.""" - -class SchemaNotFoundError(XceAdminError): - path: str -``` -- Preconditions: `schema_path` (explicit or resolved) points at `controller.capnp`; `c.capnp` sits alongside it. -- Postconditions: returned module exposes `.ClusterConfig`, `.Request`, `.Reply`, `.UpgradeConfigRequest`, `.UpgradeConfigReply`, `.ProbeConfigTagRequest`, `.ProbeConfigTagReply`. -- Invariants: no network or filesystem writes. - -#### xcecommon.mapping - -| Field | Detail | -|-------|--------| -| Intent | Convert validated YAML section dicts into `ClusterConfig` capnp field values, and back (field metadata reused by any future `config diff`). | -| Requirements | 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3 | - -**Responsibilities & Constraints** -- One declarative table (`SECTION_MAP`) drives all 7 sections — each entry states: YAML filename, top-level YAML key, target capnp field name (`@N`), and mapper kind (`list` or `scalar`). Two mapper functions (`map_list_section`, `map_scalar_section`) implement every section — no per-file bespoke mapper (Generalization, see `research.md`). -- Two mapper shapes only: **list** sections (`resources`, `queues`, `ugroups`, `hgroups`, `hostConfigs`, `hproviders`, `limits` — YAML top key is an array, mapped to `List(Struct)`) and **scalar** section (`xce.yaml` — flat key/value, mapped to 5 top-level `ClusterConfig` scalar fields). -- Enforces 9.1, 9.2, 9.3: no timestamp/hostname/username is ever read into a mapped value; list-section ordering follows YAML source order exactly, not sorted or hashed (order stability is required for byte-identical output). -- `configGeneration` (`@12`) is never touched by this module (1.9) — capnp leaves it at its default (0) unconditionally. -- `candidates`, `port`, hash-table-size fields, `xstrings` do not exist in the schema (post `clusterconfig-schema-split`) — there is nothing for this module to exclude; 1.8/1.10 are satisfied structurally, not by runtime logic. - -**Dependencies** -- Inbound: `xceadmin.build` (P0) — sole caller. -- Outbound: `xcecommon.schema` (P0) — target field types. - -**Contracts**: Service [x] - -##### Service Interface -```python -from typing import Literal, TypedDict - -class SectionSpec(TypedDict): - yaml_file: str # e.g. "queues.yaml" - yaml_key: str # e.g. "queues" - capnp_field: str # e.g. "queues" - kind: Literal["list", "scalar"] - -SECTION_MAP: list[SectionSpec] # 7 entries, one per YAML file - -def map_list_section(spec: SectionSpec, items: list[dict], target: "capnp._DynamicStructBuilder") -> None: - """Populate target.<capnp_field> as a capnp List sized to len(items).""" - -def map_scalar_section(spec: SectionSpec, values: dict, target: "capnp._DynamicStructBuilder") -> None: - """Set each scalar capnp field on target directly from values.""" -``` -- Preconditions: `items`/`values` already passed `xcecommon.validation`. -- Postconditions: `target` (a `ClusterConfig` builder) has exactly the fields for the given section set; no other field is touched. -- Invariants: idempotent given the same input dict — calling twice with identical input produces byte-identical serialized output (9.1). - -**Full field-mapping table** (Data Models has the authoritative field-by-field version — see Data Contracts & Integration). - -#### xcecommon.validation - -| Field | Detail | -|-------|--------| -| Intent | JSON Schema validation plus business rules (reserved names, duplicates, undefined references) not expressible in JSON Schema alone. | -| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5, 5.5 | - -**Responsibilities & Constraints** -- **Bundled schemas (2026-07-21 revision, Requirement 5.5, supersedes Requirement 1.6)**: `default_schema_dir()` uses `importlib.resources.files("xcecommon") / "schema_files" / "json"` — same `importlib.resources` mechanism as `xcecommon.schema.default_schema_path()`, correct for both `pip install -e .` and `pip install .`. `configs_dir` is never consulted for a `schema/` subdirectory; there is no "schema not found in configs_dir" error path anymore. No CLI override exists for this one (unlike `--schema-path` for the capnp schema) — Simplification: no requirement or use case calls for operator-swappable *validation* schemas, only for pointing at a different `controller.capnp` during cross-version troubleshooting. -- `validate_yaml(file, schema_name)` — `schema_name` is the YAML file's base name with `.yaml` stripped (e.g. `queues.yaml` → `"queues"` → `default_schema_dir() / "queues.schema.json"`), matching `SECTION_MAP`'s `yaml_file` entries and the naming convention the 7 files under `src/configs/schema/` already follow. `jsonschema.Draft202012Validator`; on failure raises with file name, JSON-pointer field path, and violated constraint (4.1). -- `check_reserved_names(data)` — rejects `"lost_and_found"` for any queue/host name (4.2), independent of schema validity. -- `check_duplicates(all_sections)` — per-file/per-section uniqueness only; a limit name may collide with a queue name (4.3, scoped intentionally). Raises like `check_reserved_names`, not return-and-let-caller-decide — 4.3, like 4.2, is an unconditional hard-fail per requirements.md, so the two share the same raise-on-violation contract (Design Review correction: an earlier draft had `check_duplicates` return messages for the caller to judge, which was an unjustified control-flow inconsistency between two checks with identical fail-semantics). -- `check_undefined_refs(all_sections)` — cross-file warnings only (queue→host-group, limit→group/`resname`); never blocks the build (4.4). `resources.yaml` is treated as the exhaustive resource catalog — every `resname` elsewhere, including the built-in/dynamic names (`slots`, `mem`, the 8 `add_hostattrs_resvals()` attributes), must resolve against it or this check warns (4.5). - -**Dependencies** -- Inbound: `xceadmin.build` (P0). -- External: `jsonschema` (P0), `xcecommon/schema_files/json/*.schema.json` (P0, bundled — synced from `src/configs/schema/` at build time, File Structure Plan → Modified Files). - -**Contracts**: Service [x] - -##### Service Interface -```python -class ValidationError(XceAdminError): - file: str - json_pointer: str - constraint: str - -class ReservedNameError(XceAdminError): ... -class DuplicateEntryError(XceAdminError): ... - -def default_schema_dir() -> str: - """importlib.resources.files('xcecommon') / 'schema_files' / 'json'.""" - -def validate_yaml(file_path: str, schema_name: str) -> dict: ... # raises ValidationError; schema_name resolved under default_schema_dir() -def check_reserved_names(sections: dict) -> None: ... # raises ReservedNameError -def check_duplicates(sections: dict) -> None: ... # raises DuplicateEntryError -def check_undefined_refs(sections: dict) -> list[str]: # returns warning messages, never raises — the one intentionally non-fatal check (4.4) -``` - -All raising exceptions here subclass `XceAdminError` (defined in `xcecommon.errors`, see `xceadmin.cli` Implementation Note) so the CLI boundary has one dispatch surface instead of catching unrelated types per module. - -#### xcecommon.wire - -| Field | Detail | -|-------|--------| -| Intent | TLS connection, the 8-byte native-order framing, and a generic single-request/single-reply exchange — opcode-agnostic so it also serves `ProbeConfigTag`'s direct-struct variant. | -| Requirements | 2.1, 2.3, 2.4, 5.2, 6.1, 7.1, 7.3, 7.6 | - -**Responsibilities & Constraints** -- `connect(host, port, cert_dir, timeout=30)` — `socket.create_connection` wrapped by `ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)` with `check_hostname=False`, `verify_mode=ssl.CERT_NONE`, loading `xce.crt`/`xce.key` from `cert_dir` (2.3, 2.4, 7.1, 7.6). This mirrors the controller's own outbound TLS posture exactly (`myapp.c:103-109`) — confirmed no CN/peer-cert check exists anywhere in the codebase to diverge from. -- `send_frame(sock, payload: bytes)` / `recv_frame(sock) -> bytes` — `struct.pack("=Q", len(payload))` / `struct.unpack("=Q", ...)`, **native byte order**, matching `netio.c`'s raw-memory-copy framing exactly (verified: no `htole64`/`htobe64` anywhere in the C framing path). -- `send_request(sock, request: "Request") -> "Reply"` — generic: encode via `.to_bytes()`, frame, send, receive frame, decode via `Reply.from_bytes()`. Used for `OP_upgradeConfig`; the same primitives (not the same function, since `ProbeConfigTagRequest`/`Reply` are direct structs, not `Request`/`Reply`-union-wrapped) serve `config push --dry-run`'s `ProbeConfigTag` client. -- Never passes an `SSLSocket` or its `fileno()` into a pycapnp read/write call — decode/encode happens in-memory (`from_bytes`/`to_bytes`) against plaintext buffers assembled by `recv_frame`/`send_frame`, so TLS is never bypassed (verified gotcha from discovery: pycapnp's `.write()` calls `fileno()` and writes to the raw fd, which would skip encryption entirely on a wrapped socket). - -**Dependencies** -- Inbound: `xceadmin.push`, `xceadmin.discovery` (P0). -- External: Python stdlib `ssl`, `socket`, `struct` (P0); `xcecommon.schema` (P0). - -**Contracts**: Service [x] - -##### Service Interface -```python -import socket, ssl - -def connect(host: str, port: int, cert_dir: str, timeout_s: float = 30.0) -> ssl.SSLSocket: ... -def send_frame(sock: ssl.SSLSocket, payload: bytes) -> None: ... -def recv_frame(sock: ssl.SSLSocket) -> bytes: ... -def send_request(sock: ssl.SSLSocket, request_bytes: bytes) -> bytes: - """Frame+send request_bytes, receive+unframe one reply. Raises - ConnectionTimeoutError / ConnectionRefusedError on failure.""" - -class ConnectionTimeoutError(XceAdminError): host: str; port: int -class ReplyDecodeError(XceAdminError): reason: str # corrupt/truncated capnp bytes, 7.3 -``` -- Preconditions: `cert_dir` contains `xce.crt`+`xce.key`. -- Postconditions: exactly one request sent, exactly one reply returned, per call — no retry, no connection pooling (one-shot CLI, not a daemon; Simplification). -- Invariants: socket closed on every exit path, including exceptions and `SIGINT` (7.5). - -### CLI Orchestration - -#### xceadmin.build - -| Field | Detail | -|-------|--------| -| Intent | `config build <dir>` command: scan → validate → map → serialize → write → print SHA-256. | -| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.9, 8.3, 8.4 | - -**Responsibilities & Constraints** -- Scans for the 7 known filenames only (`SECTION_MAP` keys) — not an open-ended directory walk (1.1). -- `-o <path>` / `-o -` (stdout) output targets (1.2, 1.3); SHA-256 always to stderr, regardless of `-o` target (1.4). -- No `schema/` subdirectory check — schemas are bundled (1.6, superseded); missing individual YAML file → empty section, continue (1.5). -- Unrecognized-but-schema-valid fields → warning, not an error (1.7). - -**Dependencies** -- Outbound: `xcecommon.validation` (P0), `xcecommon.mapping` (P0), `xcecommon.schema` (P0, transitively). - -**Contracts**: Service [x] - -##### Service Interface -```python -def build_cluster_config(configs_dir: str) -> bytes: - """Full pipeline: validate all present sections, map to ClusterConfig, - serialize. Raises ValidationError / ReservedNameError / DuplicateEntryError - on hard failures (all subclass XceAdminError).""" - -def write_output(data: bytes, output_path: str) -> str: - """Writes data to output_path ('-' => stdout). Returns hex SHA-256.""" -``` - -#### xceadmin.push - -| Field | Detail | -|-------|--------| -| Intent | `config push <file>` (commit) and `config push --dry-run <file>` (probe): connect, discover primary, send, display result. Two distinct functions, not one boolean-flagged function — the two wire messages (`Request{upgradeConfig}` vs. direct-struct `ProbeConfigTagRequest`) and their replies (`UpgradeConfigReply{versionTag, ackedCount, error}` vs. `ProbeConfigTagReply{configTag}`) share no fields, so a shared return type would misrepresent one of the two (Design Review correction — an earlier draft's single `push_config(..., dry_run: bool) -> PushResult` implied `acked_count` exists on a dry-run probe reply, which it does not). | -| Requirements | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6 | - -**Responsibilities & Constraints** -- Builds the `Request{opcode: upgradeConfig, upgradeconfigrequestmsg: {sha256, bytes}}` message exactly per `controller.capnp:628-654,768-771` (2.1). -- Delegates connection + redirect-following to `xceadmin.discovery` before sending (3.1, 3.2, 3.3, 3.4). -- `--dry-run` sends `ProbeConfigTagRequest` instead (direct struct, no `Request` wrapper — see `xcecommon.wire`), reports acceptance without committing (6.1, 6.2, 6.3). Per Boundary Commitments, this path has no live server counterpart today; it is implemented against the documented contract and exercised only by the mock-controller test suite until the controller gains a handler. -- Maps every failure mode to a specific message + non-zero exit per 7.1, 7.2, 7.3, 7.5, 7.6 (connection failure, non-zero `Reply.status`, corrupt binary, SIGINT, timeout). - -**Dependencies** -- Outbound: `xcecommon.wire` (P0), `xceadmin.discovery` (P0). - -**Contracts**: Service [x] - -##### Service Interface -```python -class PushResult: - version_tag: int - acked_count: int - error: str - -class DryRunResult: - current_generation: int # from ProbeConfigTagReply.configTag - would_be_generation: int # current_generation + 1, computed client-side for display only (6.2) - accepted: bool - reason: str # populated when accepted is False (6.3) - -def push_config(cluster_bin_path: str, controller: str, cert_dir: str) -> PushResult: ... -def probe_config(cluster_bin_path: str, controller: str, cert_dir: str) -> DryRunResult: ... -``` -- Postconditions (`push_config`): exit 0 iff `acked_count > 0` and `error == ""` (2.6); otherwise exit non-zero with `error` displayed (2.7). -- Postconditions (`probe_config`): exit 0 and `"OK: controller will accept this config (generation N→M)"` iff `accepted` (6.2); otherwise exit non-zero with `reason` displayed (6.3). - -#### xceadmin.discovery - -| Field | Detail | -|-------|--------| -| Intent | Generic, opcode-agnostic redirect-following against `RequestReply.redirectHost`/`redirectPort`. | -| Requirements | 3.1, 3.2, 3.3, 3.4 | - -**Responsibilities & Constraints** -- Inspects any `Reply` for non-empty `redirectHost`/`redirectPort`; reconnects and resends up to 3 hops (3.1, 3.2), then errors. -- Displays the redirect target when a standby redirects (3.3). -- 30s "no primary elected" timeout is a property of each connection attempt, not the overall discovery loop (3.4, delegated to `xcecommon.wire.connect`). -- Implemented generically against the `RequestReply` schema shape (not `upgradeConfig`-specific) so it was correct for `OPCode_join`-style redirects from the start and required zero client-side changes once the controller closed the `upgradeConfig` gap (bug-1700/1701, 2026-08-26) — validated in practice, not just in principle (Simplification — no opcode-specific branching was ever needed). - -**Dependencies** -- Outbound: `xcecommon.wire` (P0). - -**Contracts**: Service [x] - -##### Service Interface -```python -def follow_redirects(initial_host: str, initial_port: int, cert_dir: str, send_fn) -> tuple[str, int, bytes]: - """send_fn(host, port) -> reply_bytes. Returns (final_host, final_port, reply_bytes) - after following <=3 redirects. Raises TooManyRedirectsError / NoPrimaryError.""" - -class TooManyRedirectsError(XceAdminError): hops: int -class NoPrimaryError(XceAdminError): host: str; port: int -``` - -### xceadmin.cli — Implementation Note -Thin `click` layer: `@click.group() cli()` → `@cli.group() config()` → `@config.command() build(...)`/`push(...)`. - -**Shared exception contract** (Design Review correction — an earlier draft had 7 unrelated exception types with no unifying dispatch mechanism): `xcecommon.errors` defines one base, `class XceAdminError(Exception): message: str`, and every raising exception across `xcecommon`/`xceadmin` (`SchemaNotFoundError`, `ValidationError`, `ReservedNameError`, `DuplicateEntryError`, `ConnectionTimeoutError`, `ReplyDecodeError`, `TooManyRedirectsError`, `NoPrimaryError`) subclasses it. `xceadmin.cli` has exactly one `except XceAdminError as e: raise click.ClickException(e.message)` — a single dispatch surface, per-type message formatting happens in each exception's `__str__`/`message` construction at raise time, not in the CLI layer (7.1, 7.2, 7.3, 7.4). A `SIGINT` handler closes any open socket before re-raising as a "cancelled" message (7.5). No business logic lives here — covered by the summary row. - -## Data Models - -### Domain Model -No persistent entities or aggregates — `xceadmin` is a stateless, one-shot transform (YAML → bytes) and a stateless one-shot RPC client (bytes → controller → result). The only domain concept is the **`ClusterConfig` field set**, whose provenance (which YAML file owns which field) is the core invariant this design enforces. - -### Data Contracts & Integration - -**YAML section → `ClusterConfig` field mapping** (all 13 fields, `controller.capnp:694-708`): - -| capnp field (`@N`) | YAML source | Kind | Req | -|---|---|---|---| -| `clusterName` (@0), `heartbeatInterval` (@1), `inactiveHeartbeatCount` (@2), `fetchJobInterval` (@3), `netioTimeout` (@4) | `xce.yaml` (flat keys `cluster_name`, `heartbeat_interval`, `inactive_heartbeat_count`, `fetch_job_interval`, `netio_timeout`) | scalar | 1.11 | -| `resources` (@5) | `resources.yaml` → `resources[]` | list | 1.1 | -| `queues` (@6) | `queues.yaml` → `queues[]` | list | 1.1 | -| `ugroups` (@7) | `users.yaml` → `usergroups[]` | list | 1.1 | -| `hgroups` (@8) | `hosts.yaml` → `hostgroups[]` | list | 1.1 | -| `hostConfigs` (@9) | `hosts.yaml` → `hosts[]` | list | 1.1 | -| `hproviders` (@10) | `scaler.yaml` → `scalers[]` | list | 1.12 | -| `limits` (@11) | `limits.yaml` → `limits[]` | list | 1.1 | -| `configGeneration` (@12) | none — controller-assigned | excluded | 1.9 | - -`hosts.yaml` is the one file feeding two capnp fields (`hgroups` + `hostConfigs`) — `xcecommon.mapping`'s `SECTION_MAP` has two entries pointing at the same `yaml_file` with different `yaml_key`/`capnp_field` pairs; this is the only many-to-one case in the table and is called out explicitly here so it is not mistaken for a bug during implementation. - -Per-field YAML-key-to-capnp-field-name renames (e.g., `groupname`→`GroupConfig.name`, `exclude_members`→`exmembers`) are enumerated in `research.md` (too granular for this table) and enforced by `xcecommon.mapping`'s per-section spec, not restated here. - -### Logical Data Model -`cluster.bin` has no indexing/sharding concerns (single-file capnp blob, read once at controller startup/config-push time). Byte-for-byte determinism (9.1, 9.2, 9.3) is the only "consistency" property that matters, guaranteed by: (a) `SECTION_MAP` iterating YAML list order verbatim, never re-sorting; (b) no wall-clock/host/user value ever entering a mapped field; (c) capnp's own canonical single-segment encoding for a message built once and never mutated after `.to_bytes()`. - -## Error Handling - -### Error Strategy -Every `xcecommon`/`xceadmin` internal exception is caught once, at the `xceadmin.cli` boundary, and re-raised as `click.ClickException(message)` — giving a single place that guarantees "clear, actionable error, non-zero exit" (7.1, 7.2, 7.3, 7.4) without scattering `sys.exit()` calls through the library layer (library code raises; only the CLI layer exits). - -### Error Categories and Responses -| Category | Trigger | Response | Req | -|---|---|---|---| -| Connection | TLS connect fails | `"cannot connect to <host:port>: <reason>"`, exit non-zero | 7.1 | -| Protocol | `Reply.status != 0` | status code + reply error text, exit non-zero | 7.2 | -| Decode | corrupt/truncated capnp bytes | `"invalid cluster.bin: <reason>"`, exit non-zero | 7.3 | -| Environment | schema file missing at expected path | missing path + installation hint | 7.4 | -| Interrupt | `SIGINT` during push | close TLS socket, `"cancelled"`, graceful exit | 7.5 | -| Timeout | any network op > 30s | `"timeout connecting to <host:port>"` | 7.6 | - -### Monitoring -Out of scope — `xceadmin` is a one-shot CLI with stderr/exit-code as its only observability surface; no logging framework, no metrics endpoint (Simplification — a persistent log/metrics story belongs to a daemon, not a CLI invoked from a shell or a skill). - -## Testing Strategy - -- **Unit — schema bundling**: `default_schema_path()`/`default_schema_dir()` resolve to a real, non-empty, parseable `controller.capnp`/JSON schema set under whatever install is currently active — trivial in isolation, but only meaningful when it actually runs against a **non-editable** install; every other test in this list runs fine against an editable install even if `package_data` packaging is broken, since editable installs keep resolving into the checked-out source tree regardless. This is what `/xce-build xceadmin release` (8.2) validates that `/xce-build xceadmin` (8.1) cannot — a gap discovered post-design-approval (see `research.md`) that the schema-bundling fix itself would otherwise ship untested. -- **Unit — mapping**: `SECTION_MAP` list-section and scalar-section mappers each produce the expected capnp field values from a minimal in-memory dict (no file I/O) — one test per `kind`, plus the `hosts.yaml`-feeds-two-fields case explicitly. -- **Unit — wire framing**: `send_frame`/`recv_frame` round-trip against a fixed byte sequence, and against a loopback `socket.socketpair()` — verifies native-order packing matches `struct.pack("=Q", ...)` semantics without needing a real controller. -- **Integration — config build (golden-file)**: `minimal`/`full`/`invalid` fixture directories (File Structure Plan) — `full` now covers all 7 YAML files including the newly-added `xce.yaml`/`scaler.yaml`; byte-compares `build_cluster_config()` output against a committed `expected.bin` and its SHA-256 (1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 9.1, 9.2, 9.3). Plain `assert produced == expected_bytes` fixture comparison — `pytest-regressions` was evaluated (see `research.md`) but not adopted, since this project's nix-pinned dependency list would need independent verification and the byte-equality need here is simple enough not to warrant it (Simplification). -- **Integration — validation**: one test per `invalid/` fixture case — schema violation, reserved name, duplicate name (including a limit/queue name collision, proving 4.3's intentionally-narrow scope), undefined reference (warn, not fail). -- **Integration — config push (mock controller)**: a Python TLS server speaking just enough of the wire protocol (frame in, decode `Request` or direct-struct `ProbeConfigTagRequest`, encode a canned `Reply`/`ProbeConfigTagReply`) to exercise `push_config()` (2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, 3.3 redirect, 7.1, 7.2, 7.3) and `probe_config()` (6.1, 6.2, 6.3 — accepted and rejected probe replies) against the same mock, since both share `xcecommon.wire`'s framing/send-receive primitives. This remains valuable for dev-loop speed and `probe_config()` coverage (`ProbeConfigTag` still has no server-side handler — Out of Boundary), but it is no longer the only way to test `push_config()`: the controller-side dispatch-gate defect that once blocked live-cluster testing was fixed (bug-1468, 2026-07-25), and the standby-redirect gap that followed it was fixed too (bug-1700/1701, 2026-08-26) — see Boundary Commitments and E2E below. -- **E2E**: real-cluster `config push` round-trip, live against genuine `xce_controller` processes — `test_e2e_live_push.py` (single-node, task 6.2) and `test_e2e_live_ha_push.py` (3-node HA, task 6.3): primary success path (positive `ackedCount`, real standby ACKs), and the standby-redirect success path (`test_live_config_push_against_standby_redirects_and_succeeds`). No longer gated on the controller dispatch fix or the redirect fix — both landed; `brief.md`'s "Option B" is realized. - -## Security Considerations - -- **Transport is encryption-only, not authenticated** — no peer-certificate validation exists anywhere in XCE's controller-to-controller TLS today (`SSL_VERIFY_NONE`, no `X509_check_host`); `xceadmin` matches this exactly rather than silently adding (and thereby implying) a stronger guarantee the rest of the system doesn't have. An attacker with network access to the controller port can MITM either direction. This is a pre-existing, project-wide posture (`product.md` — "trust OS boundary"), not something this spec introduces or should unilaterally fix. -- **Input validation at the boundary** — every YAML file is JSON-Schema-validated before any byte of it reaches capnp encoding (4.1), consistent with `security.md`'s "validate at the API boundary, never trust external data." -- **No secrets in xceadmin** — no JWT, no RBAC, no credential storage; trust model is "same TLS cert bundle as the controller cluster," matching `xce_test`'s existing zero-config posture. Nothing new to rotate or leak. -- **Reply decoding must not crash on malformed input** — `xcecommon.wire.send_request`'s `Reply.from_bytes()` call is wrapped to convert a pycapnp decode exception into `7.3`'s "invalid" error path rather than an unhandled traceback, even though the reply source (a controller within the operator's own cluster) is semi-trusted. - -## Supporting References -- Full per-field YAML-key ↔ capnp-field-name rename table, pycapnp API usage notes, and the two controller-side defect write-ups: `research.md`. -- Nix package list, harness hook JSON (ruff format/check, YAML schema guard), and settings allowlist: `brief.md` Harness Enhancement section (unchanged, referenced not restated here). diff --git a/.kiro/specs/admin-config-tool/requirements.md b/.kiro/specs/admin-config-tool/requirements.md deleted file mode 100644 index 8c8cd54e..00000000 --- a/.kiro/specs/admin-config-tool/requirements.md +++ /dev/null @@ -1,185 +0,0 @@ -# Requirements Document - -## Introduction - -XCE operators need a tool to build and deploy cluster configuration. Currently, YAML config files (`configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml`) exist with JSON schemas for validation, but no tool compiles them into the `ClusterConfig` Cap'n Proto binary the controller expects. The controller-side `OP_upgradeConfig` handler exists (`config_repl.c` — receives binary, verifies SHA-256, pushes to standbys, awaits majority ACK), but no CLI sends it. - -`xceadmin` is a Python CLI that fills this gap: `config build` compiles YAML into `cluster.bin`, `config push` sends it to a controller over TLS+capnp. - -Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact — deterministic for the same inputs, never committed to git. - -## Boundary Context - -- **In scope**: - - YAML → `ClusterConfig` capnp binary compilation (`config build`) - - TLS+capnp config deployment to controller (`config push`) - - Auto primary discovery via controller redirect reply - - JSON Schema validation of YAML inputs - - Reserved name defense-in-depth (`"lost_and_found"`) - - `xcecommon` shared Python package (capnp schema, wire protocol, config models) - - Integration with xce-build and xce-deploy-ha skills - - Nix dev shell: Python package dependencies - - `config push --dry-run` — validate without committing (nice-to-have) - -- **Out of scope**: - - Controller-side config replication, snapshot, restart logic (owned by `config-upgrade-snapshot`) - - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`) - - Performance tuning / per-instance parameter extraction (separate work item) — specifically `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, and `xstrings` (xstring hash size). These are not cluster-shared: each controller instance is expected to set them locally, so `xceadmin` never reads, accepts, or writes them (see Requirement 1.8). - - `candidates` (peer/standby list) — changes independently and more frequently than the rest of cluster config (adding/removing a standby shouldn't require a `config build`/`config push` cycle); `xceadmin` never reads, accepts, or writes it (see Requirement 1.10). Peer membership is owned elsewhere (mechanism TBD — see brief.md Open Design Question #3). - - `configGeneration` management — fully controller-owned (see Adjacent Expectations below); `xceadmin` never reads, increments, or overrides it. - - xcecli Python rewrite (future, separate spec) - - Controller config version history, rollback safety validation, config activation status (owned by `config-upgrade-snapshot`, see Future Roadmap in brief.md) - - `config diff` — deferred to roadmap follow-up (2026-07-21): no operator has asked for it yet; revisit once `config push` has real-world usage (see brief.md R1) - -- **Adjacent expectations**: - - **Controller `OP_upgradeConfig` handler** (`config_repl.c`) exists and is operational. On receiving `UpgradeConfigRequest{sha256, bytes}`: - 1. SHA-256 transit verification - 2. Cap'n Proto decode + reserved name check (`"lost_and_found"`) - 3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary — the primary is the sole authority for `configGeneration`; whatever value `xceadmin` ships in `cluster.bin` is discarded and replaced - 4. Save to disk, replicate to standbys, await majority ACK - 5. Commit `WAL_CONFIG_MARKER`, signal controller restart - 6. Return `UpgradeConfigReply{versionTag, ackedCount, error}` - - **Controller redirect**: Standby returns `redirectHost`/`redirectPort` in the `RequestReply` union — xceadmin follows the redirect to reach the primary. - - **Per-instance field preservation** — RESOLVED structurally, not by controller behavior: `port`, `candidates`, the hash-table-size fields, and `xstrings` have been removed from the `ClusterConfig` capnp struct entirely (`clusterconfig-schema-split` spec, landed commit `519e11c`). `ClusterConfig` is now 13 fields (@0–@12, `controller.capnp:694-708`); these fields simply do not exist in `cluster.bin` — there is no zero-default-clobber risk to guard against. Req 1.8/1.9/1.10 remain as defense-in-depth documentation (if any of these fields is ever reintroduced, xceadmin must still not source it), not as active field-suppression logic. - - **ProbeConfigTag RPC** (`xce_probe_config_tag()`, `config_repl.c:1829`): Separate direct-capnp RPC (not in `Request` union) for reading current config tag. Used by `config push --dry-run` to validate before committing. - - `controller.capnp` schema is the single source of truth for `ClusterConfig`, `Request`/`Reply` unions, and all OP codes. - - `config-upgrade-snapshot` spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status. - - **Blocked controller-side gaps** (NOT owned by admin-config-tool): - - Entity-bearing snapshot after config upgrade (blocked on `entity-bearing-snapshots` spec) - - Config rollback safety (e.g., reject rollback while jobs reference deleted queues) - - Config version history query (`xceadmin config history`) - - **Dependency on `clusterconfig-schema-split`** — **RESOLVED, landed** (commit `519e11c`, phase=`implementation-complete`). Both subcommands are now unblocked: - - `config build`: no dependency, unaffected either way. - - `config push`: previously soft-blocked over the risk of `xceadmin` silently wiping `candidates`/`port`/perf-tuning on a zero-valued push. That risk no longer exists — those fields are physically absent from the `ClusterConfig` schema, so there is nothing for a push to clobber. `config push` may be exercised against a real cluster without this precondition. - -## Requirements - -### Requirement 1: Config Build - -**Objective:** As an XCE operator, I want to compile YAML config files into a `ClusterConfig` capnp binary, so that I can produce a deployable cluster configuration from human-editable source files. - -#### Acceptance Criteria - -1. When the operator runs `xceadmin config build <configs_dir>`, the system shall scan the directory for YAML config files (`hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`, `limits.yaml`, `xce.yaml`, `scaler.yaml`), validate each against its corresponding JSON schema bundled with `xceadmin`, map the validated data to the `ClusterConfig` capnp struct, serialize it to binary, and write the result to `cluster.bin` in the current working directory. -2. When the operator specifies `-o <path>`, the system shall write the output to the given path instead of the default `cluster.bin`. -3. When the operator specifies `-o -`, the system shall write the binary output to stdout. -4. After writing output, the system shall print the SHA-256 hash of the resulting binary to stderr. -5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors). -6. **RESOLVED, superseded (2026-07-21)**: JSON Schema validation is performed against schemas bundled with the `xceadmin` package (see Requirement 5.5), not a `schema/` subdirectory the operator supplies. `configs_dir` therefore never needs its own `schema/` folder, and there is no "missing schema" error mode — the schemas travel with the tool, version-locked to the same release as the `ClusterConfig` field mapping they validate against (`xcecommon.mapping`), the same rationale already applied to `controller.capnp` (see Requirement 5.5, brief.md). An operator-supplied `schema/` subdirectory, if present, is ignored — not read, not merged, not treated as an override. -7. Where the YAML content passes JSON Schema validation but maps to fields not present in the `ClusterConfig` capnp struct, the system shall warn the operator about unrecognized fields. -8. The system shall not source `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, or `xstrings` (xstring hash size) from YAML or CLI input under any circumstance. These are per-controller-instance performance-tuning parameters, not cluster-shared configuration, and the built `cluster.bin` shall leave them unset (capnp default). -9. The system shall not source, read, or write `configGeneration`. The built `cluster.bin` shall leave it unset (capnp default) — the primary controller is the sole authority for assigning it (see Requirement 2 note). -10. The system shall not source `candidates` (controller peer/standby list) from YAML or CLI input. Standby membership changes independently of, and more frequently than, the rest of cluster configuration, so it shall not require a `config build`/`config push` cycle; the built `cluster.bin` shall leave `candidates` unset (capnp default). Peer membership sourcing mechanism is out of scope for this spec (open question — see brief.md Open Design Question #3). -11. `clusterName`, `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness — not the HA/Raft election heartbeat, which has no `ClusterConfig` field, see `.kiro/steering/config-design.md` Rule 0), `fetchJobInterval` (scheduler job-fetch pass interval), and `netioTimeout` (network I/O timeout) SHALL be sourced from a new `xce.yaml` file and included in `cluster.bin` — these must stay consistent across all controller instances (unlike Criteria 8–10, which are intentionally excluded). **RESOLVED**: source is `xce.yaml`, a flat (non-list) YAML file holding these 5 scalar `ClusterConfig` fields. `cluster_name` is required (no sensible default). `heartbeat_interval` and `fetch_job_interval` are optional — both have controller-side defaults (`fill_default_cluster_config()`, `myconf.c:72-76`: 5 and 10 respectively). `inactive_heartbeat_count` is required and has **no** controller-side default — `cluster.c:1622` computes the worker-dead timeout as `heartbeat_interval * inactive_heartbeat_count`, so an unset (0) value would mark workers dead immediately; xceadmin must not let this ship unset. `netio_timeout` is optional; note it currently has no reader anywhere in the codebase outside the struct declaration itself (`clusterconfig.h:12`) — the field is wired on the wire/schema but not yet consumed by any controller logic. That's a controller-side gap, out of scope for this spec, but xceadmin should still source it from YAML if present rather than silently drop it. -12. `hproviders` (`List(ScalerConfig)` — autoscaler provider definitions: `name`, `type`, `conf_path`, `script_path`, `script_option`) SHALL be sourced from a new `scaler.yaml` file and included in `cluster.bin`. Scaler definitions must be identical across controller instances (a queue configured against a scaler must resolve it the same way after failover) and change rarely — both Configuration Inclusion Rules fail, so this field is included per the Default rule. - -Criteria 8–12 apply the **Configuration Inclusion Rules** (brief.md): a field is excluded from `cluster.bin` if it can change independently of the rest of cluster config, OR if it's expected to change frequently; otherwise it's included. Any future `ClusterConfig` field proposed for xceadmin's YAML mapping must be classified against these two rules first. - -### Requirement 2: Config Push - -**Objective:** As an XCE operator, I want to deploy a compiled `cluster.bin` to a controller cluster, so that the configuration takes effect on the running system. - -#### Acceptance Criteria - -1. When the operator runs `xceadmin config push <cluster.bin>`, the system shall connect to the controller via TLS, send the binary as a capnp `Request` with `OP_upgradeConfig` opcode, `upgradeconfigrequestmsg` body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result. -2. When the operator specifies `--controller <host:port>`, the system shall connect to the given controller address instead of the default `localhost:9991`. -3. ~~The system shall use the same TLS certificate bundle the controller cluster is deployed with (`xce.crt`/`xce.key`, default path `ha/data/`) to establish trust — no separate client CA flag or cert-issuance step.~~ **Superseded (2026-07-22)**: verified against the real client TLS setup (`netio.c`'s `create_ssl_context()`) that the actual controller-to-controller/CLI client presents **no** certificate at all when acting as a client (`cert_file`/`key_file` both `NULL` → bare `TLS_client_method()`) — only the server side loads a cert/key pair. `xceadmin`'s client previously diverged from this (loaded a client cert too); fixed to match exactly. No CA flag or cert-issuance step either way — `verify_mode=CERT_NONE` on both the real client and `xceadmin`. -4. ~~When the operator specifies `--cert-dir <path>`, the system shall load `xce.crt`/`xce.key` from the given directory instead of the default `ha/data/`.~~ **Superseded (2026-07-22)**: removed entirely — with no client cert to load, `--cert-dir` had no remaining purpose. `ha/data/` itself was also removed (deploy-ha.sh and the HA test harness now generate a fresh server-side cert/key per run via `openssl` instead of reading a checked-in fixture). -5. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message. -6. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0. -7. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero. - -### Requirement 3: Auto Primary Discovery - -**Objective:** As an XCE operator, I want xceadmin to automatically find the primary controller, so that I don't need to know which node is currently primary. - -#### Acceptance Criteria - -1. When the operator connects to any controller node (primary or standby), the system shall inspect the reply for `redirectHost` and `redirectPort` fields. If both are non-empty, the system shall reconnect to the redirect target and resend the request. -2. When the redirect target itself redirects, the system shall follow at most 3 redirect hops before reporting an error. -3. When a standby controller redirects the operator to the primary, the system shall display the redirect target to inform the operator where the config was actually pushed. -4. When connecting to a controller and no primary has been elected yet, the system shall time out after 30 seconds and report "no primary available" to the operator. - -### Requirement 4: Validation - -**Objective:** As an XCE operator, I want validation errors caught early, so that invalid configuration never reaches a running cluster. - -#### Acceptance Criteria - -1. When a YAML file fails JSON Schema validation, the system shall report the specific validation error with file name, line/field path, and the constraint violated, then exit non-zero. -2. When a YAML file uses the reserved name `"lost_and_found"` for a queue or host, the system shall reject it with a clear error message explaining the reservation, even if it passes schema validation. -3. When a YAML file contains duplicate entries (duplicate queue names, host names, resource names, group names, limit names — each checked within its own file/section; a limit name is not required to differ from a queue, host, or group name), the system shall report the duplicates and exit non-zero. -4. When a YAML file references an undefined entity (e.g., a queue references a host group that is not defined, or a limit references a user/queue/project/host group or a `resname` that is not defined in `resources.yaml`), the system shall warn the operator but allow the build to proceed (the controller performs final referential integrity checks at runtime). -5. `resources.yaml` is the exhaustive resource catalog: every `resname` referenced elsewhere (e.g., `limits.yaml`, `queues.yaml` `sched_stop`/`res_req`/`gpu_req`, `hosts.yaml` `sched_stop`) — including built-in/dynamic resources such as `slots`, `mem`, and the 8 per-host attributes the worker registers automatically (`type`, `model`, `cpuf`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks` — see `add_hostattrs_resvals()`, `controller/worker.c`) — shall be declared as an entry in `resources.yaml`. The system shall not treat any resource name as an implicit built-in exempt from this catalog; Criterion 4's undefined-entity warning applies equally to built-in resource names. - -### Requirement 5: xcecommon Shared Package - -**Objective:** As a future Python tool author, I want a shared `xcecommon` package providing capnp schema loading, wire protocol, and config model types, so that I don't duplicate protocol code across tools. - -#### Acceptance Criteria - -1. The `xcecommon` package shall provide a capnp schema loader that reads `controller.capnp` and exposes the compiled Python types (`ClusterConfig`, `Request`, `Reply`, `UpgradeConfigRequest`, `UpgradeConfigReply`, and all nested types). -2. The `xcecommon` package shall provide a wire protocol module that handles TLS connection, capnp message framing (8-byte uint64 size prefix in host byte order), send, and receive. -3. The `xcecommon` package shall provide YAML→capnp mapping functions that convert validated YAML data to `ClusterConfig` capnp struct fields. -4. Where the `xcecommon` package is installed for development, it shall be importable under the name `xcecommon` from any tool within the project. -5. **(2026-07-21)** The `xcecommon` package shall bundle its own copy of `controller.capnp`/`c.capnp` and the 7 JSON Schema files (one per `config build` input file), synced from `src/schema/` and `src/configs/schema/` respectively at build time (see Requirement 8.1/8.2) and packaged so they are available regardless of install mode (`pip install -e .` or `pip install .`). Neither schema is read from a path outside the installed package at runtime; an operator-supplied `configs_dir/schema/` directory, if present, is not consulted (superseded Requirement 1.6). - -### Requirement 6: Dry-Run Push (Nice-to-Have) - -**Objective:** As an XCE operator, I want to validate that the controller will accept a config before committing to the push. - -#### Acceptance Criteria - -1. When the operator runs `xceadmin config push --dry-run <cluster.bin>`, the system shall connect to the controller, probe the generation, validate SHA and version compatibility, and report whether the config would be accepted, without actually committing the config to the cluster. -2. When dry-run passes, the system shall report "OK: controller will accept this config (generation N→M)" and exit 0. -3. When dry-run fails, the system shall report the specific rejection reason and exit non-zero. - -### Requirement 7: Error Handling & Resilience - -**Objective:** As an XCE operator, I want clear, actionable error messages for all failure modes, so that I can diagnose and fix configuration issues without reading controller logs. - -#### Acceptance Criteria - -1. If the TLS connection to the controller fails, the system shall report "cannot connect to <host:port>: <reason>" and exit non-zero. -2. If the controller returns a `Reply` with non-zero `status`, the system shall report the status code and any error text from the reply, then exit non-zero. -3. If the capnp binary is corrupted or cannot be decoded, the system shall report "invalid cluster.bin: <reason>" and exit non-zero. -4. If the `controller.capnp` schema file is not found at the expected path, the system shall report the missing path and suggest checking the installation. -5. If the operator interrupts the push (SIGINT), the system shall clean up the TLS connection and exit gracefully with a "cancelled" message. -6. The system shall time out after 30 seconds on any network operation and report "timeout connecting to <host:port>". - -### Requirement 8: Skill & Harness Integration - -**Objective:** As a developer, I want xceadmin integrated into the existing development toolchain, so that building, deploying, and testing XCE clusters uses consistent tooling. - -#### Acceptance Criteria - -1. ~~When the developer runs `/xce-build xceadmin`, the system shall make the `xceadmin` CLI available on PATH and run the full test suite, reporting pass/fail.~~ **Superseded (2026-07-22)**: `xceadmin` is now meson-built (`src/xceadmin/meson.build`) — `xce-build.sh xceadmin` delegates to `meson compile -C "$BUILD_DIR" xceadmin`, and `nix build .#xcecore[_debug]` installs a standalone `result/bin/xceadmin` (no venv at runtime). See design.md's File Structure Plan Amendment. -2. ~~When the developer runs `/xce-build xceadmin release`, the system shall install `xceadmin` for production use (pinned, non-editable) and then run the full test suite against that non-editable install...~~ **Superseded (2026-07-22)**: the editable-vs-pinned distinction collapsed — every meson-driven build (`xce-build.sh xceadmin`, `meson compile`, `nix build`) now runs the pinned/non-editable install path unconditionally, a strictly stronger guarantee (exercises real `package_data` resolution every run, not just under a `release` profile). -3. ~~When the developer runs `/xce-deploy-ha` with the `--build-config` flag, the system shall invoke `xceadmin config build configs/ -o ha/data/cluster.bin` as a precondition before deploying the cluster.~~ **Superseded (2026-07-22)**: `--build-config` removed entirely — every `deploy-ha.sh` run now unconditionally builds `cluster.bin` fresh from `src/configs/*.yaml` directly into the deploy dir's own `controller/ctrl-1/conf/` (no `ha/data/` staging step, no flag needed since nothing is ever preserved across runs to opt out of rebuilding). -4. ~~When `cluster.bin` is missing from `ha/data/` during deploy, `/xce-deploy-ha` shall run `xceadmin config build` automatically (same as `--build-config`).~~ **Superseded (2026-07-22)**: `ha/data/` removed entirely (directory no longer exists). See 8.3 above — always-fresh generation directly into the deploy dir replaces the missing-file check. -5. ~~When `cluster.bin` already exists and `--build-config` is not passed, `/xce-deploy-ha` shall use the existing binary as-is (backward compatible).~~ **Superseded (2026-07-22)**: no longer applicable — nothing persists across deploys for this to preserve; every deploy is fully fresh (cert included, generated via `openssl`, self-signed, 10y validity — TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase, so a fresh pair per deploy loses no real security property versus a long-lived shared one). -6. The nix dev shell (`nix develop ./src`) shall include all Python dependencies required by xceadmin after the nix environment is updated (see Nix Environment section in brief.md). - -### Requirement 9: Deterministic Builds - -**Objective:** As an XCE operator, I want config builds to be deterministic, so that I can reproduce any historical config binary from its YAML source. - -#### Acceptance Criteria - -1. When `xceadmin config build` is run twice with the same YAML inputs and same xceadmin version, the system shall produce byte-identical `cluster.bin` output. -2. The system shall not embed build timestamps, hostnames, usernames, or any other non-deterministic data in the output binary. -3. When different operators on different machines build the same YAML with the same xceadmin version, the output shall be byte-identical. - -## Requirements Summary - -| # | Requirement | Priority | Phase | -|---|------------|----------|-------| -| 1 | Config Build | **Core** | Implementation | -| 2 | Config Push | **Core** | Implementation | -| 3 | Auto Primary Discovery | **Core** | Implementation | -| 4 | Validation | **Core** | Implementation | -| 5 | xcecommon Package | **Core** | Implementation | -| 6 | Dry-Run Push | Nice-to-Have | Follow-up | -| 7 | Error Handling | **Core** | Implementation | -| 8 | Skill & Harness | **Core** | Implementation | -| 9 | Deterministic Builds | **Core** | Implementation | diff --git a/.kiro/specs/admin-config-tool/research.md b/.kiro/specs/admin-config-tool/research.md deleted file mode 100644 index 5b0754a7..00000000 --- a/.kiro/specs/admin-config-tool/research.md +++ /dev/null @@ -1,215 +0,0 @@ -# Research & Design Decisions: admin-config-tool - -## Summary -- **Feature**: `admin-config-tool` -- **Discovery Scope**: Complex Integration — first Python component in a C/Go codebase, first external client of the controller's raw TLS wire protocol, no in-repo precedent for either. -- **Key Findings**: - - The controller's `OP_upgradeConfig` network dispatch path is currently broken for every role (standby rejected by `ha_dispatch_gate`, primary rejected by an inverted `ha_is_primary()` check in `cluster.c`) — a controller-side defect discovered during this design's discovery phase, out of this spec's boundary but blocking real E2E validation of `config push`. - - Redirect (`redirectHost`/`redirectPort`) is wired only for `OPCode_join`, never `OPCode_upgradeConfig` — auto-primary-discovery (Req 3) has no live data to act on for config push until the controller is extended. - - `pycapnp` handles `controller.capnp`'s `$C.mapname(...)` annotations without issue (declared, inert metadata under dynamic reflection) — no custom-annotation workaround needed. - - `src/frontend/xcecli/meson.build`'s `go_build`-wrapper + `custom_target()` pattern is a directly reusable template for `src/xceadmin/meson.build`, resolving brief.md's open question on Python package location/build integration. - -## Research Log - -### TLS setup and certificate validation -- **Context**: `config push` needs to establish the same trust model the controller cluster already uses (brief.md Req 2.3-2.4 assumption). -- **Sources Consulted**: `src/lib/netio.c:1208-1243` (`create_ssl_context`), `src/controller/myapp.c:94-109`, `src/controller/main.c:374-375,90`, `src/controller/join.c:13-16`. -- **Findings**: - - `create_ssl_context(cert_file, key_file)` builds a client context (`TLS_client_method()`) when no cert/key is given, a server context otherwise. `myconf->certfile`/`keyfile` resolve to `<certdir>/xce.crt`/`xce.key`. - - The controller's own outbound client context explicitly sets `SSL_CTX_set_verify(client_ssl_ctx, SSL_VERIFY_NONE, NULL)` (`myapp.c:109`). - - No `SSL_CTX_set_verify(..., SSL_VERIFY_PEER, ...)`, `X509_check_host`, or `SSL_get_peer_certificate` call exists anywhere in the codebase. `validate_host()` (`join.c:13-16`) is an unrelated stub (`return 1;`) used only in the cluster-join path, not TLS. - - No minimum TLS version or cipher-suite pinning exists (`SSL_CTX_set_min_proto_version`/`set_ciphersuites` not found). -- **Implications**: `xceadmin` should connect with `ssl.CERT_NONE`/`check_hostname=False` to match current behavior exactly — this is encryption-only transport security throughout XCE today, not a weakening introduced by this spec. Captured in design.md Security Considerations. - -### Wire framing byte order -- **Context**: brief.md asserted "8-byte uint64 size prefix in host byte order" — needed independent confirmation before committing a Python `struct.pack` format string. -- **Sources Consulted**: `src/include/netio.h:30`, `src/lib/netio.c:111-199`, `src/lib/request.c:863-938`, `src/controller/config_repl.c:1925-1968`. -- **Findings**: `io_packet_t.sz` (`uint64_t`) is written/read as raw memory (`&pkt->sz`, `sizeof(...)`) directly into `SSL_read`/`SSL_write` — no `htole64`/`htobe64`/`ntohll` anywhere in the path. `request.c:863-864` states this explicitly in a comment; `config_repl.c`'s `xce_probe_config_tag` send/recv code (`:1925-1968`) follows the identical pattern. -- **Implications**: Python must use `struct.pack("=Q", size)` (native byte order), not network byte order (`">Q"`). Documented in design.md `xcecommon.wire`. - -### OP_upgradeConfig dispatch path -- **Context**: Verifying the request would actually reach `do_upgrade_config_with_bytes()` on a real cluster before designing around it as a working dependency. -- **Sources Consulted**: `src/controller/ha.c:3794-3960` (`ha_dispatch_gate`), `src/controller/cluster.c:238-682` (`myreq_handler`), `src/controller/config_repl.c:1591-1810`, `src/controller/config_repl.h:347-349`, `src/controller/testsuite/test_config_repl.c:2042`, `test_commit_barrier_rollout.c:1484,1531`. -- **Findings**: - - `ha_dispatch_gate()`'s `CTRL_STANDBY` allow-list (`ha.c:3926-3951`) does not include `OPCode_upgradeConfig` (only `upgradeConfigAck` is listed) — standbys are rejected with `-XCE_ERR_STANDBY` before reaching the opcode switch. - - For `CTRL_PRIMARY` with quorum, the gate falls through (`ha.c:3880`) — primaries pass the gate. - - But `cluster.c:667-679`'s in-switch handler does `if (ha_is_primary(&c->ha)) { /* reject, -EAGAIN */ } else { handle_upgrade_config(...) }` — i.e., it only calls the real handler for a **non-primary**, the opposite of what passed the gate. - - Net: standby → rejected at the gate; primary → passes the gate, rejected by the inverted in-switch check. No role reaches `handle_upgrade_config()`/`do_upgrade_config_with_bytes()` via the network today. That function is exercised only by unit tests calling it directly. - - `config_repl.h:347-349`'s doc comment ("handle an OP_upgradeConfig request from an admin tool") and the handler's actual logic (WAL marker commit, `push_config_to_standbys`, restart signal) both imply it should run on the **primary** — the `cluster.c:671` guard looks inverted/stale relative to both the dispatch gate and the handler's own semantics. -- **Implications**: This is a controller-side defect, not an xceadmin gap — out of this spec's boundary (see design.md Boundary Commitments), but it makes real-cluster E2E testing of `config push` currently impossible. Testing Strategy in design.md treats the mock-controller test suite as load-bearing, not merely a dev-loop convenience, until this is fixed. Recommend logging this as a bug/blocking item outside this spec's implementation tasks. - -### Redirect wiring scope -- **Context**: Requirement 3 (Auto Primary Discovery) assumes `redirectHost`/`redirectPort` get populated on a standby's reply to any request, including `upgradeConfig`. -- **Sources Consulted**: `src/schema/controller.capnp:573-595` (`RequestReply`), `src/controller/join.c:27-31,64,277`. -- **Findings**: `fill_redirect_reply()` is defined at `join.c:27-31` and its only production call site is `join.c:277`, inside the `OPCode_join` handling path. No other opcode handler calls it. -- **Implications**: `xceadmin.discovery`'s redirect-following is implemented generically against the `RequestReply` schema shape (correct for any opcode that does populate these fields, e.g. `join`), but will not actually fire for `config push` until the controller extends `fill_redirect_reply()`'s call sites to include `upgradeConfig`. Documented as an Out-of-Boundary item and Revalidation Trigger in design.md. - -### ProbeConfigTag liveness -- **Context**: `config push --dry-run` (Requirement 6) was planned around reusing `ProbeConfigTag`. -- **Sources Consulted**: `src/controller/config_repl.c:1811-1998` (`xce_probe_config_tag` + doc comment), `src/schema/controller.capnp:821-827`, `ha.c:7409`, `ha.c:3828,3860,3889,3911,3943` (dispatch-gate allow-lists). -- **Findings**: `xce_probe_config_tag()` is a direct-struct RPC (no `Request`/`Reply` wrapper) with only test callers (`test_ha_bootstrap.c:259` and a mock in `test_ha_bootstrap_support.c`). No server-side dispatch case for `OPCode_probeConfigTag` exists anywhere in `cluster.c`, despite `ha_dispatch_gate` listing the opcode as allowed for every role — the allow-list entry is moot because nothing answers it. A stale doc comment at `ha.c:7409` claims it's "already called in myapp_run before ha_start_election_async"; no such call exists in `myapp.c` or anywhere in production code. -- **Implications**: `config push --dry-run`'s wire client can and should still be implemented against the documented contract (Requirement 6 remains valid future-facing design), but it has no live counterpart to call today. Reflected in design.md as an Out-of-Boundary item; Requirement 6 stays Nice-to-Have/Follow-up as already scoped in requirements.md. - -### No existing external-client precedent -- **Sources Consulted**: `src/tools/` (`xce-build.sh`, `deploy-ha.sh`, `go_build`, `appimage/entry.sh`), `src/controller/testsuite/test.c` (built as `xce_test`), `src/frontend/xcecli/common/request/request.go`, repo-wide `find . -iname "*.py"`. -- **Findings**: `src/tools/` has no socket/capnp client. `xce_test` is a WAL/snapshot file-inspection CLI with zero network code. `xcecli` (Go) speaks HTTP/JSON to `apiserver`, never the raw wire protocol. No Python file exists anywhere in the repository. -- **Implications**: `xceadmin` is the first client of this wire protocol from outside a C daemon — there is no reference implementation to diff against; correctness rests entirely on independently re-deriving the framing/union contract from the C source (done above), not on copying an existing pattern. - -### Build tooling precedent for a non-C, non-Meson-native component -- **Sources Consulted**: `src/tools/xce-build.sh:52,63-131`, `.claude/skills/xce-build/SKILL.md`, `.claude/skills/xce-deploy-ha/SKILL.md`, `src/nixSupport/xce.nix:18-70`, `src/frontend/xcecli/meson.build`, `src/meson.build:34-35,84-86`, `src/meson.options:3-6`. -- **Findings**: `xce-build.sh` dispatches on component name via a `case` statement; `apiserver`/`xcecli` branches additionally shell out to `make test`/`go test`. `xcecli`'s `meson.build` wraps `go build` via a `custom_target()` calling a generic `tools/go_build` script, gated by the `component` meson option (`combo`: `all|xcecore|xcecli`) and conditionally included via `if component != 'xcecore': subdir('frontend/xcecli')`. `xce.nix`'s `nativeBuildInputs` has zero Python packages today — confirmed genuinely absent, not just undocumented. -- **Implications**: `src/xceadmin/meson.build` should mirror `xcecli`'s pattern exactly (a new `tools/py_build` wrapper + `custom_target()` + a `component` choice or unconditional `subdir('xceadmin')`), resolving brief.md's open question #9 (Python package layout) in favor of `src/xceadmin/`, consistent with `structure.md`'s "each major subsystem lives under `src/<component>/`" convention. - -### pycapnp schema loading and message encoding -- **Context**: Confirm `controller.capnp`'s C-specific annotations (`$C.mapname(...)`) don't break pycapnp's schema compilation, and confirm the right API surface for non-RPC encode/decode. -- **Sources Consulted**: [pycapnp GitHub/README](https://github.com/capnproto/pycapnp), [pycapnp CHANGELOG](https://github.com/capnproto/pycapnp/blob/master/CHANGELOG.md), [Quickstart (2.2.4 docs)](https://capnproto.github.io/pycapnp/quickstart.html), [capnp.pyx source](https://github.com/capnproto/pycapnp/blob/master/capnp/lib/capnp.pyx). -- **Findings**: `$C.mapname(...)` is a properly declared annotation (via `c.capnp`'s `annotation mapname @0x...`), not an unknown construct — pycapnp's `DynamicStruct` reflection parses it as inert metadata; the only real failure mode is an unresolved `/c.capnp` import path, handled via `capnp.load(path, imports=[schema_dir])`. `msg.write(f)` calls `f.fileno()` and writes to the raw fd, bypassing any Python-level wrapper — safe for `open(path, 'wb')`, unsafe for a socket. `msg.to_bytes()`/`Struct.from_bytes()` are pure in-memory and the correct choice for the socket path. -- **Implications**: No custom annotation-stripping preprocessing needed before schema load. `xcecommon.wire` must use `from_bytes()`/`to_bytes()` exclusively (never `.write()`/`.read()` on a live socket) — captured as an explicit constraint in design.md. - -### pycapnp + Python ssl module composition -- **Findings**: pycapnp's async/KJ-loop machinery (`AsyncIoStream`, `capnp.run()`) is scoped to its RPC/vat-network layer (`TwoPartyClient/Server`) and does not intrude on plain synchronous `from_bytes()`/`to_bytes()` usage. `ssl.SSLContext().wrap_socket()` gives a normal `SSLSocket` whose `.recv()`/`.sendall()` perform the framing at the plaintext layer; pycapnp never touches the socket object directly in this design. -- **Implications**: Confirms the `xcecommon.wire` design (manual framing over `ssl.SSLSocket`, in-memory capnp encode/decode) has no hidden event-loop conflict. - -### click and pytest conventions -- **Findings**: Current stable `click` (8.3.x/8.4.x) idiom for grouped subcommands: `@click.group()` → `@cli.group()` → `@config.command()`, giving `xceadmin config build`/`xceadmin config push` naturally. `pytest-regressions`' `file_regression` fixture (binary-capable, deliberate `--force-regen` updates) is the more actively maintained golden-file library versus `pytest-golden` (YAML-only, no binary support) — evaluated but not adopted (see Design Decisions) in favor of a plain `assert produced == expected_bytes` fixture comparison, since the byte-equality need here doesn't need the extra dependency's ergonomics. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|----------------------|-------| -| Layered CLI over shared library (selected) | Thin `click` CLI (`xceadmin`) over a protocol/schema/mapping library (`xcecommon`) with no CLI dependency | Matches project's inward-dependency convention; `xcecommon` directly reusable by future Python `xcecli` (brief.md Future Roadmap); each layer independently testable | None material for this scope | — | -| Single-file script | One `xceadmin.py` doing everything | Fastest to write initially | Not reusable by future `xcecli`; violates project's "one domain per file" convention (`structure.md`); harder to unit-test wire framing in isolation | Rejected | -| Two separate pip distributions (`xceadmin`, `xcecommon`) from day one | Split packaging immediately | Cleaner dependency graph if `xcecommon` gets a second consumer soon | No second consumer exists yet; premature packaging overhead (Simplification lens) | Deferred, not rejected — revisit when Python `xcecli` work starts | - -## Design Decisions - -### Decision: `src/xceadmin/` as the package location -- **Context**: brief.md left this as Open Design Question #9 (candidates: `src/xceadmin/`, `tools/xceadmin/`, top-level `xceadmin/`). -- **Alternatives Considered**: 1) `tools/xceadmin/` alongside `deploy-ha.sh`. 2) top-level `xceadmin/`. 3) `src/xceadmin/`. -- **Selected Approach**: `src/xceadmin/`, mirroring `src/frontend/`'s existing precedent for a non-C, non-Meson-native language living directly under `src/` with its own build wrapper. -- **Rationale**: `structure.md`'s "Organization Philosophy" states "each major subsystem lives under `src/<component>/`" — `xcecli` (Go) already establishes that this applies across languages, not just C. Discovery confirmed a directly reusable `custom_target()` + wrapper-script pattern from `xcecli/meson.build`. -- **Trade-offs**: None material — this is the path of least resistance given an existing, working precedent. -- **Follow-up**: Confirm during implementation whether `component` meson option needs a new `xceadmin` choice or can use unconditional `subdir()` inclusion (both patterns exist for `xcecli` depending on option value — see `src/meson.build:84-86`). - -### Decision: plain byte-equality golden-file tests, not `pytest-regressions` -- **Context**: External research surfaced `pytest-regressions` as the more idiomatic 2025/2026-era binary golden-file library versus brief.md's unspecified "compare binary, verify SHA-256." -- **Alternatives Considered**: 1) `pytest-regressions` (`file_regression.check(..., binary=True)`). 2) `pytest-golden` (YAML-only, ruled out — no binary support). 3) Plain `assert produced_bytes == open(expected_path, 'rb').read()` in a parametrized fixture. -- **Selected Approach**: Option 3 — plain assertion. -- **Rationale**: `pytest-regressions` is not yet in this project's nix-pinned dependency list (`brief.md`'s Nix Environment section) and would need independent nixpkgs-availability verification before adoption; the actual requirement (9.1-9.3, byte-for-byte determinism against a committed fixture) doesn't need `file_regression`'s deliberate-update ergonomics for a small, stable fixture set (Simplification lens). -- **Trade-offs**: Loses `--force-regen`'s one-flag golden-file update convenience; gains zero new dependencies to verify/pin. -- **Follow-up**: Revisit if the fixture set grows large enough that manual golden-file regeneration becomes a recurring pain point. - -### Decision: generic (opcode-agnostic) redirect-following and wire exchange -- **Context**: Discovery found `fill_redirect_reply()` only wired for `OPCode_join`, and two distinct message shapes (`Request`/`Reply`-wrapped vs. `ProbeConfigTagRequest`/`Reply` direct-struct) both need send/receive. -- **Alternatives Considered**: 1) An `upgradeConfig`-specific redirect function, built now and generalized later if `dry-run`/other opcodes need it. 2) One generic `follow_redirects()` against the `RequestReply` shape, and one generic `send_request()`/framing pair reused by both message shapes. -- **Selected Approach**: Option 2, generic from the start. -- **Rationale**: Generalization lens — both consumers (upgradeConfig today, join-shaped mocks in tests, potentially dry-run's direct-struct variant) are instances of the same "read `RequestReply`-shaped fields, frame-and-exchange bytes" problem; building the opcode-specific version first and generalizing later would mean deleting code, not adding it (the redirect logic and framing logic are trivial regardless — no meaningful cost to being generic from day one). -- **Trade-offs**: None material. -- **Follow-up**: None. - -### Decision: `push_config()`/`probe_config()` split into two functions, not one boolean-flagged function (Design Review correction, 2026-07-21, second pass) -- **Context**: A second `/kiro-validate-design` pass found the first draft's `push_config(..., dry_run: bool) -> PushResult` implied a dry-run reply has `acked_count`, which `ProbeConfigTagReply` (only a `configTag` field) does not have — the shared return type would misrepresent one of the two call shapes. -- **Alternatives Considered**: 1) Keep one function, make `PushResult` fields `Optional` for the dry-run case. 2) Two functions, two result types. -- **Selected Approach**: Option 2 — `push_config() -> PushResult` and `probe_config() -> DryRunResult`. -- **Rationale**: The two wire messages and replies share no fields; a boolean flag silently switching which fields are populated is a Type Safety anti-pattern this project's `design-principles.md` implicitly warns against ("discriminated unions for error handling" — same spirit applies to result shapes). -- **Trade-offs**: None material — strictly a clarity improvement over the first draft. -- **Follow-up**: None. - -### Decision: schema files are packaged as `package_data`, not referenced from `src/schema/` at runtime -- **Context**: User question after design approval — does `pip install .` (Req 8.2, non-editable) actually work? It doesn't as originally drafted: non-editable installs copy files into `site-packages`, severing any fixed relative path back to `src/schema/controller.capnp`. A `__file__`-relative walk (e.g. `Path(__file__).parents[3] / "schema"`) only survives editable installs, where `__file__` still resolves into the checked-out repo tree. -- **Alternatives Considered**: 1) Require `--schema-path` on every invocation (no default) — rejected, bad ergonomics for the common case. 2) `__file__`-relative resolution — rejected, breaks for `pip install .`. 3) Package a synced copy of the schema as `package_data`, resolved via `importlib.resources` (packaging-aware, correct for both install modes). -- **Selected Approach**: Option 3. `src/schema/{controller,c}.capnp` synced (`cp`) into `src/xceadmin/src/xcecommon/schema_files/` by `xce-build.sh`'s `xceadmin)` case on every build, packaged via `pyproject.toml` `package_data`, resolved at runtime via `importlib.resources.files("xcecommon") / "schema_files" / "controller.capnp"`. -- **Rationale**: `importlib.resources` is the standard-library-correct way to locate packaged non-code files regardless of install mode (editable, wheel-installed, zipped) — it doesn't care where `site-packages` physically is. `src/schema/` remains the single source of truth (this spec doesn't own it, doesn't hand-edit the copy); the sync step is mechanical and runs every build, so it can't drift stale. -- **Trade-offs**: One more build-step to keep in sync (mitigated: it's a single `cp`, not a generation/compilation step); the packaged copy is momentarily stale between a `controller.capnp` edit and the next `xce-build xceadmin` run — acceptable, matches how any other build artifact in this repo already works (nothing is watched/hot-reloaded). -- **Follow-up**: None. - -## Risks & Mitigations -- **Controller `OP_upgradeConfig` dispatch defect (`cluster.c:667-679` vs `ha.c:3926-3951`)** — no role can process a push today. Mitigation: documented as an explicit Out-of-Boundary/Revalidation-Trigger item in design.md; `config push` implementation and testing proceed against the documented contract via a mock controller; real-cluster E2E is blocked until a separate fix (outside this spec) lands. Recommend raising this as its own tracked defect. -- **Redirect not wired for `upgradeConfig`** — auto-discovery (Requirement 3) is untestable end-to-end against a real cluster for this opcode specifically. Mitigation: generic implementation is correct and forward-compatible; mock-controller tests exercise the logic using synthetic `redirectHost`/`redirectPort` replies. -- **`ProbeConfigTag` has no server handler** — `config push --dry-run` cannot be validated live. Mitigation: already Nice-to-Have/Follow-up in requirements.md; client implemented against the schema contract, tested via mock only. -- **First Python component in the repo** — nix/build/CI integration has no precedent to copy exactly (only an analogous one, `xcecli`'s Go integration). Mitigation: `xcecli`'s `meson.build`/`custom_target()` pattern is close enough structurally to de-risk this; flagged for extra scrutiny during task review. -- **`pycapnp` C-extension build inside nix** — `pip install pycapnp` normally compiles a C extension against `capnproto`; getting this wrong would break `nix develop`. Mitigation: use `python3Packages.pycapnp` from nixpkgs (pre-built against nix-provided `capnproto`), not a bare `pip install pycapnp`, exactly as brief.md's Nix Environment section already specifies. - -### Decision: `xceadmin` is not wired into meson (Design Review correction, 2026-07-21) -- **Context**: `/kiro-validate-design` review flagged that the initial design draft's `src/xceadmin/meson.build` (mirroring `xcecli`'s `custom_target()`) was architecturally unsound — Go's pattern gates on a single static-binary output; `pip install -e .` mutates a venv/site-packages tree with no single trackable output for meson to gate on. -- **Alternatives Considered**: 1) Force it through meson anyway via a stamp-file `custom_target()`. 2) Skip meson entirely, dispatch purely from `xce-build.sh`'s case statement. -- **Selected Approach**: Option 2. -- **Rationale**: `brief.md`'s own harness sketch (`xce-build.sh`'s `xceadmin)` case) already specified direct `pip install`/`pytest` dispatch with no `meson compile` step — the meson integration was an unjustified addition during design drafting, not something the original harness plan called for. -- **Trade-offs**: `xceadmin` doesn't participate in `meson compile -C build <targets>` the way C/Go components do; acceptable since it has no C-extension compile step of its own that meson would need to orchestrate (pycapnp's C extension is pip/nix's concern, not this project's meson graph). -- **Follow-up**: None — resolved at design time, `component` meson option is unchanged. - -### Decision: bundle the 7 JSON Schema files too, not just `controller.capnp` (2026-07-21, post-approval user question) -- **Context**: After design approval, user asked whether the JSON schemas used to validate YAML input should also ship with `xceadmin`, the same way `controller.capnp` does. Initial answer was "no, they're operator-supplied input per Req 1.1/1.6" — but on reconsideration, that framing didn't hold up: the JSON schemas are the formal spec of what `xcecommon.mapping`'s hardcoded field-extraction logic (`groupname`, `exclude_members`, `resname`, …) actually expects, not independent operator content. An operator-managed copy that drifts from the shipped mapping code risks a *silent* failure mode — validation passes on YAML that then maps wrong (or rejects YAML that would map fine) — arguably worse than the capnp case (hard crash) because nothing raises. No requirement or brief text ever described org-custom validation schemas as an actual use case; the `configs_dir/schema/` layout in `brief.md`'s original sketch was inherited unreflectively from `src/configs/`'s example layout, not a deliberate feature. -- **Alternatives Considered**: 1) Keep operator-supplied, `configs_dir/schema/` required (original design). 2) Bundle as `package_data`, same mechanism as `controller.capnp`, `configs_dir/schema/` becomes irrelevant. -- **Selected Approach**: Option 2 — confirmed with the user directly (`AskUserQuestion`, "Yes, bundle" selected over "Keep operator-supplied"). -- **Rationale**: Consistency with the capnp decision's own rationale (version-lock a tool-internal contract to the release that consumes it); removes an entire operator-facing requirement (maintaining a `schema/` folder they didn't author) for zero documented benefit forfeited. -- **Trade-offs**: No mechanism left for an org to enforce custom/stricter YAML validation rules — acceptable, since none was ever a stated requirement; can be revisited as a follow-up feature (e.g. a future `--extra-schema` overlay) if a real need surfaces. -- **Process note**: this reopened an *approved* acceptance criterion (Requirement 1.6) — amended requirements.md directly (added 5.5, rewrote 1.6 as "resolved, superseded") rather than looping back through `/kiro-spec-requirements`, since the user's direct confirmation in this conversation constitutes the approval for this specific amendment. -- **Follow-up**: None. - -## References -- [pycapnp GitHub](https://github.com/capnproto/pycapnp) — README, quickstart usage patterns -- [pycapnp CHANGELOG](https://github.com/capnproto/pycapnp/blob/master/CHANGELOG.md) — confirms synchronous `from_bytes`/`to_bytes`/`write`/`read` remain available through 2.2.x after RPC-layer sync support removal in 2.0.0b1 -- [pycapnp Quickstart (2.2.4 docs)](https://capnproto.github.io/pycapnp/quickstart.html) -- [capnp.pyx source](https://github.com/capnproto/pycapnp/blob/master/capnp/lib/capnp.pyx) — verified `.write()`'s `fileno()` behavior directly from source -- [Click — Commands and Groups](https://click.palletsprojects.com/en/stable/commands-and-groups/) -- [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/overview.html) / [GitHub](https://github.com/ESSS/pytest-regressions) — evaluated, not adopted (see Design Decisions) -- [pytest-golden PyPI](https://pypi.org/project/pytest-golden/) — ruled out, YAML-only -- `src/schema/controller.capnp`, `src/schema/c.capnp`, `src/lib/netio.c`, `src/lib/request.c`, `src/controller/{ha,cluster,config_repl,join,myapp,main}.c`, `src/frontend/xcecli/meson.build`, `src/nixSupport/xce.nix` — primary in-repo sources for all findings above - ---- - -## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28) - -### 1. Current State Investigation - -All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles: - -| Spec | Phase | Scope | -|---|---|---| -| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` | -| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller | -| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing | - -**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.** - -**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."* - -### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker - -`admin-config-tool/spec.json`'s `blocked_on` array still lists: -> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation" - -This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity. - -- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists. -- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note). - -### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477 - -`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this. - -- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle). -- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked. - -### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration - -`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here. - -### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence - -Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways. - -### Summary Table - -| # | Finding | Classification | Effort | Risk | Action | -|---|---|---|---|---|---| -| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation | -| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable | -| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None | -| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None | - -### Research Needed (carried forward) -- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only. - -**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap. diff --git a/.kiro/specs/admin-config-tool/spec.json b/.kiro/specs/admin-config-tool/spec.json deleted file mode 100644 index 4b8f5b7f..00000000 --- a/.kiro/specs/admin-config-tool/spec.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "feature_name": "admin-config-tool", - "created_at": "2026-07-06T01:30:00Z", - "updated_at": "2026-08-16T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [ - "RESOLVED 2026-07-25 (bug-1468, commit 959b0cda): controller OP_upgradeConfig dispatch defect (cluster.c) -- myreq_handler's dispatch guard was inverted (introduced by commit fd93997f, 2026-05-31), rejecting the only role that could ever legitimately reach it and making handle_upgrade_config() permanently unreachable from any network client. Fixed by config-upgrade-snapshot task 8.6 (which independently hit the same wall building a live OP_upgradeConfig sender). The controller-side path this blocker described has since been exercised live and successfully many times via config-upgrade-snapshot's --config-upgrade soak. push_config() itself (xceadmin's TLS/redirect/framing code) remains mock-tested only -- a live `xceadmin config push` run against a deployed cluster would give first-hand confirmation of the CLI's own wire code, but the blocking condition (dispatch unreachable) no longer exists.", - "RESOLVED 2026-07-22 (bug-1461, commit e53c086): controller.capnp's struct Data shadowed the capnp builtin blob type, miscompiling UpgradeConfigRequest.sha256/.bytes — fixed at the schema root (renamed to XceData), unblocking push_config()'s request construction. The dispatch defect above (bug-1468) is a separate issue and is now ALSO resolved (see entry above) — push_config() remains mock-tested only, no live xceadmin-specific E2E run yet, but nothing blocks one." - ], - "gitea": { - "milestone_id": 1, - "issues": { - "requirements_design": 22, - "implementation": 23, - "validation": 24 - } - }, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-08-16T00:00:00Z", - "note": "Corrected stale phase metadata (was 'tasks-generated' though 20/21 tasks were already [x] complete). Task 6.3's remaining blocker -- bug-1485's ~24/25 flaky primary-push failure -- was already followed up (2026-08-01): DEVNULL stderr redirect replaced with real per-node log capture, test re-run 40x standalone with zero reproductions and zero captured crash evidence. Decision reached: mergeable as-is, bug-1485 downgraded to low-priority/environment-flake rather than a blocker. Task 6.3 now marked [x] complete.", - "known_limitations": [ - "bug-1485: primary-push E2E test (test_e2e_live_ha_push.py) had a ~24/25 flaky failure rate in one investigation round (elected primary observed <defunct> shortly after promotion) that could not be reproduced in 40 clean standalone runs; left OPEN as a low-priority environment-flake, not closed, since absence of reproduction here is not proof of absence. Real stderr capture is now permanently in place to diagnose any recurrence.", - "6.2's own finding stands: a genuinely single-node config push always exits 1 (ackedCount == 0, no standbys to ACK) even on full server-side success -- spec-consistent per cli.py/design.md's exit-0-iff-acked_count>0 postcondition, not a regression, but a future task may want a distinct single-node success signal.", - "push_config()'s TLS/redirect/framing code has had live E2E coverage only against the single-node (6.2) and 3-node (6.3) topologies exercised in this spec's own tests -- no production/CI live run yet." - ] - } -} diff --git a/.kiro/specs/admin-config-tool/tasks.md b/.kiro/specs/admin-config-tool/tasks.md deleted file mode 100644 index 9fdb1098..00000000 --- a/.kiro/specs/admin-config-tool/tasks.md +++ /dev/null @@ -1,200 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: package scaffolding, shared errors, build/toolchain integration -- [x] 1.1 Set up the xceadmin Python package skeleton - - Editable install registers a real `xceadmin` console-script entry point backed by an empty, working click group (no subcommands yet — `config build`/`config push` are added by task 4.1, not this task) - - `xcecommon` is importable as a standalone package independent of `xceadmin`'s CLI - - Observable: running the entry point after an editable install prints usage/help text without error, showing zero commands - - _Requirements: 5.4_ - -- [x] 1.2 Establish a shared error-handling foundation - - One common base exception type carries a human-readable message - - Every specific failure mode (schema not found, validation failure, reserved name, duplicate entry, connection timeout, corrupt reply, too many redirects, no primary available) has its own subclass - - Observable: raising any of these subclasses and catching the common base type recovers the message - - _Requirements: 7.1, 7.2, 7.3, 7.4_ - -- [x] 1.3 Wire xceadmin into the project's build and dev-shell tooling - - Nix dev shell provides the Python interpreter and all dependencies needed to build and test xceadmin - - Running the build command for xceadmin performs an editable install by default and a pinned non-editable install under the release profile, running the test suite after whichever install just happened - - The build step keeps the packaged copies of the wire-protocol schema and the validation schemas in sync with their source locations elsewhere in the repo, and packages both correctly under either install mode - - Build/test command documentation, permission allowlist entries, and automatic formatting/linting on Python file edits all exist for the new component - - Observable: running the build command for this component from a clean checkout succeeds and reports test pass/fail - - _Requirements: 5.5, 8.1, 8.2, 8.6_ - -- [x] 2. Core: xcecommon shared library -- [x] 2.1 (P) Validate operator-authored YAML against the bundled schemas - - Each of the seven config file types can be checked against its own bundled validation schema, independent of anything the operator supplies - - Reserved queue/host names are rejected even when otherwise schema-valid - - Duplicate names within a single file/section are rejected - - Cross-references to undefined groups or resources are flagged as warnings without blocking the build - - Observable: validating a config directory with no schema files present anywhere in it still succeeds - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 5.5_ - - _Boundary: xcecommon.validation_ - -- [x] 2.2 (P) Load the cluster wire-protocol schema - - The wire-protocol schema loads once per process and exposes every message type xceadmin needs to build or decode - - Loading works identically whether xceadmin was installed in editable or pinned mode - - A missing or corrupted schema copy produces a clear, actionable error rather than a crash - - Observable: loading the schema under a pinned (non-editable) install still resolves the bundled copy correctly - - _Requirements: 5.1, 5.5, 7.4_ - - _Boundary: xcecommon.schema_ - -- [x] 2.3 (P) Map validated YAML into cluster configuration fields - - Every field the cluster configuration binary format defines is either populated from validated YAML or deliberately left untouched, with no field silently dropped - - List-based sections preserve the exact order they appeared in the YAML - - No timestamp, hostname, or username value ever enters the mapped output - - Building the same input twice produces identical output - - _Requirements: 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3_ - - _Boundary: xcecommon.mapping_ - - _Depends: 2.2_ - -- [x] 2.4 (P) Establish the TLS connection and wire framing to the controller - - A single request can be sent and a single reply received over a TLS connection using the cluster's existing certificate trust model - - Message framing matches the controller's own byte-for-byte, verified against known-good byte sequences - - The connection always closes cleanly, including on error or interruption - - _Requirements: 2.1, 2.3, 2.4, 5.2, 6.1, 7.1, 7.3, 7.6_ - - _Boundary: xcecommon.wire_ - - _Depends: 2.2_ - -- [x] 3. Core: xceadmin CLI orchestration -- [x] 3.1 (P) Compile a configs directory into a deployable cluster binary - - Running the build command against a directory of YAML files produces a binary output file with a printed checksum - - A missing individual YAML file is treated as an empty section, not an error - - Unrecognized-but-valid YAML fields produce a warning, not a failure - - Output can be directed to a specific path or to standard output - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.9, 8.3, 8.4_ - - _Boundary: xceadmin.build_ - - _Depends: 2.1, 2.3_ - -- [x] 3.2 (P) Follow controller redirects to find the primary - - A reply pointing at a different host/port is followed automatically, up to a bounded number of hops - - Exceeding the hop limit or waiting past the connection timeout produces a clear error naming the last host contacted - - Observable: a chain of up to three redirects resolves to a final destination without operator intervention - - _Requirements: 3.1, 3.2, 3.3, 3.4_ - - _Boundary: xceadmin.discovery_ - - _Depends: 2.4_ - -- [x] 3.3 Push a cluster binary to the controller, with a dry-run mode - - A real push reports the assigned configuration version, acknowledgement count, and any error, exiting non-zero on failure - - A dry-run reports whether the controller would accept the config without ever committing it - - Every connection failure, protocol-level rejection, corrupt input, or missing schema maps to a specific, distinct error - - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_ - - _Boundary: xceadmin.push_ - - _Depends: 2.4, 3.2_ - -- [x] 4. Integration: wire the CLI entry point and hook xceadmin into cluster deployment -- [x] 4.1 Expose config build and config push as a single command-line tool - - `xceadmin config build` and `xceadmin config push` (with `--dry-run`) are both reachable from one installed command - - Any failure surfaces as a specific, human-readable message with a non-zero exit code, not a raw traceback - - Interrupting a push mid-flight leaves no dangling connection and reports a clean cancellation - - _Requirements: 1.2, 1.3, 2.2, 2.4, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 8.1, 8.2_ - - _Boundary: xceadmin.cli_ - - _Depends: 3.1, 3.3_ - -- [x] 4.2 Build cluster configuration automatically during HA deployment - - Deploying a cluster with the build-config option produces a fresh binary from the current YAML before startup - - Deploying with no binary present and no flag still produces one automatically - - Deploying with an existing binary and no flag leaves it untouched, matching prior behavior - - _Requirements: 8.3, 8.4, 8.5_ - - _Boundary: deploy-ha.sh_ - - _Depends: 4.1_ - -- [x] 5. Validation: test coverage across build, push, and packaging -- [x] 5.1 (P) Unit test the field-mapping logic in isolation - - Both mapper shapes (list sections and the scalar section) are verified against in-memory input with no file I/O - - The one file that feeds two separate output fields is verified explicitly - - _Requirements: 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3_ - - _Boundary: xcecommon.mapping_ - - _Depends: 2.3_ - -- [x] 5.2 (P) Unit test wire framing - - Framing round-trips correctly against a fixed, known byte sequence and against a local loopback connection - - _Requirements: 2.1_ - - _Boundary: xcecommon.wire_ - - _Depends: 2.4_ - -- [x] 5.3 (P) Verify bundled schemas resolve under every install mode - - Both the wire-protocol schema and the validation schemas resolve to real, parseable files under the currently active install - - A deliberately corrupted or absent packaged schema produces the expected installation-hint error rather than a crash - - Meaningful confirmation of the pinned install path requires running this under the release build profile - - _Requirements: 5.5, 7.4_ - - _Boundary: xcecommon.schema, xcecommon.validation_ - - _Depends: 2.1, 2.2, 1.3_ - -- [x] 5.4 (P) Golden-file test config build determinism - - Building from a minimal fixture, a full fixture covering all seven file types, and a set of intentionally invalid fixtures each produce the expected result - - Building the same fixture twice produces byte-identical output and an identical checksum - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 9.1, 9.2, 9.3_ - - _Boundary: xceadmin.build_ - - _Depends: 3.1_ - -- [x] 5.5 (P) Test every YAML validation failure mode - - Each invalid-fixture case (schema violation, reserved name, duplicate name including a cross-type collision, undefined reference) produces the documented outcome — hard failure or warning, matching the requirement - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_ - - _Boundary: xcecommon.validation_ - - _Depends: 2.1_ - -- [x] 5.6 (P) Test config push and dry-run against a mock controller - - A mock TLS server standing in for the controller confirms the full push flow: successful push, redirect-following, and every documented error path - - The same mock confirms dry-run reports both an accepted and a rejected outcome correctly - - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, 3.3, 3.4, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3_ - - _Boundary: xceadmin.push, xceadmin.discovery_ - - _Depends: 3.3_ - -- [x] 6. Fix reply envelope decode and add live E2E push coverage -- [x] 6.1 Unwrap the `Request` reply envelope before decoding as `RequestReply` - - Live-verified 2026-07-30 against a real deployed single-node controller: `config push` crashes every time with `capnp.lib.capnp.KjException: ... non-list pointer where text was expected`. Root cause: `xce_reply()` (`src/lib/request.c`) wraps every reply — including `config push`'s — in an outer `Request{opcode: OPCode_reply, body: union{replymsg @4: RequestReply}}` envelope (`src/schema/controller.capnp:645-651`) before writing it to the wire. Two call sites decode the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope, which misaligns every pointer offset: `xceadmin.discovery._redirect_target()` (crashes first, on every reply) and `xceadmin.push.push_config()`'s own `mod.RequestReply.from_bytes(reply_bytes)` (same latent bug, currently unreached only because discovery crashes first — fixing discovery alone is not sufficient). The controller side is not at fault: server logs confirm the same request's dispatch, commit, and activation all succeed for the exact bytes that then crash the client on decode. - - Both decode sites unwrap `Request.body.replymsg` before treating the bytes as a `RequestReply` — introduce one shared decode helper (e.g. in `xceadmin.discovery`, which `xceadmin.push` already imports from) rather than duplicating the unwrap logic at each call site - - `config push` against a real controller (mock or live) succeeds end to end: exits 0, prints `versionTag`/`ackedCount`, no `KjException` - - Redirect-following (standby → primary) still decodes correctly through the fixed unwrap — existing task 5.6 mock-based redirect tests continue to pass unmodified after their mock reply fixtures are updated to the real envelope shape - - _Requirements: 2.1, 2.5, 3.1, 3.2, 7.2, 7.3_ - - _Boundary: xceadmin.discovery, xceadmin.push_ - - _Depends: 3.2, 3.3_ - -- [x] 6.2 Add a live single-node E2E test for `config push` - - New test deploys one real `xce_controller` (no HA peers needed — reachability/decode correctness is what's under test, not consensus), runs `xceadmin config build` against `src/configs/`, then a real `xceadmin config push` against it over the actual TLS+capnp wire protocol — no mock server - - Test asserts exit 0, a parsed `versionTag`/`ackedCount` in the output, and confirms the controller's own log shows the config committed and activated - - Skips cleanly (not a hard failure) when no controller binary is available in the current build profile, matching this suite's existing cert-dependent skip convention (see task 5.3's Implementation Notes) - - _Requirements: 2.1, 2.5, 2.6, 3.1_ - - _Boundary: xceadmin (test-only, cross-cutting with the built controller binary)_ - - _Depends: 6.1_ - -- [x] 6.3 Add a live 3-node HA E2E test for `config push` (success path + redirect) - - Task 6.2 proved a single-node deployment always returns `ackedCount == 0` and exits 1 even on full server-side success (no standbys exist to ACK) — the `acked_count > 0` exit-0 success path, and Requirement 3's redirect-following, have never been exercised against a real controller, only against mocks (task 5.6) or a topology structurally unable to reach them (task 6.2). This task closes both gaps in one live 3-node deployment. - - Deploys a real 3-node HA `xce_controller` cluster (reuse `deploy-ha.sh`'s existing 3-node pattern, e.g. `localhost:19991/19992/19993`, staggered start for clean election per this repo's existing convention), waits for a primary to be elected - - Pushing to the primary directly: real `xceadmin config push` exits 0, reports `ackedCount > 0` (a real positive count from real standby ACKs, not the single-node `0` case), and the controller logs confirm commit/activate on the primary and replication/ack on at least one standby - - Pushing to a standby node: `xceadmin` follows the redirect to the primary (Requirement 3.1, 3.3), the push still succeeds end-to-end, and the redirect is visible either in `xceadmin`'s own stderr redirect message (`discovery.py`'s existing "redirected: host:port -> host:port" line) or equivalent captured evidence - - Skips cleanly (not a hard failure) when no controller binary is available, matching task 6.2's convention; reuses task 6.2's cleanup discipline (kill all node processes, remove scratch deploy dir, on both pass and failure paths) — check whether task 6.2's fixture/helpers can be extracted and reused for the multi-node case rather than duplicated - - _Requirements: 2.1, 2.5, 2.6, 3.1, 3.2, 3.3_ - - _Boundary: xceadmin (test-only, cross-cutting with the built controller binary)_ - - _Depends: 6.2_ - - _Closed: standby-redirect half is DONE and correct (round-1 reviewer independently re-verified: OP_upgradeConfig has no server-side redirect wiring today, matches design.md's own pre-existing "Out of Boundary" gap; test correctly locks in real current rejection behavior). Primary-push success-path test was flaky (~24/25 live fails in round-2, see bug-1485) — elected primary observed as a `<defunct>` zombie shortly after promotion. Per this note's own next step, `test_e2e_live_ha_push.py`'s DEVNULL stderr redirect was replaced with real per-node log capture (commit `a5560256`) and the test re-run 40 times standalone (25 idle-sandbox + 15 with a concurrent competing build): 40/40 PASSED, 0/120 captured stderr logs non-empty — no crash, no ASAN report, in any run. Could not reproduce the original failure rate under matching conditions; leading hypothesis is that the original ~24/25 rate was itself caused by OTHER concurrent sandbox sessions contending for resources during that investigation, not a deterministic controller bug reachable in isolation. Decision: **mergeable as-is** — bug-1485 stays open, downgraded to low-priority/environment-flake (absence of reproduction here is not proof of absence), with the now-permanent stderr-capture infrastructure ready to catch a real ASAN report or signal if it recurs in a genuinely contended sandbox or in CI._ - -## Implementation Notes - -- **1.3**: nix's bare `python3` does not put `pip` on PATH — `python3Packages.pip` had to be added to `nativeBuildInputs` alongside the packages design.md's Technology Stack table lists; not a spec gap, just an operational dependency the table didn't need to spell out. -- **1.3**: Claude Code `PostToolUse` hooks receive tool input via stdin JSON (`tool_input.file_path`), not a `$FILE_PATH` env var — brief.md's ruff-hook sketch used the env-var form, which would have silently no-op'd. The actual hooks added to `.claude/settings.json` follow this repo's existing `.wolf/hooks/post-write.js` stdin-JSON convention instead. Any future hook work in this repo should follow the same pattern, not brief.md's original sketch. -- **1.3**: `src/xceadmin/src/xcecommon/schema_files/` (synced capnp + JSON schema copies) is a build artifact, regenerated every `xce-build.sh xceadmin` run — never hand-edit it, never commit it (`src/xceadmin/.gitignore` excludes it). -- **2.1**: `check_undefined_refs` deliberately does not parse resource names out of `res_req`/`gpu_req` free-form LSF-style expression strings (e.g. `"mem>1000 rusage[mem=1000]"`) — design.md's `xcecommon.validation` component scopes this check to queue→host-group and limit→group/`resname` only. `limits.yaml`'s `projects` field and `scaler.yaml` name collisions are likewise out of scope — no `projects.yaml` catalog exists among the 7 config types, and Requirement 4.3 only enumerates queue/host/resource/group/limit names. `src/configs/hosts.yaml` has 2 pre-existing undefined refs (resource `mg`, host `h3`) that now surface as warnings — expected, not a regression; fixing the example fixture is outside this task's boundary. -- **2.2**: The real `controller.capnp` names its reply struct `RequestReply`, not `Reply` as design.md's postcondition states — an inexact design pseudo-type, corrected empirically (verified directly against `src/schema/controller.capnp`, not assumed). pycapnp 2.0.0/capnproto 1.2.0 abort the whole process (uncaught C++ exception, not a Python exception) on a missing cross-file import or malformed `.capnp` content passed to `capnp.load()` — a bare `try/except` cannot catch this. `xcecommon.schema.load_schema()` works around it by pre-validating in an isolated subprocess before ever calling `capnp.load()` in the caller's own process, converting any abort into a clean `SchemaNotFoundError`. Any future code that calls `capnp.load()` directly (bypassing this loader) reintroduces the crash risk — always go through `xcecommon.schema.load_schema()`. -- **2.3**: `map_all_sections()` (a `SECTION_MAP`-driven orchestrator, not explicitly named in design.md's Service Interface but consistent with its Contracts) is the single call `xceadmin.build` (task 3.1) needs — it treats a missing YAML file identically to a present-but-empty section (byte-for-byte), satisfying Requirement 1.5 structurally. `target.init(field, 0)` and never touching an empty/absent list field produce *different* serialized bytes for the same logical "nothing here" value (verified empirically) — the mapper always picks "never touch" for byte-determinism (9.1). Calling `.to_bytes()` twice on the same capnp builder triggers a spurious "message already written" `UserWarning` even when no field was mutated between calls — capture the bytes once and reuse them rather than calling `.to_bytes()` again. -- **2.4**: `errors.py`'s exception subclasses (task 1.2) all take `message` as an explicit first constructor argument in addition to their structured fields (e.g. `ConnectionTimeoutError(message, host, port)`, `ReplyDecodeError(message, reason)`) — design.md's abbreviated `Service Interface` type stubs (`class ConnectionTimeoutError(XceAdminError): host: str; port: int`) read as if only the structured fields were needed; always check the real constructor in `errors.py`, not the design doc's shorthand. `xcecommon.wire` is deliberately opcode-agnostic — `send_request`/`send_frame`/`recv_frame` move raw `bytes` only and never import a capnp type; callers own `.to_bytes()`/`.from_bytes()`. Reused `ha/data/xce.crt`/`xce.key` (the repo's existing dev cert pair, same one `deploy-ha.sh` already copies for controller TLS) as the test fixture for both the TLS test server and the client `cert_dir` — no synthetic cert generation needed. -- **3.1**: Requirement 1.7's "unrecognized-but-schema-valid field" warning is real and fires on essentially every real build using `src/configs/resources.yaml` — `resources.yaml`'s `type` key is *required* by `resources.schema.json` (`required: [name, type]`) but has no `ClusterConfig` destination (per `xcecommon.mapping`'s documented decision not to guess the `valtype`/`ordtype`/`flags` integer encoding), so every valid resource entry triggers a "field 'type' ... ignored" warning unconditionally, not just for unusual input. This is a disclosed spec tradeoff, not a bug — flagged here so a future pass over `xceadmin.build`'s warning UX (e.g. only warning once per field-name per run instead of once per item) knows the noise is expected, not a regression. `xceadmin.build`'s unmapped-field table (`_UNMAPPED_BUT_VALID_ITEM_KEYS`) was derived by cross-referencing `xcecommon.mapping`'s per-item builder functions against the real `src/configs/schema/*.schema.json` property lists directly (not from prose alone) to avoid drift. -- **3.2**: design.md's Dependencies list for `xceadmin.discovery` names only `xcecommon.wire`, but decoding a reply to inspect `RequestReply.redirectHost`/`.redirectPort` genuinely requires `xcecommon.schema` too (verified real field names against `controller.capnp:594-595`, which matched design.md exactly this time — unlike task 2.2's `Reply`/`RequestReply` struct-name slip). Treated as a design-doc omission, not license to hand-roll capnp decoding. `follow_redirects`'s `cert_dir` parameter is accepted for signature parity with design.md but unused internally — all I/O, including TLS setup, is delegated to the caller's `send_fn` closure, which is expected to already carry its own `cert_dir`. A connection timeout to any hop (including a non-initial one reached via redirect) raises `NoPrimaryError` naming that specific hop, not the original target. -- **bug-1461 (out-of-band, between 3.2 and 3.3)**: `controller.capnp` had a real, confirmed schema bug — a locally-declared `struct Data` (an `OPCode_get` query-result tagged union) shadowed Cap'n Proto's builtin `Data` blob type for every later `: Data` field in the file, miscompiling `UpgradeConfigRequest.sha256`/`.bytes` (and 5 other fields across `ReplConfigReply`/`ReplConfigPushRequest`/`ReplEnvelope`/`SnapshotTransfer`/`WalCatchupEntry`/`CtrlHelloRequest`, several HA-critical) as a struct-pointer type instead of a byte blob — blocking `push_config()` from constructing a valid request via pycapnp. Fixed at the root (renamed the local struct to `XceData`, `$C.mapname("xce_data_t")` unchanged so the C typedef is stable) across 20 files spanning controller/lib/scheduler/Go — see commit `e53c086` and `.wolf/buglog.json` bug-1461 for the full investigation and fix. This was NOT an admin-config-tool-boundary change; it's a pre-existing, cross-component schema defect that happened to block this task. `push_config()`'s real controller counterpart was, separately, blocked by a dispatch-gate defect (`ha_dispatch_gate()`/`cluster.c`) — **RESOLVED 2026-07-25 (bug-1468, commit `959b0cda`)**, fixed by `config-upgrade-snapshot` task 8.6 hitting the same wall building a live `OP_upgradeConfig` sender; that path has since been exercised live and successfully many times via that spec's `--config-upgrade` soak. Nothing blocks a live `push_config()` E2E run anymore, though `xceadmin`'s own TLS/redirect/framing code has not yet had one specifically. -- **3.3**: `probe_config()` does NOT go through `xceadmin.discovery.follow_redirects` — `ProbeConfigTagReply` is a direct struct with only `configTag`, no `redirectHost`/`.Port` fields exist on it at all, so there is no redirect signal for `follow_redirects` (which unconditionally decodes replies as `RequestReply`) to read; only `push_config()`'s `Request`/`RequestReply`-wrapped exchange supports redirects. `ProbeConfigTagReply` also carries no accept/reject signal — `DryRunResult.accepted` is a client-side inference (true on any successful round-trip), matching the real C reference client (`config_repl.c`'s `xce_probe_config_tag`, which also always sends an empty `name` and returns only a bare tag). pycapnp union field assignment for Void/Text-typed variants (e.g. `reply.data.nodata = None`, `reply.data.debugmsg = "..."`) must use direct assignment, not `.init(name)` — `.init()` without a size is only valid for struct/list/data-typed variants, confirmed via `KjException: init() without a size is only valid for struct and object fields`. -- **4.1**: `xcecommon.wire.connect()` deliberately lets `ConnectionRefusedError`/other bare `OSError`s propagate unwrapped (documented in `wire.py`'s own module docstring) — `xceadmin.cli` is the layer that catches `OSError` and formats it per Requirement 7.1, alongside its `except XceAdminError` dispatch surface; a task 4.1 implementer must not assume `XceAdminError` alone covers every network failure mode. click 8.3.1's `CliRunner` no longer accepts `mix_stderr=`; `result.stdout`/`.stderr` are separated by default, but `CliRunner`'s output capture decodes as text and is **lossy for arbitrary binary** (confirmed: `bytes(range(256))` does not round-trip through the captured `.output` string) — do not assert exact byte content of a `-o -` binary write through `CliRunner`; that's already covered at the `write_output()` unit level (task 3.1's `test_config_build.py`). A CLI test that omits `-o` falls through to the real default (`cluster.bin`, relative to CWD) — always pass an explicit `-o <tmp_path>` in tests that reach the write step, or the test will write a stray artifact into the actual repo working directory (caught via `git status` before commit, not by the test itself). -- **4.2**: `deploy-ha.sh` resolves both `xceadmin` (`$SCRIPT_DIR/../xceadmin/.venv/bin/xceadmin`) and the YAML configs source (`$SCRIPT_DIR/../configs`) relative to the script's own location (`$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)`), not the caller's CWD — matches the project's other relative-path scripts and keeps the deploy working regardless of where it's invoked from. `--build-config` is parsed out of `"$@"` before positional-arg assignment so it can appear anywhere in the argument list, not just a fixed position. Verified live against the real `xceadmin` build and `src/configs/`: missing `cluster.bin` auto-builds (8.4), an existing one with no flag is left byte-for-byte untouched — same mtime, same SHA (8.5), and `--build-config` against an existing `cluster.bin` forces a rebuild that reproduces byte-identical content (8.3, confirming 9.1's determinism guarantee holds through this path too). -- **5.1, 5.2**: Already fully satisfied by task 2.3's `test_mapping.py` and task 2.4's `test_wire.py` respectively — same requirement lists, same acceptance bullets (in-memory-only mapper coverage + the explicit two-field case for 5.1; a fixed known byte sequence + real loopback round-trips for 5.2). No new tests written; would have been pure duplication. -- **5.3**: Found and fixed a real gap while writing `test_schema_bundling.py` — `xcecommon.validation.validate_yaml()`'s `json.loads(schema_path.read_text())` call (loading the *bundled JSON validation schema itself*, not the operator's YAML) had no error handling, so a corrupted bundled JSON schema crashed with a raw `json.JSONDecodeError` instead of a clean `ValidationError` — the same "crash vs. clean error" class of bug task 2.2 already fixed for `controller.capnp`, just on the JSON-schema side, and not caught by task 2.1 since that task's tests only exercised operator-YAML failure modes, never a corrupted *schema* file. Fixed by wrapping it and raising `ValidationError(..., constraint="schema-corrupted")`, matching the existing missing-schema handling's pattern one function above it. **Gotcha for any future task touching `xcecommon`/`xceadmin` source**: editing a source file does NOT propagate to a running `.venv` until reinstalled — always re-run `xce-build.sh xceadmin` after a source edit before trusting a bare `.venv/bin/python -m pytest` result; a stale venv (especially one left in `release`/pinned mode by a prior profile run) will silently test the *old* code and produce a false failure (or false pass) that looks like a real bug. -- **5.4**: `tests/fixtures/{minimal,full}/configs/*.yaml` + checked-in `expected.bin` golden files added to `test_config_build.py` (not a new file — design.md's File Structure Plan lists the SAME requirement set for `test_config_build.py` under both 3.1 and 5.4). `full/configs/*.yaml` is a byte-for-byte copy of `src/configs/*.yaml`; its golden SHA (`860cfd4f59ef...`) independently matches the tag `deploy-ha.sh` produced in task 4.2's live verification against the same source files — an unplanned but welcome additional confirmation of Requirement 9's cross-context determinism guarantee. Golden-file tests catch representation drift (bytes change even though decoded content is still semantically correct) that purely structural/decode-based assertions elsewhere in the suite cannot detect; regenerate `expected.bin` deliberately (via `build_cluster_config()`, sanity-checked by decoding it back) whenever a real, intentional field-mapping change lands — never hand-edit the binary. -- **5.5**: Already fully satisfied by task 2.1's `test_validation.py` — every sub-case (schema violation, reserved name, cross-type duplicate collision via `test_check_duplicates_allows_limit_name_to_collide_with_queue_name`, undefined reference) already has an explicit test with the exact same requirement list. No new tests written. -- **5.6**: `probe_config()`'s dry-run has no reachable `accepted=False` path — `ProbeConfigTagReply` genuinely carries no rejection signal on the wire (see task 3.3's note). 5.6's bullet asking for both an accepted *and* a rejected outcome cannot be satisfied without inventing undocumented client-side rejection logic; instead added `test_probe_config_has_no_wire_level_rejection_signal_by_design` (parametrized over several `configTag` values) that locks in and documents the limitation explicitly rather than leaving it silently uncovered. Added two new end-to-end propagation tests through `push_config`'s real composition with `xceadmin.discovery` (not just discovery.py's own unit tests): a real 4-mock-server redirect chain proving `TooManyRedirectsError` surfaces correctly, and a `NoPrimaryError`-on-timeout test that monkeypatches `xcecommon.wire.connect` to raise `ConnectionTimeoutError` immediately rather than waiting out a real 30s socket timeout. -- **Post-implementation, out-of-band (2026-07-22)**: reopened and reversed the 2026-07-21 "not wired into meson" Design Review correction — see `design.md`'s File Structure Plan Amendment for full rationale and verification. Summary: added `src/xceadmin/meson.build` (`custom_target()`, stamp-file output, `build_always_stale: true`) + `src/xceadmin/meson-build.sh` (out-of-tree install+test, `repo-mirror/` symlink staging so `src/configs/`, `src/schema/`, `ha/data/` relative-path test assumptions still resolve from a copied location); `src/meson.build` gained `subdir('xceadmin')`; `src/tools/xce-build.sh`'s `xceadmin)` case now delegates to `meson compile -C "$BUILD_DIR" xceadmin` instead of its own pip/venv logic. Verified both `meson compile` (165/165 passed against a real dev checkout with certs present) and `nix build --option sandbox false .#xcecore_debug` (128 passed / 37 skipped — cert-dependent tests correctly skip under nix's git-tracked-only hermetic source, since `ha/data/xce.crt`/`.key` are intentionally not committed). AppImage bundling (README 3.5) explicitly deferred, not part of this change. Net effect on Requirement 8: the editable-vs-pinned distinction (8.1 vs. 8.2) collapses — every `xce-build.sh xceadmin` run is now the pinned/meson-driven path, a strictly stronger guarantee than the old default. -- **Post-implementation, out-of-band (2026-07-22, second pass)**: `meson-build.sh`/`meson-install.sh` no longer resolve `python3` via a bare PATH lookup — `meson.build` now pins it once via `import('python').find_installation('python3')` (same mechanism as `find_program('go')`/capnp/packcc elsewhere in this build, visible at `meson setup` time as `Program python3 found: YES (/nix/store/...)`) and threads the resolved path into both scripts explicitly. Verified: `meson setup` prints the pinned interpreter, rebuild stays 165/165 green, and the built venv's own `python3` resolves through to that exact store path. -- **Post-implementation, out-of-band (2026-07-22, third pass — removes `ha/data/` entirely)**: `deploy-ha.sh` no longer takes a `<data-dir>` argument at all (`--build-config` flag also removed) — every invocation now generates a fresh self-signed TLS cert/key (`openssl req -x509 -newkey rsa:2048 -nodes -days 3650`, 10y validity) directly into the deploy dir's `certs/`, and builds `cluster.bin` directly into `controller/ctrl-1/conf/` via `xceadmin`, with no intermediate staging directory. `run-ha-soak.sh` (`/xce-test-ha`) updated to match: dropped its own `XCE_HA_DATADIR`-based precondition entirely, since `deploy-ha.sh` now handles both cert and config generation itself. Updated 4 callers (`run-ha-soak.sh`, `test-heartbeat-tolerance.sh`, `test-commit-durability.sh`, plus both `xce-deploy-ha`/`xce-test-ha` SKILL.md docs) to the new 3-arg signature. `ha/data/{xce.crt,cluster.bin}` removed from git tracking (`cluster.bin` was already gitignored from the prior pass; the certs were a real checked-in TLS private key, now gone from history going forward). - -- **6.2**: A genuinely single-node (zero-peer) controller always computes `ackedCount == 0` for `config push` — no standbys exist to ACK — even when the config change commits and activates successfully server-side (`config_repl.c`'s `push_config_to_standbys`/`do_upgrade_config_with_bytes`). `xceadmin.cli`'s postcondition (`acked_count > 0` required for exit 0) can't distinguish "no standbys exist" from "push failed," so a real single-node `config push` always exits 1 with `"push rejected (ackedCount == 0)"` even on full success — reproduced live and independently re-confirmed by the reviewer against real `config_repl.c`/`repl.c` source. Verified spec-consistent (matches `cli.py`/design.md's own exit-0-iff-`acked_count>0` postcondition as written), not a regression from this task. `test_e2e_live_push.py` asserts this real exit-1/ackedCount==0 behavior directly rather than the task text's literal "exit 0," and independently proves genuine server-side success via the controller's own commit/activate log lines. Out of this task's boundary to fix; a future task should decide whether single-node pushes deserve a distinct success signal in `xceadmin.cli`/`push.py` (e.g. treating a fully-connected zero-peer topology as trivially quorate) rather than always reading as a CLI failure. -- **6.1**: `decode_reply()` (new, `discovery.py`) is the single unwrap point for the `Request{opcode: reply, body.replymsg: RequestReply}` envelope every real controller reply arrives in (`xce_reply()`, `src/lib/request.c:384-418`) — both `discovery._redirect_target()` and `push.push_config()` go through it; `probe_config()` is untouched (`ProbeConfigTagReply` is never envelope-wrapped). Fixing the two documented call sites alone was insufficient — task 5.6's own mock-server test fixtures (`test_discovery.py`, `test_config_push_mock.py`) AND `test_cli.py`'s two `config push` mock handlers all constructed bare, unwrapped `RequestReply` bytes, meaning the entire existing test suite had been silently validating the wrong wire shape since task 5.6/task 4.1. All fixed via one shared `wrap_reply` fixture in `conftest.py`. Any future test that hand-builds a controller reply for a mock server must wrap it the same way, or it will pass against a mock while still being broken against a real controller. - - Investigating this surfaced a real, independent finding: `xcecommon.wire.connect()` was loading a **client-side** TLS cert (`context.load_cert_chain(...)` on the `PROTOCOL_TLS_CLIENT` context) — verified against the actual reference implementation (`netio.c`'s `create_ssl_context()`) that the real client presents **no** certificate at all when `cert_file`/`key_file` are `NULL` (bare `TLS_client_method()`, no `SSL_CTX_use_certificate_chain_file`/`SSL_CTX_use_PrivateKey_file` call). `xceadmin`'s client diverged from the reference client it's meant to mirror. Fixed: `connect()` no longer loads a client cert or takes a `cert_dir` parameter at all. Since nothing else consumed `cert_dir` (no CA verification happens either — `verify_mode=CERT_NONE` throughout), removed it end-to-end: `xceadmin.discovery.follow_redirects()`'s `cert_dir` param (already dead/`noqa`'d — signature-parity leftover), `xceadmin.push.push_config()`/`probe_config()`, and the `--cert-dir` CLI flag on `config push`. Server-side cert loading (the mock TLS servers in `test_wire.py`/`test_cli.py`/`test_config_push_mock.py`) is unaffected and still needed — moved into a shared session-scoped `tls_certs` fixture in a new `tests/conftest.py` (openssl-generated, same approach as `deploy-ha.sh`), replacing 3 duplicated module-level `REPO_HA_DATA_DIR`/`pytest.mark.skipif(not ha/data/xce.crt...)` blocks. Net result: those 3 files' TLS-dependent tests no longer skip under a hermetic `nix build` sandbox (which never had `ha/data/` in its git-tracked-only source) — verified 165/165 passed, 0 skipped, both via `meson compile` and via a full `nix build --option sandbox false .#xcecore_debug` rebuild (previously 128 passed / 37 skipped there). Requirements 2.3/2.4 and 8.3/8.4/8.5 marked superseded in `requirements.md` with dated correction notes rather than silently rewritten. diff --git a/.kiro/specs/build-time-versioning/design.md b/.kiro/specs/build-time-versioning/design.md deleted file mode 100644 index dcdf086d..00000000 --- a/.kiro/specs/build-time-versioning/design.md +++ /dev/null @@ -1,241 +0,0 @@ -# Design Document - -## Overview -**Purpose**: This feature gives every XCE component (controller, scheduler, worker, apiserver, xcecli, xceadmin) a build-time identity — the git commit it was built from and when it was built — visible from the running artifact itself, via a `--version` flag and (for the four long-running daemons) a startup log line. - -**Users**: Operators and developers diagnosing a running deployment (the motivating incident: a stale scheduler binary kept running a known-fixed crash, and confirming that required manual `ls -la` mtime checks plus `git merge-base --is-ancestor`). - -**Impact**: Extends the existing, partial C-side mechanism (controller/scheduler already have a commit-id-only `--version` via `argp`, but `xce-build.sh` never actually sets it, so it is empty on every local build today) to carry a build timestamp too, fills the worker/Go/Python gaps, and fixes the `xce-build.sh` gap that made the motivating incident possible in the first place. - -### Goals -- Every component's build artifact carries a git commit identifier (dirty-suffixed when applicable) and a build timestamp. -- `--version` on every component prints both and exits. -- Controller, scheduler, worker, and apiserver additionally log both at INFO level on startup. -- `xce-build.sh` (the dev-loop build path implicated in the motivating incident) actually sets these values — today it does not. -- The values are byte-for-byte identical between `--version` output and the startup log line for the same artifact. - -### Non-Goals -- Automatically detecting or alerting that a *running* binary is stale relative to current source (out of scope per requirements' Boundary Context — this feature only makes the comparison possible). -- Version compatibility negotiation between components running different versions. -- Retrofitting version info into already-built/deployed binaries. -- Migrating worker's `getopt_long()` argument parsing to `argp` (a much larger, unrelated change) — worker gets a hand-added `--version` case instead. - -## Boundary Commitments - -### This Spec Owns -- The two build-time values themselves (commit identifier, build timestamp) and how each of the six components' build steps receive them. -- `--version` output and (for the four daemons) the startup log line, for all six components. -- `xce-build.sh`'s computation of both values for the dev-loop build path. -- The nix packages' (`xce.nix`, `xcecli.nix`) computation of both values for the production build path. - -### Out of Boundary -- Any runtime mechanism that compares a running binary's version against "current" source (no such comparison exists or is planned). -- `xceadmin`'s own semantic package version (`pyproject.toml`'s `version = "0.1.0"`) — untouched; build metadata is additive, displayed alongside it, not a replacement for it. -- Worker's broader argument-parsing architecture. - -### Allowed Dependencies -- The existing `commit_id` meson option and `VERSION_COMMIT_ID`/`MK_VERSION` C mechanism (`src/meson.build`, `src/meson.options`, `src/include/utils.h`) — extended, not replaced. -- The existing Nix flake commit-id derivation (`self.shortRev`/`self.dirtyShortRev`, `src/flake.nix`) — extended with a new build-time value, same wiring style. -- `src/tools/go_build`'s existing env-var-driven `go build` invocation pattern (already used for CGO flags). -- `src/xceadmin/meson.build`'s existing `custom_target` → `meson-build.sh` argument-passing pattern. -- Go's standard `-ldflags -X` package-variable injection; Cobra's built-in `Version` field (xcecli already uses Cobra); Click's built-in `version_option` (xceadmin already uses Click). - -### Revalidation Triggers -- The `commit_id` meson option is renamed, removed, or its consumption in `src/include/utils.h`'s `MK_VERSION`/`STR` macros changes shape. -- `xce-build.sh`'s `MESON_OPTS` construction is refactored away from its current per-profile array pattern. -- `src/tools/go_build`'s final `go build` invocation line changes. -- `src/xceadmin/meson-build.sh`'s step ordering changes (the generated build-info module must land before the `pip install` step that packages it). - -## Architecture - -### Existing Architecture Analysis -- Controller and scheduler already parse `--version`/`-V` for free via GNU `argp`'s `argp_program_version` mechanism (`MK_VERSION("xce v0.0.1-", STR(VERSION_COMMIT_ID))`, both `main.c`s) — this feature extends that same macro rather than adding a parallel one. -- `VERSION_COMMIT_ID` already flows into every C component uniformly via `add_project_arguments()` at the top-level `src/meson.build` — the new build-timestamp define follows the identical path, added once, inherited everywhere. -- Worker's `getopt_long()`-based parsing has no equivalent auto-version behavior and needs an explicit new case. -- `git` is deliberately absent from the nix devShell's declared buildInputs (`src/nixSupport/xce.nix`) — the *sandboxed* `nix build` path already avoids shelling out to git for exactly this reason (uses Nix's own `self.shortRev` introspection instead). The *dev-loop* `xce-build.sh` path runs inside `nix develop`, which layers onto the ambient host PATH rather than replacing it, so git IS usable there — and needs to be used there, since `xce-build.sh` today never sets `commit_id` at all (the actual gap behind the motivating incident). - -### Architecture Pattern & Boundary Map -Two independent **computation sites** (respecting the existing sandboxed-vs-ambient split) feed one **resolved value pair** into meson, which every language-specific **consumption site** reads from — never recomputes independently. - -```mermaid -graph TB - DevScript[xce-build.sh git plus date] - NixPkg[xce.nix xcecli.nix self.shortRev plus preConfigure date] - MesonOpts[meson commit_id and build_time options] - CDefine[C VERSION_COMMIT_ID and VERSION_BUILD_TIME defines] - GoLdflags[go_build ldflags -X injection] - PyGen[meson-build.sh generated build info module] - Controller[controller and scheduler argp version] - Worker[worker getopt_long version case] - Apiserver[apiserver flag version] - Xcecli[xcecli cobra version] - Xceadmin[xceadmin click version_option] - - DevScript --> MesonOpts - NixPkg --> MesonOpts - MesonOpts --> CDefine - MesonOpts --> GoLdflags - MesonOpts --> PyGen - CDefine --> Controller - CDefine --> Worker - GoLdflags --> Apiserver - GoLdflags --> Xcecli - PyGen --> Xceadmin -``` - -**Architecture Integration**: -- Selected pattern: single resolved-value-pair fan-out, matching the existing `commit_id` mechanism's own shape. -- Domain/feature boundaries: computation (2 sites, environment-specific) is fully separated from consumption (6 sites, one per component) — a consumption site never knows or cares which computation site produced its value. -- Existing patterns preserved: `argp` version macro (C), Cobra `Version` field and Click `version_option` (already the idiomatic mechanism in each respective Go/Python tool), `go_build`'s env-var plumbing, `meson-build.sh`'s positional-arg plumbing. -- New components rationale: `sim_prescan_session_count`-style small, single-purpose additions only where a gap genuinely exists (worker's version case, Go's ldflags wiring, Python's generated module) — no new abstraction layer introduced. -- Steering compliance: no new external dependency (`tech.md`); wire/build-format changes are free pre-release (no back-compat burden); TDD required for every new behavior. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| C build (controller/scheduler/worker) | Meson preprocessor defines (existing mechanism) | Carries `commit_id` (existing) + new `build_time` into every C translation unit | `src/meson.build:30` | -| C CLI (controller/scheduler) | GNU `argp` (existing) | `--version`/`-V` auto-handling via `argp_program_version` | Extend `MK_VERSION` macro | -| C CLI (worker) | `getopt_long()` (existing) | New explicit `--version` case | No parser migration | -| Go build | `-ldflags -X` (standard Go idiom, new to this repo) | Injects `main.GitCommit`/`main.BuildTime` package vars | Via `go_build` script + `Makefile` | -| Go CLI (apiserver) | `flag` package (existing) | New `-version` flag | Matches existing single-flag style | -| Go CLI (xcecli) | Cobra (existing) | `rootCmd.Version` field (built-in `--version`) | Zero new parsing code | -| Python build | Meson `custom_target` (existing) | New generated build-info module written before `pip install` | `meson-build.sh` | -| Python CLI (xceadmin) | Click (existing) | `click.version_option()` (built-in `--version`) | Zero new parsing code | -| Nix packaging | Nix flake (existing) | `self.shortRev`/`self.dirtyShortRev` (existing, eval-time) + a `preConfigure` build-phase `date -u` hook for build_time (task 2.2 round 2: `builtins.currentTime` was tried first but breaks pure-eval `nix build`, i.e. requires `--impure`, and yields non-UTC epoch seconds — rejected in review, corrected to a build-phase computation) | `xce.nix`, `xcecli.nix` | - -## File Structure Plan - -### Modified Files -- `src/meson.options` — add a `build_time` string option (default `''`), alongside the existing `commit_id`. -- `src/meson.build` — add `-DVERSION_BUILD_TIME=` to the existing `add_project_arguments()` call (line ~30). -- `src/include/utils.h` — extend `MK_VERSION`/introduce a companion macro so the version string includes the build time, not just the commit id. -- `src/controller/main.c`, `src/scheduler/src/main.c` — update `argp_program_version` construction to include build time; add the startup INFO log line (each already has one startup INFO call site to extend/follow). -- `src/worker/src/main.c` — add a `--version` long option + case (prints the same shape, exits 0); add the startup INFO log line (follows its own existing startup INFO call site). -- `src/tools/xce-build.sh` — compute `commit_id` (git, `-dirty` suffixed) and `build_time` (UTC `date`) and pass both via new `-Dcommit_id=`/`-Dbuild_time=` entries in `MESON_OPTS`, falling back to `"unknown"` if `git` fails (e.g. not a git checkout). -- `src/nixSupport/xce.nix`, `src/nixSupport/xcecli.nix` — a `preConfigure` build-phase hook runs `date -u +%Y-%m-%dT%H:%M:%SZ` and appends `-Dbuild_time=` to `mesonFlagsArray` (build-time, not eval-time, unlike `commit_id`'s `self.shortRev` wiring in `src/flake.nix`/`src/nixSupport/default.nix`, which is unchanged). -- `src/tools/go_build` — read `XCE_GIT_COMMIT`/`XCE_BUILD_TIME` env vars (empty-safe) and append `-ldflags "-X main.GitCommit=... -X main.BuildTime=..."` to the final `go build` invocation. -- `src/frontend/apiserver/meson.build`, `src/frontend/xcecli/meson.build` — pass the two new env vars into their `custom_target()` calls (mirrors existing env var passing). -- `src/frontend/Makefile` — add the same `-ldflags` construction (via `git`/`date` directly, dev-loop path) to `build-server`/`build-cli` targets. -- `src/frontend/apiserver/cmd/main.go` — declare `var (GitCommit, BuildTime string)`; add a `-version` flag; add/extend the startup `logger.Info(...)` line. -- `src/frontend/xcecli/cmd/main.go` — declare the same two vars; set `rootCmd.Version` (Cobra); extend its own startup `logger.Info(...)` line. -- `src/xceadmin/meson.build` — pass two new positional args (or env vars) to `meson-build.sh`. -- `src/xceadmin/meson-build.sh` — write a small generated Python module (e.g. `_build_info.py`) into the mirrored source tree, before the `pip install` step, containing the two values (falls back to `"unknown"` if not provided). -- `src/xceadmin/src/xceadmin/cli.py` — read the generated module (import guarded for the case it's genuinely absent, e.g. a non-meson editable install) and wire `click.version_option()` on the top-level group. - -### New Files -- `src/xceadmin/src/xceadmin/_build_info.py` — **generated at build time only** (not checked in); a committed fallback stub (all fields `"unknown"`) ships in-tree so `pip install -e .` (the documented fast-iteration path in `build-deployment.md`) still imports cleanly without having run the meson build. - -> No new C/Go/Python source files beyond the one generated-with-fallback-stub Python module above — every other change extends an existing file's existing responsibility (arg parsing gains a flag, a build script gains an option). - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1, 1.2 | Every artifact carries commit id + build time | All 6 consumption sites + both computation sites | `MK_VERSION` macro, Go ldflags vars, Python `_build_info` module | Computation → meson → consumption | -| 1.3 | Dirty tree marked | `xce-build.sh`'s git computation | commit_id string carries `-dirty` suffix | n/a | -| 1.4 | Unknown when undeterminable | `xce-build.sh`, `meson-build.sh`, `go_build` | fallback to literal `"unknown"` | n/a | -| 2.1 | `--version` prints and exits | argp (controller/scheduler), worker's new case, apiserver's `-version`, Cobra `Version`, Click `version_option` | six CLI entry points | n/a | -| 2.2 | `--version` and startup log agree | shared macro/var/module per component | same source value read twice, never recomputed | n/a | -| 3.1, 3.2 | Startup log line at INFO | controller/scheduler/worker/apiserver startup sequences | existing first-INFO-line call sites, extended | Startup → log | -| 4.1, 4.2 | Build tooling confirms embedding | `xce-build.sh`, nix packages | non-empty resolved value check before/after `meson setup`/build | Build → verify | -| 4.3 | Accurate under every C build profile | `src/meson.build`'s project-wide `add_project_arguments()` | applies uniformly to asan/debug/release | n/a | -| 5.1, 5.2 | Values reflect actual tree/build event | `xce-build.sh` (git+date at invocation time), nix (`self.shortRev` at evaluation time + `preConfigure` `date -u` at build time) | no hardcoded/stale values | n/a | - -## Components and Interfaces - -### C Daemons (controller, scheduler, worker) - -#### Version macro -| Field | Detail | -|-------|--------| -| Intent | Carry commit id + build time into every C daemon's `--version` output and startup log | -| Requirements | 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.3 | - -**Responsibilities & Constraints** -- `MK_VERSION` (`src/include/utils.h`) extended to compose `"xce v0.0.1-<commit_id> built <build_time>"` (or equivalent) from the two new preprocessor defines — a single macro, both consumers (controller/scheduler's `argp_program_version`, worker's hand-added case) call it, guaranteeing `--version` and the startup log line render from the identical string. -- Controller/scheduler: their existing single startup INFO line (`main.c:365`/`377`+`381` respectively, per discovery) gets this version string appended or logged as an immediately-preceding line — never a second, independently-formatted string. -- Worker: new `{"version", no_argument, NULL, OPT_VERSION}` long option in its existing `long_opts[]`/`getopt_long()` switch (`main.c:358-364`), printing the same macro's output and exiting 0 before any other startup work begins. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: extends `src/include/utils.h`'s existing `STR`/`MK_VERSION` macros — no new header. -- Validation: a build with `commit_id`/`build_time` unset must still compile and print `"unknown"` fields, never a blank/garbled string. -- Risks: worker's `getopt_long()` short-option string (`"n:l:vh"`) must gain a way to reach the new long-only option — use a long-option-only constant (no short equivalent), matching `--peers`'s own existing precedent in the same `long_opts[]` array. - -### Go Binaries (apiserver, xcecli) - -#### ldflags version injection -| Field | Detail | -|-------|--------| -| Intent | Inject build-time values into each Go binary at link time, exposed via each CLI's native version mechanism | -| Requirements | 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2 | - -**Responsibilities & Constraints** -- `src/tools/go_build` (consumed by both `apiserver/meson.build` and `xcecli/meson.build`'s `custom_target()`) reads `XCE_GIT_COMMIT`/`XCE_BUILD_TIME` from its environment (empty-safe — falls back to `"unknown"` when unset, e.g. a bare `go build` invoked by hand) and appends `-ldflags "-X main.GitCommit=$v1 -X main.BuildTime=$v2"` to its existing `go build` invocation. -- `src/frontend/Makefile`'s `build-server`/`build-cli` targets compute the same two values directly (git/date, dev-loop path, no meson involved) and pass the identical `-ldflags` shape — same output format as the meson path, different computation site (mirrors the C-side split). -- `apiserver/cmd/main.go`: package-level `var (GitCommit = "unknown"; BuildTime = "unknown")`, a new `-version` bool flag (checked first, before `utils.InitLogger()` — prints and exits, no logging side effects), and the existing `logger.Info("********** Initialize Logger done!")` call site extended to also log both values (same convention, `"********** <action> done!"` is untouched; the version fields ride as structured zap fields on that same line or an adjacent one). -- `xcecli/cmd/main.go`: same two vars; version string passed into `router.Execute(cm, version)`, which sets `rootCmd.Version` before `rootCmd.Execute()` runs (Cobra then handles `--version` natively — zero new flag-parsing code). Correction (task 4.5): xcecli's startup log is deliberately NOT extended — Requirement 3 scopes the startup-log obligation to "the controller, scheduler, worker, or apiserver daemon" only; xcecli is a one-shot CLI, not a daemon, and was never in Requirement 3's scope. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: `go_build`'s env-var pattern already exists for CGO flags — this reuses it, not a new plumbing mechanism. -- Validation: `--version`/`-version` output and the startup log's values must be the exact same two package vars, read twice, never two separate computations. -- Risks: none identified — `-ldflags -X` only accepts string vars, both fields are strings by construction. - -### Python (xceadmin) - -#### Generated build-info module -| Field | Detail | -|-------|--------| -| Intent | Package build-time values into the wheel so `xceadmin --version` reflects the actual build, not just `pyproject.toml`'s static semantic version | -| Requirements | 1.1, 1.2, 2.1, 4.1, 4.2 | - -**Responsibilities & Constraints** -- `meson-build.sh` writes `_build_info.py` (containing `GIT_COMMIT`/`BUILD_TIME` string constants, sourced from two new args/env vars, `"unknown"` fallback) into the mirrored source tree's `xceadmin/` package directory, *before* the `pip install` step (so it packages into both the dev venv install and the final wheel) — same ordering discipline the existing schema-sync step already follows (also runs before `pip install`). -- `cli.py`'s top-level `@click.group()` gains `@click.version_option(version=f"{pkg_version} ({_build_info.GIT_COMMIT}, {_build_info.BUILD_TIME})")`, importing `_build_info` defensively (a committed fallback stub with `"unknown"` fields ships in-tree so a non-meson `pip install -e .` — the documented fast-iteration path — still imports cleanly). -- No startup log line (Requirement 3 scopes that to the four long-running daemons only; xceadmin is a one-shot CLI). - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: extends `meson-build.sh`'s existing custom_target sequence at exactly one new step, same file-write style already used for schema sync. -- Validation: `pip install -e .` (no meson) must still import cleanly against the committed fallback stub — this is the one file in this feature that must exist in two states (checked-in fallback vs. build-time-overwritten real values) and both must be valid Python. -- Risks: the generated module must not be checked in with real values (would go stale immediately) — only the `"unknown"`-only fallback stub is committed; `.gitignore` (or equivalent) must exclude the real generated file if `meson-build.sh` ever writes it into the actual source tree rather than only the mirrored build-tree copy (verify this during implementation — the mirroring step at `meson-build.sh` lines 32-46 should already keep it out of the real source tree, matching how the schema sync step works today). - -### Build Orchestration - -#### xce-build.sh value computation -| Field | Detail | -|-------|--------| -| Intent | Close the actual gap behind the motivating incident — the dev-loop build path must set real values, not leave them empty | -| Requirements | 4.1, 4.2, 4.3, 5.1, 5.2 | - -**Responsibilities & Constraints** -- Computes `commit_id` via `git rev-parse --short HEAD`, appending `-dirty` when `git diff --quiet HEAD --` (or `git status --porcelain`) reports changes; falls back to `"unknown"` if `git` itself fails (e.g. run against an extracted source tarball with no `.git`). -- Computes `build_time` via `date -u +%Y-%m-%dT%H:%M:%SZ`. -- Appends `-Dcommit_id="$commit_id" -Dbuild_time="$build_time"` to `MESON_OPTS` (`xce-build.sh:70-72`) for every profile, before the existing `meson setup`/`meson setup --reconfigure` calls. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: a small addition right where `MESON_OPTS` is already constructed — no restructuring of the profile dispatch. -- Validation: the exact scenario from the motivating incident — build via `/xce-build scheduler debug`, run `--version`, confirm it shows a real (non-empty, non-"unknown") commit id matching `git rev-parse --short HEAD` at build time. -- Risks: none — purely additive to an existing array. - -## Testing Strategy - -### Unit Tests -- C: a new scheduler/controller/worker test (or a shared `lib` test if `MK_VERSION`'s logic is factored into a testable helper) asserting the version string contains both the commit id and build time defines when set, and `"unknown"` fallbacks when unset. -- Go: a test in each `cmd` package (or a shared helper) asserting `-version`/`--version` output contains both `GitCommit` and `BuildTime` values. -- Python: a `pytest` case (xceadmin's existing suite, run by `meson-build.sh`) asserting `xceadmin --version` output contains the generated build-info fields, and that the committed fallback stub is itself valid, importable Python. - -### Integration Tests -- `xce-build.sh`-driven build (any profile) followed by running `--version` on the resulting controller/scheduler/worker binaries, asserting a real (non-`"unknown"`) commit id is present — this is the regression test for the motivating incident itself. -- A `go_build`-driven meson build of apiserver/xcecli, followed by `-version`, asserting the same. -- `xceadmin`'s existing meson-driven build+test+wheel cycle, followed by installing the wheel and running `xceadmin --version`. - -### E2E / Consistency -- For at least one C daemon and one Go binary: start it, capture its startup log line, separately run `--version`, and assert the two report byte-identical commit id and build time (Requirement 2.2). diff --git a/.kiro/specs/build-time-versioning/requirements.md b/.kiro/specs/build-time-versioning/requirements.md deleted file mode 100644 index a09db79c..00000000 --- a/.kiro/specs/build-time-versioning/requirements.md +++ /dev/null @@ -1,52 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Build-time versioning: every component (controller, scheduler, worker, apiserver, xcecli, xceadmin) must embed and display its source git commit id and build timestamp, shown both via a --version CLI flag and a startup log line (INFO level, matching existing daemon startup log conventions). xce-build.sh should also ensure/verify this info is correctly embedded for each component it builds. No existing infrastructure for this exists today (confirmed via investigation: no version.h/version.c, no meson version defines, no Go ldflags -X injection, no --version handling in any daemon, no dynamic pyproject versioning) -- this is new, cross-cutting work spanning three toolchains: C/meson (controller/scheduler/worker, linking libxce), Go (apiserver/xcecli, via ldflags), and Python (xceadmin, via pyproject/meson custom_target). Motivating incident: a live production-like deployment kept running a stale debug-profile scheduler binary that lacked a just-landed heap-corruption fix, and diagnosing that required manual `ls -la` mtime checks plus `git merge-base --is-ancestor` -- this feature should make that kind of staleness immediately visible from the running binary/log itself, no forensic git commands needed. - -## Introduction -This feature adds build-time version identification to every XCE component so operators can determine, directly from a running binary or its log output, exactly which source commit and build time produced it — without resorting to manual filesystem or git forensics. It is motivated by an incident where a stale scheduler binary silently kept running a known-fixed heap-corruption bug because there was no way to tell, from the binary or its logs, which commit it was actually built from. - -## Boundary Context (Optional) -- **In scope**: embedding and displaying a git commit identifier and build timestamp for controller, scheduler, worker, apiserver, xcecli, and xceadmin; exposing this via a `--version` flag and, for long-running daemons, a startup log line; the build tooling confirming this information was correctly embedded for each component it builds. -- **Out of scope**: automatically detecting or alerting that a running/deployed binary is stale relative to the latest source; retroactively updating the version information of an already-running process; version negotiation or compatibility checks between components running different versions. -- **Adjacent expectations**: operators are expected to compare the displayed commit identifier against the source repository themselves (for example, via `git`) to judge staleness; this feature makes that comparison possible, it does not automate it. - -## Requirements - -### Requirement 1: Version Information Content -**Objective:** As an operator diagnosing a running deployment, I want each component to know which exact source commit and build time produced it, so that I can determine whether a binary reflects a given fix without inspecting file timestamps or git history by hand. - -#### Acceptance Criteria -1. The system shall associate a git commit identifier with every built artifact of controller, scheduler, worker, apiserver, xcecli, and xceadmin. -2. The system shall associate a build timestamp, in absolute UTC form, with every built artifact of controller, scheduler, worker, apiserver, xcecli, and xceadmin. -3. If the source tree contains uncommitted changes at build time, then the system shall mark the associated version information as modified, so an operator does not mistake it for an exact committed state. -4. If the git commit identifier cannot be determined at build time (for example, a source tree without git history), then the system shall record the version information as unknown rather than fail the build. - -### Requirement 2: Version Display via CLI -**Objective:** As an operator or script, I want to query a component's build identity directly from its binary, so that I can verify a deployed artifact without needing separate log access. - -#### Acceptance Criteria -1. When a `--version` flag is passed to the controller, scheduler, worker, apiserver, xcecli, or xceadmin command, the invoked component shall print its associated commit identifier and build timestamp and exit without performing its normal function. -2. The system shall present the same commit identifier and build timestamp values through the `--version` flag and through the startup log line (Requirement 3) for the same build artifact, so the two never disagree. - -### Requirement 3: Version Display at Startup -**Objective:** As an operator watching a daemon's log output, I want to see its build identity as soon as it starts, so that I can confirm which commit is actually running without a separate query step. - -#### Acceptance Criteria -1. When the controller, scheduler, worker, or apiserver daemon starts, the daemon shall log its commit identifier and build timestamp at INFO level, following the daemon's existing startup log format. -2. While a controller, scheduler, worker, or apiserver daemon is running, an operator shall be able to determine its build identity by reading its log without sending it any request. - -### Requirement 4: Build-Time Embedding Verification -**Objective:** As a developer running the project's build tooling, I want confirmation that version information was actually embedded correctly, so that I do not unknowingly ship or deploy a binary with missing or incorrect build identity. - -#### Acceptance Criteria -1. When the build tooling builds a component covered by this feature, the build tooling shall confirm that a commit identifier and build timestamp were embedded in the resulting artifact for that component. -2. If the build tooling cannot confirm that version information was embedded for a component it built, then the build tooling shall report this as a build problem rather than silently succeeding. -3. The system shall embed accurate version information under every supported build profile of controller, scheduler, and worker, not only a subset of profiles. - -### Requirement 5: Accuracy of Embedded Version Information -**Objective:** As an operator relying on displayed version information to make a deployment decision, I want that information to always reflect the actual source tree and build event that produced the running binary, so that I can trust it during an incident. - -#### Acceptance Criteria -1. The system shall derive the embedded commit identifier from the actual state of the source tree at build time, not from a value set independently of the build. -2. The system shall derive the embedded build timestamp from the actual time the build artifact was produced, not from the source commit's own timestamp. diff --git a/.kiro/specs/build-time-versioning/research.md b/.kiro/specs/build-time-versioning/research.md deleted file mode 100644 index 6f0f880d..00000000 --- a/.kiro/specs/build-time-versioning/research.md +++ /dev/null @@ -1,57 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `build-time-versioning` -- **Discovery Scope**: Extension (existing, partial C-side infrastructure; zero Go/Python infrastructure) -- **Key Findings**: - - Controller and scheduler already have a **working `--version`/`-V` flag** (GNU `argp` auto-generates it from `argp_program_version = MK_VERSION("xce v0.0.1-", STR(VERSION_COMMIT_ID))`, both `main.c`s) and a project-wide `VERSION_COMMIT_ID` preprocessor define wired through meson (`src/meson.build:30`, `src/meson.options`'s `commit_id` option) — but it carries **no build timestamp**, and **`xce-build.sh` never sets `-Dcommit_id=`**, so every locally-built binary (the entire `/xce-build` dev loop, including the one that produced the stale binary in the motivating incident) gets an **empty** commit id today. - - Worker uses plain `getopt_long()`, not `argp` — it has **no `--version` at all**. - - `git` is deliberately **not** a nix devShell buildInput (`src/nixSupport/xce.nix`) — the existing `commit_id` mechanism avoids shelling out to git entirely; the two production nix packages (`xce.nix:109`, `xcecli.nix:30`) derive it from the flake's own `self.shortRev`/`self.dirtyShortRev` introspection instead. Any new mechanism must respect this: no `run_command`/`custom_target` in `meson.build` may shell out to `git`. - - Go (apiserver, xcecli) and Python (xceadmin) have **zero** version infrastructure — no `-ldflags` injection anywhere in `src/tools/go_build` or `src/frontend/Makefile`, no `click.version_option`, no dynamic `pyproject.toml` version. - -## Research Log - -### Where does `xce-build.sh` (the actual dev-loop path) get its build metadata from? -- **Context**: the motivating incident happened via `/xce-build scheduler debug`, not `nix build` — need to know which computation path is actually load-bearing for fixing it. -- **Findings**: `xce-build.sh`'s `MESON_OPTS` array (set per-profile at `xce-build.sh:70-72`) never includes `-Dcommit_id=` or anything build-time related; `meson setup` is called with only `--buildtype`/`-Db_sanitize`. `git` IS available here despite not being a declared nix devShell buildInput, because `nix develop` (unlike a sandboxed `nix build`) layers onto the ambient host PATH rather than replacing it — this project's own scripts already run from inside a real git checkout. -- **Implications**: `xce-build.sh` must be the one to compute commit id (`git rev-parse --short HEAD`, `-dirty` suffixed via `git diff --quiet HEAD --` / `git status --porcelain`) and build time (`date -u`), independently of the nix-package path, and pass both via meson options. This is the fix that actually closes the incident's gap — the nix-package path already has a working (commit-id-only) mechanism. - -### Message/value plumbing per toolchain -- **Context**: three different build tool invocations (meson C custom target-free preprocessor define, Go's `go_build` shell script + `Makefile`, Python's `meson-build.sh` custom_target) need the same two values. -- **Findings**: `src/meson.build:30` already threads `commit_id` project-wide via `add_project_arguments()` for C. `src/tools/go_build` (invoked by both `apiserver/meson.build`'s and `xcecli/meson.build`'s `custom_target()`) already threads other env vars (e.g. CGO flags) into its `go build` invocation — same mechanism can carry new `XCE_GIT_COMMIT`/`XCE_BUILD_TIME` env vars for `-ldflags -X`. `src/xceadmin/meson.build`'s `custom_target` already passes several positional args to `meson-build.sh`; two more (or two new env vars) can carry the same values for a small generated Python module written before the `pip install` step. -- **Implications**: meson itself is the single source of truth for one build invocation — it resolves the two new options (`commit_id`, now actually SET by `xce-build.sh`; a new `build_time`) once, and every language-specific build step (C define, Go ldflags, Python generated module) reads from that same resolved value instead of independently recomputing it. Avoids the 3 toolchains ever disagreeing within one build. - -### Dirty-tree representation: separate flag vs. suffix convention -- **Context**: Requirement 1 AC3 requires distinguishing an uncommitted-changes build from an exact commit. -- **Findings**: both git's own `--dirty` flag and this project's existing Nix `self.dirtyShortRev` already use a suffix convention (`<hash>-dirty`) rather than a separate boolean field. -- **Implications**: reuse that convention — `commit_id`'s own string value carries `-dirty` when applicable, no third meson option/env var needed. Matches the Build-vs-Adopt lens (reuse an established convention already present twice in this codebase) and the Simplification lens (one string carries both facts). - -## Design Decisions - -### Decision: One resolved value pair, fed from two independent computation sites -- **Context**: production (`nix build`) and dev-loop (`xce-build.sh`) are architecturally different environments (fully sandboxed vs. ambient-PATH devShell) that already compute `commit_id` differently for exactly that reason. -- **Alternatives Considered**: - 1. Make `xce-build.sh` also avoid shelling out to git, deriving everything from Nix like the packages do — rejected, `xce-build.sh` doesn't invoke Nix package builds at all, it drives `meson` directly inside `nix develop`. - 2. Add a git-invoking `run_command()` inside `meson.build` itself, used by both paths — rejected, breaks the sandboxed `nix build` path where git is unavailable by design. -- **Selected Approach**: extend the existing split — `xce-build.sh` computes via `git`+`date` and passes `-Dcommit_id=`/`-Dbuild_time=`; the nix packages (`xce.nix`, `xcecli.nix`) compute `commit_id` via `self.shortRev`/`self.dirtyShortRev` (unchanged, eval-time) and `build_time` via a `preConfigure` build-phase `date -u` hook appending to `mesonFlagsArray` (build-time, not eval-time), passed the same `-Dbuild_time=` way. -- **Rationale**: matches the codebase's own existing precedent exactly; zero new dependencies; keeps the sandboxed build's reproducibility model intact. -- **Trade-offs**: two computation sites to keep in sync (same risk the existing `commit_id`-only mechanism already carries and has lived with). -- **Follow-up**: verify during implementation that a plain `meson setup` with neither `xce-build.sh` nor a nix package (e.g. `meson setup build` run by hand) degrades to the "unknown" fallback (Requirement 1 AC4), not a build failure. -- **Correction (task 2.2, round 1 → round 2)**: the nix-side value was originally attempted via `builtins.currentTime` (eval-time) as noted above — this broke Nix's flake pure-eval mode entirely (`nix build .#xcecore_debug`, the documented no-flag command, failed with `error: attribute 'currentTime' missing`, requiring `--impure`) and produced a raw epoch-seconds string rather than UTC ISO-8601. Caught by task-local review (REJECTED, round 1) before merge; corrected to the `preConfigure`/`date -u` build-phase approach described in "Selected Approach" above, verified to restore pure-eval and produce a correctly-formatted value (round 2, APPROVED). Recorded here so this decision's rationale stays accurate for anyone reading only `research.md`. - -### Decision: Worker gets a hand-rolled `--version` case, not a switch to argp -- **Context**: worker uses `getopt_long()`, not `argp`; controller/scheduler's free `--version` comes from argp's own auto-generated handling. -- **Alternatives Considered**: migrate worker's arg parsing to argp for consistency — rejected as disproportionate scope (a full parser migration to fix a missing flag). -- **Selected Approach**: add a `{"version", no_argument, ...}` long option and a `case`, printing the same `MK_VERSION`-style string then exiting 0 — mirrors argp's own output shape without adopting argp. -- **Rationale**: minimal, consistent output, zero risk to worker's existing option handling. - -## Risks & Mitigations -- Two independent computation sites (dev-loop vs. nix-package) could drift in format — mitigate by having both resolve to the exact same two meson option names and the exact same downstream consumption code, so only the *value-sourcing* differs, never the *shape*. -- `go_build`/`meson-build.sh` are shared shell scripts already carrying other env-var-driven behavior (CGO flags, schema sync) — new env vars must be additive/optional so a caller that doesn't set them still builds (falls back to "unknown"), not required. - -## References -- `src/meson.build:30`, `src/meson.options` — existing `commit_id` mechanism. -- `src/flake.nix:15-18`, `src/nixSupport/xce.nix:109`, `xcecli.nix:30` — existing Nix-side commit id derivation. -- `src/tools/xce-build.sh:70-98` — dev-loop meson invocation, current gap. -- `src/tools/go_build`, `src/frontend/apiserver/meson.build:53-62`, `src/frontend/xcecli/meson.build:9-17` — Go build plumbing. -- `src/xceadmin/meson.build:27-47`, `src/xceadmin/meson-build.sh` — Python build plumbing. diff --git a/.kiro/specs/build-time-versioning/spec.json b/.kiro/specs/build-time-versioning/spec.json deleted file mode 100644 index 22c30f87..00000000 --- a/.kiro/specs/build-time-versioning/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "build-time-versioning", - "created_at": "2026-08-20T07:02:00Z", - "updated_at": "2026-08-24T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/build-time-versioning/tasks.md b/.kiro/specs/build-time-versioning/tasks.md deleted file mode 100644 index 5860b81c..00000000 --- a/.kiro/specs/build-time-versioning/tasks.md +++ /dev/null @@ -1,132 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: meson build-time option and C project-wide define - - Add a `build_time` meson option (string, default `''`) alongside the existing `commit_id` option in `src/meson.options` - - Extend `src/meson.build`'s existing `add_project_arguments()` call to also define `VERSION_BUILD_TIME` for every C component, mirroring how `VERSION_COMMIT_ID` is already threaded through - - Observable: a plain `meson setup` with neither option set still configures successfully (both defines resolve to empty string, no build failure) - - _Requirements: 1.1, 1.2, 1.4, 4.3_ - -- [x] 2. Foundation: compute real values at both build entry points -- [x] 2.1 (P) xce-build.sh computes and passes commit id and build time - - Compute `commit_id` via `git rev-parse --short HEAD`, appended with `-dirty` when the tree has uncommitted changes (`git status --porcelain` or `git diff --quiet HEAD --`) - - Compute `build_time` via `date -u +%Y-%m-%dT%H:%M:%SZ` - - Fall back to the literal string `unknown` for `commit_id` when `git` itself fails (e.g. no `.git` present) - - Append `-Dcommit_id="$commit_id" -Dbuild_time="$build_time"` to `MESON_OPTS` for every build profile, before the existing `meson setup`/`meson setup --reconfigure` calls - - After computing both values, verify each is non-empty (`build_time` always is; `commit_id` falls back to `unknown` only on a genuine `git` failure) and print a loud warning to stderr if either resolves empty anyway -- the build tooling must not silently proceed with a value it cannot confirm - - Observable: running `/xce-build scheduler debug` followed by the resulting binary's `--version` shows a commit id matching the current `git rev-parse --short HEAD` at build time, not empty; deliberately breaking the computation (e.g. temporarily renaming `.git`) prints the warning instead of silently succeeding - - _Requirements: 1.1, 1.2, 1.3, 1.4, 4.1, 4.2, 5.1, 5.2_ - - _Boundary: xce-build.sh_ - -- [x] 2.2 (P) Nix packages compute and pass build time - - ~~Add a `builtins.currentTime`-derived build-time value in `src/flake.nix`~~ — corrected during review (round 1 REJECTED): `builtins.currentTime` breaks pure-eval `nix build` (requires `--impure`) and yields non-UTC epoch seconds. Implemented instead as a `preConfigure` build-phase `date -u +%Y-%m-%dT%H:%M:%SZ` hook in `xce.nix`/`xcecli.nix` appending to `mesonFlagsArray`, alongside the unchanged eval-time `-Dcommit_id=` wiring - - Fail loudly (non-empty guard) if `date -u` ever produces empty output, before the value reaches `mesonFlagsArray` - - Observable: `nix build .#xcecore_debug` produces a binary whose `--version` shows a non-empty, ISO-8601 UTC build time, without any `git` invocation inside the sandboxed derivation, and without requiring `--impure` - - _Requirements: 1.1, 1.2, 1.4, 4.1, 4.2, 5.1, 5.2_ - - _Boundary: Nix packaging_ - -- [x] 3. Core: C daemon version display -- [x] 3.1 Extend the shared version macro with build time and unknown fallback - - ~~Extend `MK_VERSION`/`STR`~~ — `MK_VERSION`/`STR` cannot express the empty-string-to-`unknown` substitution at the preprocessor level, so implemented as new runtime functions `xce_version_field()`/`xce_compose_version()` (`utils.h`/`utils.c`) instead; `MK_VERSION`/`STR` marked SUPERSEDED with an explicit doc comment and handoff notes for 3.2 (round 1 REJECTED for silently orphaning `MK_VERSION` + undocumented `argp_program_version` file-scope-const-initializer trap; round 2 APPROVED after adding the SUPERSEDED comment + NOTE FOR CALLERS guidance) - - Both fields render as `unknown` when their underlying define is empty (verified via `""`, not just `NULL`), never a blank/garbled string - - Observable: a build with both meson options unset produces a `--version` string containing `unknown` for both fields, not empty gaps -- 5 new unit tests in `test_utils.c` confirm this - - _Requirements: 1.4, 2.2_ - - _Boundary: utils.h_ - -- [x] 3.2 (P) Controller and scheduler: version in --version and startup log - - `argp_program_version` migrated to `xce_compose_version()` via a `static char version_buf[128]` computed and assigned before `argp_parse()` in all 3 files (controller/main.c, scheduler/main.c, test.c); zero remaining `MK_VERSION` callers repo-wide outside its own definition - - Startup log lines read the identical `version_buf`, never recomputed -- extended to BOTH of scheduler's mutually-exclusive first-INFO branches (`--peers` and no-`--peers`), not just the one literally quoted in this task's text, since Requirement 3.1's "when the daemon starts" is unconditional - - Observable confirmed live (independently re-verified by review, not just self-reported): `xce_controller --version`, `xce_test --version`, and `xce_scheduler --version` all print commit id + build time and exit; all startup log branches for both daemons are byte-identical to their own `--version` output - - _Requirements: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.3_ - - _Boundary: controller/main.c, scheduler/main.c, controller/testsuite/test.c_ - - _Depends: 3.1_ - -- [x] 3.3 (P) Worker: add --version and startup log line - - Added `OPT_VERSION=257` (long-only, beyond ASCII range, no collision with `"n:l:vh"`/`OPT_PEERS=256`); handling case computes `xce_compose_version()` into a `static char version_buf[128]` and returns before any startup work (`check_user()`/logdir/`xcecd_init`/`xcecd_start`) - - Both startup INFO log branches (`--peers` and no-`--peers`) extended to log the same buffer - - Observable confirmed live (independently re-verified by review): `xcecd --version` prints and exits; both startup log branches (captured via a real `sudo` daemon run) are byte-identical to `--version` output - - _Requirements: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.3_ - - _Boundary: worker/main.c_ - - _Depends: 3.1_ - -- [x] 4. Core: Go binary version display -- [x] 4.1 go_build script injects ldflags from environment - - `src/tools/go_build` reads `XCE_GIT_COMMIT`/`XCE_BUILD_TIME` via `[ -z ... ]` (correctly empty-safe, not just unset-safe) and appends `-ldflags "-X main.GitCommit=... -X main.BuildTime=..."` to its `go build` invocation, falling back to `unknown` - - Confirmed (research finding, independently reproduced by review): `-X` on a `main` package var that doesn't exist yet is silently ignored by the Go linker, not a build error -- this task has no real ordering dependency on 4.4/4.5 - - Observable: invoking `go_build` with neither env var set still produces a working binary whose injected vars read `unknown` - - _Requirements: 1.1, 1.2, 1.4, 4.1, 4.2_ - - _Boundary: go_build script_ - -- [x] 4.2 apiserver and xcecli meson builds pass resolved values through - - Both meson.build files pass `XCE_GIT_COMMIT`/`XCE_BUILD_TIME` (from `get_option('commit_id')`/`get_option('build_time')`) into `go_build`'s env, UNQUOTED (go_build's `fix_input_envs()` only quote-strips `CURRENT_SOURCE_DIR`/`BUILD_ROOT`, not these two -- quoting them would embed literal `"` chars and corrupt the `-ldflags` argument) - - Verified live (independently, by review): real commit id + build time reach `go build`'s actual invocation and are embedded in both binaries' Go build-info metadata (`go version -m`) - - Observable partially deferred by design: no `main.GitCommit`/`main.BuildTime` symbols exist in either binary yet (`go tool nm` confirms) since apiserver/xcecli's `main` packages don't declare those vars until tasks 4.4/4.5 -- this task's own scope (`_Depends: 4.1_` only) is the env-var plumbing itself, not the full round trip; task 6.2 owns re-verifying that once 4.4 lands - - _Requirements: 1.1, 1.2, 4.1, 4.2_ - - _Boundary: apiserver/meson.build, xcecli/meson.build_ - - _Depends: 4.1_ - -- [x] 4.3 (P) Makefile dev-loop path computes and passes the same values - - `GIT_COMMIT_HASH`/`GIT_STATUS`/`GIT_COMMIT`/`BUILD_TIME`/`VERSION_LDFLAGS` (`:=`, computed once per invocation) mirror `xce-build.sh`'s dirty-check semantics and `go_build`'s exact `-ldflags` shape; `build-cli`/`build-server` recipes reference `$(VERSION_LDFLAGS)` - - `unknown` fallback verified via empty-output detection (robust for `git rev-parse --short HEAD` specifically: never succeeds with empty stdout) - - `--version` CLI output not yet observable (tasks 4.4/4.5 not landed) -- verified instead via `go version -m`'s embedded build-info metadata, matching live `git rev-parse --short HEAD` + dirty suffix - - `make build-server`'s pre-existing CGO failure (bug-1597) independently confirmed unrelated via a stash A/B (identical failure on the unmodified Makefile) -- not a regression - - _Requirements: 1.1, 1.2, 1.3, 1.4, 4.1, 4.2, 5.1, 5.2_ - - _Boundary: Makefile_ - - _Depends: 4.1_ - -- [x] 4.4 (P) apiserver: version flag, vars, and startup log - - `GitCommit`/`BuildTime` package vars (default `unknown`); `-version` flag checked before `utils.PromptForTopDir`/`utils.InitLogger()`, zero filesystem side effects (live-verified: no logs dir created) - - Startup log extended via `zap.String("gitCommit", ...)`/`zap.String("buildTime", ...)` structured fields on the existing `"Initialize Logger done!"` line, per design.md's explicit mandate - - Observable confirmed live (independently re-verified by review, full round trip): `-version` and the startup log's first line are byte-identical (`45c9b026-dirty`, matching `git rev-parse --short HEAD` + dirty suffix, and a real UTC timestamp) - - _Requirements: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2_ - - _Boundary: apiserver/cmd/main.go_ - - _Depends: 4.1_ - -- [x] 4.5 (P) xcecli: version field and vars - - `GitCommit`/`BuildTime` vars in `main.go`; `router.go`'s `Execute()` extended with a `version string` parameter, setting `rootCmd.Version` before `rootCmd.Execute()` runs (the sole call site, confirmed by both implementer and review) - - Cobra's native `--version`/`-v` auto-registers once `rootCmd.Version != ""` (confirmed via reading Cobra v1.9.1 source directly) -- zero additional flag-parsing code, as designed - - Observable confirmed live (independently re-verified by review, both build paths): `xcecli --version` prints commit id (matching `git rev-parse --short HEAD` + dirty suffix) and build time, exits 0 - - xcecli's startup log deliberately NOT extended -- Requirement 3 scopes that obligation to the four daemons only; xcecli is a one-shot CLI (design.md corrected to match) - - _Requirements: 1.1, 1.2, 2.1_ - - _Boundary: xcecli/cmd/main.go, xcecli/router/router.go_ - - _Depends: 4.1_ - -- [x] 5. Core: xceadmin version display -- [x] 5.1 (P) meson-build.sh generates a build-info module before packaging - - `commit_id`/`build_time` passed as 2 new positional args (`$6`/`$7`, matching this custom_target's existing all-positional style); write block sits between schema-sync and `pip install`, with an independent-per-field `unknown` fallback and a file-existence guard checking the exact path it just wrote (the mirrored tree, not the real source tree) - - Observable confirmed live (full real build+test+wheel cycle, independently re-run by review): mirrored tree's `_build_info.py` and the packaged wheel both contain real, matching, non-`unknown` values; 258 tests pass, 3 skips confirmed as documented live-HA e2e tests - - _Requirements: 1.1, 1.2, 1.4, 4.1, 4.2_ - - _Boundary: xceadmin/meson.build, xceadmin/meson-build.sh_ - - _Depends: 1_ - -- [x] 5.2 (P) Committed fallback stub and CLI version wiring - - Committed `src/xceadmin/src/xceadmin/_build_info.py` with `GIT_COMMIT`/`BUILD_TIME` both `"unknown"` -- exact variable names verified identical to 5.1's `meson-build.sh`-generated module; `cli.py`'s import is a genuine `try/except ImportError` (inline fallback class), not a bare import - - `click.version_option()` wired directly on the top-level `@click.group() def cli()`; version string composed via `importlib.metadata.version("xceadmin")` plus both `_build_info` fields, no hand-parsed `pyproject.toml` - - Observable confirmed live (independently re-verified by review, both install paths): fresh scratch-venv editable install (`pip install --no-deps -e .`, no meson) -> `xceadmin --version` prints `0.1.0 (unknown, unknown)`, exit 0, no import error; the meson-built wheel installed into a separate scratch venv -> `--version` shows real, non-`unknown` commit id + build time. Full meson build/test: 261 passed, 3 skipped (258 baseline + 3 new tests) - - _Requirements: 1.1, 1.2, 2.1_ - - _Boundary: xceadmin/src/xceadmin_ - - _Depends: 5.1_ - -- [x] 6. Integration: cross-toolchain consistency and regression verification -- [x] 6.1 Motivating-incident regression test - - New standalone script `src/tools/test-build-version-regression.sh` (follows the existing `test-commit-durability.sh`/`test-heartbeat-tolerance.sh` convention: real-build regression check, not a meson `test()`) runs real `xce-build.sh scheduler` builds in fresh dirs for both `asan` and `debug`, extracts the commit id from `xce_scheduler --version`, and compares it against a live `git rev-parse --short HEAD` (+`-dirty`, using the identical dirty-detection logic as `xce-build.sh` itself) - - Deliberately tolerates only `xce-build.sh`'s own known-unrelated flaky-test exit code (bug-1560) via captured exit status, not a blanket `set -e` bypass -- a real compile failure still fails the check, since fresh build dirs mean no stale binary can survive to fool the binary-existence + `--version` gate - - Observable confirmed live (independently re-run twice, by both implementer and review, from-scratch builds each time): both asan and debug profiles PASS with a real, matching, non-`unknown` commit id; a built-in demonstration block proves the comparison logic correctly fails loudly on a wrong hash, an empty commit id, and the literal `unknown` fallback - - _Requirements: 4.1, 4.2, 4.3, 5.1, 5.2_ - - _Depends: 2.1, 3.2, 3.3_ - -- [x] 6.2 Version/startup-log consistency test - - New standalone script `src/tools/test-version-startup-consistency.sh` (same house style as task 6.1's `test-build-version-regression.sh`) builds `xce_scheduler` and `xce_apiserver` for real, starts each daemon, greps its startup log, and asserts the commit-id/build-time VALUES (not whole-line text, since the C daemon's plain-text log line and apiserver's structured zap-JSON fields are differently shaped by design) are identical between `--version`/`-version` output and the startup log - - apiserver's build routes around two pre-existing, unrelated, independently-reproduced sandbox blockers -- `make build-server`'s missing CGO env (bug-1597) and `xce-build.sh apiserver`'s `go_build`-hardcoded `GOPROXY=goproxy.cn` hanging in this sandbox (confirmed live by review via orphaned processes) -- via a direct `go build -mod=readonly -ldflags` invocation computing `commit_id`/`build_time` with the identical logic and injecting them via the identical `-X` shape `go_build` itself uses; only the orchestration layer differs, not what's injected or how task 4.4's unmodified `main.go` consumes it - - Observable confirmed live (independently re-run in full by both implementer and review, real builds + real daemon startups each time): scheduler and apiserver both PASS with matching, real commit id + build time; a demonstration block proves the check correctly rejects a commit-id mismatch, a build-time mismatch, and an empty field on either side - - _Requirements: 2.2_ - - _Depends: 3.2, 4.4_ - -- [x] 6.3 xceadmin end-to-end build/install/version cycle - - New standalone script `src/tools/test-xceadmin-build-install-version.sh` (same house style as tasks 6.1/6.2) runs the full real `xce-build.sh xceadmin` cycle (schema sync, venv install, pytest, wheel build), installs the resulting wheel into a fresh scratch venv, and separately does an editable install (`pip install -e src/xceadmin`) into a second fresh scratch venv - - Explicit differ-assertion (not just two independent pass/fail checks) that the wheel install's and editable install's build-info fields are NOT equal -- proving the generated module genuinely replaces the fallback rather than both installs coincidentally agreeing - - Observable confirmed live (independently re-run in full by both implementer and review, real build+two installs each time): wheel install shows real, non-`unknown` commit id + build time; editable install shows the committed `unknown`/`unknown` fallback; a demonstration block proves the check rejects a wheel install showing `unknown`, an editable install showing a real-looking hash, and -- the crux case -- two identical build-info outputs fed to the differ-check - - _Requirements: 1.1, 1.2, 2.1, 4.1, 4.2_ - - _Depends: 5.1, 5.2_ - -## Implementation Notes -- **Task 2.2**: `builtins.currentTime` cannot be used for anything feeding a `nix build` derivation without `--impure` — Nix's flake pure-eval mode strips it entirely, breaking the plain, documented `nix build .#xcecore`/`.#xcecore_debug` commands. It also yields a raw epoch-seconds integer, not UTC ISO-8601. Any nix-side build metadata that must vary per-build (timestamps, etc.) belongs in a derivation's `preConfigure` (or similar) build-phase shell hook using ordinary coreutils (`date -u`), appended to `mesonFlagsArray` — never in eval-time Nix expressions. `self.shortRev`/`self.dirtyShortRev` remain the correct, eval-time-safe mechanism specifically for the commit id (Nix computes those specially even under pure-eval), so this distinction is per-value, not a blanket "avoid nix expressions" rule. diff --git a/.kiro/specs/clusterconfig-schema-split/design.md b/.kiro/specs/clusterconfig-schema-split/design.md deleted file mode 100644 index 6d81b948..00000000 --- a/.kiro/specs/clusterconfig-schema-split/design.md +++ /dev/null @@ -1,272 +0,0 @@ -> Sections may be reordered when it improves clarity; this is a backend/systems (C daemon + capnp schema) change, not a web app — "Service Interface"/API/Event contract subsections are adapted to C function-signature contracts where used, and omitted where not applicable. - -## Overview - -**Purpose**: Removes eight fields from the `ClusterConfig` capnp struct (`port`, `candidates`, `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings`) that are either already dead at runtime or (in `port`'s case) load-bearing for single-node startup but conceptually misplaced in cluster-shared config. `port`'s replacement source is the already-existing `--peers` candidate-match mechanism plus a hardcoded default — no new CLI flag. - -**Users**: XCE maintainers (schema/controller code owners) and, indirectly, XCE operators running single-node or `--peers`-mismatched controllers, whose startup behavior must not change. - -**Impact**: `ClusterConfig` shrinks from 21 to 13 fields. `cluster_config_t` (the generated C struct) loses the matching members. `cluster_open_listener`'s port-resolution fallback moves from a capnp-blob field to a two-tier shape: `--peers` candidate match (unchanged, including the already-supported single self-referencing `--peers <name>=<host>:<port>` entry for single-node mode) → hardcoded constant when `--peers` was never passed at all. - -### Goals -- Remove the 8 fields and the `MasterCandidate` struct from `controller.capnp`, and their mirrors in `cluster_config_t`. -- Replace `port`'s ClusterConfig-sourced fallback with the existing `--peers` candidate-match mechanism (unchanged) plus a hardcoded default for the true zero-`--peers` case, preserving single-node startup behavior exactly with no new CLI flag. -- Prove behavior-preservation live (single-node startup, 3-node soak against a regenerated `ha/data/cluster.bin` fixture) — not just by unit tests, given the touched files (`cluster.c`, `ha.c`, `main.c`) are HA-critical. -- (Nice-to-have) Add CLI flags for the hash-tuning tunables, closing the existing `cluster.c` TODOs. - -### Non-Goals -- `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness) — unchanged, stay in `ClusterConfig`. -- `fetchJobInterval`/`netioTimeout` — unresolved, left untouched, a candidate for a future pass. -- A real HA/Raft election-heartbeat CLI parameter — separate roadmap item, not this spec. -- Peer list replication protocol / joint consensus — future, unrelated to a field deletion. -- Any change to `entity-bearing-snapshots` / `config-upgrade-snapshot`. - -## Boundary Commitments - -### This Spec Owns -- `ClusterConfig` and `MasterCandidate` struct shape in `controller.capnp`. -- `cluster_config_t`'s field list in `clusterconfig.h`. -- The port-resolution fallback branch in `cluster_open_listener` (`cluster.c`): `--peers` candidate match (unchanged) → hardcoded default constant. -- The dead-code removals that don't need a replacement: `candidates` KV-mirror loop (`cluster_build_config`), `SET_DEFAULT_IF_UNSET(c->port, ...)` call sites, and the (already-inert) `jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`/`vcpus` blob reads. -- The two known test-fixture updates (`test_commit_barrier_rollout.c`, `test_config_repl.c`) and any others a full-repo grep surfaces during implementation. -- Regenerating capnp bindings. -- (Optional, Requirement 6) CLI-configurable perf tunables — five new `OPT_*` flags mirroring the existing `--vcpus`, deferrable without blocking the rest of this spec. - -### Out of Boundary -- `heartbeatInterval`/`inactiveHeartbeatCount`, `fetchJobInterval`/`netioTimeout` — no field or code change. -- A `ClusterConfig` field or CLI flag for the HA/Raft election heartbeat — structurally disallowed (`config-design.md` Rule 0); tracked as its own roadmap item. -- Peer list replication / joint-consensus membership — unrelated, `candidates` is deleted, not replaced with a distributed mechanism. -- `config-upgrade-snapshot`, `entity-bearing-snapshots` — no dependency either direction. -- `admin-config-tool`'s `xceadmin` implementation — this spec closes the risk that tool depends on (per its requirements.md), but does not implement or block on any of that tool's own code. - -### Allowed Dependencies -- Existing build tooling: capnp compiler + `c-capnproto` codegen (schema → C bindings), already wired into the meson build. -- `argp` CLI parsing (already used by every other controller flag, including `--vcpus`). -- No new external dependencies, no new libraries. - -### Revalidation Triggers -- Any future proposal to add a field to `ClusterConfig` must be checked against `.kiro/steering/config-design.md`'s inclusion rules before landing. -- `admin-config-tool`'s `xceadmin config build`/`push` implementation, when it lands, should target the post-split (13-field) schema — if it's implemented against the pre-split schema instead, that's a signal this spec's sequencing assumption was violated. -- If the controller's invocation contract changes elsewhere (e.g., `--peers` becomes mandatory for another reason), revisit whether the hardcoded-default fallback added here is still needed. - -## Architecture - -### Existing Architecture Analysis -`ClusterConfig` today mixes three concerns in one capnp struct: cluster-wide business config (queues, resources, groups — genuinely shared), HA peer membership (`candidates`, already superseded in practice by the `--peers` CLI flag into a separate `xce_cluster_config_t`/`peers_cfg` struct), and per-controller performance tunables (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`/`vcpus`, already hardcoded in `cluster.c`'s `init_controller_tunables()` with TODOs acknowledging the blob values are dead). `port` sits in the same struct but, unlike the others, still has one live reader: `cluster_open_listener`'s fallback branch, reached whenever the local controller's name isn't matched in `--peers` (including single-node mode, which passes no `--peers` at all). - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - Schema[ClusterConfig capnp schema] - Codegen[capnp codegen] - ConfigStruct[cluster_config_t] - Blob[cluster.bin blob] - - PeersFlag[peers CLI flag] - PeersCfg[peers_cfg struct] - - Listener[cluster_open_listener] - Tunables[init_controller_tunables] - Business[cluster_init_phase2 business config] - - Schema --> Codegen --> ConfigStruct - Blob --> ConfigStruct - ConfigStruct --> Business - - PeersFlag --> PeersCfg --> Listener - Listener --> DefaultConst[hardcoded default constant] - - Tunables --> HardcodedTunables[hardcoded tunable constants] -``` - -**Architecture Integration**: -- Selected pattern: no new CLI surface — `port` resolution reuses the existing `--peers` candidate-match loop (a single self-referencing `--peers <name>=<host>:<port>` entry already gives single-node mode a custom port today) plus a hardcoded fallback constant for the true zero-`--peers` case. -- Domain/feature boundaries: `ClusterConfig` (schema) retains only fields that are genuinely cluster-shared (per `config-design.md`); `peers_cfg` (CLI-sourced, per-process, already existing) owns port resolution for everything this spec touches. -- Existing patterns preserved: `--peers` remains the sole port/membership source; `init_controller_tunables()`'s hardcoded constants are the default values and reproduce today's exact behavior when no tunable flag is passed (Requirement 2.2 — no behavior change by default). (Note: the optional Requirement 6 CLI-sourcing was in fact implemented — `init_controller_tunables()` now reads the corresponding `myconf_t` field first, falling back to the hardcoded constant only when that field is 0/unset — so the function itself is no longer literally untouched, though default behavior is unchanged.) -- New components: one new hardcoded constant (`cluster.c` or `clusterconfig.h`) — the minimum needed to satisfy Requirement 1.7 without any invocation-contract or CLI-surface change. -- Steering compliance: enforces `.kiro/steering/config-design.md`'s independence/frequency/Rule-0 classification; follows `src/controller/CLAUDE.md`'s HA protocol verification checklist for the touched files. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Schema | Cap'n Proto (existing, `controller.capnp`) | Defines `ClusterConfig`/`MasterCandidate` shape | No version change | -| Codegen | `c-capnproto` (existing) | Schema → C struct/reader/builder | Regenerated, not upgraded | -| Backend | C (gnu17, existing) | `cluster.c`/`myconf.c`/`config_repl.c` changes | No new C dependency | -| CLI | `argp` (existing) | No new flag for Req 1.7; reused unchanged for optional Req 6 tunables, same pattern as `--vcpus` | No new dependency | - -## File Structure Plan - -### Modified Files -- `src/schema/controller.capnp` — remove `port` (@1), `candidates` (@2), `vcpus` (@7), `jobmgr` (@13), `workermgr` (@14), `qmgr` (@15), `resmgr` (@16), `xstrings` (@17) from `ClusterConfig`; remove the `MasterCandidate` struct entirely; remove the now-orphaned `HashZArrayConfig`/`ResmgrConfig` structs (used only by the removed fields — discovered during `/kiro-impl` task 2.1, their `$C.mapname` bindings broke the capnp-generated code once task 1.2 removed the C typedefs they referenced). Cap'n Proto requires gapless ordinals, so the 13 retained fields are renumbered to a contiguous `0..12` range as a direct consequence — see `research.md`'s ordinal-gap finding (Req 1.1, 1.2, 1.6, 2.1, 2.5, 3.1). -- `ha/data/cluster.bin` (binary fixture, not source) — regenerated in place against the renumbered schema, preserving the retained fields' semantic values from the pre-spec blob (Req 3.2, 5.3). (Note: superseded by `admin-config-tool` (landed 2026-07-22, one day after this spec closed) — `ha/data/` was deleted entirely and `deploy-ha.sh` now builds `cluster.bin` fresh via `xceadmin config build` from `src/configs/*.yaml` on every deploy run, not from a checked-in regenerated blob. See that spec's design.md.) -- `src/include/clusterconfig.h` — remove the matching `cluster_config_t` members: `port`, `ncandidates`, `candidates`/`master_candidate_t**`, `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings` (Req 1.5, 2.4). -- `src/controller/cluster.c` — (a) remove the `candidates`/`ncandidates` KV-mirror loop in `cluster_build_config` (Req 1.4); (b) change `cluster_open_listener`'s fallback branch from reading `cluster_config->port` to a hardcoded default constant, reached only when `--peers` was never passed at all — the `--peers` candidate-match branch itself is unchanged (Req 1.3, 1.7); `init_controller_tunables()` itself is unchanged (Req 2.2). -- `src/controller/myconf.c` — remove the `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` call (the field it defaults no longer exists) (part of Req 1.7's cleanup). -- `src/controller/config_repl.c` — remove the equivalent `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` call (same reason). -- `src/controller/testsuite/test_commit_barrier_rollout.c` — drop the `cfg.port = 9990;` literal at line 1359 (Req 5.1). -- `src/controller/testsuite/test_config_repl.c` — drop the `cfg.port = 9991;` literal at line 983 (Req 5.1). -- Capnp-generated bindings (build output, not hand-edited) — regenerated as part of the normal build after the schema change (Req 1.1-1.2, 1.6, 2.1). - -### Verification Infrastructure (existing, no new files) -- `controller_unit_test` (meson target, existing) — runs the full existing suite including the two updated test files above (Req 5.2). -- A small throwaway capnp-encoding tool (written and discarded during implementation, not part of the shipped codebase) — decodes `ha/data/cluster.bin`'s retained-field values under the pre-spec schema and re-encodes them under the post-spec (renumbered) schema, overwriting the fixture (Req 3.2, 5.3). (Note: the fixture this tool targeted no longer exists — see the `ha/data/cluster.bin` note above.) -- `/xce-deploy-ha` skill (existing, `src/tools/deploy-ha.sh`) — deploys the 3-node live cluster for the config-push cycle, using the regenerated `ha/data/cluster.bin` (Req 5.4). (Note: superseded by `admin-config-tool` — `deploy-ha.sh` now generates `cluster.bin` fresh via `xceadmin config build` instead of consuming this fixture. See that spec's design.md.) -- Direct `xce_controller` invocation with no `--peers` flag (existing supported mode, `main.c:441-443` — line numbers shifted from the original `main.c:293-295` after this spec's own Requirement 6 CLI flags were added earlier in the file) — the single-node startup verification (Req 5.5). -- Direct `xce_controller` invocation with a single self-referencing `--peers <name>=<host>:<port>` entry (existing supported mode) — verifies the custom-port replacement mechanism itself (Req 5.5). - -### Optional (Requirement 6, nice-to-have) -- `src/controller/main.c` — five new `OPT_*` flags (`--jobmgr-nhash`, `--workermgr-nhash`, `--qmgr-nhash`, `--resmgr-nhash`, `--xstrings-nhash`, plus `-delta`/`-inst` variants where the tunable has more than one dimension), same pattern as the existing `--vcpus`. -- `src/include/myconf.h` — matching `myconf_t` fields. -- `src/controller/cluster.c` — `init_controller_tunables()` reads the new `myconf_t` fields with the existing hardcoded values as fallback (0 = unset), rather than always using the hardcoded constant. - -No new files are created by this spec — every change is to an existing file. - -## System Flows - -Port resolution at controller startup — corrected during design review (see `research.md` Design Decisions for the two fixes below): - -```mermaid -flowchart TD - Start[cluster_open_listener starts] - HasPeers{peers_cfg candidates_count greater than 0} - SearchName[search peers_cfg for local name] - NameFound{local name found} - PortValid{matched candidate port greater than 0} - UseMatchedPort[use matched candidate port] - UseFallback[use hardcoded default port] - RefuseStart[log error and refuse to start] - Bind[bind listener] - - Start --> HasPeers - HasPeers -->|no, --peers never passed| UseFallback --> Bind - HasPeers -->|yes| SearchName --> NameFound - NameFound -->|no| RefuseStart - NameFound -->|yes| PortValid - PortValid -->|yes| UseMatchedPort --> Bind - PortValid -->|no, malformed entry| UseFallback -``` - -**Key decisions** (corrected from the initial draft during `/kiro-validate-design`; the `--port` CLI flag was further revised away in favor of the mechanism below during `/kiro-impl` kickoff — see `research.md`'s "Design Revision" section): -- **Fallback trigger is port value, not just match presence.** A matched candidate whose port field is `0` (a malformed `--peers` entry — already WARNed about at `main.c:434-439`, was `main.c:289-291`; line numbers shifted after this spec's own Requirement 6 CLI flags were added earlier in the file) falls through to the hardcoded-default tier exactly like today's `if (my_port == 0)` check, rather than binding to literal port 0. This preserves Requirement 1.3's "exactly as it does today" for the already-tolerated malformed-entry case. -- **Error path is preserved, but only for one specific case.** `candidates_count == 0` can only happen when `--peers` was never passed at all — a passed-but-empty or malformed `--peers` string is rejected earlier by `xce_parse_peers` itself (`src/lib/cluster.c:59` returns `-1` unless `candidates_count > 0`; was cited as `:58`), which `main.c` treats as a startup error before `cluster_open_listener` is ever reached (corrected citation, 2026-07-20: this guarantee comes from `xce_parse_peers`'s contract, not `validate_peer_count`, which only rejects even non-zero counts) — that's genuine single-node mode, and gets the hardcoded-default fallback per Requirement 1.7. An operator who wants a non-default port in this mode passes a single self-referencing `--peers <name>=<host>:<port>` entry instead — an already-existing, already-supported invocation (this resolves via the unchanged `NameFound`/`PortValid` branches above, not a new mechanism). But when `--peers` WAS passed and the local name simply isn't in it, that's a likely operator misconfiguration (own name typo'd or omitted) — this spec keeps today's `-EINVAL`/refuse-to-start behavior for that case rather than silently defaulting, per the project's fail-loud convention (`src/controller/CLAUDE.md` process-lifecycle invariant). Decided interactively during design review rather than defaulting to "simplest single code path." - -## Requirements Traceability - -| Requirement | Summary | Components | Files | -|-------------|---------|-------------|-------| -| 1.1 | Remove `port` from schema | Schema Component | `controller.capnp` | -| 1.2 | Remove `candidates` from schema | Schema Component | `controller.capnp` | -| 1.3 | Listener uses `--peers` match when available | Listener/Port-Resolution Component | `cluster.c` | -| 1.4 | Debug dump stops reading `candidates` | Debug-Dump Component | `cluster.c` | -| 1.5 | `cluster_config_t` drops `port`/`candidates` members | C Struct Component | `clusterconfig.h` | -| 1.6 | Remove `MasterCandidate` struct | Schema Component | `controller.capnp` | -| 1.7 | Replacement port-sourcing mechanism (existing `--peers` self-reference + hardcoded default) | Listener/Port-Resolution Component | `cluster.c`, `myconf.c`, `config_repl.c` | -| 2.1 | Remove 6 perf-tuning fields from schema | Schema Component | `controller.capnp` | -| 2.5 | Remove orphaned `HashZArrayConfig`/`ResmgrConfig` structs | Schema Component | `controller.capnp` | -| 2.2 | Hardcoded tunables unchanged (as defaults) | Tunables Component | `cluster.c` (values unchanged; function itself later extended by the now-implemented Req 6.2 CLI-sourcing wrapper) | -| 2.3 | Controller stops attempting to read removed tunable fields | Tunables Component | `cluster.c` (implicit — field no longer exists) | -| 2.4 | `cluster_config_t` drops tunable members | C Struct Component | `clusterconfig.h` | -| 3.1 | Retained fields unchanged in type/number | Schema Component | `controller.capnp` (no change to retained fields) | -| 3.2 | Renumbering breaks pre-spec `cluster.bin` binary-compat (accepted); fixture rebuilt | Schema Component, Verification | `controller.capnp`, `ha/data/cluster.bin` regeneration | -| 3.3 | Worker-liveness behavior unaffected | Tunables/Business Component | `cluster.c` (no change) | -| 4.1 | Discard removed-field values from legacy blobs silently | Schema Component | `controller.capnp` (capnp default behavior) | -| 4.2 | No deprecate-in-place transition | Schema Component | `controller.capnp` | -| 5.1 | Build + test-fixture compile cleanly | Test Fixture Updates | `test_commit_barrier_rollout.c`, `test_config_repl.c` | -| 5.2 | No new unit test failures | Verification | controller unit test suite | -| 5.3 | Regenerate `ha/data/cluster.bin` fixture against the new schema | Verification | `ha/data/cluster.bin` (Note: `ha/data/` was later deleted entirely by `admin-config-tool`; see the "Note" on this file in File Structure Plan above) | -| 5.4 | 3-node live soak (config-push, peer/listener/worker-liveness) | Verification | live HA cluster | -| 5.5 | Single-node + bare-peers port-override + `-EINVAL` live verification | Verification | live single-node controller | -| 6.1 | (Nice-to-have) CLI tunable flags | CLI Tunables Component (optional) | `main.c`, `myconf.h` | -| 6.2 | (Nice-to-have) Fallback to hardcoded default | Tunables Component (optional) | `cluster.c` | - -## Components and Interfaces - -| Component | Domain | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|--------|--------|---------------|-------------------|-----------| -| Schema Component | capnp | Defines `ClusterConfig`/`MasterCandidate` shape | 1.1, 1.2, 1.6, 2.1, 3.1, 4.1, 4.2 | capnp compiler (P0) | State | -| C Struct Component | C headers | `cluster_config_t` field list matches schema | 1.5, 2.4 | Schema Component (P0) | State | -| Listener/Port-Resolution Component | C, `cluster.c` | Resolves listen port via `--peers` match (unchanged) with a hardcoded-default fallback | 1.3, 1.7 | C Struct Component (P0) | Service | -| Debug-Dump Component | C, `cluster.c` | Internal status/debug KV dump, no longer mirrors `candidates` | 1.4 | C Struct Component (P0) | State | -| Tunables Component | C, `cluster.c` | Hardcoded (or optionally CLI-sourced) perf tunables | 2.2, 2.3, 3.3, 6.2 | CLI Tunables Component (P1, optional path) | State | -| CLI Tunables Component (optional) | C, `main.c`/`myconf.h` | Requirement 6 only: parses tunable flags, populates `myconf_t`, same pattern as the existing `--vcpus` | 6.1 | argp (P0) | Service | -| Test Fixture Updates | C tests | Two known struct-literal sites updated to compile | 5.1 | C Struct Component (P0) | — | -| Verification | Build + live cluster | Proves behavior preservation | 5.1, 5.2, 5.3, 5.4, 5.5, 3.2, 4.1 | All components above (P0) | — | - -Only the Listener/Port-Resolution Component introduces new logic (the two-tier fallback); every other component is either schema/struct shrinkage or already-hardcoded behavior left untouched, so only that one gets a full contract block below. - -### Controller - -#### Listener/Port-Resolution Component - -| Field | Detail | -|-------|--------| -| Intent | Determine the controller's own listen port without depending on a `ClusterConfig` field | -| Requirements | 1.3, 1.7 | - -**Responsibilities & Constraints** -- Primary responsibility: resolve exactly one port value before `cluster_open_listener` binds. -- Precedence is strict and non-configurable: `--peers` match (if local name found) > hardcoded default constant. No tier is skipped or reordered by configuration. No CLI flag mediates this — an operator who needs a non-default port in the no-other-peers case supplies it via a single self-referencing `--peers <name>=<host>:<port>` entry, which resolves through the same match branch as any other `--peers` candidate. -- Must not depend on `cluster_config_t` in any form (the field this component replaces no longer exists after this spec). - -**Dependencies** -- Inbound: `main.c` startup sequence — invokes this resolution before/during `cluster_open_listener` (P0) -- Outbound: `myconf->peers_cfg` (existing, unchanged parsing) (P0) -- Outbound: hardcoded default constant (same file or `clusterconfig.h`) — no external dependency (P0) - -**Contracts**: Service [x] - -##### Service Interface -```c -/* cluster_open_listener — port resolution, updated contract */ -/* Precondition: myconf->peers_cfg is populated (possibly empty, iff - * --peers was never passed). */ -/* Postcondition, --peers never passed (peers_cfg.candidates_count == 0): - * returns 0 with the listener bound to the hardcoded default (9990). - * Always succeeds. An operator wanting a non-default port here passes - * a single self-referencing --peers entry instead (see below). - * Postcondition, --peers given: if the local name is matched with - * a nonzero port, binds to it (today's unchanged happy path — this - * is also how a single-node operator gets a custom port). If - * matched with a zero port (malformed entry), falls through to the - * hardcoded-default resolution. If the local name is NOT matched at - * all, returns -EINVAL and refuses to start — unchanged from today, - * preserved deliberately (see System Flows "Error path is preserved" - * decision). */ -int cluster_open_listener(cluster_t *c); -``` -- Preconditions: `myconf->peers_cfg` populated (empty iff `--peers` never passed). -- Postconditions: see the branches above — no-`--peers` and matched-with-zero-port always resolve to a bound listener; `--peers`-given-but-unmatched still returns `-EINVAL`. -- Invariants: precedence order (`--peers` match > hardcoded default) never changes based on runtime state; the `-EINVAL` case is determined solely by "was `--peers` passed AND local name absent," never by port value. - -**Implementation Notes** -- Integration: the match/no-match branch structure is unchanged from today's `cluster_open_listener`; only the final fallback's source changed (from `cluster_config->port` to a hardcoded `XCE_CONTROLLER_DEFAULT_PORT` constant). No new CLI flag, no new `myconf_t` field — see `research.md`'s "Design Revision" section for why this replaced the originally-designed `--port` flag. -- Validation: no new validation needed — the hardcoded constant requires no runtime validation, and `--peers` candidate validation is unchanged. -- Risks: none identified beyond what Requirement 5.5's live single-node test is designed to catch — this is the one path that removes a previously-live capability, and this component IS the requirement's specified replacement. Requirement 5.5's live verification must cover BOTH the true zero-`--peers` case AND the self-referencing single-candidate `--peers` case (the latter is now the sole way to get a non-default single-node port, so it's load-bearing, not incidental). The preserved `-EINVAL` path for "peers given, no match" should also get its own live/unit test coverage during tasks (not separately required by requirements.md, but a natural extension of Requirement 5.5's spirit — flag during task generation). - -## Error Handling - -### Error Strategy -No new error categories are introduced, and the existing `-EINVAL` error path (`cluster_open_listener`'s "no port could be determined," `cluster.c:1226` today) is **preserved**, not removed — corrected during design review, since single-code-path simplicity would have traded a clear startup error for a silent wrong-default in a likely-misconfiguration scenario (`--peers` given, local name absent). It now applies to exactly one case: `--peers` was passed and the local name isn't in it. The genuinely-new case (`--peers` never passed at all, i.e. single-node) and the pre-existing-tolerated case (name matched, that entry's port is `0`) both resolve via the hardcoded-default fallback and never error — unchanged from today's tolerance for the latter, newly-defined for the former. All other error handling (schema decode failures, capnp field-absence for legacy blobs) is capnp's existing default behavior: an old `cluster.bin` with data in now-unused field slots decodes without error, and those values are simply never read (Requirement 4.1) — no new error branch needed for that case either. - -### Monitoring -No new logging beyond what already exists for `--peers` parsing (`main.c:411-443`, was `main.c:284-291` — shifted after this spec's own Requirement 6 CLI flags were added earlier in the file) and controller startup (`INFO`/`WARN`/`ERROR` via the existing `mylog` macros). No new metrics. - -## Testing Strategy - -- **Unit**: build-time enforcement is the primary "test" for Requirements 1.1-1.2, 1.5-1.6, 2.1, 2.4 — a removed field/struct member that's still referenced anywhere is a compile error, not a silent gap (this is why Requirement 5.1's test-fixture fix is explicit rather than assumed). -- **Unit**: `controller_unit_test` full suite run (Requirement 5.2) — existing coverage for `cluster_build_config`, `init_controller_tunables`, and the two touched test files, now compiling against the shrunk struct. -- **Integration**: regenerate the `ha/data/cluster.bin` fixture (Requirement 3.2, 5.3) — since retained-field renumbering breaks binary compatibility with any pre-spec blob, decode the current fixture's semantic field values under the pre-spec schema and re-encode them under the new schema with a small throwaway tool; confirm the regenerated blob decodes correctly under the post-spec schema and `deploy-ha.sh`'s verbatim-copy usage keeps working unmodified. (Note: superseded by `admin-config-tool` — `deploy-ha.sh` no longer does verbatim-copy of a fixture at all; it builds `cluster.bin` fresh via `xceadmin config build` from YAML on every run. See that spec's design.md.) -- **Live/E2E**: 3-node HA cluster deploy + config-push cycle (Requirement 5.4) — using the regenerated fixture; peer discovery via `--peers`, listener binding, worker-liveness tracking all behave identically to pre-spec. -- **Live/E2E**: single-node controller startup with no `--peers` flag, AND with a single self-referencing `--peers` entry for a custom port (Requirement 5.5) — the actual regression risk this spec's design resolves; must be verified live, not inferred from the 3-node soak (single-node never exercises the `--peers`-match branch at all). -- **Live/E2E (optional, Requirement 6)**: if the nice-to-have CLI tunable flags are implemented, verify each flag overrides its corresponding hardcoded default and that omitting all of them reproduces today's exact hardcoded values. - -## Migration Strategy - -No data migration is required for the fields this spec targets — Requirement 4.2 explicitly rules out a deprecate-in-place transition, and Requirement 4.1 confirms retained-field renumbering is an accepted, unguarded compatibility break for this pre-1.0 product: a `cluster.bin` built before this spec is not expected to be usable as-is against a controller built after it, and no backward-read path is supported or verified for such an artifact. The one thing this spec DOES migrate is the checked-in `ha/data/cluster.bin` test fixture (Requirement 3.2, 5.3): its retained-field values are decoded under the pre-spec schema and re-encoded under the new schema with a small throwaway tool, so `/xce-deploy-ha` and dependent live-verification tooling keep working. (Note: superseded by `admin-config-tool`, landed 2026-07-22 — `ha/data/cluster.bin` was deleted entirely and `/xce-deploy-ha` now generates `cluster.bin` fresh via `xceadmin config build` from YAML, so this migrated fixture no longer exists or is consumed. See that spec's design.md.) There is no forward path either (a post-spec `cluster.bin` is not expected to be readable by a pre-spec controller, but that direction isn't a supported downgrade path in this project regardless of this spec). - -## Supporting References -- `src/docs/cluster-config-split-analysis.md` — prior team analysis; primary design input for the dead-field classification. -- `.kiro/specs/clusterconfig-schema-split/research.md` — port-replacement-mechanism investigation, alternatives considered, and the decision rationale in full. -- `.kiro/steering/config-design.md` — the general shared-vs-local config rule this spec enforces. diff --git a/.kiro/specs/clusterconfig-schema-split/requirements.md b/.kiro/specs/clusterconfig-schema-split/requirements.md deleted file mode 100644 index 45a6c86f..00000000 --- a/.kiro/specs/clusterconfig-schema-split/requirements.md +++ /dev/null @@ -1,104 +0,0 @@ -# Requirements Document - -## Introduction - -`ClusterConfig` (`src/schema/controller.capnp:711-733`, generated as C struct `cluster_config_t`) is the capnp struct pushed cluster-wide via `OP_upgradeConfig`. It currently carries several fields that were bundled in without a principled reason: `port`, `candidates` (peer/standby list), `vcpus`, and the `jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings` performance-tuning fields. This spec removes those fields from the schema and the C code that reads them, so `ClusterConfig` contains only fields that are genuinely cluster-shared, per the classification in `.kiro/steering/config-design.md`. - -**Correction (2026-07-20, second review pass)**: `candidates`, `vcpus`, and the perf-tuning fields ARE already dead weight at runtime (verified — see Adjacent Expectations). `port` is NOT — `cluster_open_listener` (`cluster.c:1216-1258`) has a live fallback to `cluster_config->port` that is the **sole port-sourcing mechanism for single-node deployments** (no `--peers` flag is passed at all in single-node mode — `main.c:293-294`). Removing the `port` field without adding a replacement mechanism would break single-node controller startup. This spec now requires that replacement as part of the `port` removal (Requirement 1.7), not as an assumed-safe deletion. - -This closes an assumption flagged in `.kiro/specs/admin-config-tool/requirements.md` (the "per-instance field preservation" risk) — not by verifying controller behavior, but by making the scenario structurally impossible: fields that don't exist in the schema can't be silently zeroed on a config push. - -## Boundary Context - -- **In scope**: - - Remove `port` (@1) and `candidates` (@2, `List(MasterCandidate)`) from `ClusterConfig`. - - Remove `vcpus` (@7) from `ClusterConfig` — already marked deprecated in the schema comment ("use --vcpus CLI flag; kept for config blob backward compatibility"); this is the schema revision that comment anticipated. - - Remove `jobmgr` (@13), `workermgr` (@14), `qmgr` (@15), `resmgr` (@16), `xstrings` (@17) from `ClusterConfig`. - - Remove the C code that reads the above fields from `cluster_config_t`: the `candidates`/`ncandidates` KV-mirror loop in `cluster_build_config` (`cluster.c`) is dead-code removal (no behavior change); the port-fallback branch in `cluster_open_listener` (`cluster.c:1216-1258`) is NOT dead code (see Introduction correction) and its removal requires a replacement mechanism per Requirement 1.7, not a bare deletion. Also remove the corresponding fields in `cluster_config_t` (`clusterconfig.h`) and the `SET_DEFAULT_IF_UNSET(c->port, ...)` calls in `myconf.c` and `config_repl.c` (only after the replacement mechanism lands). - - Regenerate capnp bindings after the schema change. - - Direct field removal (no deprecate-in-place transition, no field-number reservation) — the product has not reached 1.0, so capnp wire/on-disk compatibility for these fields is not a constraint (user decision, 2026-07-20). - -- **Out of scope**: - - `heartbeatInterval` (@3) and `inactiveHeartbeatCount` (@4) — these govern controller→**worker** liveness, not the HA/Raft election heartbeat (see Adjacent Expectations). Investigated and deliberately kept in `ClusterConfig` (operational-uniformity override — see `.kiro/steering/config-design.md`); this spec does not touch them. - - `fetchJobInterval` (@5) and `netioTimeout` (@6) — flagged by the prior `src/docs/cluster-config-split-analysis.md` as candidates for the same treatment, but not yet decided in this pass. Left untouched; a candidate for a future iteration of this spec or a follow-up. - - `clusterName` (@0), `resources`, `queues`, `ugroups`, `hgroups`, `hostConfigs`, `hproviders`, `limits` — cluster-shared business config, unaffected. - - `configGeneration` (@20) — controller-assigned protocol state, not operator config; unaffected, not part of this spec's field-removal scope. - - Adding a real HA/Raft election-heartbeat CLI parameter (`--heartbeat-interval-ms`/`--heartbeat-timeout-ms`) to replace the currently-hardcoded `HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS` — tracked as a separate roadmap item (`.kiro/steering/roadmap.md`, "Raft/HA heartbeat interval+timeout as CLI parameters"), not part of this spec. This spec does not add any new `ClusterConfig` field for HA heartbeat timing, and per Rule 0 in `config-design.md` never should. - - Peer list replication protocol (joint-consensus membership changes) — a much larger future item noted in the prior analysis doc's Phase 5; not needed here since `candidates` is simply being deleted, not replaced with a new distributed mechanism (the live peer source, `--peers` CLI, already exists and is unaffected). - - `entity-bearing-snapshots` / `config-upgrade-snapshot` — unrelated concern (snapshot consistency at config-change boundaries); this spec has no dependency on either. - -- **Adjacent expectations**: - - **Blast-radius finding (corrected 2026-07-20)**: `candidates` is fully dead in `ClusterConfig` at runtime — the live peer list comes exclusively from the `--peers` CLI flag (parsed into a separate struct, `myconf.peers_cfg`), and the blob's `candidates` is read only into a debug-only KV mirror (`cluster_build_config`, `cluster.c:851-885`) with no other readers. `jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`/`vcpus` are likewise dead — hardcoded constants in `cluster.c`'s `init_controller_tunables()` (`cluster.c:1093-1108`), with existing TODOs acknowledging the blob values are ignored; `vcpus` is sourced only from the separate CLI-set `myconf->vcpus`, never from the blob. **`port` is the exception**: `cluster_open_listener` (`cluster.c:1216-1258`) falls back to `cluster_config->port` whenever the local controller's name isn't matched in `--peers` — and single-node mode passes no `--peers` at all (`main.c:293-294`, "No --peers flag; running as single-node primary"), making this the sole port-sourcing path for that mode today. Also: `xce_parse_peers` only WARNs (not rejects) on a `--peers` candidate with `port == 0` (`main.c:289-291`), so a malformed `--peers` entry for the local node hits the same live fallback. Full analysis: `src/docs/cluster-config-split-analysis.md` (note: that doc's own effort/risk table calls this "Low risk... no fallback needed," which this spec's investigation found does not hold for single-node mode — treat the doc's file:line citations as reliable, its risk conclusion for `port` specifically as superseded by this correction). - - **Two distinct "heartbeat" concepts** (do not conflate when implementing): (1) the HA/Raft controller-to-controller election heartbeat (`ha->config.heartbeat_interval_ms`, governs failover timing, currently hardcoded, no `ClusterConfig` field, out of scope here per Rule 0 in `config-design.md`); (2) the controller→worker liveness heartbeat (`ClusterConfig.heartbeatInterval`/`inactiveHeartbeatCount`, governs when a controller marks its own worker unavailable, stays in `ClusterConfig`, out of scope here because it's unchanged, not because it's unrelated). - - **`admin-config-tool` dependency**: `xceadmin config push` is soft-blocked on this spec landing (see `admin-config-tool` requirements.md Adjacent Expectations) — not because `xceadmin`'s code depends on it, but because until these fields are removed, nobody has verified the controller preserves them correctly when a pushed config leaves them zero-valued. - - **`MasterCandidate` capnp struct**: verified (grep, 2026-07-20) as referenced only by `ClusterConfig.candidates` and nowhere else in `controller.capnp` — removal is required by Requirement 1.6, not an open design question. - - **Stale citation, verified and dropped**: the prior analysis doc cites `testsuite/test.c:124,266` for an unrelated `get_setting("port")` (scheduler/worker connect-to-primary port) as a "don't touch this" landmine. That call no longer exists in `test.c` as of this spec's discovery pass — the citation has drifted. Two other test files DO reference a removed field directly: `test_commit_barrier_rollout.c:1359` and `test_config_repl.c:983` (see Requirement 5.1). - - **Prior analysis document**: `src/docs/cluster-config-split-analysis.md` (2026-06-08) already scoped this exact change (its "Phase 1"), verified the zero-blast-radius claim, and listed the exact file:line change sites — treat it as a primary design input, not something to re-derive from scratch. Some line numbers in it have drifted; current locations are in this spec's discovery notes. - -## Requirements - -### Requirement 1: Remove Dead Peer/Listener Fields - -**Objective:** As an XCE maintainer, I want `port` and `candidates` removed from `ClusterConfig`, so that the schema no longer implies these are cluster-shared values when the live peer list is already sourced from `--peers`. - -#### Acceptance Criteria - -1. The `ClusterConfig` capnp struct shall not contain a `port` field. -2. The `ClusterConfig` capnp struct shall not contain a `candidates` field. -3. When the controller opens its listener and its name is found in the `--peers` CLI-parsed candidate list, the system shall determine the bind port from that list, exactly as it does today. -4. When the controller builds its internal debug/status config dump, the system shall not attempt to read `candidates` or `ncandidates` from `cluster_config_t` (the field no longer exists). -5. The `cluster_config_t` C struct (`src/include/clusterconfig.h`) shall not contain `port`, `ncandidates`, or a `candidates`/`master_candidate_t**` member, matching the capnp struct. -6. The `MasterCandidate` capnp struct shall be removed from `controller.capnp` (verified: referenced only by `ClusterConfig.candidates`, nowhere else in the schema — its removal is not a design-phase open question, it's a direct consequence of Requirement 1.2's `candidates` removal). -7. Where the controller's name is NOT found in `--peers` (including single-node mode, where `--peers` is not passed at all), the system shall provide a way to determine the controller's listen port that does not depend on a `ClusterConfig` field. This spec does not mandate the specific mechanism (e.g., a new `--port` CLI flag, or requiring single-node mode to pass a self-referencing `--peers` entry) — selecting it is a design-phase decision — but Requirement 1.1's removal of the `port` field SHALL NOT ship without this replacement in place, and single-node controller startup SHALL continue to work exactly as it does today once the replacement lands. - -### Requirement 2: Remove Dead Performance-Tuning Fields - -**Objective:** As an XCE maintainer, I want the already-hardcoded performance-tuning fields removed from `ClusterConfig`, so that the schema doesn't advertise configurability that doesn't exist. - -#### Acceptance Criteria - -1. The `ClusterConfig` capnp struct shall not contain `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, or `xstrings` fields. -2. While these fields are absent from the schema, the controller shall continue to use its existing hardcoded tunable values (`c->tunables.*` defaults in `cluster.c`) with no behavior change from before this spec. -3. When the controller starts, the system shall not attempt to read `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, or `xstrings` from `cluster_config_t` (the fields no longer exist). -4. The `cluster_config_t` C struct (`src/include/clusterconfig.h`) shall not contain `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, or `xstrings` members, matching the capnp struct. -5. The `HashZArrayConfig` and `ResmgrConfig` capnp structs shall be removed from `controller.capnp` (discovered during `/kiro-impl` task 2.1, 2026-07-20: verified via grep that these structs are referenced only by the removed `jobmgr`/`workermgr`/`qmgr`/`xstrings` fields and the removed `resmgr` field respectively, nowhere else in the schema — their removal is a direct consequence of Criterion 2.1's field removal, the same class of cleanup as Requirement 1.6's `MasterCandidate` removal. Left in place, their `$C.mapname(...)` bindings reference C typedefs that Requirement 2.4's struct-member removal deletes, breaking the capnp-generated bindings' compile). - -### Requirement 3: Preserve Cluster-Shared Fields Unchanged - -**Objective:** As an XCE operator, I want the fields that are genuinely cluster-shared to remain exactly as they are today, so that this schema cleanup doesn't regress any existing config-push behavior. - -#### Acceptance Criteria - -1. The `ClusterConfig` capnp struct shall retain `clusterName`, `heartbeatInterval`, `inactiveHeartbeatCount`, `fetchJobInterval`, `netioTimeout`, `resources`, `queues`, `ugroups`, `hgroups`, `hostConfigs`, `hproviders`, `limits`, and `configGeneration` unchanged in type and semantics. Field numbers (`@N`) for these fields ARE expected to change (compacted to a contiguous `0..12` range): Cap'n Proto requires struct ordinals to be a gapless sequence, so removing 8 fields via direct deletion (Requirement 4.2 — not in-place reservation) necessarily renumbers every retained field declared after a removed one. This was verified empirically during `/kiro-impl` (2026-07-20): a schema with an ordinal gap fails to compile (`error: Skipped ordinal @N. Ordinals must be sequential with no holes.`). -2. Because retained fields are renumbered per 3.1, a `cluster.bin` built before this spec is NOT expected to decode its retained fields correctly when pushed to a controller built after this spec — Cap'n Proto's field layout is ordinal-position-based, so renumbering shifts wire byte-offsets for every field after the first removed one, not just the removed fields' own bytes. This is an accepted compatibility break for this pre-1.0 product (see Requirement 4.1). Any `cluster.bin` artifact that predates this spec — including the checked-in `ha/data/cluster.bin` used by `/xce-deploy-ha` and `/xce-test-ha` — must be rebuilt against the new schema before being used with a post-spec controller (Requirement 5.3). -3. The controller→worker liveness behavior driven by `heartbeatInterval`/`inactiveHeartbeatCount` shall be unaffected by this spec. - -### Requirement 4: Compatibility Posture - -**Objective:** As an XCE maintainer, I want the compatibility expectations for this change stated explicitly, so implementation and review don't over- or under-build compatibility handling. - -#### Acceptance Criteria - -1. Because this spec performs direct field removal via ordinal renumbering rather than in-place reservation (4.2), a `cluster.bin` built before this spec is not binary-compatible with a controller built after this spec — not only are the removed fields' values (`port`, `candidates`, `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings`) discarded without error, warning, or migration, but retained-field values will not decode correctly if such a pre-spec blob is pushed as-is, since their wire byte-offsets shifted (see 3.1/3.2). This is accepted given the product has not reached 1.0; no backward-read guarantee is provided for any pre-spec `cluster.bin`, and any such artifact — including the repo's own fixture — must be rebuilt against the new schema (Requirement 5.3). -2. The system shall not implement a deprecate-in-place transition (field-number reservation, dual-read-old-and-new) for the fields removed by this spec — direct removal (with the resulting renumbering of retained fields) is the chosen approach. - -### Requirement 5: Build and Test Verification - -**Objective:** As an XCE maintainer, I want this schema change verified as behavior-preserving, so the "zero blast radius" claim from the prior analysis is proven, not assumed. - -#### Acceptance Criteria - -1. When the controller AND its test suite are built after this spec's changes, the system shall compile without errors using the regenerated capnp bindings. This includes test code that directly constructs a `cluster_config_t` literal and sets a removed field — confirmed present at `test_commit_barrier_rollout.c:1359` (`cfg.port = 9990;`) and `test_config_repl.c:983` (`cfg.port = 9991;`); both shall be updated to drop the now-nonexistent assignment. Any other direct struct-literal reference to a removed field found during implementation shall be updated the same way. -2. When the existing controller unit test suite is run after this spec's changes, the system shall show no new failures attributable to the removed fields. -3. Because field-number renumbering breaks binary compatibility with any pre-spec `cluster.bin` (Requirement 3.2/4.1), the system shall regenerate the checked-in `ha/data/cluster.bin` fixture against the new (post-renumbering) schema, preserving the semantic values of all retained fields from the pre-spec blob, so `/xce-deploy-ha` and `/xce-test-ha` continue to work for this and every future spec that relies on that fixture. -4. When a 3-node HA cluster is deployed and run through a config-push cycle after this spec's changes — using the regenerated `ha/data/cluster.bin` from Criterion 3 — the system shall behave identically to before this spec (peer discovery via `--peers`, listener binding, worker liveness tracking) — verified live, not just by unit tests, given this touches `cluster.c`/`ha.c`/`main.c`. -5. When a single-node controller (no `--peers` flag) is started after this spec's changes, the system shall successfully bind its listener and operate correctly using Requirement 1.7's replacement port-sourcing mechanism — this is the scenario Finding #2 of this spec's review identified as the actual regression risk, and it shall be verified live, not assumed from the multi-node soak in Criterion 4. - -### Requirement 6 (Nice-to-Have): CLI-Configurable Performance Tunables - -**Objective:** As an XCE operator running a high-memory or high-latency controller node, I want the previously-hardcoded hash-table-size tunables to become CLI-configurable, so I can tune an individual controller's performance without needing a schema field that was never wired up anyway. - -#### Acceptance Criteria - -1. Where this nice-to-have is implemented, the system shall accept `--jobmgr-nhash`, `--workermgr-nhash`, `--qmgr-nhash`, `--resmgr-nhash`, `--xstrings-nhash` (and corresponding `-delta`/`-inst` variants as applicable) CLI flags, following the same `OPT_*` + `struct arguments` + `myconf_t` field pattern already used for `--vcpus`. -2. Where a tunable CLI flag is not specified (or is 0), the system shall fall back to the existing hardcoded default for that tunable, with no behavior change from Requirement 2. diff --git a/.kiro/specs/clusterconfig-schema-split/research.md b/.kiro/specs/clusterconfig-schema-split/research.md deleted file mode 100644 index cb43fa91..00000000 --- a/.kiro/specs/clusterconfig-schema-split/research.md +++ /dev/null @@ -1,153 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `clusterconfig-schema-split` -- **Discovery Scope**: Extension (existing system) — light discovery per `design-discovery-light.md` -- **Key Findings**: - - `candidates`, `vcpus`, and the five perf-tuning fields are genuinely dead in `ClusterConfig` at runtime (verified across every `.c` file that touches `cluster_config_t`). - - `port` is NOT dead — its removal (Requirement 1.1) needs a replacement mechanism (Requirement 1.7), which this design resolves as a new `--port` CLI flag, matching the existing `--vcpus` pattern exactly. - - No tooling in this repo (`deploy-ha.sh`, test scripts) currently starts a controller with zero peer-related flags in a way that would be affected either way — but the codebase's own `main.c:294` message ("No --peers flag; running as single-node primary") confirms the zero-`--peers` invocation is an intentional, first-class supported mode, not incidental. The design preserves it exactly rather than forcing a CLI-contract change. - -## Research Log - -### Requirement 1.7's port-sourcing replacement mechanism -- **Context**: `requirements.md` Requirement 1.7 deliberately left the replacement mechanism as an open design question after the requirements-phase review found `cluster_open_listener` (`cluster.c:1216-1258`) depends on `cluster_config->port` as the sole port source for single-node mode (no `--peers` flag passed at all, `main.c:293-294`). -- **Sources Consulted**: `src/controller/main.c` (CLI parsing, `--vcpus` precedent at lines 65-66/114-118/252, single-node branch at 293-295), `src/controller/cluster.c:1216-1258` (port resolution), `src/include/myconf.h` (existing CLI-derived field conventions), `src/controller/ha.h` (hardcoded-default naming convention, `HA_DEFAULT_*`), prior memory of this codebase (`--peers` with a single self-referencing candidate is a confirmed, already-used alternative single-node invocation — but no repo tooling actually relies on the zero-flag form being removed). -- **Findings**: - - No standalone `--port` CLI flag exists today — confirmed via grep (zero hits for `OPT_PORT`/`"port"` in `main.c`). - - `--vcpus` establishes the exact pattern to follow: `OPT_VCPUS` argp option → `struct arguments.vcpus` → validated → copied into `myconf.vcpus` (0 = unset sentinel, documented inline in `myconf.h:37`) → consumed at the point of use with a hardcoded fallback (`cluster.c:1320`, `myapp.c:129`: `(myconf->vcpus > 0) ? myconf->vcpus : 8`). - - `deploy-ha.sh` (the only repo-internal controller launcher found) always passes `--peers`; no repo script exercises the true zero-flag single-node path. This means changing that path's invocation contract would carry no *repo-internal* tooling risk — but it's still a documented, INFO-logged, intentionally-supported mode (`main.c:294`), not dead code, so removing it outright is a real behavior change to a supported feature, not cleanup. - - `xce_parse_peers`'s side effects (CA cert path population, identity string, `main.c:278-292`) only run inside the `if (arguments.peers_str)` branch — the zero-flag path currently skips them. Preserving the zero-flag path (rather than forcing a synthetic self-referencing `--peers` entry) avoids introducing any change to that unrelated cert/identity setup. -- **Implications**: Adding `--port` as a new CLI flag (rather than mandating `--peers` for all invocations) requires slightly more new code but changes zero existing invocation behavior and directly reuses a pattern already reviewed and shipped in this exact file. Given this spec's own history of an overconfident "zero blast radius" claim being wrong once already, the lower-risk option is preferred even at the cost of a few more lines. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| A: `--port` CLI flag (chosen) | New `OPT_PORT` flag, `myconf.port` field, hardcoded fallback constant when unset and not found in `--peers` | Zero invocation-UX change; exact precedent (`--vcpus`) to copy; isolated, low-risk diff | Slightly more new code than B | Selected — see Decision below | -| B: Mandate `--peers` everywhere (self-referencing entry for single-node) | Remove the zero-flag path; single-node must pass `--peers ctrl-1=host:port` | Single source of truth for port; reuses an already-proven 1-peer self-quorum path; less code | Removes a documented, intentionally-supported invocation mode; unverifiable whether anything outside this repo (docs, operator muscle memory) depends on the zero-flag form | Rejected — the removed mode is intentional/documented, not incidental, and this spec's mandate is schema cleanup, not an invocation-contract change | - -## Design Decisions - -### Decision: Add `--port` CLI flag as the replacement for `ClusterConfig.port` -- **Context**: Requirement 1.7 requires a non-`ClusterConfig` port-sourcing mechanism before `port` can be removed from the schema, without breaking single-node controller startup. -- **Alternatives Considered**: (A) new `--port` flag, hardcoded fallback default; (B) mandate `--peers` for all invocations including single-node. -- **Selected Approach**: (A). New `OPT_PORT` argp option → `struct arguments.port` (uint32_t, default 0) → copied to a new `myconf_t.port` field → `cluster_open_listener()`'s fallback branch reads `myconf->port` instead of `cluster_config->port`; if still 0 (flag not passed), falls back to a new hardcoded constant `XCE_CONTROLLER_DEFAULT_PORT = 9990` (matching today's exact `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` value byte-for-byte, so the default behavior is unchanged for anyone relying on it). -- **Rationale**: Exact structural precedent already exists and was already reviewed/shipped for `--vcpus`; zero changes to any existing invocation (scripted or manual); avoids introducing a new port-precedence edge case between two different flags (`--port` only matters when `--peers` doesn't supply a match, so precedence is unambiguous: `--peers` match > `--port` > hardcoded default). -- **Trade-offs**: A few more lines of new code than mandating `--peers` everywhere; two flags (`--peers`, `--port`) both nominally relate to "port" now, though only one (`--peers`) is meaningful once any peer is configured. -- **Follow-up**: None — this is fully specified for the tasks phase. - -## Risks & Mitigations -- **Risk**: A future contributor reintroduces a `ClusterConfig`-sourced default for `port`/`candidates`/hash-tuning "for convenience" during an unrelated change, recreating the original problem. **Mitigation**: `.kiro/steering/config-design.md` now documents the general rule; this spec's removal is the concrete enforcement. -- **Risk**: The checked-in `ha/data/cluster.bin` might carry non-default values for the removed fields, silently masking a real backward-compat issue if the live-soak test (Requirement 5.3) doesn't actually inspect them first. **Mitigation**: Requirement 5.3 (and Testing Strategy below) mandates inspecting or synthesizing non-default legacy field values before relying on that test as proof. -- **Risk**: `test_commit_barrier_rollout.c:1359` / `test_config_repl.c:983` are the only *known* struct-literal hits, but a broader grep pattern might surface more during implementation. **Mitigation**: Requirement 5.1's catch-all clause; the build itself will fail loudly on any missed site (C struct field removal is a compile error, not a silent runtime gap). - -## References -- `src/docs/cluster-config-split-analysis.md` — prior team analysis (2026-06-08), primary design input for the dead-field classification. -- `.kiro/steering/config-design.md` — general shared-vs-local config classification rule this spec enforces. -- `.kiro/specs/admin-config-tool/requirements.md` — downstream consumer whose "per-instance field preservation" risk this spec closes structurally. - -## Design Review (`/kiro-validate-design`, 2026-07-20) - -Initial GO/NO-GO: NO-GO, 3 issues found on a fresh critical pass of the first design draft. All fixed in `design.md` directly: - -1. **Flow diagram dropped the zero-value fallback trigger.** Original diagram branched on "name found" (boolean); actual pre-spec code branches on `my_port == 0`, which also covers "found but that candidate's port field is malformed/zero" (`main.c:289-291` already WARNs on this input). As drafted, a matched-but-zero-port entry would have bound to literal port 0. Fixed: diagram and Service Interface contract now check the port value, not just match presence. -2. **Removing the `-EINVAL` path conflated two different scenarios.** Original design treated "`--peers` never passed" (true single-node) and "`--peers` passed but local name absent" (likely operator typo) identically — both silently falling to `--port`/hardcoded-default. User decision (interactive, during review): keep today's hard error for the second case; only the first gets the new fallback. Rationale: the project's fail-loud convention (`src/controller/CLAUDE.md` process-lifecycle invariant) — a silent wrong-default here would surface later as a confusing "peer unreachable" symptom instead of a clear startup error now. Fixed: System Flows, Service Interface, and Error Handling sections all updated to reflect the split. -3. **Requirement 6 missing from Boundary Commitments.** Present in File Structure Plan/Components/Traceability but not in "This Spec Owns" — mechanical inconsistency. Fixed: one bullet added. - -Re-affirmed GO after fixes — no further review pass required (fixes were direct, not requiring a requirements-phase return). - -## Design Revision (`/kiro-impl` kickoff, 2026-07-20) - -**Reversed the Architecture Pattern Evaluation decision** (Option A → Option B) before implementation started, per direct user instruction at the start of `/kiro-impl`. - -- **Original choice**: Option A, a new `--port` CLI flag + `myconf_t.port` field, mirroring `--vcpus`. -- **Revised choice**: Option B (previously rejected) — no new CLI flag. `cluster_open_listener`'s fallback becomes two-tier: `--peers` candidate match (unchanged) → hardcoded `XCE_CONTROLLER_DEFAULT_PORT` (9990) when `--peers` was never passed at all. Operators who want a non-default single-node port pass a single self-referencing `--peers <name>=<host>:<port>` entry (matching `-n`/`--myname`) — an **already-existing, already-supported** invocation mode (confirmed working: single-node primary via `--peers` omission OR a 1-peer `--peers` self-quorum entry are both live paths today, verified by grepping `cluster_open_listener`'s existing candidate-match loop against `c->myconf->name`). -- **Why the reversal holds up**: the original rejection of Option B was "removes a documented, intentionally-supported invocation mode" — but Option B as now scoped does NOT remove the zero-`--peers` mode; it only asks operators who need a *non-default port* in that mode to use the already-supported 1-peer `--peers` form instead of a brand-new flag. Zero new surface area, zero new plumbing (`main.c` argp wiring, `myconf_t.port` field, `myconf.vcpus`-style wiring all become unnecessary), and it reuses code that already exists and is already exercised. -- **Requirements impact**: none — Requirement 1.7 already listed "requiring single-node mode to pass a self-referencing `--peers` entry" as an explicit example alternative and left mechanism selection to the design phase. -- **Design impact**: `design.md`'s Overview, Goals, Boundary Commitments, Architecture diagram, File Structure Plan, Requirements Traceability, Components and Interfaces, Service Interface contract, and System Flows are all updated in place to reflect the two-tier fallback with no CLI Component for Requirement 1.7 (the CLI Component now exists only for the optional Requirement 6 tunables, reusing the pre-existing `--vcpus` pattern directly rather than a `--port` precedent). -- **Tasks impact**: `tasks.md`'s former Task 2 (`--port` CLI flag) is removed; all subsequent major tasks renumbered down by one. -- **Note**: `xce_parse_peers` also accepts a bare `host:port` candidate (no `name=` prefix), matched via `hostname` when no name is given — but `design.md`/`tasks.md` document the explicit `name=host:port` form as the canonical example for this replacement mechanism (clearer about intent than relying on hostname matching implicitly). - -## Design Revision 2: Ordinal-Gap Discovery (`/kiro-impl` task 1.1, 2026-07-20) - -The task 1.1 implementer, before touching the schema, found and empirically proved a contradiction between Requirement 3.1 (retained fields keep their literal original `@N` field number) and Requirement 4.2 (direct removal, no field-number reservation): Cap'n Proto requires struct ordinals to be a gapless `0..N-1` sequence — verified independently (`capnp compile` on a schema with a gap fails: `error: Skipped ordinal @1. Ordinals must be sequential with no holes.`). Deleting 8 fields from the middle of `ClusterConfig`'s 21 while freezing the 13 survivors' literal ordinals is not something any compiling schema can satisfy — you either renumber the survivors (breaks their wire byte-offsets for anything reading an old blob) or leave inert renamed placeholders at the old ordinals (which is field-number reservation, exactly what 4.2 forbids). - -Neither `requirements.md` nor `design.md` had anticipated this; both were written assuming Cap'n Proto allows field deletion independent of ordinal freezing, which is not the case. - -- **Decision (user, interactive)**: renumber the 13 survivors to a contiguous `0..12` range — literal direct removal, matching Requirement 4.2's exact text and the project's "still pre-release, no wire-compat obligation" posture. Requirement 3.1 amended to drop "field number" from its invariance clause (type/semantics only); Requirement 3.2 and 4.1 amended to state plainly that a `cluster.bin` built before this spec is not expected to decode correctly (not just the removed fields — every retained field after the first removed one shifts byte-offset). -- **Consequence discovered next**: the checked-in `ha/data/cluster.bin` (`deploy-ha.sh:33-36` copies it verbatim as every cluster's seed config) is itself a pre-spec artifact. Post-renumbering, its retained fields (`resources`, `queues`, `hostConfigs`, `hproviders`, `limits`, etc.) would silently decode to wrong bytes — breaking `/xce-deploy-ha` and `/xce-test-ha` for every future spec that depends on that fixture, not just this one's verification task. No existing tool rebuilds this blob (`xceadmin config build` isn't implemented yet — separate, unstarted spec). -- **Decision (user, interactive)**: add a task to regenerate `ha/data/cluster.bin` — decode its current retained-field values under the pre-spec schema, re-encode under the new schema with a small throwaway tool (written and discarded, not shipped), overwrite the fixture in place. New Requirement 5.3; old 5.3 (3-node soak) → 5.4, old 5.4 (single-node) → 5.5. `tasks.md` gained a new task 5.1 (regenerate fixture) ahead of the renumbered 3-node-soak task. -- **Rationale**: given the renumbering choice, the fixture WILL be silently wrong if left untouched — this is not optional cleanup, it's a repo-wide landmine for every HA spec's live verification, self-inflicted by this spec's own schema edit. Regenerating it once, in-scope, is far cheaper than every future spec independently rediscovering "why is deploy-ha producing garbage config." -- **Scope boundary held**: the throwaway encoding tool is explicitly NOT a permanent addition — building a real `cluster.bin` authoring tool is `admin-config-tool`'s job (`xceadmin config build`), not this spec's. This task only needs to reproduce ONE known blob's values under the new layout, not build general tooling. - -## Design Revision 3: Orphaned HashZArrayConfig/ResmgrConfig Structs (`/kiro-impl` task 2.1, 2026-07-20) - -The task 1.1 implementer deliberately left `HashZArrayConfig` and `ResmgrConfig` (the capnp struct types used by the now-removed `jobmgr`/`workermgr`/`qmgr`/`xstrings`/`resmgr` fields) in `controller.capnp`, correctly noting no requirement or task text asked for their removal — they compile fine as unreferenced-but-valid struct definitions on their own. Task 1.2 then removed their corresponding hand-written C typedefs (`hash_zarray_config_t`, `resmgr_config_t`) from `clusterconfig.h`, since nothing in `cluster_config_t` referenced them anymore. The task 2.1 implementer's partial syntax check surfaced the resulting break: capnp's C codegen still emits `HashZArrayConfig`/`ResmgrConfig` encode/decode prototypes referencing `hash_zarray_config_t`/`resmgr_config_t` via their `$C.mapname(...)` bindings — types that no longer exist anywhere in the tree. - -- **Root cause**: this is the same class of gap Requirement 1.6 already closed for `MasterCandidate` (used only by the removed `candidates` field) — just missed for the tunables side during the original requirements pass, since `MasterCandidate` was called out explicitly by name while the tunables' supporting struct types were not. -- **Decision**: extend the same treatment — remove `HashZArrayConfig`/`ResmgrConfig` from `controller.capnp`. Verified via grep (same method as the original `MasterCandidate` verification) that neither struct is referenced anywhere else in the schema. New Requirement 2.5 added (mirrors 1.6's phrasing); design.md's File Structure Plan and Requirements Traceability updated; `tasks.md` gained task 1.3 (Schema Component, same boundary as 1.1) to close this before task 4's full build. -- **Why this wasn't caught in the requirements-phase discovery pass**: the original `cluster-config-split-analysis.md` and this spec's requirements-phase grep swept for the FIELDS being removed, not for supporting/nested struct TYPES those fields' capnp types depended on. A useful pattern for future field-removal specs: after identifying fields to remove, also check whether any of their types are struct types used nowhere else in the schema. - -## Correction: Pre-Spec Single-Node Startup Was Already Broken (discovered during task 5.3's review, 2026-07-20) - -Task 5.3's reviewer, verifying the live single-node scenarios, built and ran the actual pre-spec baseline (commit `94f50ce`, HEAD before this spec's first commit) for comparison — and found single-node startup (`xce_controller` with no `--peers`) was **already exiting with `-EINVAL`**, unconditionally, before this spec began. Root cause: `myapp_run()` (`src/controller/myapp.c`) calls `cluster_open_listener()` before the primary config (`cluster_config`) is loaded from disk, so the pre-spec fallback branch (`cluster_config->port`, only reachable when `cluster_config != NULL`) always hit its `else` branch. Logged as `bug-1457` (unrelated to this spec, not fixed as its own change — task 2.1's rewrite incidentally repaired it, verified live in task 5.3). - -**This invalidates part of this spec's own premise, stated in the Introduction and Requirement 1.7's original correction**: "`cluster_open_listener`... has a live fallback to `cluster_config->port`... the sole port-sourcing mechanism for single-node deployments" and "single-node controller startup SHALL continue to work exactly as it does today." In reality, it did not work today (pre-spec) — for a reason entirely unrelated to `ClusterConfig`'s schema. The spec's actual behavior (task 2.1's hardcoded-default fallback, independent of `cluster_config` load state) is still correct and was still the right thing to build — it just fixed a real bug as a side effect, rather than merely "preserving" already-working behavior as originally framed. Not correcting the historical Introduction/Requirement 1.7 text itself (it accurately reflects what was believed and reviewed at design time); this note is the record of what live verification actually found. - ---- - -## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28) - -### 1. Current State Investigation - -All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles: - -| Spec | Phase | Scope | -|---|---|---| -| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` | -| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller | -| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing | - -**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.** - -**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."* - -### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker - -`admin-config-tool/spec.json`'s `blocked_on` array still lists: -> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation" - -This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity. - -- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists. -- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note). - -### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477 - -`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this. - -- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle). -- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked. - -### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration - -`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here. - -### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence - -Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways. - -### Summary Table - -| # | Finding | Classification | Effort | Risk | Action | -|---|---|---|---|---|---| -| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation | -| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable | -| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None | -| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None | - -### Research Needed (carried forward) -- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only. - -**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap. diff --git a/.kiro/specs/clusterconfig-schema-split/spec.json b/.kiro/specs/clusterconfig-schema-split/spec.json deleted file mode 100644 index a5fceb4d..00000000 --- a/.kiro/specs/clusterconfig-schema-split/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "clusterconfig-schema-split", - "created_at": "2026-07-20T00:29:00Z", - "updated_at": "2026-07-21T01:10:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/clusterconfig-schema-split/tasks.md b/.kiro/specs/clusterconfig-schema-split/tasks.md deleted file mode 100644 index c5edebae..00000000 --- a/.kiro/specs/clusterconfig-schema-split/tasks.md +++ /dev/null @@ -1,109 +0,0 @@ -# Implementation Plan - -- [ ] 1. Schema & Struct Foundation -- [x] 1.1 Remove obsolete fields and the `MasterCandidate` struct from the `ClusterConfig` schema, regenerate bindings - - Remove `port` (@1), `candidates` (@2), `vcpus` (@7), `jobmgr` (@13), `workermgr` (@14), `qmgr` (@15), `resmgr` (@16), `xstrings` (@17) from `ClusterConfig` in `controller.capnp` - - Remove the `MasterCandidate` struct entirely - - Confirm `clusterName`, `heartbeatInterval`, `inactiveHeartbeatCount`, `fetchJobInterval`, `netioTimeout`, `resources`, `queues`, `ugroups`, `hgroups`, `hostConfigs`, `hproviders`, `limits`, `configGeneration` remain unchanged in type and semantics; renumber them to a contiguous `@0..@12` range (Cap'n Proto forbids ordinal gaps — verified empirically, see `research.md`'s "Design Revision 2" — so direct removal of the 8 fields necessarily renumbers every retained field declared after a removed one) - - Regenerate the capnp C bindings via the existing meson codegen step; do not hand-edit generated output - - Observable: capnp compiler succeeds against the edited schema; the generated `ClusterConfig` reader/builder exposes exactly 13 fields, ordinals `@0..@12`, and no `MasterCandidate` type - - _Requirements: 1.1, 1.2, 1.6, 2.1, 3.1, 4.2_ - - _Boundary: Schema Component_ - -- [x] 1.2 Remove the matching members from `cluster_config_t` - - Delete `port`, `ncandidates`, `candidates`/`master_candidate_t**`, `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings` members from `cluster_config_t` in `clusterconfig.h` - - Leave all retained members (matching the schema's surviving 13 fields) untouched - - Observable: `clusterconfig.h`'s field list matches the regenerated schema field-for-field; any remaining reference to a removed member elsewhere in the tree is now a compile error, not a silent gap - - _Requirements: 1.5, 2.3, 2.4_ - - _Depends: 1.1_ - - _Boundary: C Struct Component_ - -- [x] 1.3 Remove the orphaned `HashZArrayConfig`/`ResmgrConfig` capnp structs - - Discovered during task 2.1's partial compile check: these structs (used only by the fields task 1.1 already removed) were deliberately left in `controller.capnp`, but task 1.2 removed their corresponding hand-written C typedefs (`hash_zarray_config_t`, `resmgr_config_t`) — the capnp-generated bindings still reference those now-nonexistent types via `$C.mapname(...)`, breaking compilation - - Remove the `HashZArrayConfig` and `ResmgrConfig` struct definitions entirely from `controller.capnp` (grep-verify first that neither is referenced anywhere else in the schema, same method used for `MasterCandidate` in task 1.1) - - Regenerate the capnp C bindings - - Observable: capnp compiler succeeds; the generated bindings no longer emit any `HashZArrayConfig`/`ResmgrConfig`/`hash_zarray_config_t`/`resmgr_config_t` reference anywhere - - _Requirements: 2.5_ - - _Depends: 1.1, 1.2_ - - _Boundary: Schema Component_ - -- [ ] 2. Core: cluster.c Port Resolution, Debug-Dump & Port-Defaulting Cleanup -- [x] 2.1 Update `cluster_open_listener`'s port-resolution fallback - - Change the fallback branch from reading `cluster_config->port` to a new hardcoded `XCE_CONTROLLER_DEFAULT_PORT` constant (9990), reached only when `--peers` was never passed at all (`peers_cfg.candidates_count == 0`) - - Do NOT add a new CLI flag or `myconf_t` field — the `--peers` candidate-match branch (including its existing single self-referencing `--peers <name>=<host>:<port>` usage for single-node mode) is left completely unchanged; this is the only mechanism an operator uses to get a non-default single-node port - - Preserve the exact precedence and branch structure: a `--peers` match with nonzero port binds to it unchanged; a matched candidate with port == 0 falls through to the hardcoded-default tier exactly as today's tolerated case; `--peers` passed but local name absent still returns `-EINVAL` and refuses to start, unchanged - - Observable: a controller started with no `--peers` binds to 9990; a controller started with a single self-referencing `--peers` entry binds to that entry's port; a controller started with `--peers` not containing the local name still exits with the existing `-EINVAL` startup error - - _Requirements: 1.3, 1.7_ - - _Depends: 1.2_ - - _Boundary: Listener/Port-Resolution Component (cluster.c)_ - -- [x] 2.2 Remove the `candidates` debug-dump mirror - - Delete the `candidates`/`ncandidates` KV-mirror loop from `cluster_build_config` - - Confirm `init_controller_tunables()` itself is left unchanged — the hardcoded tunable constants and their consumption are untouched by this spec - - Observable: `cluster_build_config` compiles without referencing `candidates` or `ncandidates`; the controller's internal debug/status config dump no longer includes those keys - - _Requirements: 1.4, 2.2_ - - _Depends: 1.2_ - - _Boundary: Debug-Dump Component (cluster.c)_ - -- [x] 2.3 Remove the now-dead `ClusterConfig` port defaulting in `myconf.c` and `config_repl.c` - - Delete the `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` call in `myconf.c` - - Delete the equivalent `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` call in `config_repl.c` - - Observable: `myconf.c` and `config_repl.c` compile without referencing `cluster_config_t`'s `port` member - - _Requirements: 1.7_ - - _Depends: 2.1_ - - _Boundary: myconf.c, config_repl.c_ - -- [x] 3. (P) Update struct-literal test fixtures for the shrunk `cluster_config_t` - - Drop the `cfg.port = 9990;` literal at `test_commit_barrier_rollout.c:1359` - - Drop the `cfg.port = 9991;` literal at `test_config_repl.c:983` - - Grep the full tree for any other direct `cluster_config_t` struct-literal reference to a removed field and update each the same way - - Observable: both known test files compile against the shrunk struct with no remaining reference to a removed field; a repo-wide grep for the removed field names against struct-literal assignments returns no further hits - - _Requirements: 5.1_ - - _Depends: 1.2_ - - _Boundary: Test Fixture Updates_ - -- [x] 4. Build the controller and run the full unit test suite - - Build `controller_unit_test` (meson target) with the regenerated capnp bindings and all updated source/test files - - Run the full existing suite and compare against the pre-spec baseline - - Observable: build completes with zero compile errors; `controller_unit_test` passes with no new failures attributable to the removed fields - - _Requirements: 5.1, 5.2_ - - _Depends: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3_ - -- [ ] 5. Validation: Live HA Verification -- [x] 5.1 Regenerate the `ha/data/cluster.bin` fixture against the renumbered schema - - Decode the current (pre-spec) `ha/data/cluster.bin`'s retained-field values under the OLD (pre-renumbering) schema — build against a pre-1.1 checkout or otherwise capture the pre-spec field values before they become unreadable - - Write and run a small throwaway capnp-encoding tool (not part of the shipped codebase — discard it after use) that constructs a `ClusterConfig` with those same retained-field values and serializes it under the NEW (post-task-1.1) schema - - Overwrite `ha/data/cluster.bin` with the newly-encoded blob - - Observable: the regenerated `ha/data/cluster.bin` decodes correctly under the post-spec schema with every retained field matching the pre-spec blob's semantic values; `deploy-ha.sh`'s existing verbatim-copy usage (`deploy-ha.sh:33-36`) requires no changes and keeps working - - _Requirements: 3.2, 4.1, 5.3_ - - _Depends: 4_ - -- [x] 5.2 3-node HA cluster config-push verification - - Deploy a 3-node HA cluster using the existing deploy tooling (which seeds ctrl-1 from the regenerated `ha/data/cluster.bin`) and confirm peer discovery via `--peers`, listener binding, and worker-liveness tracking behave identically to pre-spec - - Worker-liveness tracking (Req 3.3) is verified indirectly: `heartbeatInterval`/`inactiveHeartbeatCount` are retained, unmodified fields, and this spec touched no worker-liveness code path (`cluster.c`'s delay-task scheduling consuming them, e.g. around the `check_worker_timeout_`/liveness-poll call sites) — the observable is that the regenerated config activates cleanly on all 3 nodes with no error from that subsystem, not a live worker daemon exercising a timeout - - Observable: the 3-node cluster reaches steady state with unchanged peer/listener/worker-liveness behavior, using the regenerated fixture with no error or warning - - _Requirements: 3.3, 5.4_ - - _Depends: 4, 5.1_ - -- [x] 5.3 Single-node startup, self-referencing-peers port override, and preserved error-path verification - - Start a controller with no `--peers` flag and confirm it binds its listener to the hardcoded default port (9990) from task 2.1 - - Start a controller with a single self-referencing `--peers <name>=<host>:<port>` entry (own `-n`/`--myname` matching, custom port) and confirm it binds to that port instead of the default — this is the sole replacement mechanism for a non-default single-node port under this spec - - Start a controller with a `--peers` list that does not contain the controller's own name and confirm it still refuses to start with the existing `-EINVAL` error, unchanged from pre-spec behavior - - Observable: the no-peers controller reaches a running state bound to port 9990; the self-referencing-peers controller reaches a running state bound to its specified custom port; the name-absent-from-peers controller exits with the same startup error it produced before this spec - - _Requirements: 5.5_ - - _Depends: 4_ - -- [ ] 6. (Optional — Requirement 6, nice-to-have, deferrable without blocking the rest of this spec) CLI-Configurable Performance Tunables -- [x] 6.1 Add CLI flags for the hash-table tunables - - Add `--jobmgr-nhash`, `--workermgr-nhash`, `--qmgr-nhash`, `--resmgr-nhash`, `--xstrings-nhash` (and corresponding `-delta`/`-inst` variants where the tunable has more than one dimension) following the same `OPT_*` + `struct arguments` + `myconf_t` pattern already used for `--vcpus` - - Observable: `xce_controller --help` lists all new flags; passing one populates the corresponding `myconf_t` field - - _Requirements: 6.1_ - - _Depends: 1.2_ - - _Boundary: CLI Tunables Component (main.c, myconf.h)_ - -- [x] 6.2 Wire the tunables into `init_controller_tunables()` - - Read the new `myconf_t` fields in `init_controller_tunables()`, falling back to the existing hardcoded default for any field left at 0 (unset) - - Observable: passing a tunable flag changes the corresponding controller tunable at startup; omitting all of them reproduces today's exact hardcoded values - - _Requirements: 6.2_ - - _Depends: 6.1_ - - _Boundary: Tunables Component (cluster.c)_ diff --git a/.kiro/specs/component-failover-adoption/design.md b/.kiro/specs/component-failover-adoption/design.md deleted file mode 100644 index 0824cbf9..00000000 --- a/.kiro/specs/component-failover-adoption/design.md +++ /dev/null @@ -1,574 +0,0 @@ -# Design Document: component-failover-adoption - -## Overview - -Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile `libxce` already ships a complete, tested, transparent multi-master mechanism (`XCECLI_FAILOVER_CALL` and friends in `src/lib/client.c`) — the same one `xce_test`'s `masters` setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect. - -**Users**: HA cluster operators who configure multiple controller candidates (`--peers` for scheduler/worker, `backend.peers` YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation `xce_test` already meets. - -**Impact**: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in `controller_connector.c` and `scheduler.c`); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to `libxce`'s failover mechanism itself. - -### Goals -- Scheduler, worker, and apiserver each hand the full configured candidate list to `xce_client_config_t` on the very first connection attempt (Req 1). -- All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2). -- Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3). -- apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4). -- Every connect/reconnect attempt and outcome is logged per-candidate (Req 5). -- apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6). -- Scheduler decision publishing (`xcecli_jobdispatch`/`xcecli_set_pend_reason`) never silently drops a decision on retry-budget exhaustion — it keeps retrying that decision until success or shutdown, so scheduler-local and controller-observed job state cannot permanently diverge (Req 7). - -### Non-Goals -- Changing `xce_test` (it is already the reference-correct client) — its clamp already references `N_MAX_MASTERS` symbolically (`test.c:372-378`), so it auto-absorbs the constant reconciliation below with zero code change. -- Completing or adopting the separate, unused `xce_connect_cluster()`/`xce_reconnect_*` primitive in `cluster.c` — superseded by the mechanism this design adopts; left untouched. -- Wiring CA-cert/TLS trust into the real connect path — `xce_client_config_t` carries no such field today and this feature does not add one; TLS is established the same way it already is. -- Controller-side changes (redirect-reply population, peer authentication) — already shipped by `libxce-failover`. -- Recovering worker job-state lost during a disconnect window — owned by `worker-rejoin-resync`. -- Persisting queued-but-unpublished scheduler decisions across a scheduler process crash/restart (Req 7 AC6) — Req 7's fix keeps a running scheduler process from giving up early on a network retry; it adds no WAL/persistence for the in-memory `decisions`/`jbuf` queues, so a scheduler crash mid-outage still loses whatever was queued, exactly as it does today. - -## Boundary Commitments - -### This Spec Owns -- A new shared libxce helper that converts a validated `xce_cluster_config_t` (parsed candidate list) into a fully-populated `xce_client_config_t`. -- Worker's, scheduler's, and apiserver's integration code that calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` with the full candidate list and registers a rejoin callback. -- apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation). -- apiserver's client-facing fail-fast behavior while its backend connection is degraded. -- Connect/reconnect diagnostic logging in all three components. -- Scheduler's decision-publish retry-until-success-or-shutdown wrapper around `xcecli_jobdispatch`/`xcecli_set_pend_reason` (Req 7) — closes the silent-drop gap found in the 2026-08-10 design validation. -- **Reconciling `N_MAX_MASTERS` (`client.h`) to derive from `MAX_HA_CANDIDATES` (`config.h`)** (Req 1 AC7, scope expanded 2026-08-10 at explicit user direction — supersedes this design's earlier Non-Goal treating the mismatch as pre-existing/out-of-scope): `config.h` becomes the single source of truth for "max configured controller candidates." Confirmed safe by source audit — no wire-protocol array assumes 3 (`masters[]` is never serialized; the controller's own peer list is a dynamic capnp `List(CtrlPeer)`), and `config.h` has no include-cycle risk (it depends only on `<stdint.h>`; `client.h` already transitively sees `MAX_HA_CANDIDATES` via `xceapi.h`/`xce/types.h` → `config.h`). -- **Eliminating the legacy single-host code path in worker, scheduler, and apiserver in favor of a single unified pipeline** (Req 1 AC5, redefined 2026-08-10): today all three synthesize a `nmasters=1` `xce_client_config_t` directly from legacy fields, bypassing `xce_cluster_config_t`/`xce_parse_peers`/`xce_cluster_config_validate`/ConfigConverter entirely (`controller_connector.c:129-160`'s `init_client_config()`, `scheduler.c:613`'s single-host `else` branch, `clib.go:79-101`'s `tryConnect()` direct-CGO-struct path). This spec now closes that parallel path: a single controller is the `candidates_count=1` case of the exact same validated pipeline, not a separately-maintained mechanism. - -### Out of Boundary -- `xce_test` and its `masters` setting (unchanged, reference implementation). -- `libxce`'s failover **mechanism** (`client.c`'s connect/rotate/retry/backoff logic, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified. Distinct from `client.h`'s `N_MAX_MASTERS` **constant**, which this spec does change (above) — the mechanism and the struct-layout constant are different surfaces with different boundaries. -- `xce_connect_cluster()`/`xce_reconnect_*` (unused, untouched). -- Controller-side membership/redirect logic. -- Worker job-state resync content on rejoin (`worker-rejoin-resync`). -- TLS/CA-cert plumbing into the real connect path. - -### Allowed Dependencies -- `src/lib/client.c` / `client_internal.h` / `client_failover.h` (shipped `libxce-failover` mechanism) — read-only dependency, no modification. -- `src/lib/cluster.c` / `xce/cluster.h` (`xce_parse_peers`, `xce_cluster_config_validate`) — existing, unmodified; this spec adds one new function alongside them. -- Existing per-component candidate-list CLI/YAML surfaces (`--peers`, `--ca-cert`, `backend.peers`) — unmodified externally; only how they're consumed changes. - -### Revalidation Triggers -- Any change to `xce_client_config_t`'s layout, or any future change to `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` that reintroduces divergence between them (this spec establishes them as one value — a regression that redefines `N_MAX_MASTERS` independently of `config.h` again should re-trigger review). -- Any change to `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL` retry/classification semantics (`libxce-failover`). -- Any change to `xce_parse_peers`/`xce_cluster_config_validate` syntax or validation rules (apiserver's CGO-based parser would need re-verification). -- `worker-rejoin-resync` landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that. - -## Architecture - -### Existing Architecture Analysis -Today each of the three components independently drives its own `xcecli_ctx_t` lifecycle with `nmasters` hardcoded to `1` per connection attempt, and layers its own outer candidate loop on top: -- **worker**: `controller_connector.c`'s `xcecd_join_cluster` loops candidates itself, but the very first connection (inside `xcecd_start()`) happens before `cc_ctx->cluster_cfg` is even populated (`main.c` assigns it after `xcecd_start()` returns) — the candidate list is silently ignored on startup. -- **scheduler**: `scheduler.c`'s `sched_init` loops candidates correctly at startup, but there is no reconnect logic anywhere afterward. -- **apiserver**: `clib.go`'s `RegisterToBackend` loops candidates once via a hand-written Go parser, only at process startup. - -None of the three ever set `nmasters > 1` on a single `xce_client_config_t`, so the library's own rotation/retry (`client.c`, wrapped by `XCECLI_FAILOVER_CALL`) never actually engages for them. - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph Daemons - Worker[worker xcecd] - Scheduler[xce_scheduler] - Apiserver[apiserver backend package] - end - - subgraph NewSharedCode - Convert[xce_cluster_config_to_client_config] - end - - subgraph ExistingLibxce - Parse[xce_parse_peers] - Validate[xce_cluster_config_validate] - CtxInit[xcecli_ctx_init xcecli_start xcecli_join] - Failover[XCECLI_FAILOVER_CALL rotate retry backoff] - Rejoin[rejoin callback xcecli_set_reconnect_handler] - end - - subgraph Cluster - Primary[Controller Primary] - Standby1[Controller Standby] - end - - Worker --> Parse - Scheduler --> Parse - Apiserver --> Parse - Parse --> Validate - Validate --> Convert - Convert --> CtxInit - CtxInit --> Failover - Failover --> Rejoin - Failover --> Primary - Failover --> Standby1 -``` - -**Architecture Integration**: -- Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem). -- Domain boundaries: `libxce` owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining). -- Existing patterns preserved: the `xce_parse_peers`/`xce_cluster_config_validate`/`xce_client_config_t` split; the CGO bridge pattern already used throughout `backend/`; worker's dedicated connector thread. -- New components: one C helper (candidate-list → client-config conversion, generalized out of `xce_test`'s inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need). -- Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered. - -### Rejoin callback constraint: avoiding self-deadlock on `ac->failover.lock` - -Verified directly against `client_failover.c:341-411`: `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` (a plain, non-recursive `pthread_mutex_t` — `mylock_init` calls `pthread_mutex_init(l, NULL)`, no `PTHREAD_MUTEX_RECURSIVE` attribute) for the **entire** retry loop, and `xcecli_invoke_rejoin()` — which calls the registered rejoin callback — is invoked **synchronously from inside that loop, while the lock is held** (`client_failover.c:397`). Any rejoin callback that itself calls a public `xcecli_*` wrapper on the *same* `xcecli_ctx_t` (e.g. `xcecli_join`, which is itself `XCECLI_FAILOVER_CALL`-wrapped and tries to re-lock the same mutex) **self-deadlocks** — the calling thread blocks forever on its own already-held lock. This was never caught because `xcecli_set_reconnect_handler` has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client. - -This directly constrains every component's rejoin implementation in this spec: -- The callback **registered with the library** (worker's `checking_connection`→`xcecli_hello` path, scheduler's `xcecli_jobdispatch` path, apiserver's write-call path) must do **no I/O and call no `xcecli_*` function** — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return `0` so the interrupted operation resumes normally. -- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks that flag and, if set, issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then. **The flag is cleared only once that `xcecli_join` call itself succeeds, never merely because it was attempted** (Req 2 AC5, gap found in the 2026-08-10 design validation's second pass): `xcecli_join` is itself `XCECLI_FAILOVER_CALL`-wrapped and can exhaust its own retry budget and return `-E_XCE_NOT_READY` just like any other wrapped call. Clearing the flag unconditionally before or regardless of that outcome would leave the component transport-reconnected but not cluster-joined, with nothing left to retry the join until the next full disconnect/reconnect cycle happens to fire the rejoin callback again — an indefinite, silent orphaning, not a bounded delay. On failure the flag stays set, so the component's next naturally-occurring opportunity (worker: next `xcecli_hello` return; scheduler: next decision-publish return, itself now retried indefinitely per Req 7; apiserver: next backend-call wrapper return) re-attempts the join. -- apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic **after** `xcecli_heartbeat` has returned (`XCECLI_FF_CALL` also releases its lock before returning), so it was never at risk. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Shared library | C (gnu17), `src/lib/cluster.c` | New candidate→client-config conversion helper | Extends existing file alongside `xce_parse_peers` | -| Worker / Scheduler | C (gnu17) | Consume the existing `xcecli_ctx_t` mechanism directly | No new dependency | -| apiserver | Go 1.23.4 + CGO | Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard | `sync/atomic`, consistent with existing `heartbeat.go` pattern | - -## File Structure Plan - -### Modified / New Files -``` -src/include/client.h # fix: N_MAX_MASTERS derives from - # config.h's MAX_HA_CANDIDATES (3->4) - # instead of an independent literal (Req 1.7) -src/include/xce/cluster.h # + declare xce_cluster_config_to_client_config() -src/lib/cluster.c # + implement xce_cluster_config_to_client_config() -src/lib/testsuite/test_cluster.c # + unit tests for the new helper -src/lib/testsuite/test_libxce_retry.c # fix: fo_make_client_masters() currently - # hardcodes 3 literal host/port entries then - # copies N_MAX_MASTERS slots -- update to match - # the raised N_MAX_MASTERS, not silently copy - # an uninitialized 4th slot - -src/worker/src/controller_connector.c # rewrite: delete init_client_config()'s - # standalone xce_client_config_t construction - # and xcecd_join_cluster()'s separate - # single-host-fallback branch (both bypassed - # xce_cluster_config_t entirely) -- single - # init+start+join via the one synthesize- - # validate-convert-connect sequence, whether - # the candidate list came from --peers or was - # synthesized as one candidate (Req 1.5); - # register rejoin callback -src/worker/src/main.c # fix: populate cc_ctx->cluster_cfg BEFORE - # xcecd_start() (removes init-order bug); - # fix hardcoded `if (candidates_count > 3)` - # (main.c:414) to reference N_MAX_MASTERS - # symbolically instead of a literal 3, so it - # tracks the raised value automatically -src/worker/include/controller_connector.h # + rejoin callback function declaration - -src/scheduler/src/scheduler.c # rewrite: sched_init uses the new helper + - # single init+start+join; delete inline - # per-candidate loop AND the separate - # "single-host path (original)" else branch - # (scheduler.c:613, bypassed - # xce_cluster_config_t entirely) -- single-host - # is now candidates_count=1 through the same - # sequence (Req 1.5); register rejoin callback; - # sched_publish_dispatch/sched_publish_pend_reason - # gain a retry-until-success-or-shutdown wrapper - # around their xcecli_* call (Req 7) -src/scheduler/src/main.c # fix hardcoded `if (candidates_count > 3)` - # (main.c:314) to reference N_MAX_MASTERS - # symbolically, same as worker's fix above; - # no ordering change needed (already correct) - -src/frontend/apiserver/backend/clib.go # rewrite RegisterToBackend: builds exactly ONE - # xce_cluster_config_t regardless of source - # (config.Peers parsed via xce_parse_peers, or - # a synthesized single candidate from - # config.Address/config.Port when Peers is - # empty -- Req 1.5) then the same - # validate/xce_cluster_config_to_client_config/ - # connect sequence either way; deletes - # tryConnect()'s direct C.xce_client_config_t - # construction and the Go-level per-peer loop - # entirely; cgo preamble gains a static C shim - # (mirrors the existing - # xce_backend_handler/go_push_handler pattern) - # that calls the reconnect.go export, passed to - # xcecli_set_reconnect_handler -src/frontend/apiserver/backend/peers.go # DELETED: hand-written parsePeer() retired in - # favor of the CGO-routed xce_parse_peers (Req 6.1) -src/frontend/apiserver/backend/health.go # NEW: backend-degraded circuit-breaker state -src/frontend/apiserver/backend/reconnect.go # NEW: exported Go rejoin logic (re-sends - # xcecli_join for HostRole_apiserver, clears - # health.go state); called from clib.go's C - # shim AND directly from the heartbeat callback -src/frontend/apiserver/backend/heartbeat.go # MODIFIED: tick callback sets a needs_rejoin - # flag on a failure->success transport - # transition (FF_CALL never sets it itself), - # then on every subsequent tick where - # xcecli_heartbeat succeeds AND needs_rejoin is - # still true (not gated on - # ApiserverBackendHealth.IsDegraded(), which has - # an independent TTL -- Req 2.5) retries - # reconnect.go's rejoin function until it - # actually succeeds -src/frontend/apiserver/utils/config.go # no schema change (Peers/Address/Port already exist) - -src/worker/testsuite/test_e2e_multimaster_join.c # NEW: candidate-list join + reconnect E2E; - # also covers the unified single-host path - # (no --peers) hitting the same code (Req 1.5) -src/scheduler/testsuite/test_e2e_multimaster_join.c # NEW: same, scheduler -src/frontend/apiserver/backend/health_test.go # NEW: circuit-breaker unit tests -src/frontend/apiserver/backend/clib_test.go # + candidate-list parity tests (existing file) -``` - -> `controller/` and `libxce`'s failover mechanism itself are not modified — only consumed. - -## System Flows - -### Startup connect (Req 1) - -```mermaid -sequenceDiagram - participant Daemon - participant Convert as xce_cluster_config_to_client_config - participant Ctx as xcecli_ctx (client.c) - participant CandA as Candidate A (standby) - participant CandB as Candidate B (primary) - - Daemon->>Convert: full candidate list (validated) - Convert-->>Daemon: xce_client_config_t (nmasters=N) - Daemon->>Ctx: xcecli_ctx_init + xcecli_start - Ctx->>CandA: connect - CandA-->>Ctx: connected, not primary - Ctx->>Daemon: xcecli_join (role) - Ctx->>CandA: join request - CandA-->>Ctx: standby redirect - Ctx->>Ctx: rotate to next candidate - Ctx->>CandB: connect + join - CandB-->>Ctx: joined as primary - Ctx-->>Daemon: xcecli_join success -``` - -### Runtime failover (Req 2, Req 3) - -```mermaid -sequenceDiagram - participant Daemon - participant Ctx as xcecli_ctx lock held - participant OldPrimary as Old Primary dies - participant NewPrimary as New Primary - - Daemon->>Ctx: xcecli_* call existing traffic - Ctx->>OldPrimary: send - OldPrimary--xCtx: connection lost - Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off - Ctx->>NewPrimary: reconnect - NewPrimary-->>Ctx: connected - Ctx->>Daemon: invoke rejoin callback lock still held - Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0 - Ctx->>NewPrimary: resume original call - NewPrimary-->>Ctx: reply - Ctx-->>Daemon: original call result, lock released - Note over Daemon: after the call returns, lock is free - Daemon->>Daemon: check and clear needs-rejoin flag - Daemon->>Ctx: xcecli_join role separate call - Ctx->>NewPrimary: join - NewPrimary-->>Ctx: joined - Ctx-->>Daemon: rejoin complete -``` - -Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: the main driver-loop thread, inside `sched_publish_dispatch` immediately after `xcecli_jobdispatch` returns — **not** the allocator thread, see SchedulerControllerClient below; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way. - -### apiserver fail-fast while degraded (Req 4) - -```mermaid -sequenceDiagram - participant Client as API Client - participant Handler as apiserver handler - participant Health as health.go - participant Ctx as xcecli_ctx (gCtx) - - Client->>Handler: HTTP request needing backend - Handler->>Health: is backend degraded? - alt degraded - Health-->>Handler: yes - Handler-->>Client: 503 service unavailable (fast) - else healthy - Health-->>Handler: no - Handler->>Ctx: backend call - alt call exhausts retry budget - Ctx-->>Handler: error - Handler->>Health: mark degraded (with TTL) - Handler-->>Client: 503 service unavailable - else call succeeds - Ctx-->>Handler: result - Handler-->>Client: 200 with result - end - end -``` - -### apiserver idle-period reconnect (Req 2.4) - -```mermaid -sequenceDiagram - participant HB as heartbeat.go tick - participant Ctx as xcecli_ctx (gCtx, FF_CALL) - participant Rejoin as reconnect.go - participant Health as health.go - participant NewPrimary as New Primary - - Note over HB: no client requests in flight - HB->>Ctx: xcecli_heartbeat (fire-and-forget) - Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect - Ctx->>NewPrimary: reconnected transport only - Ctx-->>HB: rc success (transport reconnected) - Note over HB: was failing, now succeeds -> set needs_rejoin=true (FF_CALL never sets it for us) - HB->>HB: needs_rejoin still true? - HB->>Rejoin: trigger rejoin - Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver) - alt join succeeds - NewPrimary-->>Rejoin: joined - Rejoin->>HB: clear needs_rejoin - Rejoin->>Health: MarkHealthy() - else join's own retry budget exhausted - NewPrimary--xRejoin: -E_XCE_NOT_READY - Note over HB: needs_rejoin stays true (independent of Health's TTL-governed state) — next tick retries - end -``` - -This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (`XCECLI_FAILOVER_CALL`, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1-1.3 | Try each candidate in order at startup | libxce (unchanged), ConfigConverter | `xcecli_start`, `xcecli_join` | Startup connect | -| 1.4 | Worker uses full list on FIRST attempt | WorkerControllerConnector | `controller_connector.c` init order fix | Startup connect | -| 1.5 | Single controller is `candidates_count=1` of the one mechanism, not a separate fallback (redefined 2026-08-10, was "unchanged single-host mode") | All three | ConfigConverter (`nmasters=1` case, no branch) — worker's `init_client_config()`/single-host branch, scheduler's single-host `else` branch, and apiserver's `tryConnect()` all deleted | Startup connect | -| 1.6 | Clear startup failure if all candidates fail | All three | existing `xcecli_start`/`xcecli_join` return codes | Startup connect | -| 1.7 | `MAX_HA_CANDIDATES` (`config.h`) is the single source of truth for max candidates; `N_MAX_MASTERS` derives from it | ConfigConverter (clamp becomes defensive, not normally triggered) | `client.h`'s `N_MAX_MASTERS` `#define` | — | -| 2.1-2.3 | Automatic reconnect + retry until success or shutdown | libxce (unchanged), all three's rejoin callbacks | `xcecli_set_reconnect_handler` | Runtime failover | -| 2.4 | apiserver resumes push notifications after reconnect (incl. idle-period recovery) | ApiserverBackendBridge | `reconnect.go`'s actual-rejoin function re-sends `xcecli_join(HostRole_apiserver)`, triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from `heartbeat.go` on idle-period recovery | Runtime failover, apiserver idle-period reconnect | -| 2.5 | Re-join call itself retries on failure, not treated as one-shot | WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge | flag/degraded-state clears only on `xcecli_join` success, not on mere attempt; apiserver's idle-path checks "still degraded" every tick, not just the failure→success edge | Runtime failover, apiserver idle-period reconnect | -| 3.1-3.3 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation, full isolation per AC1), SchedulerControllerClient (bounded, occupancy-dependent buffering via `jbuf`/`decisions` cbufs per AC2 — narrower than worker's guarantee by design, no job lost per Req 7, resolved 2026-08-10) | existing thread structure | Runtime failover | -| 4.1-4.2 | apiserver fails fast to clients while degraded | ApiserverBackendHealth | `health.go` | apiserver fail-fast flow | -| 5.1-5.2 | Per-candidate connect/reconnect logging | All three | log statements at each rotation | Startup connect, Runtime failover | -| 6.1-6.2 | Consistent candidate-list syntax | ApiserverConfigParity | CGO call to `xce_parse_peers`/`xce_cluster_config_validate` | Startup connect | -| 7.1-7.6 | Scheduler decision publish survives retry-budget exhaustion without loss | SchedulerControllerClient | `sched_publish_dispatch`/`sched_publish_pend_reason` retry-until-success-or-shutdown wrapper | Runtime failover | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|--------------|--------|---------------|-------------------|-----------| -| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.7, 6.1 | `xce_cluster_config_validate` (P0) | Service | -| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service | -| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler; decision-publish never silently drops on retry-budget exhaustion | 1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6 | ConfigConverter (P0), `xcecli_*` (P0) | Service | -| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service | -| ApiserverBackendHealth | apiserver (Go) | Fail-fast circuit breaker for backend-down state | 4.1-4.2 | ApiserverBackendBridge (P0, consumes rejoin/error signals) | State | - -### Shared Library - -#### ConfigConverter - -| Field | Detail | -|-------|--------| -| Intent | Turn a validated `xce_cluster_config_t` into a fully-populated `xce_client_config_t`, clamped to `N_MAX_MASTERS` with a WARN on truncation | -| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 1.7, 6.1 | - -**Responsibilities & Constraints** -- Pure, socket-free transformation (mirrors `xce_test`'s existing inline logic at `test.c:360-372`, generalized into one reusable function). -- Does not call `xce_cluster_config_validate()` itself — callers validate first (existing contract, unchanged) and pass a pre-validated config. -- Does not own `from`/`key` (identity/auth) — caller sets those on the output struct after conversion, matching how `xce_test`, worker, and scheduler already source them independently (CLI/env-specific). -- **Handles `nmasters=1` as the ordinary case, not a special one** (Req 1 AC5): a one-candidate `cfg` produces a one-master `out` through the identical copy loop as any other count — no branch distinguishes "single controller" from "multi controller" anywhere in this function. -- **The `N_MAX_MASTERS` clamp is now a defensive safety net, not a normally-triggered path** (Req 1 AC7): since `client.h`'s `N_MAX_MASTERS` now derives from `config.h`'s `MAX_HA_CANDIDATES`, and `xce_cluster_config_validate()` already rejects `candidates_count > MAX_HA_CANDIDATES` before a config ever reaches this function, `cfg->candidates_count > N_MAX_MASTERS` cannot occur via the validated-input contract this function documents as its precondition. The clamp+WARN stays in place as a guard against a caller that skips validation or a future re-divergence of the two constants — it is intentionally not removed, just no longer load-bearing in the normal path. - -**Dependencies** -- Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0) -- Outbound: none (leaf function) - -**Contracts**: Service [x] - -##### Service Interface -```c -/* xce/cluster.h */ -/** - * xce_cluster_config_to_client_config() — convert a validated candidate list - * into an xce_client_config_t's master fields (nmasters/masters/master_ports). - * - * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order. - * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and - * logs a WARN identifying the truncated count (mirrors xce_test's existing - * clamp behavior). - * - * Does NOT set out->from / out->key — caller sets those separately. - * Does NOT validate cfg — caller must have already called - * xce_cluster_config_validate() successfully. - * - * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1. - */ -int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg, - xce_client_config_t* out); -``` -- Preconditions: `cfg` already passed `xce_cluster_config_validate()`. -- Postconditions: `out->nmasters` in `[1, N_MAX_MASTERS]`; `out->masters[i]`/`out->master_ports[i]` populated for `i < out->nmasters`, in the same order as `cfg->candidates[]`. -- Invariants: never writes beyond `N_MAX_MASTERS` entries regardless of `cfg->candidates_count`. - -### Worker - -#### WorkerControllerConnector - -| Field | Detail | -|-------|--------| -| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect | -| Requirements | 1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2 | - -**Responsibilities & Constraints** -- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:239-312`) with: build ONE `xce_cluster_config_t` — from `cc_ctx->cluster_cfg` when `--peers` was given, or synthesized as a single-candidate `xce_cluster_config_t` (`candidates_count=1`, from the existing hardcoded `"localhost":9990` default, `main.c:333-334`) when it was not — run it through `xce_cluster_config_validate()`, then ConfigConverter, then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`. **Deletes `init_client_config()`'s standalone direct-construction of `xce_client_config_t`** (`controller_connector.c:129-160`) and `xcecd_join_cluster()`'s separate "single-host fallback (original path)" branch (`controller_connector.c:297-311`) — both were a wholly parallel mechanism that never touched `xce_cluster_config_t`/`xce_parse_peers`/`xce_cluster_config_validate` at all (confirmed by source audit, 2026-08-10). Per the redefined Req 1 AC5, single-host is not a separately-maintained fallback; it is the `candidates_count=1` input to the exact same synthesize-validate-convert-connect sequence multi-candidate configs use. -- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4) — this fix composes naturally with the single-path unification above: since there is no longer a separate "first connect uses the single-host branch" code path to accidentally fall into, the only remaining ordering requirement is that whichever `xce_cluster_config_t` gets built (real `--peers` list or synthesized single-candidate) is fully assembled before the one synthesize-validate-convert-connect sequence runs, on both the first attempt and any later one. -- Registers a rejoin callback (`xcecli_set_reconnect_handler`) that only sets a `needs_rejoin` flag and returns `0` — per "Rejoin callback constraint" above, it must not call `xcecli_join` (or any `xcecli_*` function) directly, since the library invokes it while `ac->failover.lock` is still held. -- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup. **The flag clears only on that join call's success** (Req 2 AC5): if `xcecli_join` itself returns `-E_XCE_NOT_READY` (its own retry budget exhausted), `needs_rejoin` stays set, so the next `xcecli_hello` tick's return retries the join — `checking_connection()`'s existing periodic cadence is what makes this retry naturally, no new timer needed. -- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from the main-loop `worker_process_update()` and `resource_sync_thread` liveness paths (Req 3.1) — no new thread introduced. - -**Dependencies** -- Inbound: `xcecd_start()` (P0) -- Outbound: ConfigConverter (P0), `xcecli_*` (P0) - -**Contracts**: Service [x] - -**Implementation Notes** -- Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification. -- Validation: implementation must confirm (code read, not test-only) that the connector thread is isolated from the main-loop `worker_process_update()` and the dedicated `resource_sync_thread` — the two recurring liveness paths — flagged as a Validation Hook, not re-verified in this design pass. (The three `*_update` calls previously cited — `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` — are one-shot init calls at `xcecd.c:476-480` on the main thread, not recurring liveness duties; corrected per 2026-08-10 gap re-validation.) -- Risks: deleting the hand-rolled loop must not silently drop the `MAX_RETRY_COUNT`/`RETRY_INTERVAL` bounded-retry-then-`handle_disconnect()` behavior in `checking_connection()` — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous. - -### Scheduler - -#### SchedulerControllerClient - -| Field | Detail | -|-------|--------| -| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect | -| Requirements | 1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6 | - -**Responsibilities & Constraints** -- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:564-624`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call. **Deletes `sched_init`'s "single-host path (original)" `else` branch** (`scheduler.c:613`, confirmed by source audit 2026-08-10 to be a wholly separate direct-construction mechanism when `--peers` is not given) — when no `--peers` is given, `sched_init` now synthesizes a single-candidate `xce_cluster_config_t` (`candidates_count=1`, from `init_config()`'s existing `"localhost":9990` default, `main.c:143-152`) and runs it through the identical `xce_cluster_config_validate()` → ConfigConverter → connect sequence as any real `--peers` list, per the redefined Req 1 AC5: single controller is `candidates_count=1` of the one mechanism, not a parallel fallback. -- `--sim` mode early-returns before `sched_init()` (`main.c:296`), so the peers-parsing block only reaches `sched_init()` for normal-mode operation — the design's assumption that candidate-list wiring always runs before `sched_init()` in the normal-mode path is confirmed correct; only the line numbers shifted. -- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). **Corrected call site** (supersedes an earlier, incorrect claim that this runs on `sched_allocator`): `xcecli_jobdispatch` is called from `sched_publish_dispatch` (`scheduler.c:1257`), which is reached only via the decision-drain loop in `xce_sched_driver_run_normal_mode_round` (`driver.c:765-786`) — a loop that runs on the single main-loop thread (`main.c`'s `while(1)`, `main.c:348-349`), not on `sched_allocator`. `sched_allocator` only pushes decisions onto the `decisions` cbuf (`scheduler.c:336,387`); it never calls `xcecli_jobdispatch` directly. The check-`needs_rejoin`-then-`xcecli_join` call must therefore live in `sched_publish_dispatch`, immediately after its `xcecli_jobdispatch` call returns (lock released) — same thread, same call frame — **not** in `sched_allocator`. **The flag clears only once that `xcecli_join` call succeeds** (Req 2 AC5): on its own `-E_XCE_NOT_READY`, `needs_rejoin` stays set, so the next decision's publish-return retries the join — since Req 7's retry-until-success wrapper means `sched_publish_dispatch` returns (successfully) far more reliably now than before, this composes cleanly rather than needing its own separate retry loop. -- **Corrected thread/backpressure model** (Critical Issue from 2026-08-10 design validation, resolved by narrowing below — supersedes the prior claim that `sched_job_picker` runs "unaffected"): the scheduler's real pipeline is three stages joined by two `cbuf`s: `sched_job_picker` → `jbuf` (capacity 1, `core.c:161` — the scheduler-core determinism invariant, never resizable without re-deriving `scheduler-core`'s Req 5 AC7) → `sched_allocator` → `decisions` (capacity `ndecisions`=5, `core.c:84`) → the main-loop thread's decision-drain loop → `xcecli_jobdispatch`. When that call blocks inside `XCECLI_FAILOVER_CALL`'s retry/backoff (lock held), the main thread stops draining `decisions`. `sched_allocator` can still produce more decisions before its own `cbuf_put(&core.decisions, ...)` blocks, and `sched_job_picker` can still produce jobs into `jbuf` for as long as allocator keeps draining it. So picker and allocator are **buffered, not isolated**: they continue briefly past the point the outage begins, then both stall until connectivity (and dispatch) resumes. **The exact size of that grace window is occupancy-dependent, not a fixed constant**: if `decisions` had Q items queued (1≤Q≤5) at the moment the main thread's pop grabs the decision that turns out to be un-publishable, allocator can push (6-Q) more decisions before it stalls, and picker's total new picks during the window come out to (7-Q) — ranging from 2 (Q=5, `decisions` was already near-full, common under load) up to 6 (Q=1, `decisions` was nearly empty). Earlier text in this section claimed "at least 6" as a guaranteed floor — that was backwards: 6 is the best-case ceiling (`ndecisions`+1), the guaranteed floor is as low as 1-2 depending on `jbuf`'s own occupancy at the same instant. This is a materially different, weaker guarantee than "unaffected," and a materially different, more accurate one than the original "at least 6" claim. This "stall until resumes" description is only actually true once the Req 7 fix below is in place — see that bullet for why, unfixed, the main thread can instead silently unstick itself by dropping a decision rather than genuinely staying stalled. -- **Decision-publish retry-until-success-or-shutdown wrapper (Req 7, gap found in 2026-08-10 design validation)**: `xcecli_jobdispatch`/`xcecli_set_pend_reason` are `XCECLI_FAILOVER_CALL`-wrapped, which retries a *single call* up to a bounded budget (`max_retries`=20, backoff capped at `max_retry_ms`=5000ms — roughly 1-2 minutes total, `client_failover.c:281-282,312-317`) before returning `-E_XCE_NOT_READY`. Today's driver decision-drain loop (`driver.c:783-789`) does not inspect this return code — it logs and advances to the next queued decision regardless of outcome. Since `sched_commit_decision` already mutates scheduler-local job state (`job->allocs`, accounts) *before* the publish attempt (`scheduler.c` `sched_allocator`), a budget-exhausted publish today silently and permanently diverges scheduler-local state from what the controller was told — not a delay, a loss. Fix: `sched_publish_dispatch`/`sched_publish_pend_reason` (`scheduler.c:1257`/`1251`) wrap their `xcecli_*` call in `while (!sched->core.stop) { rc = XCECLI_FAILOVER_CALL(...); if (rc != -E_XCE_NOT_READY) break; WARN(extended-retry, ...); }` — checking `core.stop` **before** each attempt, not only after (a plain `do`/`while` checking the stop condition only at the bottom would let every already-queued decision behind the one currently stuck each burn one full ~100s budget cycle discovering shutdown was requested, compounding to minutes-per-queued-decision instead of a single bounded delay — caught in the 2026-08-10 design validation's third pass). Re-invoking the whole macro on each loop iteration resets its own internal retry state and starts a fresh budget, rather than adding a second retry mechanism. This is a straightforward extension of Req 2 AC3's existing "keep retrying, never permanently give up" language, applied per-decision instead of only at the connection level. **Shutdown interaction**: `stop_scheduler()`'s existing `cbuf_tryput(&core.decisions, NULL)` wake-up (`main.c:163`) only unblocks a thread parked in `cbuf_get(&core.decisions)` — it does nothing for a thread already inside this new retry loop, so the loop's own `core.stop` check is the only shutdown path. With the check moved to the top, only the ONE decision actively in-flight when shutdown is requested can cost up to one full budget cycle (~1-2 minutes); every decision still queued behind it in `decisions` sees `core.stop` already true and returns immediately without attempting a call, so total shutdown latency stays bounded to a single cycle regardless of queue depth — an accepted bounded tradeoff (same posture as scheduler-core's accepted 5s `OpCode_schedAdmin` blocking-wait case) since `libxce`'s retry macro itself is out of this spec's boundary to modify for a faster mid-attempt shutdown check. **Ordering**: decisions remain published strictly in `decisions` cbuf FIFO order (Req 7 AC3) because the drain loop still only advances to `cbuf_get()` the next decision after the current one's publish call returns — a stuck decision blocks the whole loop, it does not get skipped and revisited later. **Rejoin-check interaction**: the existing check-`needs_rejoin` step (Responsibilities bullet above) sits after this whole `while` loop, not after each inner attempt — it observes the loop's final, successful return, exactly the same as it would for a single non-retried call. - -**Dependencies** -- Inbound: `sched_init()` (P0) -- Outbound: ConfigConverter (P0), `xcecli_*` (P0) - -**Contracts**: Service [x] - -**Implementation Notes** -- Integration: preserve the existing `WARN`/`INFO` log lines around candidate parsing (`main.c`) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally). -- **Req 3 AC2 scope note (Critical Issue from 2026-08-10 design validation, resolved 2026-08-10)**: the original merged AC1 wording ("shall continue performing ... allocation/decision processing ... without being blocked") was not met unconditionally by the scheduler — only for a bounded, occupancy-dependent grace window (floor as low as 1-2 jobs, ceiling `ndecisions`+1=6, see the corrected backpressure-model bullet above), after which `sched_allocator` and then `sched_job_picker` do block. `jbuf`'s capacity is a fixed invariant owned by the `scheduler-core` spec (out of this spec's boundary to change) and `decisions`' capacity is an existing config constant; redesigning either is out of scope here. Resolution: Requirement 3's single merged AC1 was split into a worker-only AC1 (unconditional, unchanged guarantee) and a scheduler-only AC2 (bounded — "continues briefly ... then blocks until connectivity is restored, at which point processing resumes automatically ... no job picked or decided during that bounded window is ever lost"), with the former heartbeat AC renumbered to AC3. `requirements.md` deliberately does not hardcode a specific job count — see the corrected backpressure-model bullet above for why that count is occupancy-dependent (floor 1-2, ceiling `ndecisions`+1=6), not a fixed constant an initial draft of this note incorrectly stated as a guaranteed "at least 6." `requirements.md` reflects this split and the corrected framing; the design and its bounded-buffering behavior now match the requirement it's held to, with Requirement 7 supplying the actual load-bearing guarantee (no loss, regardless of how many jobs the buffer absorbs). -- Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting. - -### apiserver - -#### ApiserverBackendBridge - -| Field | Detail | -|-------|--------| -| Intent | Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker | -| Requirements | 1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2 | - -**Responsibilities & Constraints** -- `RegisterToBackend()` (`clib.go`) is rewritten to build exactly ONE C `xce_cluster_config_t` regardless of source, then run it through the single synthesize-validate-convert-connect sequence worker/scheduler use: if `config.Peers` is non-empty, format it into the same `[name=]host:port[,...]` syntax and parse via `xce_parse_peers` (CGO, not the existing hand-written Go `parsePeer`); if empty, synthesize a one-candidate `xce_cluster_config_t` (`candidates_count=1`) directly from `config.Address`/`config.Port` — **either way** the result is validated with `xce_cluster_config_validate`, converted with ConfigConverter, then `xce_backend_init` is called **once** with the resulting config (`nmasters` in `[1, N_MAX_MASTERS]`, not necessarily `>1`). **Deletes `tryConnect()`'s direct `C.xce_client_config_t` construction** (`clib.go:79-101`) and the `for _, peer := range config.Peers { tryConnect(...) }` Go-level loop (`clib.go:116-135`) entirely, along with `parsePeer()` (`backend/peers.go:11`, retired per Req 6.1) — both `tryConnect()`'s Peers-loop call site and its no-Peers `Address`/`Port` call site (`clib.go:138`) were separate mechanisms that never touched `xce_cluster_config_t` at all (confirmed by source audit, 2026-08-10); per the redefined Req 1 AC5, a bare `Address`/`Port` config is `candidates_count=1` of the same mechanism, not a separate fallback. -- `reconnect.go` holds two plain Go functions: (1) a `//export`-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a `needs_rejoin` atomic flag and returns `0` (per "Rejoin callback constraint" above: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join`/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears `ApiserverBackendHealth`'s degraded state on success. -- The flag-setter is **registered with the C library from `clib.go`'s own cgo preamble**, via a small static C shim (`xce_backend_rejoin_handler`, mirroring the existing `xce_backend_handler`/`go_push_handler` pattern already used for push dispatch) passed to `xcecli_set_reconnect_handler` — cgo symbol resolution is per-file (see `wire_dispatch_support.go`'s documented rationale for the identical `go_push_handler` wrapper), so the shim and the registration call must live in the same file as the `extern` declaration, not standalone in `reconnect.go`. -- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper. **The flag clears only once the follow-up `xcecli_join` call succeeds** (Req 2 AC5): if that call itself returns `-E_XCE_NOT_READY`, `needs_rejoin` stays set so the next backend-call wrapper's return retries the join, and `MarkHealthy()` is correspondingly only called on that eventual success — not on the mere attempt. -- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: on a failure→success transport transition, `heartbeat.go`'s tick callback sets the **same `needs_rejoin` atomic flag the write-path's C-side callback sets** — since `XCECLI_FF_CALL` never invokes that callback for this path, `heartbeat.go` sets the Go-side flag itself. On every subsequent tick where `xcecli_heartbeat` succeeds **and `needs_rejoin` is still true** (checked every tick, not only on the transition edge — Req 2 AC5 gap found in the 2026-08-10 design validation's second pass), it calls `reconnect.go`'s rejoin function; the flag (and `ApiserverBackendHealth`'s degraded state) clears only when that rejoin call itself succeeds. **Deliberately not gated on `ApiserverBackendHealth.IsDegraded()`** (a design-validation third-pass correction over an earlier draft of this fix): that state has its own independent TTL-based auto-clear (see ApiserverBackendHealth below) for request-path circuit-breaker purposes — if the TTL expired before the rejoin actually succeeded, gating the retry on "still degraded" would stop retrying the join even though it never succeeded, silently reintroducing the same starvation this fix exists to close. `needs_rejoin` and `ApiserverBackendHealth`'s degraded-until timestamp are two independent pieces of state serving two different purposes (one tracks "have I re-joined yet," the other "should request handlers fail fast right now") and must not be conflated. A transition-edge-only trigger would fire the rejoin attempt exactly once per outage and, if that one attempt's own `xcecli_join` hit its retry budget and failed, never retry — every subsequent tick would see success→success (transport already up) and never re-fire; checking the persistent flag instead of the transient edge makes each tick a natural retry opportunity for as long as apiserver remains un-joined. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code. - -**Dependencies** -- Inbound: `cmd/main.go` (P0) -- Outbound: ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome) - -**Contracts**: Service [x] - -##### Service Interface -```go -// backend/clib.go -// RegisterToBackend initialises the client using the full configured -// candidate list — config.Peers if set, otherwise a single-candidate list -// synthesized from config.Address/config.Port (not a separate code path, -// see Req 1 AC5) — and registers the rejoin callback. Single call — no -// per-peer Go loop. -func RegisterToBackend(config *BackendConfig) error -``` -- Preconditions: `StartNotificationListener` already called (unchanged from today). -- Postconditions: on success, `gCtx.ctx` is joined with `nmasters` reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered. -- Invariants: candidate-list parsing/validation errors surface the same way `xce_parse_peers`/`xce_cluster_config_validate` report them in C (Req 6.2) — not a separately-worded Go error. - -**Implementation Notes** -- Integration: `parsePeer` (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1). -- Integration: the rejoin C shim must be defined in `clib.go`'s cgo preamble (not `reconnect.go`) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise. -- Risks: CGO error surfaces (C `-EINVAL` etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code. - -#### ApiserverBackendHealth - -| Field | Detail | -|-------|--------| -| Intent | Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded | -| Requirements | 4.1, 4.2 | - -**Responsibilities & Constraints** -- New `health.go`: an atomic "degraded-until" timestamp. `IsDegraded() bool` — true if `now < degraded-until`. `MarkDegraded(ttl time.Duration)` — called by backend-call wrapper functions in `clib.go` when a call's returned error indicates retry-budget exhaustion (e.g. `E_XCE_NOT_READY`). `MarkHealthy()` — called by the rejoin callback (`reconnect.go`) on a successful rejoin. -- Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing `gCtx.ctx`-based calls in `clib.go`) check `IsDegraded()` first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch `gCtx`). - -**Dependencies** -- Inbound: ApiserverBackendBridge (sets state on outcomes) -- Outbound: none - -**Contracts**: State [x] - -##### State Management -- State model: single atomic degraded-until timestamp (int64 unix-nano via `sync/atomic`, same pattern as existing `heartbeatIntervalSecs` in `heartbeat.go`). -- Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per `tech.md`). -- Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request. - -**Implementation Notes** -- Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag. -- Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit `MarkHealthy()`. -- **Deliberately independent of `needs_rejoin`** (Req 2 AC5, correction from the 2026-08-10 design validation's third pass): `IsDegraded()`'s TTL-based auto-clear exists so request handlers eventually probe again even without an explicit `MarkHealthy()` — a legitimate circuit-breaker half-open pattern. It must **not** be used as the retry-trigger condition for the heartbeat-idle-path's rejoin attempts (ApiserverBackendBridge above) — the TTL could expire before an actual rejoin succeeds, and gating the rejoin retry on "still degraded" would then silently stop retrying the join while apiserver is still not actually joined. The rejoin-tracking `needs_rejoin` flag has no TTL and clears only on an actual `xcecli_join` success, independent of whatever `ApiserverBackendHealth`'s degraded-until timestamp is doing. - -## Error Handling - -### Error Strategy -No new error taxonomy — reuse the existing controller status codes (`E_XCE_REDIRECT`/`E_XCE_STANDBY`/`E_XCE_NOT_READY`, `errnox.h`) that `client_failover.c` already classifies, and the existing `xce_cluster_config_validate()` `-EINVAL` cases for malformed candidate-list config. - -### Error Categories and Responses -- **Malformed candidate-list config** (Req 1.6, 6.2): `xce_parse_peers`/`xce_cluster_config_validate` failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route). -- **All candidates unreachable at startup** (Req 1.6): `xcecli_start`/`xcecli_join` returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: `RegisterToBackend` returns an error, `cmd/main.go`'s existing handling applies unchanged). -- **Runtime disconnect** (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes. -- **Scheduler decision-publish retry-budget exhaustion** (Req 7): `-E_XCE_NOT_READY` from `xcecli_jobdispatch`/`xcecli_set_pend_reason` is not treated as terminal — the caller re-invokes the same publish call rather than surfacing an error or dropping the decision (see SchedulerControllerClient's retry-until-success-or-shutdown wrapper). This is scheduler-specific: worker's periodic `xcecli_hello` carries no payload to lose on the same error, and apiserver's request/response calls correctly surface `-E_XCE_NOT_READY` to `ApiserverBackendHealth` since there the caller (the original HTTP client) is expected to retry, unlike scheduler's decisions which have no external retry source. -- **Candidate count exceeds `N_MAX_MASTERS`**: ConfigConverter clamps and WARNs (not an error) — matches `xce_test`'s existing behavior exactly. -- **apiserver backend degraded** (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error. - -### Monitoring -Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (`XCECD_LOG` for worker, the scheduler's existing `INFO`/`WARN` macros, `zap` for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path. - -## Testing Strategy - -- **Unit Tests**: - - `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`); a `candidates_count=1` input produces `nmasters=1` through the identical code path as any other count (Req 1.5) — no branch, no special case. - - `N_MAX_MASTERS == MAX_HA_CANDIDATES` (Req 1.7): a `_Static_assert` (or equivalent compile-time check) in `client.h` or a `test_cluster.c` case asserting the two constants stay equal — a regression guard against reintroducing the divergence this design pass reconciled. Also re-run `test_libxce_retry.c`'s `fo_make_client_masters()` fixture with its corrected slot count and confirm it doesn't silently exercise an uninitialized `masters[3]`. - - apiserver `health.go`: degraded→healthy via `MarkHealthy()`, via TTL expiry; concurrent goroutine safety (`health_test.go`). - - apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (`clib_test.go`). - - Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real `ac->failover.lock` held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found. - - Rejoin retry-on-failure (Req 2 AC5, worker/scheduler/apiserver): using a test double for `xcecli_join` that fails N times before succeeding, assert the pending-rejoin indicator (flag or degraded-state) stays set/degraded after each failed attempt and is cleared only on the eventual success; assert the next naturally-occurring opportunity (next `xcecli_hello`/decision-publish/backend-call return, or next heartbeat tick for apiserver's idle path) is what re-attempts it, not a dedicated new retry loop — a regression guard against the transition-edge/clear-before-success versions of this bug found in the 2026-08-10 design validation's second pass. -- **Integration Tests**: - - Scheduler decision-publish redelivery (Req 7): using a test double for `xcecli_jobdispatch`/`xcecli_set_pend_reason` that returns `-E_XCE_NOT_READY` for N calls before succeeding, assert the same decision is retried (not dropped, not skipped) and eventually published; assert decisions queued behind it publish afterward in original order (Req 7 AC1-3); assert setting `sched->core.stop` mid-retry breaks the loop within one budget cycle (Req 7 AC2); assert a WARN-level log line distinguishes extended-retry success from first-attempt success (Req 7 AC4); with multiple decisions queued (`decisions` at or near its 5-item capacity) when `core.stop` is set mid-retry on the first, assert total shutdown latency stays bounded to roughly one budget cycle, not one per queued decision — a regression guard against the compounding-shutdown-latency bug found in the 2026-08-10 design validation's third pass. - - Unified single-host path (Req 1.5): starting each of worker, scheduler, and apiserver with no `--peers`/empty `backend.peers` reaches the identical `xce_cluster_config_validate` → ConfigConverter → connect call sequence as a single-entry `--peers` config — assert via the same log-line signature both paths now share (there is no longer a distinct "single-host mode" log message), not just "it connects." - - Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's `testsuite/`. - - Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI. -- **E2E Tests**: - - Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new `test_e2e_multimaster_join.c` per C component plus an apiserver-side check in the existing HA soak harness. - - apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4). - - apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1). - -## Security Considerations -No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced. diff --git a/.kiro/specs/component-failover-adoption/requirements.md b/.kiro/specs/component-failover-adoption/requirements.md deleted file mode 100644 index 2c53a8f6..00000000 --- a/.kiro/specs/component-failover-adoption/requirements.md +++ /dev/null @@ -1,93 +0,0 @@ -# Requirements Document - -## Project Description (Input) -`component-failover-adoption` — pre-scoped roadmap backlog item (`roadmap.md:64`, recorded 2026-06-16, HARD-dependent on `libxce-failover`, which shipped GO 2026-07-18 — this item is now unblocked). Make the C daemons (scheduler, worker) and the Go apiserver actually exercise libxce's transparent multi-master failover, matching xce_test's `masters`-setting behavior. Two roadmap-defined integration items: -1. Each component must hand libxce the **full master/candidate list**, not a single endpoint (today even the CLI uses only `conf->masters[0]`). -2. Bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons this must be verified **liveness-safe**: it must never stall a thread that also drives heartbeats (the project-wide "never block iomgr/delaymgr" thread-role contract, `ha-async-io.md`). - -Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. `xce_test` is explicitly OUT of scope — it is the already-correct in-spec verification client, not a target. - -### Background (verification findings, 2026-07-19) - -`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates. - -None of the three production daemons match this pattern today; each has a different gap: - -- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:537`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:129-160`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:239-312`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover). -- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:299-324`, `scheduler.c:564-624`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate. -- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only. - -In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead. - -## Boundary Context - -- **In scope**: scheduler, worker, and apiserver each using their *full* configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established. -- **Out of scope**: `xce_test` (already the correct reference client, not a target of this feature); controller-side changes (peer authentication, `validate_host()`, redirect-reply population — already shipped by `libxce-failover`); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate `worker-rejoin-resync` item — this feature restores the *connection*, not job-state content). Reconciling the candidate-count limit (`N_MAX_MASTERS` deriving from `config.h`'s `MAX_HA_CANDIDATES`, Req 1 AC7) is now explicitly **in** scope — see Requirement 1; an earlier draft of this document listed raising it as out of scope, superseded 2026-08-10. -- **Adjacent expectations**: depends on `libxce-failover` (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list. - -## Requirements - -### Requirement 1: Full candidate list used on initial connection -**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to attempt their initial controller connection using the complete configured candidate list, so that startup succeeds even when the first-listed candidate is not the current primary. - -#### Acceptance Criteria -1. When a component is started with more than one controller candidate configured, the component shall attempt connection to each configured candidate, in configured order, until one accepts the join as primary. -2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart. -3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error. -4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects. -5. Where only one candidate is configured — whether from a single-entry `--peers`/`backend.peers` list, or synthesized from the legacy single-host default when no candidate list is given at all — the component shall treat it as a one-element candidate list and drive it through the exact same validate/convert/connect pipeline used for a multi-candidate configuration. A single controller is not a separately-maintained code path or a limit case of "up to 3 controllers" — it is the `candidates_count=1` case of the one HA-candidate mechanism, and observable behavior (log lines, error handling, retry/reconnect semantics) shall be identical in shape to the multi-candidate case, just with one candidate. -6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted. -7. The maximum number of controller candidates a component can accept shall be defined by a single project-wide constant (`MAX_HA_CANDIDATES` in `src/include/config.h`), not independently duplicated or re-defined elsewhere. `N_MAX_MASTERS` (`src/include/client.h`) shall derive its value from `MAX_HA_CANDIDATES` rather than being a separately-maintained constant that can silently diverge from it, as it does today (3 vs. 4). - -### Requirement 2: Automatic reconnect after controller failover -**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons. - -#### Acceptance Criteria -1. When a component's active connection to its controller is lost, the component shall attempt to reconnect using the complete configured candidate list, not only the previously connected candidate. -2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator. -3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again. -4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart. -5. If the application-level re-join call issued after a successful transport-level reconnect itself fails (including its own retry-budget exhaustion), the component shall retry that re-join rather than treat the failure as final; a pending-rejoin indicator shall be cleared only once the re-join call succeeds, not merely once it has been attempted, so a component is never left transport-connected but silently un-joined to the cluster. - -### Requirement 3: Reconnect retry does not block other liveness-critical work -**Objective:** As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive. - -#### Acceptance Criteria -1. While a worker is retrying a controller connection (including any backoff delay between attempts), the worker shall continue performing its other liveness-critical duties (hardware/process monitoring and job-state tracking) without being blocked by the retry attempt or backoff delay. -2. While a scheduler is retrying a controller connection (including any backoff delay between attempts), the scheduler's allocation/decision processing shall continue briefly past the point the retry begins, then block until connectivity is restored, at which point processing shall resume automatically from exactly where it left off. This guarantee is bounded, not indefinite: the scheduler's picker and allocator threads are buffered against a blocked retry only by the depth of its internal `jbuf`/`decisions` queues — a `scheduler-core` determinism invariant (`jbuf` capacity fixed at 1) that this feature does not adjust — so allocation/decision processing does stall for the remainder of an outage once that buffer is exhausted. The exact number of jobs this buffering absorbs before stalling depends on buffer occupancy at the moment the outage begins (as few as 1-2, as many as `ndecisions`+1=6 under today's buffer sizes) — an implementation detail of `scheduler-core`, not a fixed guarantee this requirement makes. What this requirement does guarantee is that no job picked or decided during that bounded window is ever lost, whether it is still buffered when the outage begins or blocks mid-publish — see Requirement 7. -3. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover. - -### Requirement 4: apiserver behavior toward API clients during backend disconnection -**Objective:** As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served. - -#### Acceptance Criteria -1. When apiserver's connection to the controller is down and a reconnect is in progress, apiserver shall respond to API requests that require the backend with a clear service-unavailable style error rather than hanging until a connection is restored. -2. While the backend reconnect is in progress, apiserver shall continue attempting to serve API requests that do not require the backend connection. - -### Requirement 5: Operator-visible connect/reconnect diagnostics -**Objective:** As an HA cluster operator, I want visibility into which controller candidates each component tried and why each attempt failed, so that I can distinguish a real outage from a configuration mistake. - -#### Acceptance Criteria -1. When a component attempts a connection or reconnection to a controller candidate, the component shall log the candidate attempted and the outcome (success, unreachable, or standby-redirect). -2. If a component exhausts all configured candidates during a connection or reconnection attempt, then the component shall log a summary identifying every candidate attempted and the reason each failed. - -### Requirement 6: Consistent candidate-list configuration across components -**Objective:** As an HA cluster operator, I want to configure the controller candidate list the same way for scheduler, worker, and apiserver, so that I do not need to learn or maintain three different configuration formats for the same setting. - -#### Acceptance Criteria -1. The candidate-list configuration accepted by apiserver shall use the same `[name=]host:port[,...]` syntax already accepted by scheduler and worker. -2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration. - -### Requirement 7: Scheduler decision delivery survives extended outages without loss -**Objective:** As an HA cluster operator, I want the scheduler to never silently lose a job dispatch or pend decision because the controller was unreachable longer than a single retry budget, so that scheduler-local job state and controller-observed job state cannot permanently diverge after a failover. - -#### Background -`xcecli_jobdispatch`/`xcecli_set_pend_reason` are wrapped in `XCECLI_FAILOVER_CALL`, which retries a *single call* up to a bounded budget (`max_retries`, default 20; backoff capped at `max_retry_ms`, default 5000ms — roughly 1-2 minutes total) before returning `-E_XCE_NOT_READY` (`client_failover.c:281-282,312-317`). This is a per-call budget, not indefinite retry. Today, the scheduler's decision-drain loop (`driver.c`'s decision-drain loop calling `xce_sched_driver_on_decision`) does not inspect this return code — it logs and moves on to the next queued decision regardless of outcome. Because `sched_commit_decision` already mutates scheduler-local job state (`job->allocs`, accounts) *before* the publish attempt (`scheduler.c` `sched_allocator`), a budget-exhausted publish leaves the scheduler believing a job is dispatched/pending while the controller was never told — a silent, permanent state divergence, not a delay. - -#### Acceptance Criteria -1. When publishing a decision (`xcecli_jobdispatch` or `xcecli_set_pend_reason`) returns `-E_XCE_NOT_READY` (single-call retry budget exhausted) while the scheduler has not been shut down, the scheduler shall retry publishing that same decision rather than discarding it or advancing to the next queued decision. -2. The scheduler shall keep retrying a given decision's publish across repeated retry-budget exhaustion until either the publish succeeds or the scheduler is shut down by an operator, consistent with Requirement 2 AC3's "never permanently give up" guarantee applied at the level of an individual decision, not only the connection. -3. Decisions already queued behind a stuck decision shall not be published out of order — the scheduler's existing decision-ordering guarantee is preserved while a decision is being retried. -4. When a decision publish enters extended retry (one or more budget exhaustions) and when it eventually succeeds, the scheduler shall log this distinctly from an ordinary single-attempt success, so operators can distinguish "delayed but delivered" from "delivered on the first try." -5. This requirement applies to the scheduler's own decision-publishing direction only; it does not extend to worker job-state resync on rejoin (owned by the separate `worker-rejoin-resync` backlog item) or to any controller-side redelivery mechanism. -6. This requirement guarantees delivery only for as long as the scheduler process itself stays running through the outage — decisions held in the in-memory `decisions`/`jbuf` queues (including the one actively being retried) do not survive a scheduler process crash or restart, since no persistence mechanism for queued-but-unpublished decisions exists today or is added by this requirement. A scheduler restart during an outage loses the same in-flight decisions it would have lost before this requirement, and requires a fresh scheduling pass after reconnecting — this requirement closes the silent-drop-while-still-running gap, not a durability gap. diff --git a/.kiro/specs/component-failover-adoption/research.md b/.kiro/specs/component-failover-adoption/research.md deleted file mode 100644 index 1bd8cdc6..00000000 --- a/.kiro/specs/component-failover-adoption/research.md +++ /dev/null @@ -1,170 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `component-failover-adoption` -- **Discovery Scope**: Extension (light discovery — libxce, worker, scheduler, apiserver already exist; this wires three consumers onto an existing library mechanism) -- **Key Findings**: - - `libxce-failover` (shipped, GO 2026-07-18) already built a **complete, transparent, per-call multi-master retry/rotate mechanism** (`XCECLI_FAILOVER_CALL` / `XCECLI_FAILOVER_CALL_COUNT` / `XCECLI_FF_CALL` in `client_internal.h`, wired into every public `xcecli_*` call in `client.c`, including `xcecli_join` and the initial `xcecli_start` connect). It requires only `xce_client_config_t.nmasters > 1` to activate. **No production code sets `nmasters` above 1.** - - A **rejoin hook** (`xcecli_set_reconnect_handler`) exists specifically so a consumer can re-send its join after a transparent reconnect. It has **zero production callers** — only its own setter and a unit test reference it. - - A second, older, **half-built and unused** primitive (`xce_connect_cluster()` / `xce_reconnect_init/step/fini()` in `src/lib/cluster.c`) also exists. Its production connect path (`do_connect()`) is a stub that always returns `NULL` outside unit tests, and it operates on raw `endpoint_t*`, not `xcecli_ctx_t`. No production code calls it. It predates/overlaps `libxce-failover` and is superseded by it. - - This is a genuine "wire it up" feature, not a "build failover" feature — confirmed by the roadmap: `component-failover-adoption` (roadmap.md:64) was already recorded 2026-06-16 with this exact framing, HARD-blocked on `libxce-failover`, now unblocked. - -## Research Log - -### Why do worker/scheduler/apiserver each hand-roll a candidate loop instead of using the library mechanism? -- **Context**: Initial verification (this conversation) found worker, scheduler, and apiserver each independently loop over configured candidates, calling `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` fresh for each candidate with `nmasters` hardcoded to `1` per attempt. -- **Sources Consulted**: `src/lib/client.c`, `src/lib/client_internal.h`, `src/worker/src/controller_connector.c`, `src/scheduler/src/scheduler.c`, `src/frontend/apiserver/backend/clib.go`, `roadmap.md`. -- **Findings**: `xcecli_start` (client.c:125-188) already rotates through ALL configured masters on the very first connect (bug-854 fix, comment at line 136). `xcecli_join` (client.c:291-300) is wrapped in `XCECLI_FAILOVER_CALL`, so it already retries and rotates on standby/redirect/transport-error responses. The three daemons' hand-rolled loops predate this (or were never updated after it landed) and now duplicate — imperfectly — what the library does correctly. -- **Implications**: The fix is subtractive as much as additive: delete the per-daemon candidate loops, populate `nmasters`/`masters[]`/`master_ports[]` fully once, and let the existing library mechanism do the rest. This directly satisfies Requirements 1 and most of Requirement 2 without new retry/rotation logic. - -### Does the library mechanism, once enabled, also cover runtime reconnect (Requirement 2) automatically? -- **Findings**: Yes for any component whose normal operation already calls `xcecli_*` functions periodically or on demand — every one of those calls is wrapped in `XCECLI_FAILOVER_CALL`/`XCECLI_FAILOVER_CALL_COUNT`, so a transport failure on any call transparently triggers rotate+reconnect+retry within that same call. Fire-and-forget calls (`xcecli_nop`, `xcecli_heartbeat`) use `XCECLI_FF_CALL` instead: a bounded single best-effort reconnect attempt, never the full retry budget — this is the library's own liveness-safety design for exactly the concern in Requirement 3. -- **Implications**: Scheduler's `xcecli_jobdispatch` calls (scheduler.c:1259, called from `sched_publish_dispatch` on the main driver-loop thread — see "Thread-affinity" correction below, not `sched_allocator`) and apiserver's per-request backend calls (`gCtx.ctx` in clib.go) get transparent reconnect for free once `nmasters` is set correctly — no new polling/reconnect loop needs to be written for either component. - -### Does anything need a NEW re-join after a transparent reconnect, beyond what the library does automatically? -- **Findings**: The library's rejoin hook exists precisely because reconnecting the transport does not re-establish application-level cluster membership — a new primary does not automatically know a worker/scheduler/apiserver rejoined unless it re-sends its join (role) message. `xcecli_reconnect_cb` "runs on the calling thread" and returns 0 to resume the interrupted op or <0 to treat the rejoin as failed (loop rotates/backs off again). -- **Implications**: All three components must register a rejoin callback that re-issues `xcecli_join(ctx, <role>, ...)` (with worker also re-sending `hostattrs_t`). Without it, a component would reconnect at the transport level but the new primary would not recognize it as joined — likely surfacing as silently-dropped pushes/notifications rather than an obvious error. - -### Thread-affinity of blocking retry calls (Requirement 3) -- **Findings**: Worker already isolates all controller-connection work (`controller_connector_start`, `xcecd_join_cluster`, and the periodic `checking_connection()` health-check) onto one dedicated thread (`start_controller_connector_thread` / `connector_to_controller`), separate from `hardware_monitor`/`system_monitor`/`process_manager` update calls in `xcecd_start()`. This isolation already satisfies Requirement 3 structurally for the worker — it is not something this feature needs to newly build, only preserve while removing the redundant candidate loop. -- **Corrected 2026-08-10**: `xcecli_jobdispatch` (`scheduler.c:1259`) does NOT execute from `sched_allocator`. It is called from `sched_publish_dispatch`, reached only via the decision-drain loop inside `xce_sched_driver_run_normal_mode_round` (`driver.c:765-786`), which runs on the single main-loop thread (`main.c`'s `while(1)`, `main.c:348-349`). `sched_allocator` (`scheduler.c:295-392`) only does `cbuf_get(&core.jbuf, ...)` then `cbuf_put(&core.decisions, ...)` — it never calls `xcecli_jobdispatch` itself. The scheduler's real pipeline is three stages joined by two `cbuf`s: `sched_job_picker` → `jbuf` (capacity 1, `core.c:161`, the scheduler-core determinism invariant) → `sched_allocator` → `decisions` (capacity `ndecisions`=5, `core.c:84`) → the main-loop thread → `xcecli_jobdispatch`. When that call blocks in retry/backoff, the main thread stops draining `decisions`; `sched_allocator` and `sched_job_picker` can still produce a few more jobs before their own `cbuf_put` calls block. So the two threads are **buffered for a bounded, occupancy-dependent grace window, not indefinitely isolated** — the earlier "benign, does not stall the job picker" claim was wrong; both threads do eventually stall for the remainder of an outage. **Corrected 2026-08-10 (second pass)**: the grace window's size depends on how full `decisions` was at the moment of the stall (guaranteed floor as low as 1-2 jobs, ceiling `ndecisions`+1=6) — an earlier version of this note claimed "~6-job" as if it were the guaranteed size; that was the best-case ceiling, not a floor. See design.md's SchedulerControllerClient section for the full derivation, the Req 3 AC1→AC2 scope-narrowing, and Requirement 7 (the actual load-bearing guarantee: no job lost regardless of how many the buffer absorbs). -- apiserver: unlike the C daemons, ALL request-handling goroutines that touch the backend share the same `gCtx.ctx`, serialized by the library's per-context `failover.lock`. Under a real outage this could queue every concurrent API request behind the full retry budget, which is a client-experience problem (Requirement 4), not a cluster-liveness problem. Needs a lightweight fail-fast guard in the Go layer — see Design Decisions. -- **Implications**: No new dedicated thread is required for worker (existing isolation already suffices). Scheduler needs no new thread either, but Req 3 AC1 needs its wording narrowed to match the bounded (not indefinite) buffering the existing `jbuf`/`decisions` cbufs actually provide — see design.md Critical Issue note. apiserver needs a new, small piece of state (not a thread) to short-circuit blocking calls when the backend is known-down. - -### Configuration surface mismatch: `N_MAX_MASTERS` vs `MAX_HA_CANDIDATES` -- **Findings**: `src/include/client.h:28` defines `N_MAX_MASTERS (3)` — the hard cap on `xce_client_config_t.masters[]`. `src/include/config.h:78` defines `MAX_HA_CANDIDATES 4` — the cap on `xce_cluster_config_t.candidates[]` used by `xce_parse_peers`. These are different constants. `xce_test` (`test.c:360-367`) already handles the mismatch correctly: parse up to 4 candidates, then clamp to `N_MAX_MASTERS` (3) with a `WARN` if truncated. -- **Implications**: Any new shared conversion helper must replicate this exact clamp-and-warn behavior (not silently drop entries, not error out) so all components behave identically when 4 candidates are configured. **Scope expanded 2026-08-10 (explicit user direction)**: reconciling the two constants is now IN scope — `N_MAX_MASTERS` is redefined to derive from `MAX_HA_CANDIDATES` (`config.h` becomes the single source of truth, Req 1 AC7), raising it 3→4. Source audit confirmed this is safe: no wire-protocol array assumes 3 (`masters[]` is never serialized), no header-include cycle (`config.h` depends only on `<stdint.h>`), and `xce_test`'s clamp already references `N_MAX_MASTERS` symbolically so it absorbs the raise automatically. Two latent bugs found by this audit that the raise would otherwise silently trip: `src/worker/src/main.c:414` and `src/scheduler/src/main.c:314` both hardcode a literal `> 3` check instead of referencing the macro — need fixing to `> N_MAX_MASTERS` alongside the raise. `src/lib/testsuite/test_libxce_retry.c`'s `fo_make_client_masters()` fixture also hardcodes exactly 3 literal host/port entries then copies `N_MAX_MASTERS` slots — needs a 4th literal added or it silently copies an uninitialized slot post-raise. - -### TLS/CA-cert wiring gap (pre-existing, out of boundary) -- **Findings**: `xce_client_config_t` (the struct actually consumed by `xcecli_ctx_init`/`xcecli_start`, i.e. the real connection path) has no CA-cert field at all — only `nmasters`/`masters`/`master_ports`/`from`/`key`. The `ca_cert_path` that worker/scheduler's `--ca-cert` flag populates on `xce_cluster_config_t` is validated by `xce_cluster_config_validate()` but is only ever read by the orphaned `xce_connect_cluster()` stub, never by the real connect path. -- **Implications**: This is a pre-existing gap unrelated to candidate-list wiring — the real connection already establishes TLS today (single-master) via whatever global SSL context setup exists outside this struct. Not this feature's concern; noted here so it isn't mistaken for something this design should fix. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Adopt existing `xce_client_config_t` / `XCECLI_FAILOVER_CALL` mechanism (chosen) | Populate the full candidate list once per component, register a rejoin callback, delete hand-rolled loops | Already built, tested (`libxce-failover`), and exactly what `xce_test` uses today; minimal new code | None material — this is the reference-verified path | Matches roadmap's own framing of this backlog item | -| Complete and adopt `xce_connect_cluster()`/`xce_reconnect_*` | Finish the stubbed `do_connect()`, wire it under each daemon | N/A | Duplicate mechanism to the one already shipped; raw-`endpoint_t` API doesn't carry the join/request/reply protocol the daemons need — would need its own substantial new integration layer on top | Rejected — would build a second, redundant failover mechanism | -| Build a new per-component reconnect loop | Fix each daemon's existing hand-rolled loop in place, independently | Minimal library changes | Perpetuates 3 divergent implementations (the root problem this feature exists to fix); does not gain the library's already-tested retry/backoff/classification logic | Rejected — directly contradicts the roadmap item's stated goal | - -## Design Decisions - -### Decision: Adopt the existing `xcecli_ctx_t` / `XCECLI_FAILOVER_CALL` mechanism; do not build new failover logic -- **Context**: Requirements 1-3 need multi-candidate connect + runtime reconnect + liveness-safe retry. -- **Alternatives Considered**: see Architecture Pattern Evaluation above. -- **Selected Approach**: Each component populates a fully-formed `xce_client_config_t` (all configured candidates, not just the first) once, calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once, and registers a rejoin callback via `xcecli_set_reconnect_handler`. All subsequent `xcecli_*` calls already retry/rotate/reconnect transparently. -- **Rationale**: The mechanism is shipped, tested, and is the exact one `xce_test`'s `masters` setting already uses correctly. Building anything new would duplicate it. -- **Trade-offs**: None material for worker/scheduler. For apiserver, moving off the current Go-level per-peer loop means the CGO layer becomes responsible for candidate rotation instead of Go — acceptable since it is strictly more capable (transparent runtime reconnect, not just startup). -- **Follow-up**: Confirm during implementation that worker's hardware/process-monitor update calls truly run off the connector thread (Validation Hook, not re-verified byte-for-byte in this research pass). - -### Decision: One new shared C helper converts `xce_cluster_config_t` → `xce_client_config_t` -- **Context**: All three components need to turn a parsed/validated candidate list into the `nmasters`/`masters[]`/`master_ports[]` triple, replicating `xce_test`'s clamp-to-`N_MAX_MASTERS`-with-WARN behavior (see mismatch finding above). Today this logic exists ad hoc in `test.c` only. -- **Alternatives Considered**: (1) Duplicate the clamp/copy logic in worker, scheduler, and apiserver's CGO layer separately. (2) One shared helper in `src/lib/cluster.c` (where `xce_parse_peers`/`xce_cluster_config_validate` already live), declared in `xce/cluster.h`. -- **Selected Approach**: (2) — a single `xce_cluster_config_to_client_config()` helper, reused by worker (C), scheduler (C), and apiserver (via CGO). -- **Rationale**: DRY; the clamp-and-warn behavior must be identical across components (Requirement 6's config-syntax-parity spirit extends naturally to the derived struct, not just the input string); one function is one thing to test. -- **Trade-offs**: None — this is pure generalization of logic that would otherwise be copy-pasted three times. - -### Decision: apiserver calls the existing C `xce_parse_peers`/`xce_cluster_config_validate` via CGO instead of maintaining a second Go parser -- **Context**: Requirement 6 requires apiserver's candidate-list syntax to match scheduler/worker exactly. apiserver currently has its own hand-written Go `parsePeer` function. -- **Alternatives Considered**: (1) Keep/harden the Go parser to match C syntax by inspection. (2) Call the real C parser via CGO (apiserver already links libxce via CGO for everything else). -- **Selected Approach**: (2). -- **Rationale**: Guarantees byte-identical parsing forever (no drift risk), and apiserver already pays the CGO cost for every other backend interaction. -- **Trade-offs**: One more CGO surface, consistent with the existing `backend/` package pattern. - -### Decision: apiserver gets a small Go-side "backend health" circuit breaker for Requirement 4 -- **Context**: Requirement 4 needs apiserver to fail fast toward its own HTTP clients when the backend is down, rather than queue every request behind the library's full retry budget. -- **Alternatives Considered**: (1) Let every request block on the existing per-ctx lock and rely on Gin/client-side timeouts. (2) Track backend health from call outcomes + the rejoin callback, and check it before issuing a blocking backend call. -- **Selected Approach**: (2) — new `backend/health.go`: an atomic "degraded since" state, set when a call's classified outcome is retry-budget-exhaustion, cleared when the rejoin callback fires (successful reconnect+rejoin). Backend-call wrappers in `clib.go` check it first and return a clear error immediately while degraded, instead of entering the blocking call. -- **Rationale**: Small, localized, reuses signals the library already produces (call rc, rejoin callback) rather than adding new polling. -- **Trade-offs**: A short window exists where the flag is stale (a request arrives just as the backend goes down, before any call has failed) — acceptable, matches typical circuit-breaker behavior; the request in flight simply experiences the one blocking retry cycle before the flag is set for subsequent requests. - -### Decision: rejoin callbacks must be lock-free flag-setters, never direct `xcecli_join` calls -- **Context**: A second `/kiro-validate-design` pass, cross-checking the library-invoked rejoin path against `client_failover.c` directly rather than only its header comments, found that `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` — a plain, non-recursive `pthread_mutex_t` (`mylock_init` → `pthread_mutex_init(l, NULL)`) — for its entire retry loop, and `xcecli_invoke_rejoin()` (which calls the registered rejoin callback) runs synchronously *inside* that loop, lock held (`client_failover.c:397`). Every component's original design had its rejoin callback call `xcecli_join(...)` directly — itself `XCECLI_FAILOVER_CALL`-wrapped, so it would try to re-lock the same mutex on the same thread: a guaranteed self-deadlock on the very first real failover. This was never caught because `xcecli_set_reconnect_handler` has zero production callers and the existing unit test's callback never calls back into the client (it's a no-op probe). -- **Alternatives Considered**: (1) Have the callback call `xcecli_join` directly (original design — deadlocks). (2) Expose a new, lock-free/raw join primitive from libxce for rejoin callbacks to use (would require modifying `libxce-failover`, violating this spec's Allowed Dependencies boundary of consuming it read-only). (3) Callback only sets a lightweight flag and returns immediately (no I/O, no lock); the daemon's own calling code checks and clears the flag *after* the triggering call returns (lock free by then) and issues the real `xcecli_join` as a separate call. -- **Selected Approach**: (3), applied uniformly to worker (`checking_connection`), scheduler (`sched_publish_dispatch`, the main-loop thread's actual `xcecli_jobdispatch` call site — not the allocator, corrected 2026-08-10), and apiserver (each backend-call wrapper, after its `C.xcecli_*` call). apiserver's heartbeat-triggered path (Task 4.4) was already structured this way by construction (it acts after `xcecli_heartbeat` returns) and needed no change. -- **Rationale**: Fixes the deadlock without touching `libxce` at all, staying inside this spec's stated boundary; the fix is a small, uniform pattern applied identically across all three components rather than a special case. -- **Trade-offs**: One extra check-and-clear step after each relevant call site, in exchange for correctness; negligible cost. -- **Follow-up**: Worth flagging upstream to `libxce-failover`'s owners that the rejoin-hook contract (`client.h`'s doc comment for `xcecli_reconnect_cb`) doesn't document this lock-held constraint — a future consumer without this spec's context could reintroduce the deadlock. Not fixed here (out of boundary), but the regression-guard unit test added in design.md's Testing Strategy exists specifically because of this gap in the upstream contract's documentation. - -### Decision: apiserver's rejoin registration uses a C shim in `clib.go`, not a standalone `reconnect.go` export -- **Context**: `/kiro-validate-design` review flagged that design.md originally described `reconnect.go` as "a `//export`ed Go function registered via `xcecli_set_reconnect_handler`," implying direct registration. This codebase already documents (`wire_dispatch_support.go:154-159`, explaining why `go_push_handler` needs the `xce_backend_handler` C shim) that cgo symbol resolution is per-file — a `C.<symbol>` reference only resolves in the file holding its own matching `extern` declaration. -- **Alternatives Considered**: (1) Register the exported Go function directly from `reconnect.go`. (2) Add a static C shim + `extern` declaration in `clib.go`'s own cgo preamble (mirroring the proven `xce_backend_handler` pattern) and register that. -- **Selected Approach**: (2). -- **Rationale**: Matches the codebase's own documented, working pattern instead of risking the same resolution failure `xce_backend_handler` was built to avoid. -- **Trade-offs**: None — same amount of code, just correctly placed. -- **Follow-up**: none; resolved by construction in design.md. - -### Decision: apiserver's heartbeat prober explicitly triggers rejoin on recovery -- **Context**: `/kiro-validate-design` review found, by direct inspection of `client_internal.h`/`client_failover.c`, that `XCECLI_FF_CALL` (backing `xcecli_heartbeat`, apiserver's only idle-period liveness probe) never calls `xcecli_invoke_rejoin` — only `XCECLI_FAILOVER_CALL` (the full-retry path used by write calls) does. Left unaddressed, apiserver would repair its TCP connection via the heartbeat but never re-join, so push notifications (Req 2.4) would stay dead during any failover where no write request happens to arrive first. -- **Alternatives Considered**: (1) Switch the periodic prober from `xcecli_heartbeat` (fire-and-forget) to `xcecli_hello` (full retry, rejoin-bearing) — the same mechanism worker's `checking_connection()` already uses. (2) Keep `xcecli_heartbeat` (preserves its existing wire semantics/side effects on the controller) and have the Go-side tick callback explicitly detect a failure-to-success transition and manually invoke the same rejoin logic used elsewhere. -- **Selected Approach**: (2). -- **Rationale**: Preserves `xcecli_heartbeat`'s existing wire behavior (unknown/unverified whether the controller treats the heartbeat opcode differently from hello for apiserver accounting) while closing the gap with a small, self-contained Go-side check — no library changes, no behavior change to any other component. -- **Trade-offs**: A few extra lines in `heartbeat.go`; the rejoin logic now has two call sites (the cgo shim and the heartbeat callback), both invoking the same `reconnect.go` function, so no duplicated join-payload logic. -- **Follow-up**: none; resolved by construction in design.md. - -### Decision: Do not touch `xce_connect_cluster()`/`xce_reconnect_*` -- **Context**: This second, unused, stubbed primitive exists in the same area of the codebase. -- **Selected Approach**: Leave as-is; do not adopt, complete, or remove it in this spec. -- **Rationale**: Out of this feature's boundary — removing dead code is a separate hardening decision, not required to satisfy any requirement here, and touching it risks scope creep into an unrelated cleanup. -- **Follow-up**: Worth a future hardening backlog note (superseded/dead code) — not filed as part of this spec. - ---- - -## Gap Re-Validation (2026-08-10) - -Re-verified all code citations against the current `scheduler-framework` branch (post `scheduler-core` refactor, post worker CLI cleanup commits `9ffb1d5`–`0df1cf5`). - -### Worker (xcecd) - -**CLI changes orthogonal to spec.** The 6 worker commits removed `-c`/`-p`/`-m` flags, config-file loading, `xcecd_conf.c/h`, and `worker.yaml`. Added `-v` verbose logging, fixed `logdir` default and `-h` ordering. None touched `controller_connector.c`/`controller_connector.h` — the failover-relevant code is unchanged. - -**Bug-1510 confirmed still present.** `cc_ctx->cluster_cfg = cluster_cfg` at current `main.c:537` still runs AFTER `xcecd_start()` at `main.c:531`. `controller_connector_start()` at `xcecd.c:515` performs the first join via `init_client_config()` (`controller_connector.c:129-160`, `nmasters=1` at line 144) before `cluster_cfg` is populated. Spec tasks 2.1–2.3 correctly target this. - -**Line drift (minor):** -| spec cite | current | -|-----------|---------| -| `main.c:610` | `main.c:537` | -| `controller_connector.c:127-158` | `controller_connector.c:129-160` | -| `controller_connector.c:237-311` | `controller_connector.c:239-312` | - -**Task 2.3 validation hook stale.** Task says to verify `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` don't run on connector thread. These three are one-shot init calls at `xcecd.c:476-480` — never called again. The actual recurring liveness duties are `worker_process_update()` (main loop, line 577) and `resource_sync_thread` (dedicated thread, `resource_sync.c:293-330`). The architectural property (connector thread isolated) IS correct, but the named evidence is wrong. Recommend rewording to reference main loop/`resource_sync_thread`. - -### Scheduler - -**HA candidate loop untouched.** `scheduler.c:564-624` (was ~462-495 at spec-write). Loop logic unchanged; only line numbers shifted by `scheduler-core` refactor (doc comments, `core.c` extraction, `--sim` CLI additions in `main.c`). - -**No-reconnect claim confirmed.** No `xcecli_set_reconnect_handler` call in any scheduler file. No reconnect path after `sched_init()`. - -**Line drift (minor):** -| spec cite | current | -|-----------|---------| -| `scheduler.c:462-495` | `scheduler.c:564-624` | -| `scheduler.c:1289` (`xcecli_jobdispatch`) | `scheduler.c:1259` | -| `scheduler.c:511,516` (pthread_create) | `scheduler.c:626,632` | -| research.md L22: `scheduler.c:1289` | `scheduler.c:1259` | -| research.md L30: `scheduler.c:511,516` | `scheduler.c:626,632` | - -**New structural note for design:** `main.c` now early-returns for `--sim` mode (line 296), inserted between peers-parsing block and `sched_init()`. Doesn't affect normal-mode path — peers block still runs before `sched_init()` for non-`--sim` — but design should note this guard exists. - -### Apiserver - -Not re-verified in depth (no flagged changes in this area). Light sanity check confirms `RegisterToBackend` still loops `config.Peers` with per-peer `tryConnect`. - -### Requirements.md pre-existing issues - -**Duplicate Background section.** Lines 10-22 and 24-34 are byte-identical — copy-paste artifact at spec-creation time. Should deduplicate. - -### Summary - -All architectural findings and target bugs in the spec remain substantively accurate. Only citation line numbers and task 2.3 wording need refreshing. No design decisions invalidated. Spec is ready for implementation. - -## Risks & Mitigations -- Deleting worker's/scheduler's hand-rolled candidate loops could regress subtle existing behavior (e.g., worker's specific WARN logging on standby redirect) — mitigate by preserving equivalent log statements around the new single-call path, verified in code review. -- apiserver's circuit breaker could mask a fixed backend as still-degraded if the clearing signal (rejoin callback) doesn't fire in some edge case — mitigate with a bounded TTL on the degraded flag (re-probe after a short timeout) rather than relying solely on the rejoin callback. -- `N_MAX_MASTERS` (3) vs `MAX_HA_CANDIDATES` (4) mismatch could silently drop a 4th configured candidate on any component — mitigate by making the shared helper's WARN-and-clamp behavior loud and identical everywhere (already the plan). - -## References -- `src/lib/client.c`, `src/lib/client_internal.h`, `src/include/xce/client_failover.h` — the shipped `libxce-failover` mechanism this feature adopts. -- `src/controller/testsuite/test.c:345-382` — the reference (`xce_test`) usage pattern. -- `.kiro/steering/roadmap.md:64` — `component-failover-adoption` backlog item this spec formalizes. -- `.kiro/steering/ha-async-io.md` — thread-role contract informing Requirement 3. diff --git a/.kiro/specs/component-failover-adoption/spec.json b/.kiro/specs/component-failover-adoption/spec.json deleted file mode 100644 index ef4c98cd..00000000 --- a/.kiro/specs/component-failover-adoption/spec.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "feature_name": "component-failover-adoption", - "created_at": "2026-07-20T04:23:58Z", - "updated_at": "2026-08-15T14:20:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-08-15T14:20:00Z", - "known_limitations": [], - "resolved_limitations": [ - "bug-1548/bug-1549: apiserver never re-registered with a new primary after a fast controller failover. Fixed via tasks.md Phase 6 -- endpoint_t.reconnect_gen counter (netio.h/xcectx.c) bumped on every successful transport reconnect, heartbeat.go's runHeartbeat replaced its failure-to-success transition detector with a per-tick generation comparison. Confirmed via live E2E re-run of task 5.2's exact scenario: rejoin fired 22s after primary kill (12s after new election) with zero write traffic for the preceding 5.5 minutes; a post-recovery diagnostic job submit correctly appeared via GET /xce/v1/jobs.", - "bug-1559: xce-build.sh's scheduler case didn't build/run 4 newer scheduler test binaries. Fixed directly (tooling, not a spec task) -- now runs `meson test --suite scheduler`, exercising all 41 scheduler test binaries.", - "bug-1558: make test-backend broken standalone. Fixed directly (tooling, not a spec task) -- self-derives CGO_CFLAGS/CGO_LDFLAGS via pkg-config, confirmed working with zero manual env setup." - ], - "prior_decision": { - "decision": "NO-GO", - "validated_at": "2026-08-15T13:10:00Z", - "reopened_at": "2026-08-15T13:10:00Z", - "reopen_reason": "component-failover-adoption's own /kiro-validate-impl run found Req 2 AC4 unmet in the common case (fast controller election, apiserver idle) -- a defect inside this spec's own ApiserverBackendBridge/task 4.4 boundary, not legitimately out of scope. Remediated as tasks.md Phase 6 (6.1/6.2) plus two non-blocking build-tooling fixes (bug-1558, bug-1559) found during the same validation pass." - } - } -} diff --git a/.kiro/specs/component-failover-adoption/tasks.md b/.kiro/specs/component-failover-adoption/tasks.md deleted file mode 100644 index 8e39b187..00000000 --- a/.kiro/specs/component-failover-adoption/tasks.md +++ /dev/null @@ -1,205 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: candidate-count constant reconciliation and shared conversion helper -- [x] 1.1 Reconcile the candidate-count limit constant across libxce - - Derive `client.h`'s `N_MAX_MASTERS` from `config.h`'s `MAX_HA_CANDIDATES` instead of an independent literal, so the two constants can never silently diverge again - - Fix the two hardcoded `> 3` candidate-count checks (worker's and scheduler's startup argument validation) to reference the constant symbolically instead of a magic number, so they track the raised value automatically - - Update the client-retry test fixture that currently hardcodes exactly 3 literal host/port entries so it doesn't silently exercise an uninitialized 4th slot once the constant is raised - - Observable completion: the project builds clean with the candidate-master array now sized to 4; the two updated startup checks and the updated test fixture all reference the same symbolic constant, not a literal - - _Requirements: 1.7_ - -- [x] 1.2 Implement the candidate-list-to-client-config conversion helper in libxce - - Add a pure, socket-free helper that converts a validated candidate list into a fully-populated client config, copying candidates in order and leaving identity/auth fields for the caller to set - - A one-candidate input is handled through the identical code path as any other count — no special-cased branch for "single controller" - - The truncate-with-warning safety net for an over-count input stays in place as a defensive guard, even though it can no longer be reached via a properly validated input now that the two limits match - - Observable completion: given a validated candidate list with 1, 3, and 4 candidates, the helper returns a correctly populated client config in each case, exercising the defensive truncation path only when called with an artificially invalid over-count input - - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 1.7_ - - _Depends: 1.1_ - -- [x] 1.3 Unit test the conversion helper and the constant reconciliation - - Cover: candidate order preserved; one-candidate input produces a one-master output with no branch taken; truncation-with-warning exercised directly against the defensive path; `NULL`/empty-candidate-list rejection - - Add a compile-time or unit-level assertion that the two candidate-count constants stay equal, guarding against reintroducing the divergence - - Observable completion: new test cases pass under the existing libxce unit test run, including the constant-equality assertion - - _Requirements: 1.5, 1.7_ - - _Depends: 1.2_ - -- [x] 2. Worker: unified candidate-list connect and transparent reconnect -- [x] 2.1 (P) Replace worker's dual connect mechanisms with the single unified pipeline - - Delete the standalone client-config construction used for worker's default connection and the separate single-host branch used when no candidate list is configured — both bypass the shared candidate-list validation and conversion helper entirely today - - Build exactly one validated candidate list before connecting, whether it came from an operator-supplied peer list or was synthesized as a single candidate from the existing default when none was given, then run it through the shared helper and connect once - - Register a rejoin callback that only sets a flag and returns immediately — it must not call back into the client library itself, since the library invokes it while still holding its own lock - - Reconcile the periodic liveness probe's own bounded give-up behavior (today it tears the worker's connection down for good after a handful of failed probes) with the library's own indefinite retry, so an outage longer than that old bound no longer causes the worker to give up permanently — the library now owns connection recovery - - Observable completion: starting the worker with no candidate list configured and starting it with a single-entry candidate list produce the identical connect log sequence; a three-candidate config where the first is a live standby joins correctly, showing the standby redirect and the successful candidate; a simulated outage longer than the old give-up bound no longer tears the connection down for good; invoking the registered rejoin callback while the client's lock is held returns immediately without blocking and performs no I/O - - _Requirements: 1.1, 1.2, 1.3, 1.5, 2.1, 2.2, 2.3, 5.1, 5.2_ - - _Boundary: WorkerControllerConnector_ - - _Depends: 1.2_ - -- [x] 2.2 Fix worker's candidate-list startup ordering and add rejoin retry-on-failure - - Move the candidate-list assignment to before the connector starts, so the very first connection attempt already sees the full configured list instead of only a synthesized single candidate - - After each periodic liveness probe returns, check whether a rejoin is pending; if so, issue the rejoin call as a separate step and clear the pending indicator only if that call succeeds — on its own failure, leave the indicator set so the next liveness probe retries the rejoin rather than leaving the worker connected but never actually re-registered - - Observable completion: starting the worker with a peer list where the first listed candidate is unreachable succeeds on the very first connect attempt, not only on a later reconnect; a rejoin call that fails once before succeeding is retried on the next liveness-probe cycle rather than abandoned - - _Requirements: 1.4, 2.5_ - - _Depends: 2.1_ - -- [x] 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage - - Confirm the connector thread is isolated from the worker's other recurring liveness duties, and that a blocked connector thread during retry/backoff does not stall them - - Add end-to-end coverage: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join to a surviving candidate without a worker restart - - Add coverage for the case where every configured candidate is unreachable or rejects the join at startup: assert the worker reports a clear failure identifying every candidate it attempted, not a generic error - - Confirm the worker's periodic liveness probe to a connected controller keeps its regular cadence and keeps succeeding even while a separate reconnect is in progress elsewhere, so a stalled reconnect attempt cannot itself look like a missed heartbeat and trigger an unwanted failover - - Observable completion: the new test passes, and the worker's other liveness duties continue observably updating while a connect/reconnect retry is in progress on the connector thread; the all-candidates-unreachable case produces a failure message naming each attempted candidate; the liveness probe's cadence to the connected controller is unaffected by a concurrent reconnect elsewhere - - _Requirements: 1.6, 3.1, 3.3_ - - _Depends: 2.1, 2.2_ - -- [x] 3. Scheduler: unified candidate-list connect, decision-publish redelivery, and runtime reconnect -- [x] 3.1 (P) Replace scheduler's dual connect mechanisms with the single unified pipeline - - Delete the scheduler's inline per-candidate loop and its separate single-host branch used when no candidate list is configured — both bypass the shared candidate-list validation and conversion helper entirely today - - Build exactly one validated candidate list before connecting, whether from an operator-supplied peer list or synthesized as a single candidate from the existing default when none was given, then run it through the shared helper and connect once - - Register a rejoin callback that only sets a flag and returns immediately, for the same self-deadlock reason as worker's - - Observable completion: starting the scheduler with no candidate list and with a single-entry candidate list produce the identical connect log sequence; a three-candidate config where the first is a live standby joins correctly; invoking the registered rejoin callback while the client's lock is held returns immediately without blocking and performs no I/O - - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.7, 2.1, 2.2, 2.3, 5.1, 5.2_ - - _Boundary: SchedulerControllerClient_ - - _Depends: 1.2_ - -- [x] 3.2 Add decision-publish retry-until-success-or-shutdown, composed with rejoin retry-on-failure - - Wrap the scheduler's per-decision publish call so that a retry-budget-exhausted outcome is retried rather than treated as final and the decision discarded — checking whether the scheduler is shutting down before each attempt, not only after, so decisions still queued behind the one actively retrying do not each burn a full retry cycle just to discover shutdown was requested - - After a publish finally succeeds, check whether a rejoin is pending; if so, issue it as a separate step and clear the pending indicator only on that call's success - - Log an extended-retry success distinctly from an ordinary first-attempt success - - Observable completion: a publish call that fails repeatedly before succeeding results in the same decision retried — never dropped, never skipped to the next queued one; setting the shutdown flag mid-retry breaks the loop within one retry cycle - - _Requirements: 7.1, 7.2, 7.3, 7.4, 2.5_ - - _Depends: 3.1_ - -- [x] 3.3 Validate scheduler allocation/decision backpressure behavior and add connect/reconnect test coverage - - Confirm the bounded, occupancy-dependent buffering behavior: the scheduler's allocation/decision processing continues briefly past the point a retry begins, then blocks until connectivity is restored and resumes automatically — assert the worst-case (least-slack) occupancy scenario explicitly, not only the most-generous one, since that is the guarantee actually being made - - Add end-to-end coverage: scheduler starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join without a scheduler restart - - Add coverage for the case where every configured candidate is unreachable or rejects the join at startup: assert the scheduler reports a clear failure identifying every candidate it attempted - - Confirm by code inspection that the scheduler makes no periodic liveness/heartbeat call of its own to a connected controller — the reconnect-retries-don't-delay-heartbeat-responses guarantee is satisfied by construction here, not by a dedicated runtime assertion, since there is no such response to delay - - Observable completion: the new test passes, showing allocation/decision processing continuing briefly during a simulated reconnect delay, then stalling for the remainder of the outage, then resuming automatically once the connection recovers; the all-candidates-unreachable case produces a failure message naming each attempted candidate - - _Requirements: 1.6, 3.2, 3.3_ - - _Depends: 3.1_ - -- [x] 3.4 Add scheduler decision-redelivery test coverage - - Assert decisions already queued behind a stuck one publish afterward in their original order, not out of order and not dropped - - With several decisions queued when scheduler shutdown fires mid-retry on the first, assert total shutdown latency stays bounded to roughly one retry cycle rather than accumulating one cycle per queued decision - - Confirm this fix is scoped to the scheduler's own decision-publishing direction only — it does not touch worker job-state resync and adds no persistence for queued-but-unpublished decisions, so a scheduler process restart mid-outage still loses whatever was queued, same as before this work - - Observable completion: all decision-redelivery acceptance criteria are exercised by passing tests, including the bounded-shutdown-latency case and a restart-loses-in-flight-state confirmation - - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 2.5_ - - _Depends: 3.2_ - -- [x] 4. apiserver: unified candidate-list connect via CGO, config parity, rejoin, and fail-fast -- [x] 4.1 (P) Route apiserver's candidate-list parsing through the real C parser - - Replace the hand-written Go candidate-list parser with the shared C parser used by worker and scheduler, retiring the Go parser and its call sites entirely - - Map validation errors to clear Go error messages naming the malformed entry - - Observable completion: the same malformed-candidate-string cases worker/scheduler already reject are rejected identically when passed through apiserver's config path - - _Requirements: 6.1, 6.2_ - - _Boundary: ApiserverBackendBridge_ - - _Depends: 1.2_ - -- [x] 4.2 Replace apiserver's dual connect mechanisms with the single unified pipeline - - Delete apiserver's per-peer Go connect loop and its separate direct-construction path used when no peer list is configured — both bypass the shared candidate-list validation and conversion helper entirely today - - Build exactly one validated candidate list before connecting, whether from the configured peer list (via Task 4.1's parser) or synthesized as a single candidate from the legacy address/port fields when no peer list was given, then run it through the shared helper (via CGO) and connect once - - Observable completion: starting apiserver with no peer list and with a single-entry peer list produce the identical connect log sequence; a three-peer config where the first is a live standby registers against the actual primary in one call - - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.7, 5.1, 5.2_ - - _Depends: 4.1_ - -- [x] 4.3 Add apiserver's rejoin logic with a correctly-placed cgo registration and retry-on-failure - - Add a flag-setter registered as the library's rejoin callback that does no I/O, only sets a pending-rejoin flag and returns immediately, and a separate function that performs the actual rejoin call - - Register the flag-setter's C shim in the same file as its `extern` declaration (cgo symbol resolution is per-file) - - Add one shared helper wrapping every backend call: after the call returns, check the pending-rejoin flag; if set, issue the rejoin as a separate step, clearing the flag only if that call succeeds — on its own failure, leave it set so the next backend call's return retries the rejoin - - Observable completion: after killing and restoring connectivity while a write request is in flight, apiserver resumes receiving backend push notifications without a process restart; a rejoin call that fails once before succeeding is retried on the next backend call rather than abandoned; invoking the registered flag-setter while the client's lock is held returns immediately without blocking and performs no I/O - - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_ - - _Depends: 4.2_ - -- [x] 4.4 Trigger apiserver's rejoin from the idle-period heartbeat path - - On a transport failure-to-success transition detected by the heartbeat prober, set the same pending-rejoin flag the write path uses (the heartbeat's own probe mechanism never sets it itself) - - On every subsequent heartbeat tick where the probe succeeds and a rejoin is still pending, retry the rejoin — checked every tick, not only on the transition edge, so a rejoin attempt that itself fails still gets retried on a later tick - - Do not gate this retry on the separate backend-degraded circuit-breaker state, since that state clears itself on its own timer independent of whether a rejoin actually succeeded - - Observable completion: killing the primary while apiserver has no in-flight write request still results in apiserver rejoining and resuming push notifications; a rejoin that fails on the first post-recovery tick before succeeding on a later tick confirms the retry, not just the single-attempt case - - _Requirements: 2.4, 2.5_ - - _Depends: 4.3_ - -- [x] 4.5 Add apiserver's backend-health circuit breaker - - Add a degraded-until state that lets request handlers fail fast instead of blocking on the full retry budget when the backend is known-degraded, clearing on either a successful rejoin or its own timeout - - Wire it into the shared backend-call helper: check before every call, mark degraded when a call's outcome indicates retry-budget exhaustion, mark healthy on a successful rejoin from either the write path or the idle-period path - - Keep this state's own timeout independent from the pending-rejoin flag — the circuit breaker may reset optimistically on its timer, but that must never suppress the pending-rejoin flag's own retry-until-actually-joined behavior - - Observable completion: with the state forced degraded, a wrapped backend call returns immediately with a clear unavailable error instead of invoking the backend call - - _Requirements: 4.1, 4.2_ - - _Depends: 4.4_ - -- [x] 4.6 apiserver unit test coverage - - Cover: degraded-to-healthy transitions via both an explicit successful rejoin and timeout expiry; concurrent-request safety; candidate-list parity cases from Task 4.1 - - Observable completion: the full apiserver test suite passes with the new cases included - - _Requirements: 1.5, 4.1, 4.2, 6.1, 6.2_ - - _Depends: 4.5_ - -- [x] 5. Integration and cross-component validation -- [x] 5.1 E2E: full-cluster reconnect after primary failover - - Extend the project's existing HA soak pattern to a scenario with worker, scheduler, and apiserver all configured with the full candidate list; kill the primary mid-session with a write request in flight - - Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted - - Observable completion: the soak scenario passes with all three components showing a successful re-join to the new primary in their logs and no manual intervention - - _Requirements: 2.1, 2.2, 2.3, 2.4_ - - _Depends: 2.3, 3.3, 3.4, 4.6_ - -- [x] 5.2 E2E: apiserver reconnects during an idle period - - Kill the primary while apiserver has only its heartbeat prober active, no pending write requests; assert apiserver still rejoins and resumes push notifications via the heartbeat-triggered path - - Observable completion: the scenario passes without requiring any client-triggered write request during the outage window - - _Requirements: 2.4, 2.5_ - - _Depends: 4.6_ - -- [x] 5.3 E2E: apiserver fails fast while backend is degraded, and keeps serving requests that don't need it - - With apiserver's backend connection killed and not yet reconnected, issue an API request that requires the backend and assert it returns a clear service-unavailable error within a bounded, sub-second time — not after the full retry budget - - In the same degraded window, issue a request that does not require the backend connection and assert it still succeeds normally, confirming degradation is scoped to backend-dependent requests only - - Observable completion: both timed assertions pass against the same live apiserver instance while it remains degraded - - _Requirements: 4.1, 4.2_ - - _Depends: 4.6_ - -- [x] 5.4 Regression: unified single-candidate path is observably correct across all three components - - Run the full worker, scheduler, and apiserver test suites, plus the startup path for each component with no candidate list configured - - Confirm each component still starts and operates normally with no candidate list configured, and that this now takes the identical code path as a validated single-entry candidate list rather than a separately-maintained fallback - - Observable completion: all existing tests pass unchanged, and each component's no-candidate-list startup log sequence matches its single-entry-candidate-list startup log sequence - - _Requirements: 1.5_ - - _Depends: 5.1, 5.2, 5.3_ - -## 6. Post-validation remediation (bug-1548/bug-1549, NO-GO 2026-08-15) - -- [x] 6.1 Fix apiserver's idle-period rejoin detection gap - - Add a monotonic reconnect-generation counter to the client library's endpoint struct, incremented on every successful transport reconnect regardless of trigger path - - Replace the heartbeat prober's failure-to-success transition check with a per-tick comparison against this generation counter, triggering the same pending-rejoin flag on any change - - Observable completion: a new regression test proves the pending-rejoin flag is set on a generation change even when every heartbeat tick the Go loop observes reports success (the exact shape XCECLI_FF_CALL's silent inline reconnect produces) — its own doc comment states this proves detector logic, not reachability - - _Requirements: 2.4, 2.5_ - - _Depends: 4.4_ - -- [x] 6.2 E2E re-validation: re-run task 5.2's idle-period scenario against the fix - - Re-run the exact task 5.2 scenario (apiserver only, no worker/scheduler, 3-node controller cluster, default heartbeat interval, primary killed with zero write traffic during the observation window) against the fixed code - - Observable completion: identical to task 5.2's own bar — rejoin log lines appear within the observation window, and GET /xce/v1/jobs populates correctly after a post-recovery submit, with no client-triggered write request required - - _Requirements: 2.4, 2.5_ - - _Depends: 6.1_ - -## Implementation Notes (Phase 6) -- Task 6.1: fix is a monotonic `endpoint_t.reconnect_gen` counter (`src/include/netio.h`), bumped once, unconditionally, on the success path of `xce_ctx_reconnect()` (`src/lib/xcectx.c`) — confirmed the sole reconnect call-site funnel for both the write-path retry loop and the heartbeat fire-and-forget path, so it observes every reconnect regardless of trigger. Read from Go via the existing, previously-unused `xcecli_get_my_endpoint()` accessor (`client.h`) — no new C getter needed, and none of the frozen mechanism files (`client.c`/`client_internal.h`/`client_failover.c`/`client_failover.h`) were touched. `heartbeat.go`'s `runHeartbeat()` fully replaced its `prevFailed` failure-to-success transition detector with a per-tick generation comparison (user's explicit direction: no compatibility shim needed, pick the objectively better approach) — the old detector is logically a strict subset of the new one under this codebase's own documented state machine (`ctx->state == XCE_CTX_ERR` is the single known-dead signal gating `XCECLI_FF_CALL`'s reconnect branch), and the new one additionally catches a reconnect triggered by a concurrent write-path call between heartbeat ticks. New regression tests (`heartbeat_test.go`, `heartbeat_rejoin_test.go`) prove the detector fires on a generation change even when every observed tick reports success — the literal shape of bug-1548 that the old detector's own green test suite could never catch. -- Task 6.2: live E2E re-run of task 5.2's exact scenario (apiserver alone, no worker/scheduler, 3-node cluster, default 30s heartbeat interval, primary killed, zero write traffic during the observation window) — **PASS**, closing the gap. Primary killed 14:13:46 UTC; new primary elected 14:13:56 UTC (10s); `checkAndPerformRejoin: rejoin after transparent reconnect succeeded` logged at 14:14:08 UTC (22s after the kill, 12s after the election), with zero write/backend traffic in the preceding 5.5 minutes — the heartbeat path alone triggered it, exactly the case that previously showed zero rejoin evidence across a 268s+ window. bug-1549 downstream symptom also confirmed resolved: a diagnostic job submitted after recovery correctly appeared via `GET /xce/v1/jobs` (count:1, matching job data), proving apiserver's push/registration state genuinely recovered with the new primary, not just the transport. -- Non-blocking tooling fixes made alongside (not spec tasks — `xce-build.sh`/the frontend `Makefile` are general repo tooling, not spec-owned files per design.md's File Structure Plan; tracked via buglog only): bug-1559 (`xce-build.sh`'s `scheduler` case now runs `meson test --suite scheduler`, exercising all 41 scheduler test binaries instead of just `sched_test` directly by path) and bug-1558 (`make test-backend` now self-derives `CGO_CFLAGS`/`CGO_LDFLAGS` via `pkg-config`, works standalone with zero manual env setup — one iteration needed: the first attempt's `-I../../include` was relative and silently resolved wrong under cgo's actual build cwd, fixed to an absolute `-I$(CURDIR)/../include`). - -## Implementation Notes -- Task 1.1: `src/lib/cluster.c:99-102`'s `xce_cluster_config_validate()` still has a third hardcoded `>3` WARN check (message says "<=3 only") of the same magic-number class Req 1.7 targets, and `src/include/xce/cluster.h:31`'s doc comment repeats the stale "3" figure. Task 1.1's text named only "the two" worker/scheduler checks, so this was left untouched there — task 1.2 already touches `cluster.c` to add the new conversion helper, so fold this fix in there rather than letting it silently persist as the one remaining un-reconciled magic number. -- Task 1.1 follow-up resolved in 1.2: the `>3`→`>MAX_HA_CANDIDATES` rename initially proposed for `xce_cluster_config_validate()`'s WARN check turned out to be dead code — after task 1.1's constant reconciliation, that check tests the identical field against the identical constant as an earlier `-EINVAL` return in the same function, so it can never fire. Deleted rather than renamed (with the matching stale doc-comment line also removed), caught by task 1.2's review round 1. -- Task 1.2: its 8 new tests in `test_cluster.c` substantively cover task 1.3's stated checklist already (candidate order, one-candidate-no-branch, truncation-with-warning, NULL/empty rejection) — task 1.3 should scope down to just the constant-equality assertion (`N_MAX_MASTERS == MAX_HA_CANDIDATES`) rather than duplicating this coverage. -- Task 1.3: closed with zero new code. Verified fresh (source read + a clean `xce-build.sh lib asan` run) that every item in its checklist was already satisfied by prior tasks: the `_Static_assert(N_MAX_MASTERS == MAX_HA_CANDIDATES, ...)` landed in task 1.1 (`test_libxce_retry.c:44`), and the four behavioral cases (order/no-branch/truncation/NULL-empty) landed in task 1.2's 8 tests (`test_cluster.c`). No implementer/reviewer cycle was dispatched since there was no diff to review — dispatching one would have either duplicated existing test names (build conflict) or been a no-op review of an empty diff. -- Task 2.1: `xce_cluster_config_to_client_config()` (task 1.2) aliases `out->masters[i]` directly to `cfg->candidates[i].hostname` — no `strdup`. Any caller building the source `xce_cluster_config_t` on the stack and passing it through the helper then using the resulting client config beyond that stack frame's lifetime has a use-after-scope bug. Worker's fix keeps the source config in a file-scope `static` for this reason; tasks 3.1 (scheduler) and 4.2 (apiserver) hit the identical hazard and must do the same (or otherwise guarantee the source config outlives the client config's use). -- Task 2.2: fixed the startup-ordering bug by threading `cluster_cfg` by value through `xcecd_config_t` → `xcecd_context_t` → `cc_ctx` (set in `main.c` before `xcecd_init()`/`xcecd_start()` run, instead of after). Safe as a plain struct copy because `xce_candidate_t.hostname` is a fixed `char[]` array, not a pointer — no aliasing hazard here (unlike the `xce_cluster_config_to_client_config()` output struct noted above). Scheduler (3.1) likely has the same ordering gap (`sched_init` loops candidates correctly already per design.md, so may not need this fix — verify against current source, don't assume). Rejoin-retry pattern established: `checking_connection()` checks `needs_rejoin` after every probe return, calls `xcecli_join()` directly (not full re-entry into the join-cluster function), clears the flag only on success — same pattern scheduler/apiserver rejoin tasks (3.1, 4.3) should follow. -- Task 2.3 (real E2E caught what unit tests missed — 2 production bugs slipped through tasks 2.1/2.2's approved reviews): (1) `xcecli_start()` (`src/lib/client.c`) unconditionally resets `ctx->failover.rejoin_cb = NULL` as part of its own init — any `xcecli_set_reconnect_handler()` call before `xcecli_start()` is silently wiped. Register the reconnect handler AFTER `xcecli_start()` succeeds, not before. Scheduler/apiserver's own reconnect-handler registration (3.1, 4.3) must follow this same ordering or they'll hit the identical silent-wipe bug. (2) A successful `xcecli_join()`/worker join returns the assigned slot index as `reply.status` (any `rc >= 0`, not always exactly `0`) — checking `rc == 0` for success is wrong; use `rc < 0` as the only failure test (matches `xcecd_join_cluster()`'s own established convention). Any new code checking a join/rejoin call's success must use this same `rc < 0` convention, not `rc == 0`. (3) Security lesson: an env-var runtime bypass of a security check (`XCECD_ALLOW_NONROOT`) added for test/E2E convenience was rejected on review — the correct pattern for skipping a production-only check (root privilege, etc.) in a test/virtual build is a compile-time guard against the project's existing `VIRTUAL_CLUSTER` or `XCE_TESTING` flags, never a runtime environment variable reachable in the shipped production binary. -- Task 3.2: implemented as a shared `sched_publish_retry_loop()`/`sched_publish_check_rejoin()` pair called from both `sched_publish_dispatch()` and `sched_publish_pend_reason()`, not literally inlined into `sched_publish_dispatch()` alone as design.md's prose names it — reviewed and judged compliant (same thread, fires once after the whole retry loop's successful return, not per-attempt, not in `sched_allocator`). Testing lesson for any future scheduler.c unit test needing to intercept `xcecli_*` calls: `xcecli_jobdispatch`/`xcecli_set_pend_reason`/`xcecli_join`/`xcecli_start`/`xcecli_set_reconnect_handler` all live in the single `client.c` translation unit inside `libxce.a` alongside symbols scheduler.c genuinely needs — the "test double linked ahead of the archive" trick (used by test_driver_decision_cb.c one layer up) causes duplicate-definition errors here. Use `-Wl,--wrap=<symbol>` linker interposition instead (see `test_scheduler_decision_publish_retry.c`/its `meson.build` target) when a test needs scheduler.c itself linked live alongside intercepted libxce calls. Also confirmed: `sched_admin_request_test`'s `test_recordstart_symlink_escape_replies_error` fails independent of this spec's changes (its white-box `#include` list doesn't pull in `scheduler.c` at all) — a pre-existing, unrelated, environment-specific (root-in-sandbox symlink-permission) failure; do not re-investigate it as a regression in later tasks. -- Task 4.1: first apiserver-group (Go+CGO) task. `xce_parse_peers()` alone (`src/lib/cluster.c`) does syntax parsing only — no semantic checks (accepts port 0, non-numeric port via silent `atoi()`-to-0, empty host). Worker/scheduler always pair it with a second `xce_cluster_config_validate()` call in their `main.c`s; round-1 review caught `parseCandidates()` (`candidates.go`) skipping that second call, silently accepting malformed candidates worker/scheduler reject — a reminder that "the C function I called is correct" isn't the same as "I called every C function the precedent calls." Fix: call `xce_cluster_config_validate(&cCfg)` after `xce_parse_peers()`, stamping a local placeholder into `cCfg.identity` first since that function also checks a non-empty top-level `identity` (unrelated to per-candidate validation, never propagated to the real connection identity elsewhere in `clib.go`). `xce_cluster_config_validate()` checks `hostname`/`port`/top-level `identity` only, never per-candidate `name` — so the `name=` prefix stays legitimately optional. Established convention confirmed pre-existing (not invented this task): a `//go:build cgo` test file with no `import "C"` of its own, pairing with a plain-Go-signature function in a separate cgo-preamble file (`_test.go` files can never `import "C"` themselves — a Go-toolchain restriction, not just a project style choice) — see `wire_dispatch_test.go`/`wire_dispatch_support.go` for the pre-existing precedent this task's `candidates_test.go`/`candidates.go` split follows. Task 4.1 is parsing-only by design — `RegisterToBackend()`'s connect loop and `tryConnect()` stay untouched; task 4.2 is the one that collapses them into a single connect call, so don't expect this diff to touch that territory. Behavior change (deliberate, disclosed, not a regression): a malformed entry now rejects the WHOLE candidate list (matches worker/scheduler's all-or-nothing precedent, `xce_parse_peers()` has no partial-success mode by construction) instead of the old Go parser's skip-with-warning leniency — surfaces via `RegisterToBackend`'s error → `cmd/main.go`'s `logger.Error`+`os.Exit(1)`, loud not silent. Tooling gap for future apiserver tasks (4.2-4.6): `make test-backend`'s documented CGO env-var setup doesn't actually export `CGO_CFLAGS`/`CGO_LDFLAGS` to a general shell (they only exist as meson `custom_target` env vars) — reconstruct via `pkg-config` against the deps `apiserver/meson.build` lists, or run the full `xce-build.sh apiserver asan` (rebuilds `libxce.a` fresh too) rather than a bare `go test`. -- Task 4.2: found and fixed a genuine, independently-verified use-after-free while collapsing apiserver's two connect mechanisms into one — the deleted `tryConnect()` did `C.CString(host)` then `defer C.free(cAddr)` right after its single `xce_backend_init()` call returned, but `xcecli_ctx_init()` stores `ctx->conf = cfg` as a raw pointer (`src/lib/client.c:38`, never copied), and `client_failover.c`'s rotation path (`xcecli_retry_next`) re-dereferences `ac->conf->masters[cur]` on *every* failover attempt, not just at init — so any reconnect after the first would read freed heap memory. (`from`/`key` were NOT part of this bug — `xce_ctx_init()` `strdup()`s those synchronously before the defer ever fires.) Fix mirrors worker's/scheduler's own file-scope-`static` pattern for the Go/CGO side: two package-level `var`s (`gClusterCfg C.xce_cluster_config_t`, `gClientCfg C.xce_client_config_t`, alongside the pre-existing `gCtx`/`gIDCtx`) give the CGO structs a stable, process-lifetime address, and the connect path now writes host/from/key into fixed-size C arrays (`writeCCharArray()`) instead of `C.CString`/`C.free`-ing heap pointers the C library retains forever. General lesson for any future CGO code in this codebase that hands a struct to a C API storing a bare pointer to it (check the C function's own source, not just its header, for whether it copies or aliases): a Go function-local var is NOT safe once the call returns unless the C side genuinely copies everything reachable from it — verify via source read, the same standard task 2.1's original aliasing-hazard note set for the C-side daemons. `parseCandidates()` (task 4.1's deliverable) is now production-dead code — `buildClusterConfig()` calls `xce_parse_peers()` directly into the package-level `gClusterCfg` instead, since routing through `parseCandidates()`'s Go-slice return would mean re-parsing/re-copying into a fresh C struct that doesn't have the required stable address; `parseCandidates()` is exercised only by its own tests now, worth a follow-up cleanup task if it stays unused. -- Task 4.3: rejoin flag-setter (`go_backend_rejoin_handler`), its C shim, `extern` declaration, and `xcecli_set_reconnect_handler()` registration call all live together in `clib.go`'s `xce_backend_init()` — confirms design.md's "cgo symbol resolution is per-file" constraint (see `wire_dispatch_support.go`'s own documented rationale): a `C.<exported-Go-fn>` reference only resolves against an `extern` declared in THAT file's own cgo preamble, even though the Go `//export` mechanism makes the function callable package-wide — splitting the shim into a separate `reconnect.go` preamble while the extern lived in `clib.go` (or vice versa) would silently fail to link. Registration placed strictly AFTER `xcecli_start()`'s success check (task 2.3's ordering lesson ported to Go: `xcecli_start()` unconditionally wipes `rejoin_cb` to NULL as part of its own init). `needsRejoin` is a proper `sync/atomic` flag, not a plain bool — apiserver serves concurrent HTTP requests (unlike worker's/scheduler's single-threaded daemons), so this is a real correctness requirement here, not just style. `checkAndPerformRejoin()` wired into all 5 `gCtx.ctx`-based backend-call wrappers (SubmitJob/ControlJob/ControlQueue/ControlHost/FetchFile) — note `xcecli_fetchfile()` itself isn't `XCECLI_FAILOVER_CALL`-wrapped in libxce (never triggers the C-side rejoin invoke), so wrapping it here is safe but less load-bearing than the other four; that's a libxce gap, out of this task's `ApiserverBackendBridge` boundary to fix. Reviewer found (non-blocking, correctness-safe): concurrent goroutines can each observe `needsRejoin==1` after a failover and each call the real `xcecli_join()`, serialized by the C library's own per-ctx lock (no corruption, no incorrect flag state) but each can block up to the ~100s retry budget on a redundant join — a latency/traffic concern, not a Requirement-2 violation; a CAS-based single-flight guard is a reasonable fast-follow, not required. Scope reminder for task 4.4/4.5: `ApiserverBackendHealth`/`IsDegraded()`/`MarkHealthy()` deliberately do not exist yet — `checkAndPerformRejoin()` is the POST-call half only; task 4.5 adds a pre-call check to the same helper, not a new one. -- Task 4.4: thin adapter, not a second rejoin mechanism — `runHeartbeat()`'s new `onFailureToSuccess`/`onSuccessTick` callback params are wired directly to task 4.3's real `setNeedsRejoin`/`checkAndPerformRejoin` (`reconnect.go`) from `StartHeartbeat()`, zero duplicated flag-check logic added in `heartbeat.go` itself. Necessary because `xcecli_heartbeat()` is `XCECLI_FF_CALL`-wrapped (`client_internal.h`), a deliberately non-retrying fire-and-forget path that does its own best-effort one-shot reconnect but NEVER calls `xcecli_invoke_rejoin()` (that only fires from `xcecli_retry_next`, the write-call retry path) — so nothing sets `needsRejoin` on a pure idle-period recovery unless the heartbeat loop does it itself. `rc >= 0` success classification for `xcecli_heartbeat` confirmed against `xceapi.c`/`request.c`: it only ever returns 0 or a negative errno, never a positive value, so this is equivalent to (not just consistent with) the `rc<0`-only-failure convention used elsewhere. `onSuccessTick` fires unconditionally on every successful tick (no pending-check duplicated in `heartbeat.go`) — safe because `checkAndPerformRejoin()` is itself a cheap single-atomic-load no-op when nothing is pending. `prevFailed` (transition-tracking state) is correctly a plain non-atomic local, not `sync/atomic` — verified `runHeartbeat()` has exactly one call site (`StartHeartbeat()`, called once from `cmd/main.go`), one goroutine for the process lifetime; contrast with `needsRejoin` itself, which stays atomic since concurrent HTTP-handler goroutines reach it via the write-path wrappers. Known, accepted, already-documented (task 4.3) tradeoff carried forward: a rejoin attempt from the heartbeat goroutine shares `ac->failover.lock` with concurrent HTTP-handler backend calls, so one can block the other for up to the ~1-2 minute retry budget — not a new issue this task introduces, same mechanism task 4.3 already reviewed and accepted, task 4.4 just adds one more caller of it. -- Task 3.4: pure test-coverage task, zero production changes to `scheduler.c`/`driver.c` (correct outcome — task 3.2 already implemented the retry/shutdown/rejoin behavior correctly; 3.4 proves it holds at multi-decision QUEUE granularity through the real drain loop, which task 3.2's own tests never exercised, all 6 of which call `sched_publish_dispatch()`/`sched_publish_pend_reason()` directly on one decision). New `test_scheduler_decision_redelivery.c` links real `scheduler_srcs` with `-Wl,--wrap=xcecli_jobdispatch` (task 3.2's established technique) and drives the real `xce_sched_driver_on_decision()` (driver.c) over 3 queued, distinct-jobid decisions. Two load-bearing tests, both RED→GREEN-verified against the real code by two independent reviewer rounds (not just the implementer's claim): ordering (decision 1 stuck-then-succeeds fully completes before decision 2's call ever fires) and bounded shutdown latency (decisions queued behind one that triggers `core.stop` mid-retry get ZERO publish calls — the concrete, fast, deterministic proof that `sched_publish_retry_loop()`'s check-`stop`-before-each-attempt placement is correct, verified by reviewers moving the check to bottom-of-loop and confirming only this one test catches it). Test-fixture gotcha for any future test building a bare `xce_sched_core_t` via `xce_sched_core_init()` alone (not full `sched_init()`): do NOT call `xce_sched_core_fini()` on it — `fini()` calls `dataq_fini()` on `pend_updates[0/1]`, which only `sched_init()` (not `xce_sched_core_init()`) ever `dataq_init()`s, so `fini()` on a bare core SEGVs on a NULL dataq; omit the `fini()` call entirely, matching `test_core_events.c`'s established "fixtures aren't torn down" convention for this exact minimal-fixture pattern. -- Task 4.5: subtle design question resolved and worth remembering for task 4.6/5.x — `callBackend()`'s (`clib.go`) degraded fast-path deliberately skips `checkAndPerformRejoin()` ENTIRELY (not just the underlying `C.xcecli_*` call), meaning write-path requests contribute NOTHING to rejoin-retry cadence while `IsDegraded()` is true; only task 4.4's heartbeat goroutine keeps retrying during that window. Reviewer independently confirmed this is correct, not a violation of "must never suppress the pending-rejoin flag's own retry-until-actually-joined behavior": that constraint is one-directional (the circuit breaker's own optimistic TTL-based state must never be READ BY the rejoin mechanism to gate it — confirmed `checkAndPerformRejoin()` never references `IsDegraded()` anywhere), not a requirement that every possible call site attempt a rejoin check regardless of cost. The alternative (calling `checkAndPerformRejoin()`, which can itself block ~100s inside `xcecli_join()`'s own retry budget, on every fast-failed request) would directly violate THIS task's own fail-fast acceptance criterion — the two constraints (fail-fast + never-suppress-retry) are jointly satisfiable only because `heartbeat.go`'s goroutine runs unconditionally for the process lifetime, independent of circuit-breaker state, and `backendDegradedTTL` (10s, chosen well below the ~100s `XCECLI_FAILOVER_CALL` budget) is shorter than the heartbeat interval (default 30s) so write traffic resumes contributing once the TTL clears anyway. `callBackend()` is a genuine refactor consolidating what tasks 4.3/4.4 left as duplicated inline `rc := C.xcecli_*(...); checkAndPerformRejoin()` pairs across 5 wrapper functions into one shared helper — `checkAndPerformRejoin()` now appears exactly once in `clib.go`, inside `callBackend`, matching design.md's explicit "not copy-pasted per wrapper" instruction. `E_XCE_NOT_READY`-triggered `MarkDegraded()` uses a plain-Go `notReadyRC` mirror var (`= -int(C.E_XCE_NOT_READY)`, computed live via cgo, not hardcoded) since `_test.go` files can't reference `C.*` symbols directly (task 4.1's established Go-toolchain restriction) — keep this pattern for any future test needing a C errno constant. -- Task 4.6 (closes out apiserver group 4.1-4.6): audit task, same "zero new code is a valid outcome when genuinely true" precedent as scheduler's task 1.3 — 2 of 3 coverage areas (degraded-to-healthy transitions, candidate-list parity) were already fully satisfied by tasks 4.1/4.5's own tests, verified by reading the cited tests directly rather than trusting prior notes' claims. Found and closed exactly one genuine gap: `reconnect.go`'s `needsRejoin`/`checkAndPerformRejoin` (task 4.3) had zero concurrent-access test coverage despite being reached from 5 real concurrent HTTP-wrapper call sites plus the heartbeat goroutine — `health.go`'s own 50-goroutine stress test (task 4.5) only covered the degraded-until timestamp, not this separate atomic flag. One new test closes it. `gClusterCfg`/`gClientCfg` (task 4.2's CGO structs) correctly judged NOT a concurrency gap — written once at startup, single-threaded, before the HTTP server starts, never mutated again. Confirmed apiserver's Go-level `xce_parse_peers()` malformed-string coverage (`candidates_test.go`) is broader than any C-side component's own test — worker's/scheduler's C tests, and even `test_cluster.c`, only exercise `xce_cluster_config_validate()` against pre-built structs, never `xce_parse_peers()` against malformed raw strings at all. `go test -race` is confirmed, again, flatly unusable in this ASAN-linked CGO environment ("too many address space collisions") — this is now the third task (4.2, 4.5, 4.6) to independently hit and document this; don't waste time trying `-race` here, rely on atomic-correctness-by-construction plus deterministic concurrent-stress assertions instead. `xce-build.sh apiserver asan`'s binary-compile step can hit transient `goproxy.cn` network flakiness unrelated to any code change — when that happens, fall back to `cd src/frontend && make test` (CGO_ENABLED=0, whole apiserver module + xcecli) and `make test-backend` (CGO_ENABLED=1, `./backend/...`) directly, which is what the full pipeline's own `RUN_TESTS=1` branch invokes anyway once binaries are built. A pre-existing, unrelated `apiserver/controllers` build failure under a whole-module `CGO_ENABLED=1 go test ./...` (predates this spec entirely, `integration_test.go`'s `//go:build !cgo` tag vs. `output_test.go`'s unconditional references to its helpers) doesn't affect either authoritative Makefile target — don't chase it as a regression in future apiserver work. -- Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): `xcecd_virtual` (worker's own task 2.3 meson target, `-DVIRTUAL_CLUSTER -DDBUS_MANAGER`) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT `unshare -r`, which hits the unresolved bug-1519 ASAN double-join abort. Worker's reconnect proved cleanly (real `checking_connection: rejoin after transparent reconnect succeeded` log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. **Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-1548, see buglog for full detail)**: `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so `heartbeat.go`'s cross-tick `onFailureToSuccess` transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: `needsRejoin` may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. **Also found bug-1549** (job pushes never reach apiserver's store, `GET /xce/v1/jobs` empty despite controller-confirmed commits) — very likely a downstream symptom of bug-1548, not independent: `on_apiserver_join()` (`apiserver.c`) only bulk-resyncs jobs when a fresh join is actually processed, and `c->apiservers[]` (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-1548 should be verified against bug-1549 clearing as confirmation they share a root cause. Controller-side `join.c` logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside. -- Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-1548 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: **bug-1548 independently reproduced a second time, more cleanly** — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed `checkAndPerformRejoin()` is called unconditionally after every backend call in `callBackend()`, so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side `ctrl-2.log` corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. **bug-1548 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis** — a dedicated follow-up spec/task to fix `heartbeat.go`'s `onFailureToSuccess`/`onSuccessTick` wiring (make `XCECLI_FF_CALL`'s internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-1549 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-1548 and bug-1549's own objects (occurrences/last_seen) after review caught bug-1549's object being out of sync with its own re-reproduction narrated inside bug-1548's text. -- Task 5.3: unlike 5.1/5.2 (killed only the primary, letting rotation succeed), this task needs a call to genuinely EXHAUST its retry budget — kill ALL 3 controller nodes, not just the primary. Corrected timing assumption worth remembering: `client_failover.c`'s `xce_failover_classify()` puts `ECONNREFUSED`/`ETIMEDOUT`/`ECONNRESET` (and the dead-ctx fast path) on the ZERO-backoff `XCE_FO_RETRY_NOW` track, not the ~70-100s exponential-backoff `XCE_FO_RETRY_BACKOFF` track (reserved for `E_XCE_NOMASTER`/`E_XCE_NOT_READY` responses from a reachable-but-electing controller) — so a fully-dead-cluster trigger call exhausts its 20-attempt budget in milliseconds, not minutes. Clean, strongly-corroborated PASS: exact literal string match on `ErrBackendUnavailable` (`clib.go`) proves the degraded fast-fail path genuinely skips the C call rather than the C call happening to also be fast; `E_XCE_NOT_READY`'s numeric value (10013 = `XCE_ERRNO_BASE` 10001 + 12 enum positions, `errnox.h`) independently confirmed the trigger call's `-10013` response was the real thing, not a fabricated failure; the raw C-library log (`client_failover.c`'s own WARN/error lines, a format-distinct second logging subsystem from apiserver's Go zap log) cross-corroborated the whole sequence to the millisecond — this "check two independently-generated log streams agree" technique is a strong, reusable evidence-quality pattern for any future live-validation task in this spec. `GET /ping` confirmed via source read to never touch `gCtx`/`callBackend` — genuinely meaningful Req 4.2 evidence, not a coincidence. Bonus (not required by task text but strong corroboration): waited past the 10s TTL and confirmed `IsDegraded()`'s half-open auto-clear correctly re-attempts the real call. Noted, judged non-blocking: `design.md`'s sequence diagram literally says "503 service unavailable" but the actual HTTP status is 500 (`common.ERROR_CODE`) — `api-standards.md`'s error-code table has no 503 entry anywhere in this codebase's convention, and design.md's own Error Handling section explicitly disclaims introducing new error codes, so 500 + a distinct message body is the correct, convention-following choice, not a gap; the diagram itself is just imprecise. -- Task 5.4 (final task, closing regression gate): full from-scratch rebuild + retest of the entire stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli) found zero new regressions — same audit-first, "zero new code when genuinely covered" precedent as tasks 1.3/4.6, independently re-verified by two full review rounds (one implementer, one reviewer, both rebuilding from scratch) rather than trusted from a single run. The one known failure (`sched_admin_request_test`'s `test_recordstart_symlink_escape_replies_error`) is confirmed the same pre-existing root-in-sandbox symlink-permission issue by exact message match and source-level explanation (`driver.c`'s `-EINVAL` vs `-EACCES` classifier branches) — cross-corroborated against two independent `.wolf/memory.md` entries predating this task and a peer AI session's unprompted, independently-run sweep landing on identical numbers. The "identical single-candidate path" property (Req 1.5) is proven for all three components by real, meaningful, individually-reread-and-rerun tests (`test_build_cluster_config_single_entry_peers_matches_synthesized_shape` for worker, `test_no_peers_and_single_entry_peers_produce_identical_cluster_cfg_shape` for scheduler, `TestRegisterToBackend_NoPeersAndSingleEntryPeers_IdenticalLogSequence` for apiserver — the last one the most literal proof of the task's own "log sequence matches" wording, via genuine zap log-message-sequence equality) — no live binary-startup comparison needed since these already exercise the real production entry points end-to-end. Found (non-blocking, worth a future ticket, not this spec's job to fix): `xce-build.sh`'s `all` case runs under `set -euo pipefail`, so the one known C-test failure aborts the script before it ever reaches the Go/CGO test phases in a single invocation — and even when reached, that path only runs `make test` (`CGO_ENABLED=0`), never `make test-backend` (`CGO_ENABLED=1`), meaning the CGO backend suite (home of the load-bearing apiserver identical-log-sequence test) is structurally never exercised by `all` regardless of the known failure. Both reviewer and implementer independently worked around this by running the Go/CGO suites as separate commands — do the same for any future full-stack validation in this repo until the script itself is fixed. - -## Spec Complete (superseded below — see Phase 6) -Tasks 1.1–5.4 (21 tasks) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via `kiro-verify-completion` before each commit. Two real production bugs were found during live E2E validation (bug-1548: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-1549: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-1548) and were logged in `.wolf/buglog.json` with detailed, cross-verified root-cause analysis. **Correction (2026-08-15, `/kiro-validate-impl` NO-GO):** the sentence that originally stood here — calling both bugs "explicitly out of this spec's own scope to fix... flagged for a dedicated follow-up spec/task" — was wrong. bug-1548 lives in `heartbeat.go`, task 4.4's own deliverable, inside `ApiserverBackendBridge`'s declared boundary, against the already-existing Req 2 AC4/AC5 — a defect surfacing during cross-component *validation* (task 5.x) does not make it out of the *spec's* boundary, only out of that validation task's own boundary to fix in-flight. Task 5.2 was also incorrectly marked `[x]` despite explicitly not meeting its own stated observable-completion criterion (no rejoin evidence appeared during its heartbeat-only observation window) — a task must not be marked complete when its own pass bar isn't met, even when the validation work itself was rigorous and valuable. Both are now remediated as Phase 6 below, mapped to the same Req 2.4/2.5 task 4.4 was already scoped to. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history. -- Task 3.3: fixed a real production gap found along the way — `sched_init()`'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added `sched_format_candidate_list()`, threading `sched->cluster_cfg` through as a parameter rather than a new file-scope global (unlike worker's `g_cluster_cfg`) since `test_scheduler_connect.c` links `scheduler_srcs` as separate objects, not `#include`, so the `static` helper isn't directly callable from tests — coverage requires a real `dup2()`-based stdout capture around a live `sched_init()` call hitting the failure path, not a white-box direct call (see `capture_stdout_during_sched_init()` in `test_scheduler_connect.c` — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (`test_scheduler_backpressure.c`) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's `-E_XCE_NOT_READY` retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node `/xce-deploy-ha` cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, `test_e2e_live_ha_push.py`, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-1519 (`xcecd` ASAN "already joined thread" abort under `unshare -r`, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's `test_rejoin_composition_retries_across_publish_calls`, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-1519 ever gets resolved, not a blocker for this spec. diff --git a/.kiro/specs/component-rejoin-resync/brief.md b/.kiro/specs/component-rejoin-resync/brief.md deleted file mode 100644 index 3d002b7a..00000000 --- a/.kiro/specs/component-rejoin-resync/brief.md +++ /dev/null @@ -1,53 +0,0 @@ -# Brief: component-rejoin-resync - -## Problem - -When the apiserver or scheduler disconnects and reconnects to the controller (network blip, or reconnect to a new primary after controller failover), each does a full re-join and receives **full** state again (e.g. the whole job list, full queue/resource/config state) rather than a delta. For a large cluster this resend is sizable — bandwidth and re-processing cost scale with cluster size on every reconnect, not just on first join. - -This is a **different problem from `worker-rejoin-resync`**: that spec recovers worker-reported job-*execution facts* (start/finish/fail) that would otherwise be silently lost during a disconnect window. This spec is about the *cost/efficiency* of apiserver/scheduler's full-state resend on reconnect — no data is lost today (full resend is correct), it is potentially just expensive at scale. - -## Current State - -- Full-state rejoin is already accepted, working behavior (`libxce-failover` design review decision): "simple, correct, and still cheaper than a full-history replace with no snapshot" (cf. IBM LSF). -- Recorded as a **deferred YAGNI backlog item** ("lightweight rejoin re-attach", 2026-06-16, roadmap.md Future Specs/Backlog) — explicitly *not* promoted previously because no measured problem existed yet. -- No delta-resync or session-token re-attach mechanism exists for either component today. - -## Desired Outcome - -Not yet decided — this spec starts at the discussion/investigation stage, not a committed design. Candidate outcome: apiserver and/or scheduler reconnect without re-receiving state that is unchanged since their last known position (e.g. keyed on last-acknowledged committed LSN), reducing reconnect cost at scale — IF a real problem is measured. Equally plausible outcome: investigation confirms full-state resend remains fine and this spec closes as "no action needed." - -## Approach - -Not yet chosen. Prior design review flagged the general shape (session-token re-attach hello, or delta-only transfer keyed on last-known committed LSN) but flagged delta reconciliation as "non-trivial (the primary must know what each consumer already holds)" — a real design cost, not a rubber stamp. First step for this spec is likely measurement/investigation (does reconnect cost actually hurt at realistic cluster sizes?) before committing to a mechanism. - -## Scope - -- **In**: apiserver rejoin content/cost. Scheduler rejoin content/cost. Investigation of whether full-state resend is actually a measured problem before designing a fix. -- **Out**: Worker job-execution-fact resync (→ `worker-rejoin-resync`, separate spec, different data shape — facts vs. list-state). Any change to the pending-facts queue. New failover mechanics beyond what `libxce-failover`/`component-failover-adoption` already provide. - -## Boundary Candidates - -- apiserver reconnect path (Go, `src/frontend/apiserver/backend/clib.go` cgo boundary + libxce failover hook). -- scheduler reconnect path (`src/scheduler/src/handler.c`, controller-side `on_scheduler_join`/`handle_scheduler_join`, `join.c`). -- Measurement/instrumentation to quantify current reconnect cost — likely a prerequisite sub-task before any mechanism design. - -## Out of Boundary - -- Worker-side job-fact resync (separate spec, `worker-rejoin-resync`). -- Redesigning the join protocol wholesale — this is about reconnect efficiency, not join correctness. - -## Upstream / Downstream - -- **Upstream**: `libxce-failover` (rejoin hook — implemented). `component-failover-adoption` (daemon rejoin content — currently tasks-generated, in progress) — this spec's scheduler/apiserver reconnect paths overlap with that spec's territory; sequence after or alongside it, not independently duplicating its liveness-safety work. -- **Downstream**: none yet. - -## Existing Spec Touchpoints - -- **Adjacent**: `worker-rejoin-resync` — sibling, same trigger (reconnect/failover) but disjoint data (job facts vs. list state). Do not merge scope — different information, different mechanism. -- **Adjacent**: `component-failover-adoption` — covers making the daemons *use* libxce failover safely (liveness, master-list config); this spec covers *what* they resync once reconnected, a distinct concern layered on top. -- **Supersedes discussion for**: the "lightweight rejoin re-attach" backlog item (roadmap.md) — this spec is that item promoted, scoped to apiserver+scheduler only (worker's share of that item is now `worker-rejoin-resync`). - -## Constraints - -- Not started — investigation/discussion first, no committed mechanism yet. -- Should not be pursued as a speculative optimization — prior YAGNI decision stands until this spec's own investigation finds a measured cost, per the same discipline that deferred it originally. diff --git a/.kiro/specs/component-rejoin-resync/spec.json b/.kiro/specs/component-rejoin-resync/spec.json deleted file mode 100644 index 483a4bd5..00000000 --- a/.kiro/specs/component-rejoin-resync/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "component-rejoin-resync", - "created_at": "2026-08-10T00:00:00Z", - "updated_at": "2026-08-10T00:00:00Z", - "language": "en", - "phase": "brief", - "description": "apiserver and scheduler reconnect/rejoin resync efficiency -- full-state resend today; investigate whether a measured cost problem justifies delta/session-token re-attach. Sibling of worker-rejoin-resync (disjoint data: list-state vs job-execution facts).", - "approvals": { - "requirements": { - "generated": false, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "blocked_on": [] -} diff --git a/.kiro/specs/config-upgrade-snapshot/brief.md b/.kiro/specs/config-upgrade-snapshot/brief.md deleted file mode 100644 index 88623921..00000000 --- a/.kiro/specs/config-upgrade-snapshot/brief.md +++ /dev/null @@ -1,24 +0,0 @@ -# Brief: config-upgrade-snapshot - -## Problem -XCE's state machine (jobs, queues, hosts, resources) is **interpreted against the active configuration**. Today a configuration upgrade does config-push + restart + **per-node WAL replay under the new config** (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state. Because replay is config-dependent (the `lost_and_found` queue/host re-homes entities whose config entity vanished), members can **materialize divergent state** after an upgrade — amplified by any WAL tail divergence. Logged as **bug-857** (identified via the `raft.tla` gap analysis; not yet empirically reproduced). - -This was originally Requirement 8 of `ha-term-safety`; it is **descoped into this independent spec** (2026-06-21) because it has a hard prerequisite that does not exist yet. - -## Approach -A configuration upgrade establishes a **full, entity-bearing state snapshot at the configuration-change boundary** (the `(term, lsn)` of the config-change marker that `ha-term-safety` introduces), **embeds the new configuration** in that snapshot, and **installs the same snapshot on every member** through the out-of-band snapshot-install transport. No member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — installing the snapshot delivers configuration + materialized state together as one consistent artifact. This supersedes `ha-term-safety` R7's interim config-push byte delivery for the upgrade case. - -## HARD dependency — entity-bearing snapshots -This spec **cannot proceed without entity-bearing snapshots**. Today snapshots are a 16-byte LSN sentinel (`"XCESNAP\0"` + checkpoint_lsn; `persist.c` write ~1068-1101, load ~1280-1307) carrying **zero entity state**. The COW machinery needed (`xzarray_freeze`/`thaw`, `zarray.c:695-757`) is **built and unit-tested but dormant** (`kvmgr.c:297` `vfreeze=NULL`, no production callers) — the roadmap's unstarted **`entity-bearing-snapshots` ("Task 11")** backlog item. That work must land first (or be co-scoped). The snapshot *transport* (`snapshot_transfer_s`: chunked, `(term,lsn)`-anchored, hash-verified, install-before-activate) is already reusable; only the entity-bearing *payload* is missing. - -## Scope -- **In**: full entity-bearing state snapshot materialized at the config-change marker boundary; embed the new configuration in that snapshot; install on every member via the snapshot-install path; activation gating (a member does not serve under the new configuration until it has installed the upgrade snapshot and verified the embedded configuration against the marker's content hash); post-upgrade consistency invariant (every member's materialized state + active configuration identical); supersede `ha-term-safety` R7's interim config-push byte delivery for upgrades. -- **Out**: the config-change marker / consensus ordering itself (owned by `ha-term-safety` R7); the entity serialization machinery (owned by `entity-bearing-snapshots` — this spec consumes it); the Raft protocol (the `Raft TLA+ Conformance` library series); controller voting-set/membership changes (`ha-config-bootstrap`). - -## Verification -Empirically reproduce bug-857 first (upgrade config during the chaotic-multi-kill repro and diff materialized state across nodes via `waldump`/state dump) to establish the RED, then prove every member converges to identical materialized state + active config after an upgrade interleaved with failover. - -## Dependencies -HARD (both now satisfied — 2026-07-21): `entity-bearing-snapshots` → satisfied by `ha-entity-snapshots` (21/21 tasks). `ha-term-safety` (the config-change marker it anchors on) → `implementation-complete`, GO 2026-07-17. Consumes the `snapshot_transfer_s` transport. Closes bug-857. - -**Prepare work (decided 2026-07-21):** fix **bug-1418** (apply-lag promotion-gate liveness deadlock, still open, no owning spec) before or alongside this spec's implementation. It reliably fires inside the same chaotic-multi-kill soak Requirement 6 needs, so left unfixed it contaminates that test's signal; its own (unimplemented) fix — become primary, self-apply, then serve, instead of hard-blocking promotion — is also the template Requirement 3's activation gate must mirror to avoid reintroducing the same liveness-deadlock shape for materialization-lag. Scope as task 0 / a small prerequisite fix, not optional cleanup. diff --git a/.kiro/specs/config-upgrade-snapshot/design.md b/.kiro/specs/config-upgrade-snapshot/design.md deleted file mode 100644 index e1d722d7..00000000 --- a/.kiro/specs/config-upgrade-snapshot/design.md +++ /dev/null @@ -1,432 +0,0 @@ -# Design Document: config-upgrade-snapshot - -## Overview - -**Purpose**: This feature makes a configuration upgrade on an already-running XCE cluster materialize one identical, entity-bearing state snapshot at the configuration-change marker's `(term, lsn)` boundary, embed the new configuration in it, and install that same snapshot on every member — closing bug-857 (post-upgrade materialized-state divergence) by construction, rather than by per-node WAL replay under the new configuration. - -**Users**: Cluster operators pushing a configuration change (via `xceadmin`/the upgrade-config admin path) and, transitively, every job/queue/host in flight at the moment of the change — this feature is what keeps their materialized state identical across all controller members through the transition. - -**Impact**: Retires the current "commit marker → restart every controller → each member replays its own pre-upgrade WAL under the new configuration" behavior (`do_upgrade_config_with_bytes`, `config_repl.c:1591-1773`, step 9) in favor of a synchronous, in-process materialization trigger on marker apply. No controller restart is required for a configuration upgrade after this change. - -### Goals -- Materialize a full entity-bearing snapshot (jobs, queues, hosts, embedded configuration) at the exact `(term, lsn)` of the committed configuration-change marker, on every member, without any member re-deriving state by WAL replay. -- Gate a serving primary's acceptance of state-changing client requests on its OWN materialization-and-hash-verification for the current marker having completed — independent of Raft's promotion eligibility. -- Survive crash, restart, and leadership change at any point in the upgrade sequence, with deterministic, idempotent resumption. -- Prove the fix with a regression that reproduces bug-857 on the old code path and passes on the new one, including chaotic multi-kill failover interleaved with the upgrade. - -### Non-Goals -- Initial/bootstrap configuration replication (config tag zero → first config) — owned by `ha-config-bootstrap`; not unified with this mechanism (requirements.md Boundary Context, decided and rejected 2026-07-23). -- The configuration-change marker's consensus ordering itself, or the Raft protocol — owned by `ha-term-safety` / the Raft TLA+ Conformance library series. -- Controller voting-set/membership changes — `ha-config-bootstrap`. -- Fixing the pre-existing, out-of-scope `durable_lsn`-not-advanced-by-snapshot-install gap (research.md, this session) — real but bounded to self-promotion-eligibility bookkeeping, predates this spec, applies to any snapshot install (not upgrade-specific). - -## Boundary Commitments - -### This Spec Owns -- The `WAL_CONFIG_MARKER` apply-time side effect: triggering a forced, synchronous checkpoint at marker-apply time (new `apply_committed_entry` case). -- Embedding full configuration bytes (not just a `{tag, sha}` reference) in the upgrade snapshot, and in every subsequent checkpoint until rotated away. -- The host admin-state snapshot frame (new) and the queue admin-state frame's migration to Cap'n Proto (existing raw layout retired) — one `SNAPSHOT_FORMAT_VERSION` bump covers both. -- The activation fence: a new, independent condition gating primary request-serving and promotion on this member's own materialization-and-hash-verification for the current marker. -- Retiring the per-node restart/replay upgrade path (`do_upgrade_config_with_bytes` step 9). -- Regression coverage: bug-857 reproduction, crash/leadership-change injection at the marker→snapshot-durable and marker→all-installed windows, host-placeholder/rejoin, overlapping upgrades, and format-mismatch fail-loud behavior. - -### Out of Boundary -- Entity-serialization machinery itself (job/queue capnp encode, checkpoint file layout, chunked transport) — owned by `ha-entity-snapshots`, already shipped; this spec extends it in the two narrow, explicit ways above, not otherwise. -- The configuration-change marker's append/commit and `(term, lsn)` ordering — owned by `ha-term-safety` Requirement 7; this spec only reacts to the marker once committed. -- Raft election, log replication, term safety — the Raft TLA+ Conformance library series. -- Fixing `durable_lsn` non-advancement on snapshot-install (see Non-Goals). -- Initial/bootstrap config replication — `ha-config-bootstrap`. - -### Allowed Dependencies -- `ha-entity-snapshots`'s `write_snapshot`/`load_snapshot`/`persist_checkpoint` machinery (`persist.c`) — extended, not replaced. -- `ha-term-safety`'s committed configuration-change marker (`config_committed_marker`, `WAL_CONFIG_MARKER`) — consumed as-is. -- The existing `snapshot_transfer_s` chunked install transport (`ha_handle_snapshot_chunk`) — reused as-is; this spec does not modify chunk transfer mechanics. -- The existing `config_serve_fence` pattern (`ha.h:307-322`, `config_repl.c:304-377`) — mirrored structurally for the new activation fence, not merged into the same field. -- The existing `lost_and_found` placeholder-worker precedent (`makeWorker()` without join, `config_repl.c:1114-1130`) — mirrored for host-frame restore. - -### Revalidation Triggers -- Any change to `apply_committed_entry`'s dispatch shape or to the fact that it is the single unified apply seam for all members (currently confirmed — `apply_wal_entry` has zero live callers) would require re-verifying the marker-apply hook still fires for every member. -- Any change to the on-disk snapshot header/frame layout (`SNAPSHOT_FORMAT_VERSION`) beyond this spec's own bump must re-check `snapverify`/`snapdiff`/this spec's config-bytes section for continued validity. -- Any change to `config_serve_fence`'s semantics or to `ha_dispatch_gate`'s composition order must re-check the new activation fence's placement alongside it. -- If `entity-bearing-snapshots`/`ha-entity-snapshots` ever changes its frame format, this spec's host/queue frame additions must be re-verified against the new format. - -## Architecture - -### Existing Architecture Analysis - -Three existing subsystems already do almost everything this feature needs; the actual new-work surface is narrow: - -1. **The apply seam is already unified.** `apply_committed_entry` (`persist.c:2751-3132`) is the single dispatch point for entity mutation on every member — primary online-apply (vtask-pool dispatched via `persist_apply_pump_kick`), standby normal apply (`ha_standby_advance_committed_and_apply`, heartbeat-learned commit position, since the 2026-07-10 commit-then-apply change), and boot/restart replay. `WAL_CONFIG_MARKER` currently falls into its generic `default:` no-op. The sibling dispatcher `apply_wal_entry` has zero live callers — fully superseded, not a second seam to worry about. -2. **The checkpoint/snapshot pipeline is already entity-bearing.** `write_snapshot`/`load_snapshot` (`persist.c`) already serialize job and queue frames at every checkpoint, guarded by a single-in-flight CAS (`checkpoint_in_progress`) and already stamping a config *reference* (`cfgref_tag`/`cfgref_sha`) via `scan_latest_committed_config_ref`. `persist_checkpoint()` is a plain, already-force-callable, synchronous function (a second direct call site already exists at `ha.c:3308`). -3. **The activation-fence pattern already exists for a materially identical problem.** `config_serve_fence` already gates client-request-serving (via `ha_dispatch_gate`) and promotion on a hash-verify condition, with a fully-wired non-blocking retry loop — this is a direct structural precedent for the gate 3.2/3.3 need, not a new pattern to invent. - -The upgrade path being retired (`do_upgrade_config_with_bytes` step 9, `config_repl.c:1753-1772`) already carries a NOTICE comment (`config_repl.c:1737-1751`) naming this exact spec and its target end-state — the maintainers who wrote the interim restart/replay path already anticipated this design. - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - Marker[WAL_CONFIG_MARKER committed] - Apply[apply_committed_entry] - Trigger[Forced checkpoint trigger] - Checkpoint[persist_checkpoint] - Snapshot[write_snapshot entity plus config bytes] - Fence[Upgrade materialization fence] - Gate[ha_dispatch_gate] - Promo[Promotion gate] - Transport[snapshot_transfer_s install] - Behind[Behind or joining member] - - Marker --> Apply - Apply --> Trigger - Trigger --> Checkpoint - Checkpoint --> Snapshot - Snapshot --> Fence - Fence --> Gate - Fence --> Promo - Behind --> Transport - Transport --> Fence -``` - -**Architecture Integration**: -- Selected pattern: extend the existing per-member apply-and-checkpoint pipeline in place; no new distribution/orchestration component. Each member independently materializes on its own apply of the marker (primary and standby alike, since both route through `apply_committed_entry`); a member that is behind receives the same entity-bearing snapshot through the existing generic catch-up transport, unrelated to upgrade-specific push logic. -- Domain/feature boundaries: persistence (`persist.c`) owns materialization and on-disk format; HA (`ha.c`/`ha.h`) owns the activation fence and dispatch-gate integration; config replication (`config_repl.c`) owns config-bytes sourcing and retiring the old restart path; worker/queue restore owns entity-specific create-vs-update semantics. -- Existing patterns preserved: single-in-flight checkpoint CAS, fail-loud format-version mismatch, never-self-terminate fence-and-retry, `#ifdef XCE_TESTING` fault-hook shape. -- New components rationale: a new `apply_committed_entry` case (nothing existed for this entry type); a new fence field (the existing one is logically a different condition); new capnp structs (existing ones bundle unwanted dynamic/static fields); a create-if-missing host restore path (queues' update-only assumption doesn't hold for hosts). -- Steering compliance: never blocks a vtask handler (the new checkpoint trigger runs inside `apply_committed_entry`'s own vtask-pool context, not on iomgr/delaymgr); never self-terminates (fence-and-retry, mirroring `config_serve_fence`). - -**Key simplification found during design**: 5.4 ("new leader resumes driving materialization and distribution") needs no new distribution-tracking state machine. Because every member independently materializes on its own apply of the marker (there is no primary-driven push loop — install is receiver-pulled via the existing generic catch-up mechanism), a new leader satisfies 5.4 automatically by continuing to apply its own committed log (which already includes the marker) — the same `apply_committed_entry` hook that handles this for any member handles it for a newly-elected one, with no leader-specific "resume" logic to write. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Persistence / Serialization | Cap'n Proto (existing, `src/schema/controller.capnp`) | Encodes `HostAdminState`/`QueueAdminState` frames and embedded config bytes in the snapshot | Same library already used for `Job`/`Worker` messages; no version change | -| Backend / Runtime | C (gnu17), existing controller daemon | All new logic lives in `src/controller/` | No new language/runtime | -| Async I/O | Existing vtask pool / delaymgr (`ha-async-io.md`) | New checkpoint trigger runs on the existing apply-pump vtask worker; new fence retry mirrors `config_schedule_activate_retry`'s delaymgr→vtask trampoline | No new concurrency primitive | - -No new external dependency is introduced anywhere in this spec. - -## File Structure Plan - -### Modified Files -- `src/controller/persist.c` — new `case WAL_CONFIG_MARKER:` in `apply_committed_entry` (calls `persist_checkpoint` directly); `write_snapshot`/`load_snapshot`/`load_snapshot_entities` extended with a host-frame section and a config-bytes section; `collect_queue_frames`/queue-frame decode migrated from raw layout to `QueueAdminState` capnp; new `collect_host_frames`/`restore_host_state`; `SNAPSHOT_FORMAT_VERSION` bumped 1→2; new `#ifdef XCE_TESTING` fault hook for the marker-apply→checkpoint-durable window. -- `src/include/persist.h` — `SNAPSHOT_FORMAT_VERSION` bump declaration site (if defined here rather than persist.c); no new persistent fields needed beyond what already exists (`cfgref_*` reused for the reference; the embedded bytes are read fresh from disk at write time via `myconf`'s own fields, not cached on `persist_t` — see the config_repl.c bullet below for why this avoids a new cross-module call). -- `src/controller/ha.h` — new `_Atomic int upgrade_materialization_fence` field on `ha_state_t` (sibling to `config_serve_fence`, `ha.h:307-322`); new accessor declaration. -- `src/controller/ha.c` — new `upgrade_materialization_verify()` (mirrors `config_activate_committed`'s hash-verify-then-set/clear shape); new accessor `upgrade_materialization_fenced()`; `ha_dispatch_gate`'s `CTRL_PRIMARY` branch extended with the new fence check (composed alongside, not merged with, `config_serve_fenced()`); promotion gate (`ha.c:1425` area) extended the same way `config_serve_fenced()` already gates it; new `schedule_upgrade_materialization_retry()` mirroring `config_schedule_activate_retry`'s trampoline. -- `src/controller/config_repl.c` — `do_upgrade_config_with_bytes` step 9 (`config_repl.c:1753-1772`, restart_requested/cbuf_put) removed; the NOTICE comment (`config_repl.c:1737-1751`) replaced with a short comment pointing at the new `apply_committed_entry` case; **no new accessor function added here** — `persist.c`'s snapshot writer reads the active config bytes directly off disk via `p->cluster->myconf->workdir`/`myconf->name` (plain struct fields on the shared, public `cluster_t`/`myconf_t`, `cluster_def.h`/`myconf.h`), the same `workdir/name/conf` path `conf_save_config` (`config_repl.c:776`) already writes to — this deliberately mirrors `scan_latest_committed_config_ref`'s own existing precedent of avoiding a `persist.c → config_repl.c` function-call dependency (that function's comment states this explicitly), rather than introducing the layering violation a new accessor would create; new `#ifdef XCE_TESTING` fault hook for the leadership-change-mid-distribution window, mirroring `xce_snapshot_chunk_hook`'s shape. -- `src/controller/worker.c` — new `restore_host_state(cluster_t*, const char*, uint32_t admin_status)`: create-if-missing via a small `makeWorker()`-based placeholder helper (valid non-NULL `hostattrs_t`, mirroring the `lost_and_found` precedent in `config_repl.c:1114-1130`), idempotent (look up first, only create if missing, OR-in admin bits either way — 5.8). -- `src/controller/queue.c` — no signature change to `restore_queue_state`; only its caller's decode step (in `persist.c`) changes from raw-layout parsing to `QueueAdminState` capnp decode. -- `src/schema/controller.capnp` — new `struct HostAdminState { name @0: Text; status @1: UInt32; }` and `struct QueueAdminState { name @0: Text; status @1: UInt32; }`. -- `.claude/skills/xce-test-ha/run-ha-soak.sh` — extended with a new scenario covering config upgrade interleaved with chaotic multi-kill failover (requirement 6) and back-to-back overlapping upgrades (7.2); exact flag/mode naming left to tasks.md. - -### New Files -- `src/controller/testsuite/test_upgrade_snapshot_apply.c` — bare in-memory `cluster_t` fixture (mirrors `test_apply_committed_entry.c`'s pattern), drives the new `WAL_CONFIG_MARKER` case directly, asserts a forced checkpoint fires exactly once and is idempotent under retry. -- `src/controller/testsuite/test_upgrade_snapshot_restore.c` — disk-backed fixture (mirrors `test_replay_apply_seam.c`'s pattern), covers host-placeholder creation, later real-join-through-`rejoin_worker` admin-bit preservation, and idempotent retry of a partially-failed restore. -- `src/controller/testsuite/test_upgrade_activation_fence.c` — exercises `upgrade_materialization_verify`, the dispatch-gate composition, the promotion-gate composition, and the retry trampoline. -- `src/controller/testsuite/configcompare.c` / `configcompare.h` — new sibling tool to `walcompare` (not an extension of it — `walcompare`'s internal model is WAL-entry-indexed `(lsn → term, pcrc)`; active-configuration identity across members is a single `(tag, sha)` per member with no LSN dimension, an awkward fit for `walcompare`'s report shape). **Offline/file-based, mirroring `snapdiff_paths`'s exact signature shape**: `int configcompare_paths(const char* const* conf_paths, int n, FILE* out, configcompare_report_t* rep)`, each path being a member's `workdir/name/conf` file (the same path this spec's snapshot-embed step reads, so both consumers agree on where a member's active config lives) — no live RPC to each cluster member, consistent with every other tool in this family. Compares each member's config content hash (recomputed from the file bytes, not trusted from a cached tag) for 4.2, usable directly by the HA soak scenario against each node's own data directory. -- `src/controller/testsuite/snapverify.c` (modified, not new) — extended to decode and assert `cfgref_tag`/`cfgref_sha` (not just presence), and to report host-frame count and config-bytes-section presence. - -## System Flows - -```mermaid -sequenceDiagram - participant Admin - participant Primary - participant Standby - participant ApplySeam as apply_committed_entry - participant Checkpoint as persist_checkpoint - participant Fence as Activation Fence - - Admin->>Primary: upgrade-config request - Primary->>Primary: config_commit_change (append+commit marker) - Primary->>Standby: replicate WAL_CONFIG_MARKER - par Primary applies - Primary->>ApplySeam: apply marker (own apply pump) - ApplySeam->>Checkpoint: trigger forced checkpoint at marker (term,lsn) - Checkpoint->>Fence: verify hash, clear fence - and Standby applies - Standby->>ApplySeam: apply marker (heartbeat-learned commit) - ApplySeam->>Checkpoint: trigger forced checkpoint at marker (term,lsn) - Checkpoint->>Fence: verify hash, clear fence - end - Fence-->>Primary: serve under new configuration -``` - -**Key decisions visible above**: no primary-orchestrated push step exists — both members reach the same materialized-and-verified state independently through the same seam, which is what makes leadership change mid-upgrade (5.4) self-resolving rather than requiring resume logic. - -```mermaid -stateDiagram-v2 - [*] --> Fenced: marker applied, materialization not yet verified - Fenced --> Fenced: hash mismatch or materialization incomplete, retry via delaymgr trampoline - Fenced --> Verified: hash-verify succeeds against current marker - Verified --> [*]: fence cleared, serving and promotion allowed -``` - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|---|---|---|---|---| -| 1.1 | Materialize at marker apply, not commit | `apply_committed_entry` new case | `persist_checkpoint` | Sequence diagram | -| 1.2 | Every member ends up holding the snapshot | Per-member apply seam + existing transport | `ha_handle_snapshot_chunk` (reused) | Sequence diagram | -| 1.3 | No member re-derives via replay | `do_upgrade_config_with_bytes` step 9 removed | — | — | -| 1.4 | Transfer out-of-band, not in WAL | Existing `snapshot_transfer_s` (reused) | — | — | -| 1.5 | Synchronous, non-blocking trigger | New `apply_committed_entry` case calling `persist_checkpoint` directly | — | Sequence diagram | -| 1.6 | Host admin-status in snapshot | `collect_host_frames`/`HostAdminState` | capnp struct | — | -| 1.7 | Exclude dynamic host metrics | `collect_host_frames` field selection | — | — | -| 1.8 | Apply does not itself re-home entities | Existing `lost_and_found` re-homing untouched; apply vs activation separation | Activation Fence | State diagram | -| 1.9 | Host/queue frames extensible (capnp) | `HostAdminState`/`QueueAdminState` | `controller.capnp` | — | -| 1.10 | Create-if-missing host placeholder | `restore_host_state` | `makeWorker()`-based helper | — | -| 1.11 | Real join routes through `rejoin_worker` | `restore_host_state` placeholder + existing `join.c` dispatch (unchanged) | `rejoin_worker` (reused) | — | -| 2.1 | Config embedded in snapshot | New config-bytes section in `write_snapshot` | `p->cluster->myconf` direct read | — | -| 2.2 | Supersede R7 interim byte-push for upgrades | `do_upgrade_config_with_bytes` retirement | — | — | -| 2.3 | Config bytes out of WAL | Unchanged (`WAL_CONFIG_MARKER` already `{tag,sha}` only) | — | — | -| 2.4 | Every checkpoint embeds current config | Config-bytes section added to every `write_snapshot` call, not just marker-triggered | — | — | -| 3.1 | No new promotion-eligibility mechanism | Existing Raft log-freshness gate (unchanged) | — | — | -| 3.2 | Gate serving until materialization+hash-verify | Activation Fence | `ha_dispatch_gate` | State diagram | -| 3.3 | Keep serving under last-verified config while incomplete | Activation Fence (allowlist composition) | `ha_dispatch_gate` | State diagram | -| 3.4 | Hash mismatch keeps serving/fences, retries | `upgrade_materialization_verify` | `schedule_upgrade_materialization_retry` | State diagram | -| 4.1 | Every member's materialized state identical | Per-member apply seam + `snapdiff_paths` (test) | — | Sequence diagram | -| 4.2 | Every member's active config identical | New `configcompare` tool | `configcompare_paths` | — | -| 4.3 | Invariant holds across upgrade x failover | Activation Fence + existing promotion gate composition | — | — | -| 4.4 | Overlapping upgrades supersede cleanly | Per-marker deterministic content (5.1) + fence re-verify on newer marker | — | — | -| 5.1 | Deterministic snapshot content | `write_snapshot` keyed on `(term,lsn,config-hash)` | `snapdiff_paths` (test) | — | -| 5.2 | Crash before durable write resumes from marker | Existing `checkpoint_in_progress` CAS + apply-pump re-apply on restart | — | — | -| 5.3 | Forced checkpoint shares single-in-flight guard | `checkpoint_in_progress` CAS (reused) | `persist_checkpoint` | — | -| 5.4 | Leadership change resumes materialization | Per-member apply seam (see Key simplification) | — | Sequence diagram | -| 5.5 | Never serve without durable+verified snapshot | Activation Fence | `ha_dispatch_gate`, promotion gate | State diagram | -| 5.6 | Rolling-binary-upgrade format skew fail-loud | Existing `fmt != SNAPSHOT_FORMAT_VERSION` hard-reject (reused) | — | — | -| 5.7 | Old raw queue-frame layout rejected fail-loud | `SNAPSHOT_FORMAT_VERSION` bump 1→2 | — | — | -| 5.8 | Idempotent retry of placeholder host restore | `restore_host_state` (find-before-create) | — | — | -| 6.1 | Chaotic upgrade+failover: identical state/config | HA soak new scenario | `configcompare`, `snapdiff_paths` | — | -| 6.2 | Reproduce bug-857 on old code, pass on new | `test_bug857_regression.c` (unit-level, `controller_unit_test`) — see correction note below | `snapdiff_paths` | — | -| 6.3 | Inject crash/leadership-change at marker-to-install window | New fault hooks (`persist.c`, `config_repl.c`) | — | — | -| 6.4 | Host admin-status explicitly asserted post-upgrade+failover | HA soak new scenario assertion | — | — | -| 7.1 | Placeholder-then-rejoin admin status preserved | `test_upgrade_snapshot_restore.c` | `restore_host_state`, `rejoin_worker` | — | -| 7.2 | Overlapping upgrades cleanly superseded | HA soak new scenario | — | — | -| 7.3 | Format-mismatch fail-loud (rolling upgrade + queue-layout) | `test_upgrade_snapshot_apply.c` / manual format-mismatch fixture | — | — | -| 7.4 | Marker-triggered checkpoint's WAL trim does not strand a mid-catch-up standby | `test_upgrade_snapshot_apply.c` extended + HA soak new scenario | `trim_wal_segments`, `repl_trim_lsn` (reused, unmodified) | — | - -**Correction to row 6.2 (task 8.5, post-implementation):** this row originally assigned Requirement 6.2 to "HA soak new scenario," under the same bug-857 mechanism hypothesis as the buglog entry — "config-push + restart + per-node WAL replay, config-dependent via `lost_and_found`." Task 8.5's investigation found that mechanism was never implemented as code, on any commit: `create_lost_and_found` (config_repl.c) only ever creates the sentinel `lost_and_found` queue/host objects; no code path in `job.c`/`queue.c`/`worker.c` ever reassigns an EXISTING job's queue reference onto them (`restore_job`/`restore_job_array` never consult `c->qmgr`; `restore_queue_state` explicitly skips — returns `-ENOENT` — rather than re-homes when a queue is absent). `create_lost_and_found`'s body is also unconditionally `#ifndef XCE_TESTING`'d out. The buglog's own bug-857 entry already flagged this as "NOT empirically reproduced yet" — the re-homing detail was TLA+-derived aspiration, not observed behavior. - -The real, code-backed mechanism 6.2 reproduces instead is **checkpoint-content divergence from marker-vs-threshold decoupling**: pre-task-5, `apply_committed_entry`'s `WAL_CONFIG_MARKER` case was the generic no-op `default:` case, so a checkpoint's LSN/content was determined solely by `schedule_checkpoint`'s independent per-member WAL-record-count threshold — completely decoupled from the config-change marker's position in the shared committed log. Two members reaching that threshold at different real-world moments (ordinary job-submission activity interleaved around the upgrade) materialize genuinely different job/queue content, with neither checkpoint anchored to the marker — exactly "no snapshot pinning an identical post-upgrade state" (bug-857's own root-cause wording). Post-task-5, the marker-triggered checkpoint fires synchronously, inline, the instant the marker entry is dispatched, anchoring every member to identical content regardless of processing pace. - -This mechanism is fully reproducible at the `controller_unit_test` level (no `cluster.c`/live-multi-process linkage needed), so 6.2 was moved off the "HA soak new scenario" assignment onto `testsuite/test_bug857_regression.c` — a two-scenario regression (threshold-triggered divergence pre-fix; marker-triggered convergence post-fix, verified via `snapdiff_paths` byte-identity) landed entirely within this one file. Requirement 6.1's full live chaotic-soak-with-upgrade (task 8.6, not yet started) remains the legitimate home for live multi-process testing of the broader upgrade+failover interaction — it will need to build live config-upgrade-trigger infrastructure regardless of what 8.5 did, so nothing is lost by 8.5 staying at the unit level. Not rewriting the original row's history — see task 8.5's Status Report for the full investigation trail. - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|---|---|---|---|---|---| -| Marker Apply Trigger | Persistence (`persist.c`) | Fire a forced checkpoint synchronously when `WAL_CONFIG_MARKER` is applied | 1.1, 1.5, 1.8, 7.4 | `persist_checkpoint` (P0), `checkpoint_in_progress` CAS (P0) | State | -| Snapshot Entity/Config Payload | Persistence (`persist.c`) | Embed host/queue admin frames (capnp) and config bytes in every checkpoint | 1.6, 1.7, 1.9, 2.1, 2.4, 5.1, 5.7 | `HostAdminState`/`QueueAdminState` capnp (P0), `myconf` fields (P0, direct read) | State | -| Host Restore (create-if-missing) | Worker/Restore (`worker.c`) | Restore a host admin-state frame safely whether or not the host has ever joined | 1.10, 1.11, 5.8 | `makeWorker` (P0), `rejoin_worker` (P0, unmodified) | Service | -| Activation Fence | HA (`ha.c`/`ha.h`) | Gate primary serving and promotion on this member's own materialization-and-hash-verify | 1.8, 3.2, 3.3, 3.4, 4.3, 5.5 | `ha_dispatch_gate` (P0), promotion gate (P0), `config_serve_fence` pattern (P1, mirrored not shared) | State | -| Upgrade Path Retirement | Config Replication (`config_repl.c`) | Remove the restart/replay fallback now superseded by the Marker Apply Trigger | 1.2, 1.3, 1.4, 2.2, 5.4 | Marker Apply Trigger (P0) | — | -| Regression Tooling | Testsuite | New fault hooks, `configcompare`, extended `snapverify`, HA soak scenarios | 4.2, 6.1-6.4, 7.1-7.4 | All above (P0) | Batch | - -### Persistence (`persist.c`) - -#### Marker Apply Trigger - -| Field | Detail | -|---|---| -| Intent | Add the missing `WAL_CONFIG_MARKER` case to the single unified apply seam so every member materializes at the marker boundary | -| Requirements | 1.1, 1.5, 1.8, 7.4 | - -**Responsibilities & Constraints** -- Carve a new explicit `case WAL_CONFIG_MARKER:` out of `apply_committed_entry`'s current `default:` no-op. -- Call `persist_checkpoint()` directly — safe because `apply_committed_entry` already runs on the apply-pump's own vtask-pool worker, never the iomgr/delaymgr thread; no further async hand-off is needed at this call site (unlike `schedule_checkpoint`'s WAL-threshold trigger, which fires from arbitrary vtask handlers and genuinely needs delaymgr's non-blocking return). -- Must NOT change how entities are interpreted/re-homed (`lost_and_found`) — that remains the activation step's concern (1.8), never a side effect of apply. -- `apply_wal_entry`'s matching `default:` case is left untouched — it has zero live callers; adding symmetry there would be speculative, unjustified work (Design Invariant 5 / Simplification lens). - -**Dependencies** -- Inbound: apply pump (`persist_apply_pump_run`) and boot replay (P0) -- Outbound: `persist_checkpoint` (P0) — already tolerates `-EBUSY` from a concurrent scheduled checkpoint via the shared `checkpoint_in_progress` CAS -- Outbound: new `#ifdef XCE_TESTING` fault hook, mirroring `ha_stream_body_read_fault_hook`'s shape, for 6.3 and 7.3's deterministic marker-to-checkpoint-durable injection - -**Contracts**: State [x] - -##### State Management -- State model: `checkpoint_lsn` is pinned to `last_applied_lsn` inside `persist_checkpoint` — since this case runs exactly when the marker is applied, `checkpoint_lsn` naturally equals the marker's own LSN with no new LSN-targeting parameter needed. -- Persistence & consistency: guarded by the existing `checkpoint_in_progress` CAS; a concurrent WAL-threshold-triggered checkpoint and this marker-triggered one cannot both proceed simultaneously. -- Concurrency strategy: runs to completion on the calling vtask-pool worker; no new thread or lane. - -**Implementation Notes** -- Integration: one new `case` label plus one function call; `do_upgrade_config_with_bytes` step 9 becomes dead code and is removed in the same change (Upgrade Path Retirement, below). -- Validation: `test_upgrade_snapshot_apply.c` asserts the checkpoint fires exactly once per marker apply and is safe under a forced retry (idempotent — a second apply of an already-applied marker, e.g. during replay, must not double-checkpoint incorrectly; `checkpoint_lsn` monotonicity already guards this at the `persist_checkpoint` level). -- Risks: the narrow window between `schedule_checkpoint`'s CAS release and `persist_checkpoint`'s own re-acquire (pre-existing, `persist.c:2222`/`2265`) now has a second caller; not a new race class, but worth a regression assertion that a marker-triggered and WAL-threshold-triggered checkpoint racing each other never corrupts the snapshot (covered by the existing CAS, asserted for completeness). -- **Overlapping-marker `-EBUSY` handling (4.4, 7.2)**: if a second `WAL_CONFIG_MARKER` is applied while an earlier marker's forced checkpoint is still in flight, this case's `persist_checkpoint()` call returns `-EBUSY` — this is NOT an error to surface or retry specially. The in-flight checkpoint (for the older marker) completes on its own, and the Activation Fence's own verify step (which re-reads the *latest committed* marker, not whichever one triggered the checkpoint) naturally re-evaluates against the newer marker once that checkpoint's `write_snapshot` call runs — either from the older checkpoint's own completion re-triggering a fresh comparison, or from the newer marker's normal WAL-threshold checkpoint eventually firing. No special retry loop is added at this call site for `-EBUSY`; it is logged at `TRACE`, not `WARN`/`ERROR`, since it is an expected, self-resolving outcome, not a failure. -- **Trim safety under the new trigger path (7.4)**: `persist_checkpoint()` (this case's own call) already runs `trim_wal_segments` afterward, unchanged — the retention-floor guard (`repl_trim_lsn`, bug-1147: floor = min(active segment LSN, every connected standby's acked LSN, every recently-disconnected slot's last-disconnected LSN)) is not modified by this spec, only invoked from a new trigger point. Six existing tests (`test_checkpoint_wal_trim_after_checkpoint`, `test_trim_guard_preserves_standby_segments`, `test_trim_without_standbys_proceeds_normally`, `test_repl_trim_lsn_no_standbys`, `test_repl_trim_lsn_disconnected_slot_expiry`, `test_trim_lsn_with_active_standby_at_lsn_5`) already cover this guard through the periodic WAL-threshold trigger — none exercise it through this new marker-triggered path, or under the chaotic multi-kill conditions Requirement 6 exercises, which is a real coverage gap this spec must close, not assume covered by proxy. - -#### Snapshot Entity/Config Payload - -| Field | Detail | -|---|---| -| Intent | Extend the on-disk checkpoint format with host admin-state frames, migrate queue frames to capnp, and embed full configuration bytes | -| Requirements | 1.6, 1.7, 1.9, 2.1, 2.4, 5.1, 5.7 | - -**Responsibilities & Constraints** -- `collect_host_frames`: walk `c->workermgr` with the same lock/iterate idiom `collect_job_frames`/`collect_queue_frames` already use; encode only `w->host.status & XCE_WORKER_ADMIN_MASK` per host as a `HostAdminState` capnp frame — never the full `xce_worker_t` (excludes `resvals_to_sched`, `dead_since`, dynamic status bits, per 1.7). -- `collect_queue_frames`: migrated to emit `QueueAdminState` capnp frames instead of the current raw `{name_len, name, status}` layout — same field selection (name + status), new wire encoding only. -- Config-bytes section: `write_snapshot` reads the locally-held active configuration bytes directly off disk via `p->cluster->myconf->workdir`/`myconf->name` (the same `workdir/name/conf` path `conf_save_config` writes, and the same bytes `config_activate_committed` already hash-verifies), appended after the host frames. No new `config_repl.c` function is added for this — see the File Structure Plan note on why (mirrors `scan_latest_committed_config_ref`'s existing avoid-a-config_repl-dependency precedent). -- `SNAPSHOT_FORMAT_VERSION` bumps 1→2 — this is a genuine breaking layout change (queue frames change shape), so the existing fail-loud version-mismatch behavior (`load_snapshot`, `fmt != SNAPSHOT_FORMAT_VERSION → -EINVAL`) applies unchanged; no compatibility shim is added (Design Invariant 4). -- Determinism (5.1): content is a pure function of `(term, lsn, config-hash)` — independent materialization by any member (including a newly-elected leader) must be byte-identical, verified via `snapdiff_paths`. - -**Dependencies** -- Inbound: Marker Apply Trigger (P0, calls `persist_checkpoint` which calls `write_snapshot`); periodic WAL-threshold checkpoint (P0, same `write_snapshot` gets the same new sections on every checkpoint per 2.4) -- Outbound: `controller.capnp`'s new `HostAdminState`/`QueueAdminState` structs (P0) -- Outbound: `p->cluster->myconf` fields (P0, plain struct-field read on the shared `cluster_t`/`myconf_t` — not a `config_repl.c` function call, preserving the existing layering precedent) - -**Contracts**: State [x] - -##### State Management -- State model: additive frame sections after the existing job/queue frames; config-bytes section is new, appended last. -- Persistence & consistency: same single-commit-point discipline `load_snapshot` already enforces (restore fully, THEN advance position; a partial restore leaves `persist_t`'s fields untouched). -- Concurrency strategy: unchanged — collection happens under the same table read-locks already used, no new locking model. - -**Implementation Notes** -- Integration: `HostAdminState`/`QueueAdminState` capnp structs added to `controller.capnp`; regenerate bindings. -- Validation: `snapverify` extended to decode/assert `cfgref_tag`/`cfgref_sha` (not just presence) and report host-frame count and config-bytes-section presence; `snapdiff_paths` used as-is for the determinism check. -- Risks: none new beyond the format-version bump's inherent one-time old-snapshot invalidation, already an accepted, existing pattern. - -### Worker/Restore (`worker.c`) - -#### Host Restore (create-if-missing) - -| Field | Detail | -|---|---| -| Intent | Restore a host admin-state frame correctly whether the target host has ever joined on this member or not | -| Requirements | 1.10, 1.11, 5.8 | - -**Responsibilities & Constraints** -- Classify hosts as join-provisioned (not config-provisioned, unlike queues): `key_find(c->workermgr, name)` may legitimately miss for a member receiving the snapshot via the out-of-band install path, since that is precisely the "hasn't locally replayed this host's join record yet" case. -- On miss: create a minimal placeholder via `makeWorker()` with a valid, non-NULL `hostattrs_t` (even if minimal/zeroed) — mirroring the `lost_and_found` sentinel host precedent exactly, not a new mechanism. A NULL/absent `attrs` here would crash `rejoin_worker`'s later direct dereference on the first real join (1.11) — this is the one hard implementation constraint carried over from discovery. -- On hit (placeholder or previously-joined real host): OR-in the restored admin-status bits, same as today's behavior. -- Idempotency (5.8): a retried restore after a partial failure must find the already-created placeholder and update it, never treat its existence as a create conflict — `key_find`-then-create-or-update already gives this for free as long as the create step itself is the only thing guarded (i.e., find-first, not create-first-then-handle-EEXIST). -- The later real join is NOT this component's concern: `join.c`'s existing `key_find`-based dispatch (found `w != NULL` → `rejoin_worker`, else → `init_worker`) already routes a real join against an existing placeholder through `rejoin_worker`, which already preserves admin bits (`worker.c:374,390`) — no change needed there. - -**Dependencies** -- Inbound: Snapshot Entity/Config Payload's restore path (P0, calls this per host frame) -- Outbound: `makeWorker` (P0, existing) -- Outbound: `rejoin_worker` (P0, existing, unmodified — the later real join's correctness depends on this component creating a valid placeholder, not on any change to `rejoin_worker` itself) - -**Contracts**: Service [x] - -##### Service Interface -```c -int restore_host_state(cluster_t* c, const char* name, uint32_t admin_status); -``` -- Preconditions: `name` non-empty; `c->workermgr` initialized. -- Postconditions: a `worker_t` named `name` exists in `c->workermgr` with `host.status` including `admin_status`'s bits; if newly created, `host.attrs` is non-NULL and heap-owned (via `makeWorker`'s underlying `copy_worker` deep-copy). -- Invariants: never destroys a pre-existing worker's non-admin state; never leaves a partially-constructed placeholder on any error path (mirrors the existing collect/restore all-or-nothing discipline). - -**Implementation Notes** -- Integration: called from `persist.c`'s host-frame restore loop, one call per frame. -- Validation: `test_upgrade_snapshot_restore.c` covers create-then-later-real-join (asserting admin status survives through `rejoin_worker`), and retry-after-partial-failure (asserting no duplicate/conflict). -- Risks: none beyond the `attrs`-non-NULL constraint already called out — mitigated by construction (always pass a valid, even if zeroed, `hostattrs_t` to `makeWorker`). - -### HA (`ha.c`/`ha.h`) - -#### Activation Fence - -| Field | Detail | -|---|---| -| Intent | Never let a member serve client requests or become primary under a configuration whose upgrade-snapshot materialization it has not itself completed and hash-verified | -| Requirements | 1.8, 3.2, 3.3, 3.4, 4.3, 5.5 | - -**Responsibilities & Constraints** -- A new, independent atomic fence field on `ha_state_t` — NOT merged into `config_serve_fence`, since the two conditions are logically distinct (byte-mismatch vs. materialization-incomplete) and 3.4 needs to distinguish "keep serving under last-verified configuration" from "no verified configuration exists yet," which requires knowing which fence is active. -- Verify function mirrors `config_activate_committed`'s exact shape: read the current committed marker, compare against this member's own materialized-and-hash-verified state for that marker; set the fence and log loud on mismatch/incomplete, clear it on success — never `exit`/`abort` (`controller/CLAUDE.md`'s never-self-terminate invariant). -- **Verify against LIVE on-disk config bytes each attempt, never a frozen snapshot's embedded copy.** The WAL marker's raft-commit and the R7 config-bytes push (`push_config_to_standbys`/`handle_repl_config_push`) are two independent channels with no strict ordering guarantee between them — a member could apply the marker and trigger its forced checkpoint before its own `workdir/name/conf` reflects the pushed bytes, meaning the snapshot's embedded config section could be momentarily stale. This is not a correctness risk (a stale embed is just one more mismatch the hash-verify catches, same as any other), but it IS a liveness trap if the verify function checks the snapshot's frozen embed instead of re-reading current disk bytes fresh on every retry — `config_activate_committed` already gets this right (recomputes the hash from current bytes each call, `config_repl.c:275-302`, never from a cached/frozen copy), and this new verify function must do the same, or the fence could stay stuck until the next unrelated periodic checkpoint happens to re-embed fresher bytes. -- Composed into `ha_dispatch_gate`'s `CTRL_PRIMARY` branch the same way `config_serve_fenced()` already is — a sequential additional check with its own allowlist (replication + read-only ops stay admitted), not a merge into the existing check's logic. -- Also gates promotion, the same way `config_serve_fenced()`/`ha_config_promotion_fenced()` already do — a member whose own materialization is incomplete must not become primary and immediately fail 3.2 the moment it wins an election. -- Self-clearing retry: a new `schedule_upgrade_materialization_retry()` mirroring `config_schedule_activate_retry`'s delaymgr-trampoline-then-vtask shape exactly — non-blocking trigger, blocking verify work done off the delaymgr thread. - -**Dependencies** -- Inbound: Marker Apply Trigger (P0, clears the fence once materialization+hash-verify succeeds) -- Outbound: `ha_dispatch_gate` (P0, existing, extended) -- Outbound: promotion gate (P0, existing, extended) -- Outbound: `config_schedule_activate_retry`'s trampoline pattern (P1, mirrored, not shared code) - -**Contracts**: State [x] - -##### State Management -- State model: two states per member — fenced (materialization incomplete or hash-mismatched for the current marker) and verified (cleared). -- Persistence & consistency: in-memory only, re-derived on restart from the actual on-disk snapshot state vs. the current committed marker (no separate durable flag needed — the fence is a function of comparing two already-durable facts). -- Concurrency strategy: same single-writer-plus-atomic-read pattern `config_serve_fence` already uses. - -**Implementation Notes** -- Integration: one new field, one new verify function, one new accessor, two call-site additions (dispatch gate, promotion gate), one new retry trampoline. -- Validation: `test_upgrade_activation_fence.c` covers fence-set-on-incomplete-materialization, fence-clear-on-verify-success, dispatch-gate allowlist behavior while fenced, and promotion refusal while fenced. -- Risks: none beyond what the mirrored `config_serve_fence` pattern already carries in production today. - -### Config Replication (`config_repl.c`) - -#### Upgrade Path Retirement - -| Field | Detail | -|---|---| -| Intent | Remove the now-superseded restart/per-node-replay fallback | -| Requirements | 1.2, 1.3, 1.4, 2.2, 5.4 | - -**Responsibilities & Constraints** -- Delete `do_upgrade_config_with_bytes` step 9 (`restart_requested`/`cbuf_put`) — materialization is now driven entirely by the Marker Apply Trigger, decoupled from this request-handling function entirely; the function's job now ends at step 8 (ACK the admin tool once the marker+bytes are majority-committed). -- Replace the NOTICE comment naming this spec with a short pointer at the new `apply_committed_entry` case, so the historical context isn't lost. -- No new distribution-tracking logic is added here (see Architecture's "Key simplification found during design") — 5.4 is satisfied by the per-member apply model, not by anything this function does. - -**Dependencies** -- Inbound: none (this is a deletion, not a new caller) -- Outbound: Marker Apply Trigger (P0) now owns what this code used to trigger via restart - -**Contracts**: none (behavioral removal) - -**Implementation Notes** -- Integration: this change and the Marker Apply Trigger must land together — deleting step 9 without the new case first would silently leave the marker with no materialization trigger at all (a real ordering dependency for tasks.md to sequence correctly). -- Validation: the bug-857 regression (6.2) exercises this path directly — reproduces divergence on the pre-change code (restart+replay), passes on the post-change code (no restart, synchronous materialization). -- Risks: none new; this is strictly subtractive once the replacement is proven. - -## Data Models - -### Logical Data Model - -**Checkpoint file layout (on-disk), `SNAPSHOT_FORMAT_VERSION = 2`**: -1. Legacy sentinel header (unchanged): magic `"XCESNAP\0"` (8 bytes) + `checkpoint_lsn` (8 bytes) + config-ref tail (`present`/`tag`/`sha`, 41 bytes). -2. `format_version` (4 bytes) — now `2`. -3. Job frames (unchanged): `job_count` + per-job `[payload_len][capnp Job bytes]`. -4. Queue frames (changed): `queue_count` + per-queue `[payload_len][capnp QueueAdminState bytes]` — was `[name_len][name][status]` raw. -5. Host frames (new): `host_count` + per-host `[payload_len][capnp HostAdminState bytes]`. -6. Config-bytes section (new): `config_bytes_len` + raw config bytes (the same bytes `config_activate_committed` hash-verifies against the marker's sha256). - -**Referential integrity**: restore order (job → queue → host) does not matter functionally — re-homing decisions (`lost_and_found`) are already baked into each entity's encoded content at materialization time, never re-derived at restore time (Design Invariant 2). - -**Versioning**: the `format_version` field is the sole discriminator; any mismatch is a hard fail-loud reject (`-EINVAL`), consistent with the existing, deliberate no-silent-reinterpretation policy. A short file (nothing past the legacy header) is still detected as a pre-entity-bearing legacy checkpoint, unchanged from today. - -## Error Handling - -### Error Strategy -Every new failure path follows the codebase's existing convention: fail loud (log the actionable boundary), never `exit`/`abort` on an operational condition, stay alive, keep retrying when self-healing is possible. - -### Error Categories and Responses -- **Format-version mismatch** (rolling binary upgrade skew on snapshot-install, or an old raw-queue-frame snapshot post-migration): hard reject, no partial install, no silent reinterpretation — reuses the existing `fmt != SNAPSHOT_FORMAT_VERSION` path unchanged. -- **Hash-verify mismatch** (embedded config bytes don't match the marker's sha256, or materialization incomplete): Activation Fence sets, member keeps serving under its last-verified configuration (or a loud non-serving fence if none exists yet), retries via the delaymgr trampoline — never activates mismatched configuration. -- **Host placeholder creation failure** (`makeWorker` allocation failure): treated as any other collect/restore failure today — discard partial state, fail the checkpoint/restore attempt, retriable on the next cycle; never a crash. -- **Overlapping upgrades** (a second marker commits before the first's distribution completes): the newer marker's materialization supersedes the older one's in-flight state cleanly, since content is a deterministic function of `(term, lsn, config-hash)` — an in-progress older materialization simply becomes moot, not corrupting, once the newer one completes. - -### Monitoring -Reuses existing structured-log conventions (`INFO`/`WARN`/`ERROR` per the codebase's HA logging pattern) — a loud `ERROR` on fence-set (naming the marker's `(term,lsn)` and the mismatch reason), `INFO` on fence-clear, matching `config_serve_fence`'s existing log shape. - -## Testing Strategy - -- **Unit Tests** (`test_upgrade_snapshot_apply.c`): `WAL_CONFIG_MARKER` case fires a checkpoint exactly once per apply (1.1, 1.5); idempotent under a forced retry (5.2, 5.3); does not itself re-home entities (1.8). -- **Unit Tests** (`test_upgrade_snapshot_restore.c`): host-frame create-if-missing via placeholder (1.10); admin status survives through a later real join via `rejoin_worker` (1.11); idempotent retry after partial failure (5.8). -- **Unit Tests** (`test_upgrade_activation_fence.c`): fence set/clear on hash-verify outcome (3.2, 3.4); dispatch-gate allowlist while fenced (3.3); promotion refusal while fenced (4.3, 5.5). -- **Integration Tests**: `snapdiff_paths` determinism check — two independent materializations at the same `(term, lsn, config-hash)` boundary are byte-identical (5.1). `configcompare` cross-member active-config identity (4.2). Extended `snapverify` decodes/asserts the embedded config-ref tag/hash and reports host-frame/config-bytes-section presence. bug-857 reproduction (6.2) — see the Requirements Traceability table row and the correction note above this section: reassigned off "E2E / HA soak" to `test_bug857_regression.c`, a unit-level demonstration that `schedule_checkpoint`'s pre-existing WAL-threshold trigger (decoupled from the config-marker boundary) diverges two members' checkpoint content, and that the marker-triggered checkpoint added by this spec converges them regardless of per-member processing pace. -- **E2E / HA soak** (`.claude/skills/xce-test-ha`): config upgrade interleaved with chaotic multi-kill failover, asserting identical materialized state and active config across all members afterward, including host admin-status specifically (6.1, 6.4); deterministic fault injection at the marker-apply→checkpoint-durable window and the leadership-change-mid-distribution window via the two new `#ifdef XCE_TESTING` hooks, rather than relying on chaotic-soak timing probability (6.3); overlapping/back-to-back upgrades before the first's distribution completes, asserting clean supersession (4.4, 7.2); host admin-status change issued while the host is down, followed by an upgrade snapshot restore on a member with no local join record, followed by the host's real first join, asserting preservation end-to-end (7.1); format-mismatch fail-loud for both rolling-binary-upgrade skew and the old raw-queue-frame layout (5.6, 5.7, 7.3); a standby held genuinely mid-catch-up (its acked LSN behind the marker's) while the marker-triggered checkpoint's WAL trim runs, asserting its still-needed segments are not stranded — extending `test_upgrade_snapshot_apply.c` with a fixture mirroring `test_trim_guard_preserves_standby_segments`'s existing pattern but driven through the new marker-apply case, plus one HA soak assertion under chaotic conditions specifically (7.4). - -## Migration Strategy - -A single `SNAPSHOT_FORMAT_VERSION` bump (1→2) covers the queue-frame capnp migration and the new host-frame/config-bytes sections together, so this spec introduces exactly one rollout discontinuity, not several staged ones. An old-format on-disk snapshot becomes unreadable by the new binary until that node's next checkpoint cycle regenerates one — consistent with the existing, deliberate fail-loud version-mismatch policy (not a new risk this spec introduces). No phased rollback path is defined beyond redeploying the prior binary version, matching how every prior `SNAPSHOT_FORMAT_VERSION` change in this codebase has been handled. - -## Supporting References -- `research.md` — full discovery findings, exact file:line citations for every function referenced above, and the design-synthesis rationale (generalization/build-vs-adopt/simplification) behind each decision in this document. -- `requirements.md` — the 7 requirements / 38 acceptance criteria this design traces to, plus the Design Invariants section this design's component choices were checked against. diff --git a/.kiro/specs/config-upgrade-snapshot/requirements.md b/.kiro/specs/config-upgrade-snapshot/requirements.md deleted file mode 100644 index 82e3f3d4..00000000 --- a/.kiro/specs/config-upgrade-snapshot/requirements.md +++ /dev/null @@ -1,119 +0,0 @@ -# Requirements Document - -## Introduction - -XCE's controller state machine (jobs, queues, hosts) is interpreted against the active configuration. Today a configuration upgrade pushes the new configuration, restarts each controller, and lets each member **re-derive** its post-upgrade state by replaying its pre-upgrade WAL under the new configuration — with no snapshot pinning an identical materialized state across members. Because replay is configuration-dependent (the `lost_and_found` queue/host re-homes entities whose configuration entity vanished), members can diverge in materialized state after an upgrade (bug-857), amplified by any WAL tail divergence. - -This feature makes a configuration upgrade establish a **full, entity-bearing state snapshot at the configuration-change boundary**, embed the new configuration in it, and install that same snapshot on every member — so post-upgrade state is identical by construction and no member re-derives it. It was originally Requirement 8 of `ha-term-safety` and is split out because it has a hard prerequisite (entity-bearing snapshots) that does not yet exist. - -## Boundary Context - -- **In scope**: a full entity-bearing state snapshot (the upgrade snapshot) materialized at the configuration-change marker's `(term, lsn)`; embedding the new configuration in that snapshot; every member ending up holding it, via local materialization (already applied the marker) or the out-of-band snapshot-install transport (behind or freshly joining); an activation gate on the serving primary tied to its own materialization-and-hash-verification completing, not to Raft election eligibility; the post-upgrade materialized-state + active-configuration consistency invariant; superseding `ha-term-safety` R7's interim config-push byte delivery for the upgrade case. -- **Out of scope**: - - The configuration-change marker and consensus ordering of the change — owned by `ha-term-safety` (Requirement 7). - - The entity-serialization machinery that produces an entity-bearing snapshot — owned by `entity-bearing-snapshots`; this spec **consumes** it, with one explicit exception: this spec adds the new host admin-state frame and migrates the existing queue admin-state frame to Cap'n Proto (see "Host and queue admin-state frames now extensible" below) — a deliberate, narrow modification of that spec's shipped frame *encoding*, not a re-opening of its entity-serialization design. - - The Raft protocol (term/log/commit) — owned by the `Raft TLA+ Conformance` library series. - - Controller voting-set / membership changes — `ha-config-bootstrap`. - - **Initial/bootstrap configuration replication (config tag zero → first config) — owned by `ha-config-bootstrap`, deliberately NOT unified with this spec's marker+snapshot mechanism (considered and rejected 2026-07-23).** Bootstrap has no pre-existing entity state to protect (a genuinely blank node) and a chicken-and-egg constraint this spec's mechanism cannot resolve: a configuration-change marker is a committed log entry, but at cold-start there is no log or elected leader yet — election itself must work with zero config (`ha-config-bootstrap` Req 2.9/3.3). Forcing bootstrap through marker-commit + snapshot-materialize + hash-verify would also regress single-node self-elect's immediate-serve behavior (Req 2.9). This spec's mechanism activates only for a configuration change on an already-running cluster with live entity state. -- **Adjacent expectations**: - - **Dependency on entity-bearing snapshots — SATISFIED (2026-07-09) by `ha-entity-snapshots` (21/21 tasks, fully approved).** That spec landed `write_snapshot`/`load_snapshot` (persist.c) as a direct Cap'n-Proto entity-encode/decode of job and queue records at checkpoint time, restoring before advancing position — not the COW `xzarray_freeze`/`thaw` mechanism this line originally anticipated (that machinery remains dormant; `ha-entity-snapshots` took a different, already-shipped implementation path). This spec no longer needs to wait; verify `write_snapshot`'s actual on-disk entity format (persist.c, `ha-entity-snapshots` design.md) before writing this spec's own design so it builds on the real interface, not the COW one originally assumed here. - - Anchors on the `(term, lsn)` configuration-change marker from `ha-term-safety`. - - Reuses the existing `snapshot_transfer_s` snapshot-install transport (chunked, `(term, lsn)`-anchored, hash-verified, install-before-activate). - - Reuses the existing checkpoint machinery (`persist.c` `persist_checkpoint`/`write_snapshot`/`schedule_checkpoint`) as the materialization mechanism rather than a bespoke path: it already runs off the raft apply path via `delaymgr` (never blocks a vtask handler — see `ha-async-io.md`), already guards a single in-flight checkpoint with a CAS (`checkpoint_in_progress`), and already pins `checkpoint_lsn` to `last_applied_lsn`. It already caches a committed-config *reference* (`cfgref_present`/`cfgref_tag`/`cfgref_sha`, `ha-term-safety` task 4.2) in the snapshot header — this spec extends that to a full *embedded* configuration payload, it does not replace the mechanism. - - **Gap identified 2026-07-21, scope resolved 2026-07-23 — host frames are missing from the entity-bearing payload; this spec owns adding them.** `write_snapshot` as shipped by `ha-entity-snapshots` calls only `collect_job_frames`/`collect_queue_frames` (persist.c) — there is no `collect_host_frames` and no host section in the on-disk format. This spec's own tasks add it, not a follow-up on `ha-entity-snapshots`'s owners. Confirmed content scope: record **only admin-issued status changes** (open/closed/locked/tagged, via the existing `XCE_WORKER_ADMIN_MASK` — `worker.c:275`), never the full `xce_worker_t` (host attrs and resource values are worker-reported and change dynamically; Requirement 1.7 already excludes them, and a full-struct encode would also break Requirement 5.1 determinism). The existing `capnp_encode_worker`/`Worker` capnp message (`persist.c:82`, used today for `WAL_WORKER_CTRL` replication) is NOT the right vehicle — it bundles static attrs, admin status, and dynamic state (`resvals_to_sched`, `dead_since`) together. Restore does NOT mirror `restore_queue_state`'s update-only assumption — queues are always config-provisioned before restore runs (`collect_queue_frames`'s own doc comment), but hosts are NOT: a `worker_t` is created only by a live join (`join.c:100-102`, `init_worker`), never from `hosts.yaml` config directly, and `worker_ctrl` (`worker.c:655-658`) itself hard `-ENOENT`s an admin command against a host that has never joined. A member receiving the upgrade snapshot via the out-of-band install path (Requirement 1.2's "behind" case) may specifically not have locally replayed that host's original join record yet — that is the whole reason it needs the snapshot instead of local replay. So `restore_host_state` must be **create-if-missing**: if `key_find(c->workermgr, name)` misses, create a minimal placeholder via `makeWorker()` (unjoined/`XCE_WORKER_UNAVAIL` until the physical worker actually connects and `apply_join_state` populates the rest) and set only the admin-status bits on it — mirroring the exact `makeWorker()`-without-join pattern the synthetic `lost_and_found` sentinel host already uses (`config_repl.c:1114-1130`), not inventing a new mechanism. Design must also confirm this restore path is idempotent under a retried partial failure (a second attempt's `makeWorker` call must find and update the placeholder a prior failed attempt already created, not treat it as a duplicate-create conflict — mirroring `restore_job`'s own `-EEXIST`-is-success idempotent-replay handling). `resources` is dropped from scope entirely (see Introduction) — `c->resmgr` entries (`xce_resource_t`, `controller.capnp:200`) are pure config-derived definitions with no admin-mutable field (no open/close/lock action exists for a resource definition), and the dynamic per-load values (`r15s`/`r1m`) already live in the same `w->host.resvals` bucket Requirement 1.7 excludes — there is no distinct "resources" entity needing its own admin-state capture. - - **Host and queue admin-state frames now extensible (capnp), decided 2026-07-23.** Neither frame should be a fixed-shape dead end for future admin-state fields (e.g. a lock reason or tag added later). The queue frame today is a hand-rolled raw layout — `{ uint32_t name_len; name; uint32_t status }` (`snap_queue_frame_t`, `persist.c:1759`) — with zero schema-evolution story: `load_snapshot` hard-rejects any `SNAPSHOT_FORMAT_VERSION` mismatch (`persist.c:2541`, by deliberate fail-loud design, task 5.1), so every future field addition would otherwise force another all-nodes format bump. Cap'n Proto already solves exactly this for job frames (`Job` message, `persist.c:60`) — new fields get new ordinals; old encoded blobs keep decoding under capnp's own schema-evolution rules, no format bump needed per field. Decision: define `HostAdminState` and `QueueAdminState` Cap'n Proto structs (`{ name @0: Text; status @1: UInt32; }` each, independently extensible) and use them for both frame types, migrating the queue frame off its current raw layout in the same pass. This requires exactly **one** `SNAPSHOT_FORMAT_VERSION` bump (covering host+queue together, not two separate migrations later) — an old on-disk snapshot written under the prior raw queue-frame layout becomes unreadable by the new binary until that node's next checkpoint cycle regenerates one, consistent with the existing fail-loud version-mismatch behavior (not a new risk this decision introduces). See the Boundary Context "Out of scope" note above: this is the one explicit, deliberate exception to "this spec consumes, not modifies, `ha-entity-snapshots`'s machinery." - - **bug-1418 — RESOLVED, no longer a blocker.** The 2026-07-21 note here (and the matching `spec.json.blocked_on` entry) treated bug-1418 as a still-open, unowned liveness deadlock requiring a task-0 prerequisite fix. That was stale: `.wolf/buglog.json` already recorded it fixed on 2026-07-09 (resolved via upstream `raft-apply-decoupling` tasks 5.1/5.3/7.2 + bug-1421, not a promotion-gate code change), and it was independently re-verified fresh on 2026-07-23 with a 10-round `--chaotic` soak (`result=PASS`, zero committed divergence, zero committed job loss) against a freshly rebuilt controller. `spec.json.blocked_on` has been cleared. Requirement 6's chaotic-multi-kill soak signal is not at risk of this bug's noise. See project memory `project_bug1418_prework_decision.md`. - -## Design Invariants - -Standing rules for design.md and implementation, distilled from this spec's own requirements discussion (2026-07-23) — not requirements themselves, but constraints any design/implementation must respect: - -1. **Classify every entity type as config-provisioned or join-provisioned before writing its restore path.** Queues are config-provisioned (exist once config loads, independent of any external actor) — update-only restore is safe (`restore_queue_state`). Hosts are join-provisioned (a `worker_t` exists only after a live actor registers, `join.c:100-102`) — restore must be create-if-missing (Requirement 1.10). Assuming update-only for a join-provisioned entity is the exact mistake this spec's own design caught and corrected; any future entity type added to the snapshot must be classified explicitly, not defaulted to whichever pattern was written last. -2. **Frame restore order (job/queue/host) must not matter.** Re-homing (`lost_and_found`) is decided at materialization time and baked into the encoded bytes; restore never re-derives it. Each entity type's restore must be independent of the others having already run. -3. **Every restore path must be idempotent under partial-failure retry.** Established for jobs (`restore_job`'s `-EEXIST`-is-success) and required for the new host placeholder path (Requirement 5.8): a retried restore after a partial failure must find-and-continue, never duplicate-or-conflict. -4. **A format/version mismatch is always fail-loud, never a silent reinterpretation.** Already this codebase's convention for corruption (`persist.c:2541`, task 5.1); now explicitly extended to rolling-binary-upgrade skew on snapshot-install (Requirement 5.6) and the queue-frame layout migration itself (Requirement 5.7). Do not add a compat-shim decode path later that silently reinterprets old bytes under a new schema. -5. **Never reuse an existing wire message wholesale just because it is convenient.** `capnp_encode_worker`/`Worker` (`persist.c:82`) was the readily-available option for host frames and would have been wrong — it bundles static attrs, admin status, and dynamic state together. A snapshot frame's field set must be justified field-by-field against Requirements 1.6/1.7 (admin-state only, dynamic excluded), never inherited by convenience from a message built for a different purpose (replication). -6. **A placeholder entity created out-of-band (Requirement 1.10) must be safe for its eventual real counterpart to adopt.** The `lost_and_found` sentinel host precedent (`config_repl.c:1114-1130`) and the host-frame placeholder both create a `worker_t` via `makeWorker()` outside the normal join flow; the later real join must route through the same admin-bit-preserving path (`rejoin_worker`) an ordinary reconnect uses (Requirement 1.11), and the placeholder must carry a valid, non-NULL `hostattrs_t` so that path's direct dereferences don't crash. Note this codebase already has two independent admin-bit-preserving mechanisms (`wobj.c`'s `override_worker` and `controller/worker.c`'s `rejoin_worker`, duplicating the same `XCE_WORKER_ADMIN_MASK` list) — integrate with the one `join.c` actually dispatches to, don't assume the other's semantics apply. - -## Requirements - -### Requirement 1: Full snapshot at the configuration-change boundary - -**Objective:** As an operator, I want a configuration upgrade to materialize a full state snapshot at the change boundary, so that every member adopts the same post-upgrade state rather than re-deriving it. - -#### Acceptance Criteria -1. When a member applies a committed configuration-change marker, the Controller shall materialize a full state snapshot (the "upgrade snapshot") at that marker's `(term, lsn)` boundary — apply, not commit, since `checkpoint_lsn` is already defined as the apply cursor (`last_applied_lsn`), and apply is decoupled from and can lag commit under load (`raft-apply-decoupling`). -2. The Controller shall ensure every member ends up holding the upgrade snapshot for the current marker — via that member's own local materialization once it has applied the marker itself, or via the out-of-band snapshot-install transport for a member that has not yet applied the marker (behind or freshly joining) — with both paths producing identical content per Requirement 5.1, never a mix of stale local state and a differently-sourced remote snapshot. -3. The Controller shall not require any member to re-derive post-upgrade state by replaying pre-upgrade WAL entries under the new configuration. -4. The Controller shall transfer the upgrade snapshot through the out-of-band snapshot-install path, not through WAL entry payloads. -5. The Controller shall trigger snapshot materialization synchronously upon applying the configuration-change marker (a forced checkpoint at that exact `(term, lsn)`), executed through the existing asynchronous checkpoint mechanism rather than inline within the raft apply path, so that the trigger is immediate and deterministic while the write itself never blocks a vtask handler. -6. The Controller shall include each host's administratively-set status (e.g., open/closed, locked, tagged) in the materialized snapshot, so that admin-set host state survives a configuration upgrade the same way admin-set queue status already does. -7. The Controller shall exclude dynamic, non-administratively-set host metrics (e.g., load averages, real-time resource utilization samples) from the materialized snapshot, since these naturally re-populate after restart and pinning them would make the snapshot's content non-deterministic (Requirement 5.1) for no correctness benefit. -8. Applying the configuration-change marker shall advance the Controller's apply position and trigger materialization (Requirement 1.5) but shall NOT itself change how the Controller interprets or re-homes entities (e.g., `lost_and_found` re-homing) under the new configuration — that interpretive switch is the activation step gated by Requirement 3.2, never an immediate side effect of apply. Without this separation, Requirement 3.3's "keep serving under the last-verified configuration" is incoherent: apply and activation must be distinct steps, not one. -9. The Controller shall encode the host and queue admin-state frames (Requirement 1.6) as independently-extensible Cap'n Proto structs rather than fixed raw byte layouts, so that a future admin-state field addition (e.g. a lock reason or tag) does not require another snapshot format-version migration. -10. If a host admin-state frame refers to a host with no existing entry in the local worker table at restore time (a member that has not yet locally replayed that host's original join record), the Controller shall create a minimal placeholder entry (unjoined, dead-until-connected) to carry the admin status, rather than failing the restore or discarding the host's admin state — mirroring the existing `makeWorker()`-without-join pattern already used for the synthetic `lost_and_found` sentinel host, never inventing a parallel mechanism. -11. When the physical worker for a placeholder host (Requirement 1.10) later joins for the first time, the Controller shall route that join through the same rejoin path (`rejoin_worker`) an already-known host uses — never the first-time-join path (`init_worker`) — so the placeholder's restored admin-status bits are preserved exactly as `rejoin_worker` already preserves them across an ordinary disconnect/reconnect (`worker.c:374,390`). The placeholder shall therefore be created with a valid, non-NULL `hostattrs_t` (even if minimal/zeroed): `rejoin_worker`'s `apply_join_state` step directly dereferences the existing attrs struct with no null check (`worker.c:376`), and the underlying `copy_worker` allocation path (`wobj.c:51-81`) itself requires a non-NULL source `attrs` — omitting it crashes or `-EINVAL`s on the very first real join, not merely on the placeholder-creation call. - -### Requirement 2: Configuration embedded in the upgrade snapshot - -**Objective:** As an operator, I want the new configuration delivered together with the materialized state, so that a member never holds state and configuration from different points. - -#### Acceptance Criteria -1. The Controller shall embed the new configuration in the upgrade snapshot so that installing the snapshot delivers configuration and materialized state as one consistent artifact. -2. The Controller shall make this upgrade snapshot supersede the interim out-of-band config-push byte delivery (`ha-term-safety` Requirement 7) for the upgrade case. -3. The Controller shall keep the configuration bytes out of WAL entry payloads (the WAL holds only the `(term, lsn)` configuration-change marker). -4. The Controller shall embed the currently active configuration in every checkpoint taken after a configuration change, not only the marker-triggered upgrade checkpoint, so that a member joining or catching up after the upgrade snapshot has been rotated away still receives configuration and materialized state together from whichever checkpoint it installs. - -### Requirement 3: Activation gating - -**Objective:** As an operator, I want the serving primary — whichever member that is, self-promoted or freshly elected — to never serve client requests under a configuration whose materialized snapshot it has not itself completed and verified, so that Raft's log-completeness guarantee (who CAN be elected) is never mistaken for materialization-completeness (who SHOULD start serving). - -#### Acceptance Criteria -1. The Controller shall rely on Raft's existing log-freshness election gate (a candidate's log must be at least as up-to-date as the voter's) to prevent a member missing the committed configuration-change marker from being elected; this spec introduces no separate promotion-eligibility mechanism for WAL/log completeness. -2. Regardless of whether a member authored the configuration-change marker itself or inherited it via replication and a later election, the Controller shall not let that member serve client requests under the new configuration until its own upgrade-snapshot materialization for the current marker has durably completed and the embedded configuration has been hash-verified against the marker. -3. While its own materialization for the current marker is incomplete, a member that is otherwise the serving primary shall continue serving under its own last-verified configuration rather than stopping service, consistent with the never-self-terminate / stay-alive invariant (`controller/CLAUDE.md`). -4. If the embedded configuration does not hash-verify against the marker, the Controller shall keep serving under the last-verified configuration (or remain in a loud non-serving fence if none exists yet) and continue retrying materialization rather than activating mismatched configuration. - -### Requirement 4: Post-upgrade consistency invariant - -**Objective:** As a client and operator, I want every member identical after a configuration upgrade, so that an upgrade can never silently diverge materialized state across members. - -#### Acceptance Criteria -1. When members complete a configuration upgrade, every member's materialized state shall be identical. -2. When members complete a configuration upgrade, every member's active configuration shall be identical. -3. While configuration upgrades interleave with failover, the Controller shall preserve the post-upgrade identical-state invariant across leadership changes. -4. If a second configuration-change marker commits before every member has completed installing the previous upgrade snapshot, the Controller shall have the new marker's materialization (Requirement 1.1) cleanly supersede the previous upgrade's in-flight distribution, never leave it partially in-flight or conflated with the new one. - -### Requirement 5: Crash-safety and idempotent resumption - -**Objective:** As an operator, I want a configuration upgrade to survive a controller crash or leadership change at any point, so that correctness after failure is never traded away for upgrade progress. - -#### Acceptance Criteria -1. The Controller shall make the upgrade snapshot's content a deterministic function of the configuration-change marker's `(term, lsn)` and the embedded configuration's content hash, so that independent materialization by any member — including a newly elected leader — yields identical content. -2. If a member crashes after the configuration-change marker commits but before that member's upgrade snapshot write completes durably, then upon restart the Controller shall resume materialization from the marker rather than serving under the new configuration with an incomplete or absent snapshot. -3. The Controller shall guard the forced upgrade checkpoint with the same single-in-flight-checkpoint idempotency mechanism used by periodic checkpoints, so that a post-crash retry or a concurrent periodic checkpoint cannot corrupt or partially overwrite the upgrade snapshot. -4. If leadership changes after the configuration-change marker commits but before every member has installed the upgrade snapshot, then the new leader shall resume driving materialization and distribution rather than treating the upgrade as complete or abandoning it. -5. Under no sequence of crashes, restarts, or leadership changes shall a member serve under the new configuration without having durably installed and hash-verified the matching upgrade snapshot (a failure-mode restatement of Requirement 3.2). -6. If the out-of-band snapshot-install transport would deliver a snapshot encoded in a format-version the receiving member's own binary does not recognize (e.g. a member not yet restarted onto the new binary during a rolling software upgrade), the Controller shall reject the install and fail loud rather than partially installing or misinterpreting the bytes, and shall retry once the receiving member is running a compatible binary/format — consistent with the never-self-terminate / stay-alive invariant (`controller/CLAUDE.md`). -7. After the host/queue admin-state frame migration to Cap'n Proto (Requirement 1.9), the Controller shall reject a snapshot encoded in the prior raw queue-frame layout with the same fail-loud, non-corrupting behavior as any other format-version mismatch, never silently reinterpreting old bytes under the new layout. -8. If a prior restore attempt already created a placeholder host entry (Requirement 1.10) before failing partway through the snapshot, a retried restore shall find and reuse that placeholder rather than treating its existence as a conflict — the same idempotent-replay guarantee `restore_job`'s `-EEXIST`-is-success handling already provides for jobs. - -### Requirement 6: Reproduction and regression - -**Objective:** As a maintainer, I want the bug-857 divergence reproduced and guarded, so that the fix is proven. - -#### Acceptance Criteria -1. When the regression scenario interleaves a configuration upgrade with chaotic multi-kill failover, the test shall assert that every member's materialized state and active configuration are identical afterwards. -2. The test shall reproduce the bug-857 materialized-state divergence on the pre-fix code (config-push + per-node replay) and pass on the fixed code. -3. The test shall inject a controller crash/restart and a leadership change between configuration-change marker commit and upgrade-snapshot-install completion, and assert the post-upgrade consistency invariant (Requirement 4) and the crash-safety invariant (Requirement 5.5) still hold. -4. The regression test shall explicitly assert that a host's administratively-set status (open/closed/locked) survives the configuration-upgrade-plus-chaotic-failover scenario, not merely rely on the aggregate materialized-state equality check (Requirement 6.1) to imply it. - -### Requirement 7: Regression coverage for the host-placeholder, overlapping-upgrade, format-mismatch, and WAL-trim mechanisms - -**Objective:** As a maintainer, I want the new mechanisms this spec introduces — host-placeholder creation and rejoin, overlapping upgrades, format-version mismatch handling, and the marker-triggered checkpoint's interaction with WAL-segment trim — exercised by their own regression tests, so they cannot silently regress the way bug-857 did before it had one. - -#### Acceptance Criteria -1. The test suite shall exercise a host admin-status change (e.g. close) issued while that host is down, followed by a configuration upgrade whose snapshot restores on a member with no local record of that host's join, followed by the host's real first join, and assert the admin status is preserved through the placeholder-then-rejoin path (Requirements 1.10, 1.11). -2. The test suite shall exercise a second configuration-change marker committing before every member has finished installing the previous upgrade snapshot, and assert the previous upgrade's distribution is cleanly superseded rather than left partially in-flight or conflated (Requirement 4.4). -3. The test suite shall assert that a snapshot encoded in a format-version the receiving binary does not recognize — whether from a rolling binary upgrade (Requirement 5.6) or the prior raw queue-frame layout (Requirement 5.7) — is rejected fail-loud, never partially installed or silently reinterpreted. -4. The test suite shall exercise the marker-triggered forced checkpoint's WAL-segment trim (Requirement 1.1) while a standby is genuinely mid-catch-up (has not yet acked the marker's LSN), and assert that standby's still-needed WAL segments are not stranded — the existing retention-floor guard (`repl_trim_lsn`/`trim_wal_segments`, bug-1147) must hold identically under this new trigger path as it already does under the periodic WAL-threshold trigger, including under the chaotic multi-kill conditions Requirement 6 exercises. diff --git a/.kiro/specs/config-upgrade-snapshot/research.md b/.kiro/specs/config-upgrade-snapshot/research.md deleted file mode 100644 index 8e6b4c4f..00000000 --- a/.kiro/specs/config-upgrade-snapshot/research.md +++ /dev/null @@ -1,422 +0,0 @@ -# Gap Analysis — config-upgrade-snapshot (2026-06-24) - -## Scope - -Make a configuration upgrade install one identical, **entity-bearing** state snapshot -(with the new config embedded) on every member at the config-change marker's -`(term, lsn)` boundary, replacing today's config-push + restart + per-node WAL -replay (which can materialize divergent state — **bug-857**). Closes bug-857. - -> **GATING BLOCKER (hard dependency): `entity-bearing-snapshots`.** -> Today snapshots carry **zero entity state** (an LSN sentinel). This spec -> *consumes* an entity-bearing snapshot it cannot produce. It **cannot reach a -> testable GREEN until `entity-bearing-snapshots` lands or is co-scoped.** Every -> "Missing" below traces back to this one absence. See §"Gating Blocker". - ---- - -## Requirement-to-Asset Map - -### R1 — Full snapshot at the config-change boundary - -| Need | Existing asset (file:line) | Gap | -|------|----------------------------|-----| -| Materialize a snapshot at the marker's `(term,lsn)` | `persist_checkpoint` / `write_snapshot` (`persist.c:1247-1291`) writes a checkpoint snapshot at `checkpoint_lsn` | **Missing** — snapshot is a 16-byte sentinel (`"XCESNAP\0"` + LSN + optional 41-byte config-ref tail); carries no entities. No way to pin it to the marker boundary vs. an arbitrary checkpoint LSN. | -| Snapshot pins identical materialized state | dual-zarray COW: `xzarray_freeze`/`xzarray_thaw` (`zarray.c:695-757`), `test_zarray_cow.c` | **Missing (dormant)** — built + unit-tested but **no production callers**: `kvmgr.c:297` `vfreeze=NULL /* wired in P0-2 */`, `vthaw=NULL`. | -| No member re-derives via replay | upgrade path `do_upgrade_config_with_bytes` (`config_repl.c:1495-1677`); NOTICE at step 9 (`config_repl.c:1641-1655`) | **Constraint (current behavior is the bug)** — step 9 sets `restart_requested`; on restart each node replays its own pre-upgrade WAL under the new config (config-dependent via `lost_and_found`). This is exactly what R1.3 forbids. | -| Transfer out-of-band, not in WAL payloads | `ha_handle_snapshot_chunk` (`ha.c:2424-2515`) + `persist_load_snapshot` (`persist.c:1594`); `OPCode_snapshotTransfer` dispatch-gated (`ha.c:2842/2874/2962`) | **Exists (reusable transport)** — chunked, size-checked, atomic `snap.part`→rename, then `persist_load_snapshot`. Reusable as-is; it only ever moves a sentinel today. | - -### R2 — Configuration embedded in the upgrade snapshot - -| Need | Existing asset (file:line) | Gap | -|------|----------------------------|-----| -| Embed new config in snapshot (one artifact) | snapshot config-ref tail (`persist.c:1263-1274`, read-back `1581-1590`): `{present, version_tag(8), sha256(32)}` | **Partial / Missing** — only a `{tag, sha}` **reference** is embedded, not the config **bytes** and not the entity state. Need the actual config bytes + entities in one installable artifact. | -| Supersede ha-term-safety R7 config-push for upgrades | R7 out-of-band byte push driven from `config_commit_change` (`config_repl.c:478-510`) | **Missing** — upgrade still uses the R7 interim byte-push path; no branch that delivers config via the snapshot instead. | -| Keep config bytes out of WAL payloads | `WAL_CONFIG_MARKER` is `{version_tag, sha256}` only — `config_marker_encode` (`config_repl.c:40-69`), bytes never in WAL | **Already satisfied** — marker carries no bytes; invariant holds today. | - -### R3 — Activation gating - -| Need | Existing asset (file:line) | Gap | -|------|----------------------------|-----| -| Don't serve/activate new config until snapshot installed | install-before-activate already structural in `ha_handle_snapshot_chunk` (rename+load on full receipt) | **Partial** — transport installs before load, but there is **no activation gate** tying "config active" to "upgrade snapshot for this `(term,lsn)` installed." Today activation = restart-then-replay. | -| Verify embedded config vs marker content hash | `config_committed_marker` (`config_repl.c:216`) yields `{tag, sha}`; hash-verify-before-activate already the rule for R7 byte push (steering `ha-async-io.md` §per-entry-term) | **Missing for snapshot** — the verify-against-marker logic exists for the byte-push artifact; must be extended to the snapshot-embedded config. | -| Reject + keep retrying on mismatch (no self-terminate) | loud-fence-and-stay-alive invariant (`controller/CLAUDE.md` "Process Lifecycle") | **Constraint** — reject path MUST be a loud serve-fence + retry, never `exit`/`abort`. New code must honor this. | - -### R4 — Post-upgrade consistency invariant - -| Need | Existing asset (file:line) | Gap | -|------|----------------------------|-----| -| Identical materialized state across members | none — no entity snapshot to compare | **Missing** — depends entirely on R1/R2 landing. | -| Identical active config across members | `config_committed_marker` derives committed config from `(term,lsn)` | **Partial** — config identity is reachable via the marker; **state** identity is not. | -| Invariant holds across upgrade × failover interleave | Raft `(term,lsn)` reconcile + promotion-truncate (`ha-term-safety`, `raft.c`) | **Unknown** — needs design: how a mid-upgrade snapshot interacts with promotion-truncate and catch-up streaming when a leader dies mid-install. | - -### R5 — Reproduction and regression - -| Need | Existing asset (file:line) | Gap | -|------|----------------------------|-----| -| Reproduce bug-857 divergence (RED) | `waldump` (`testsuite/waldump.{c,h}`), `/xce-test-ha` chaotic-multi-kill soak, planned `snapdump`/`walcompare` (roadmap Hardening §) | **Partial** — waldump exists; `snapdump`/`walcompare` (entity-less-snapshot + WAL-divergence detectors) are **unbuilt** and are the natural assertion tools here. | -| Pass on fixed code | — | **Missing** — fix does not exist yet. | - ---- - -## Gating Blocker — entity-bearing-snapshots (hard, explicit) - -This is the single dependency that gates the whole spec. - -- **Evidence it's absent:** `write_snapshot` (`persist.c:1247`) and `load_snapshot` - (`persist.c:1555`) handle only magic + LSN + a `{tag,sha}` config-ref tail — - **no job/queue/host/resource entities**. The header comment literally says - *"Full Cap'n Proto serialization is Task 11 when recovery needs to read it back."* -- **Why it blocks:** R1.1, R2.1, R4.1 all require an artifact that carries - materialized entity state. Without it, "install the same snapshot on every member" - delivers an empty sentinel — divergence is unavoidable. -- **What's already paid down (de-risks it):** the COW substrate exists - (`xzarray_freeze`/`thaw`, `zarray.c:695-757`, O(1) freeze, GC-filtered thaw, - unit-tested) and the **transport** is reusable (`ha_handle_snapshot_chunk` + - `persist_load_snapshot`). What's missing is the **payload**: serialize entities - under per-object locks during the freeze window into the snapshot file, and - deserialize them on `load_snapshot`. COW is single-generation/slot-level only — - one consumer owns freeze/thaw (roadmap §entity-bearing-snapshots). - -**Consequence for this spec:** requirements/design can proceed, but **tasks cannot -reach implementation** until `entity-bearing-snapshots` is implemented or -co-scoped. `spec.json.blocked_on` already records this. - -Also depends on **`ha-term-safety` R7** for the `(term,lsn)` config-change marker -that the snapshot anchors on — that marker machinery **is implemented** -(`config_commit_change`, `config_committed_marker`, persisted config-ref). - ---- - -## Implementation Approach Options - -### Option A — Extend the existing checkpoint/upgrade path - -Wire `vfreeze`/`vthaw` into `kvmgr`, teach `write_snapshot`/`load_snapshot` to -serialize/deserialize entities, and in `do_upgrade_config_with_bytes` (after the -marker commits, before restart) checkpoint an entity-bearing snapshot at the -marker `(term,lsn)`, embed the config bytes, stream it via `snapshot_transfer_s`, -and gate activation on install. Drop the per-node-replay restart for the upgrade case. - -- ✅ Reuses the live transport + COW; minimal new files; one consistent snapshot format. -- ✅ Naturally supersedes the R7 byte-push for upgrades. -- ❌ Bloats `persist.c` snapshot codec and the already-large `config_repl.c` upgrade path. -- ❌ Couples the generic checkpoint snapshot format to upgrade-specific embedding - (config bytes) unless cleanly factored. - -### Option B — New "upgrade snapshot" component - -A dedicated module owns: materialize-at-marker, embed config, drive transport, -activation gate, and the mismatch fence — leaving `persist.c` to expose only an -entity (de)serialize primitive and `config_repl.c` to call into it. - -- ✅ Clean separation; upgrade-specific logic isolated and unit-testable. -- ✅ Keeps `config_repl.c`/`persist.c` from growing further. -- ❌ More wiring across persist ↔ new module ↔ ha transport ↔ config_repl. -- ❌ Risk of a second snapshot-format path diverging from the generic checkpoint one. - -### Option C — Hybrid (recommended) - -1. **Foundation (in/with `entity-bearing-snapshots`):** wire COW into kvmgr and add - the generic entity (de)serialize to `write_snapshot`/`load_snapshot` — *Option A - style*, shared by recovery + state transfer. -2. **Upgrade layer (this spec):** a thin new helper (Option B style) that, given an - entity-bearing snapshot capability, materializes at the marker `(term,lsn)`, - embeds the config bytes, streams via the existing transport, and adds the - activation gate + hash-verify-vs-marker + loud-fence-on-mismatch. Replace the - step-9 restart/replay for the upgrade case; keep restart/replay as the fallback - only until the foundation lands. - -- ✅ Generic snapshot work is reused (recovery, faster rebuild) — not upgrade-only. -- ✅ Upgrade-specific gating/verification stays isolated and testable. -- ✅ Phased: foundation can land + be validated before the upgrade layer flips on. -- ❌ Requires careful coordination across the two specs (format ownership, ordering). - ---- - -## Effort & Risk - -| Dimension | Rating | One-line justification | -|-----------|--------|------------------------| -| **This spec alone** (assuming entity-bearing snapshots exist) | **L (1–2 wks)** | Transport + marker + COW exist; new work is materialize-at-marker, config embed, activation gate, hash-verify, mismatch-fence, and the failover-interleave regression. | -| **Including the gating dependency** | **XL (2+ wks)** | `entity-bearing-snapshots` is itself non-trivial (entity serialization under per-object locks, COW wiring, recovery read-back) and must land first. | -| **Risk** | **High** | Hard blocker not yet started; touches consensus-adjacent state at a `(term,lsn)` boundary interleaved with failover (R4.3); snapshot/promotion-truncate/catch-up interaction is unverified; must preserve the never-self-terminate invariant. | - ---- - -## Recommendations for Design Phase - -**Preferred approach:** **Option C (Hybrid)** — land entity-bearing snapshots as a -shared foundation (COW into kvmgr + generic entity (de)serialize in -`write_snapshot`/`load_snapshot`), then add a thin upgrade layer that materializes -at the marker, embeds the config, reuses `snapshot_transfer_s`, and adds activation -gating + hash-verify + loud-fence. - -**Key decisions to settle in design:** -1. **Spec sequencing** — co-scope `entity-bearing-snapshots` into this spec's - delivery, or hard-gate this spec behind it? (Recommend gate; promote to design - only once the foundation has a tasks plan — matches `roadmap.md`.) -2. **Snapshot format** — one unified entity-bearing format used by both recovery and - upgrade, with the config-bytes embed as an optional section? Avoid a second format. -3. **Materialize-at-marker semantics** — snapshot must reflect state at *exactly* the - marker `(term,lsn)`, not "latest checkpoint." Define how the freeze window aligns - to the committed marker LSN. -4. **Activation gate** — bind "new config active" to "upgrade snapshot for this - `(term,lsn)` installed + hash-verified vs `config_committed_marker`"; mismatch ⇒ - loud serve-fence + retry (never exit — `controller/CLAUDE.md`). -5. **Retire the per-node restart/replay** for the upgrade case (step 9, - `config_repl.c:1657-1676`) once the snapshot path is proven. - -**Research-Needed (carry forward):** -- Upgrade × failover interleave (R4.3): how does a mid-upgrade entity snapshot - interact with promotion-truncate, `(term,lsn)` reconcile, and ACK-driven catch-up - if the leader dies mid-install? Define the recovery story. -- COW single-generation constraint vs. a checkpoint racing an upgrade snapshot - (only one freeze/thaw consumer allowed — `zarray.c` / roadmap note). -- Build `snapdump` + `walcompare` (roadmap Hardening §) first — they are the RED/GREEN - assertion tools for R5 (entity-less snapshot detection + cross-node WAL/state diff). - ---- - -## Update — 2026-07-21 (dependencies resolved; requirements extended; harness gap analysis) - -This section is current as of 2026-07-21 and supersedes the stale parts of the -2026-06-24 analysis above (kept for history). Do not re-derive the gating -status or tool inventory from the sections above without reading this first. - -### Gating blocker — RESOLVED - -- **`entity-bearing-snapshots`** — satisfied 2026-07-09 by `ha-entity-snapshots` - (21/21 tasks, fully approved). It shipped `write_snapshot`/`load_snapshot` - (`persist.c`) as a direct Cap'n-Proto entity-encode/decode of job and queue - records at checkpoint time — **not** the COW `xzarray_freeze`/`thaw` path this - research doc's Option A/C assumed (that machinery is still dormant, - `kvmgr.c:297`). Design must build on the real `write_snapshot`/`load_snapshot` - interface, not the COW one. -- **`ha-term-safety`** (owns the `(term,lsn)` config-change marker this spec - anchors on) — `implementation-complete`, GO-validated 2026-07-17, 46/46 - acceptance criteria mapped, live chaotic soak clean. -- Net effect: this spec is **unblocked**. `spec.json.blocked_on` should be - cleared and the phase can move to design once requirements are approved. - -### Tool inventory — CORRECTED (contradicts the "unbuilt" note above) - -`snapverify`, `snapdiff`, and `walcompare`/`walcompare_report` all **already -exist** (`src/controller/testsuite/`), built 2026-07-02 during bug-1386/1389 -triage. The line-57/176 framing above ("planned", "unbuilt", "build first") is -stale — do not re-propose building them. - -- `snapverify_path` — structural-only checkpoint-file validator (magic, - format_version, frame walk to EOF). Reports `has_cfgref` as a bool but does - **not** decode or assert the tag/hash value. -- `snapdiff_paths` — byte-for-byte comparator between two checkpoint files, - with chunk-boundary diagnosis. Built for source-vs-received comparison but - directly reusable for a determinism check (two independently materialized - snapshots at the same boundary should be byte-identical). -- `walcompare`/`walcompare_report` — cross-member committed-prefix divergence - detector, compares `(term, pcrc)` per LSN, and already has a **per-member - checkpoint floor** concept (an LSN below a member's own installed-snapshot - floor is "legitimately compacted," never flagged divergent — `ha-entity- - snapshots` task 7.1 / `raft-ha-core-safety` task 5.10a). It does **not** - compare active configuration across members. - -### Requirements extended this session (2026-07-21) - -`requirements.md` gained content not reflected in this doc's "Recommendations" -section (items 1-5 above predate these): -- **Req 1.5** — snapshot materialization is triggered synchronously by - marker-apply but executed through the *existing* async checkpoint mechanism - (`persist_checkpoint`/`schedule_checkpoint`/`write_snapshot`, delaymgr-driven, - CAS-guarded single-in-flight), not inlined into the raft apply path — answers - the "should this be one atomic Raft FSM operation" question: the marker - commit already is one FSM op; materialization should be triggered by it but - run via the existing decoupled checkpoint mechanism (never block a vtask - handler); cross-node install must never be — it's inherently async. -- **Req 2.4** — every checkpoint taken after a config change embeds the - *current* config, not only the marker-triggered one, so a member joining - after the upgrade snapshot has rotated away (`rotate_snapshots`, `snap_keep_n`) - still gets config + state together. Piggybacks on the existing - `scan_latest_committed_config_ref` (`persist.c:1999`), which already re-stamps - the latest committed config *reference* into every checkpoint — Req 2 extends - that from a reference to the full embedded bytes. -- **Req 5 (new)** — Crash-safety and idempotent resumption: deterministic - snapshot content (any member's independent materialization at the same - `(term,lsn,config-hash)` is identical), crash-safe resumption if a member - dies mid-materialization, idempotency guard reusing the existing - `checkpoint_in_progress` CAS, and leader-change-mid-distribution resumption. -- **Req 6.3 (was Req 5.2)** — regression test must inject a crash/restart and a - leadership change specifically between marker-commit and snapshot-install- - complete, not just rely on general chaotic-kill timing. - -### Invariant risk carried forward from this session's controller-invariant check (2026-07-21) - -Checked the new requirements (esp. Req 3's activation gate) against -`controller/CLAUDE.md`'s TLA+ invariants and Protocol Verification Checklist. -One real precedent-risk, already fixed at the requirements level (Req 1.8) but -flagging for design/implementation attention: - -- **bug-1418 (open, NOT FIXED)** is the same failure shape this spec's Req 3 - risks reintroducing in a new flavor. bug-1418: a newly-elected primary whose - *apply* is behind (`apply_divergent=1`) hits a **liveness deadlock** — the - existing promotion gate blocks it hard, repeatedly, rather than degrading - gracefully; the (unimplemented) proposed fix is "become primary, self-apply, - then serve." Req 3.2/3.3 of this spec assume exactly that graceful pattern - for *materialization*-lag instead of *apply*-lag. Since the apply-lag version - of this pattern is proven broken in production today, design must not - implement Req 3's gate as a hard promotion block (Req 3.1 already commits to - this — no separate promotion-eligibility mechanism) and must verify the - "keep serving under last-verified config" fallback (Req 3.3) is genuinely - implemented, not degrade into the same stall bug-1418 exhibits. -- Root cause of the *need* for Req 1.8 (apply must not itself re-home entities; - activation is a separate, later-gated step): without that split, Req 3.3 - ("keep serving under last-verified configuration" while materialization is - incomplete) is incoherent — there would be no old interpretation left to - serve under once apply had already mutated live entity state. This is the - same family of apply-ordering subtlety behind bug-1417/1418/1419. -- **Design-verification item (unconfirmed, not yet checked in code):** does - `persist_load_snapshot`/`load_snapshot` advance `durable_lsn` (not just - `checkpoint_lsn`/`current_lsn`) when a member catches up via the transport- - install path (Req 1.2's second branch)? If not, a snapshot-caught-up member - may not correctly count toward commit majority for LSNs at/below the - snapshot boundary even though it durably holds that state via the snapshot - file rather than raw WAL — a possible Commit Durability invariant gap - specific to the transport-install path. Verify in design before assuming - Req 1.2's two paths are equally safe. - -### Harness gaps for design phase (new, from this session's analysis) - -Three items, only two are genuinely new build work: - -1. **No cross-member config comparator (new gap).** `walcompare` diffs WAL only. - Req 4.2 ("every member's active configuration identical") has no assertion - tool. Extend `walcompare` (or add a sibling) to decode+compare the embedded - config tag/hash (and eventually bytes) per member, mirroring how it already - compares `(term,pcrc)`. Also extend `snapverify` to decode/assert the - cfgref tag+hash value, not just report its presence. -2. **No deterministic fault-injection at the marker-apply → forced-checkpoint- - durable window (new gap, needed for Req 5.2/6.3).** `/xce-test-ha --chaotic` - kills at random points — could hit this narrow window eventually, too flaky - for a CI regression gate. Precedent for the right shape of hook already - exists twice in `config_repl.c`: `ha_stream_body_read_fault_hook`, - `xce_snapshot_chunk_hook` (both `#ifdef XCE_TESTING` seams). Add an - equivalent hook at the marker-apply/forced-checkpoint boundary. -3. **No deterministic leadership-change-mid-distribution injection (new gap, - Req 5.4/6.3).** Same precedent pattern as #2 — a targeted "kill/step-down - primary exactly after marker commits, before all standbys ACK install," - not reliance on chaos-soak probability. -4. **Reusable, no new work needed:** `snapdiff_paths` for the Req 5.1 - determinism check; `/xce-test-ha --force-snapshot` for Req 2.4's - fresh-join-after-rotation (already exercises wiped-node-rejoins-empty via - bug-1386's own test mode — just needs its assertions extended once config - embedding lands); `config_commit_change`/`do_upgrade_config_with_bytes` - (bug-970) already wires `WAL_CONFIG_MARKER` into the live upgrade path, so - the bug-857 repro (Req 6.2) is triggerable today with existing tooling — - no blocker there. - ---- - -## Update — 2026-07-23 (design discovery; requirements extended to 7 requirements/36 ACs) - -Requirements went through several rounds of review this session (host/queue admin-state frames, bug-1418 re-verified fixed, bootstrap-vs-upgrade boundary, Design Invariants section, Requirement 7 added for new-mechanism regression coverage). This update captures the DESIGN-level discovery done to write `design.md`, current as of 2026-07-23, extension-scope (light discovery — existing system, no new external dependency). - -### Discovery: exact hook points for the marker → materialization trigger (Req 1.1, 1.5, 1.8) - -**The single unified apply seam is `apply_committed_entry`** (`persist.c:2751-3132`), confirmed as the ONLY live path for entity mutation on every member — primary online-apply (`persist_apply_pump_run` → `persist_apply_pump_kick`, vtask-pool dispatched, `persist.c:3383`), standby normal apply (`ha_standby_advance_committed_and_apply`, heartbeat-learned commit position, `ha.c:2596-2613/2688-2700` — supersedes the older "apply on receipt" bug-1043 fix as of 2026-07-10), and boot/restart replay (`persist.c:3920`). The sibling dispatcher `apply_wal_entry` (`persist.c:2583-2714`) has **zero live callers** (grep-confirmed) — fully superseded, dead code, not a second seam to worry about. `WAL_CONFIG_MARKER` currently falls into `apply_committed_entry`'s generic `default:` no-op (`persist.c:3126-3131`) and `apply_wal_entry`'s equivalent no-op (`persist.c:2697-2707`, irrelevant given no callers). **Design decision: add one new explicit `case WAL_CONFIG_MARKER:` in `apply_committed_entry` only** — it fires identically for primary and standby, satisfying Req 1.2's "every member... via that member's own local materialization once it has applied the marker itself" for free, with no per-role branching needed. - -**Today's restart+replay behavior being replaced** lives in `do_upgrade_config_with_bytes` (`config_repl.c:1591-1773`, step 9, `config_repl.c:1753-1772`): `atomic_store(&c->restart_requested, 1)` + `cbuf_put(&c->wbuf, &stop_val)`. An existing NOTICE comment at `config_repl.c:1737-1751` already names this exact spec by path and describes the target end-state verbatim — a strong signal the upgrade path's own maintainers already anticipated this design. No function-pointer hook exists at that NOTICE site (grepped, none) — it is a plain fall-through, confirming the real trigger point is the `apply_committed_entry` case above, not this step-9 comment site itself. **Once the new case lands, step 9's restart/replay becomes dead code for the upgrade path and should be removed** (not left as a parallel fallback — Design Invariant "never leave two mechanisms doing the same job"). - -### Discovery: checkpoint triggering — no forced/immediate entry point exists yet (Req 1.5) - -`persist_checkpoint()` (`persist.c:2262-2321`) is unconditionally synchronous/blocking (fdatasync-backed `write_snapshot`, directory scans) and has exactly one production trigger path today — `schedule_checkpoint()` (`persist.c:2238-2256`), fired only by a WAL-record-count threshold inside `persist_wal_async()` (`persist.c:921-928`). No wall-clock periodic timer exists in the live code (an older design doc's `checkpoint_requested`/`persist_checkpoint_timer_fn` was never implemented — grep-confirmed absent). **No `force_checkpoint()`/`persist_checkpoint_now()` entry point exists** — but a second direct, unconditional call to `persist_checkpoint()` already exists at `ha.c:3308` inside `ha_snapshot_do_send()` ("Req 4.1: create a FRESH checkpoint..."), proving the function itself already tolerates being force-invoked outside the record-count path (it already handles the `-EBUSY` a concurrent scheduled checkpoint can return, via the shared `checkpoint_in_progress` CAS, `persist.h:415`). - -Two existing patterns already solve "trigger without blocking the caller" in this codebase — `delay_task`/delaymgr enqueue (what `schedule_checkpoint` does; the *actual* checkpoint I/O then runs on the delaymgr thread itself, per direct trace of `delay_worker`/`run_delay_tasks`, `delay.c:60`/`cluster.c:988-991` — delaymgr has no thread-pool hand-off) and the dedicated **slow-lane executor** (what `ha_snapshot_do_send` uses, `ha.c` ~6120-6160, specifically because `persist_checkpoint` is blocking/unbounded — this is the pattern `ha-async-io.md:29` documents as the bug-1198 fix for inline-on-iomgr checkpoint I/O). **Design decision: the new `WAL_CONFIG_MARKER` case in `apply_committed_entry` calls `persist_checkpoint()` directly, not via either existing indirection.** This is safe specifically because `apply_committed_entry` (via the apply pump) already runs on its own vtask-pool worker thread, never the iomgr or delaymgr thread — it IS the async-dispatched context, so no further hand-off is needed for THIS call site (unlike `schedule_checkpoint`'s WAL-threshold trigger, which fires from arbitrary vtask handlers appending WAL entries and genuinely needs delaymgr's async return). This gives Req 1.5's "immediate and deterministic" trigger for free — no new delay/lane plumbing required — while the CAS (`checkpoint_in_progress`) already guards against a racing periodic checkpoint. - -**Note for design (not a requirements gap, an implementation-order note):** `scan_latest_committed_config_ref`/`write_snapshot`'s existing `persist_checkpoint` call sequence (`persist.c:2301/2305`) always stamps `checkpoint_lsn = last_applied_lsn` (`persist.c:2292-2295`) — i.e. a marker-triggered forced checkpoint calling `persist_checkpoint()` at the moment `apply_committed_entry` handles the marker will naturally pin `checkpoint_lsn` to the marker's own `(term,lsn)`, satisfying Req 1.1's "materialize at that marker's boundary" without any new LSN-targeting parameter — `persist_checkpoint()` doesn't need to accept an explicit target LSN; it just needs to run at the right moment, which the new apply-time case guarantees by construction. - -### Discovery: the activation gate has a direct, reusable precedent (Req 3) - -`ha_dispatch_gate` (`ha.c:3809-3975`, single call site `cluster.c:276`, gating every wire request — client and peer alike, both HA replication opcodes and apiserver-relayed job/queue/host opcodes) already composes multiple independent boolean fences in sequence for the `CTRL_PRIMARY` role: role check, then a **config-serve-fence check** (`config_serve_fenced(ha->raft_c)`, `ha.c:3827`), then a write-quorum check. This is a **direct structural precedent already in production** for exactly what Req 3 needs — a config-dependent, state-changing-request fence that is orthogonal to role/quorum, with its own opcode allowlist (replication + read-only ops stay admitted while fenced). - -The existing `config_serve_fence` (`ha.h:307-322`, set/cleared by `config_activate_committed` in `config_repl.c:304-377`, hash-verifies committed marker bytes against locally-held config bytes) is the closer of the two fence precedents found (the other being `recovery_unrecoverable`, which only gates *promotion*, not live serving) because it already: (i) blocks client-request serving via `ha_dispatch_gate`, (ii) also blocks promotion (`ha.c:1425-1440`, `ha_config_promotion_fenced()`), and (iii) self-clears via a fully-wired, non-blocking retry loop (`config_schedule_activate_retry` → delaymgr trampoline → vtask does the actual blocking verify I/O, `config_repl.c:415-507`), triggered from the commit-advance callback (`ha.c:2197-2198`) — the exact "loud ERROR once, stay alive, keep retrying, never self-terminate" shape `controller/CLAUDE.md` mandates, already proven in production. **Design decision: add a new atomic fence field on `ha_state_t` (sibling to `config_serve_fence`, e.g. `upgrade_snapshot_fence`), a matching setter/verify function mirroring `config_activate_committed`'s hash-verify-then-clear shape, and one more fence check inside `ha_dispatch_gate`'s `CTRL_PRIMARY` branch** (composed the same way the existing config-serve-fence check already is — not folded into that same field, since the two conditions are logically independent: config-serve-fence is about byte-mismatch, the new fence is about materialization-not-yet-complete for an otherwise-matching config) — reusing `config_schedule_activate_retry`'s exact trampoline shape for the new fence's own retry loop, and `ha_config_promotion_fenced()`'s pattern for gating promotion the same way. - -### Discovery: durable_lsn is NOT advanced by snapshot-install — real but bounded gap (Req 1.2, 5.x) - -Traced definitively (grep + direct read of every function in the call path): `ha_handle_snapshot_chunk` (`ha.c:3014-3171`) → `persist_load_snapshot`/`load_snapshot` (`persist.c:2456-2575`) never touches `p->durable_lsn` anywhere — only `checkpoint_lsn`/`current_lsn`/`cfgref_*` get set. This is asymmetric with the boot-recovery path (`persist_recover`, `persist.c:4338-4353`), which DOES re-derive `durable_lsn` from the recovered WAL tip. **Verdict: real, confirmed gap, but bounded to this node's OWN promotion-eligibility/self-advertised-freshness bookkeeping during a narrow live-catch-up window** (`persist_durable_floor` consumers: pre-promotion gate `ha.c:1318-1352`, promotion truncate-floor clamp `ha.c:1259-1262`, self-advertised candidate position `ha.c:4540-4546`) — the existing fence-and-retry machinery (`ha_arm_promotion_fence_timer`) already handles this safely (conservative refusal to self-promote, never corruption). It is **NOT** a commit-durability/majority-counting safety gap for OTHER members counting a snapshot-caught-up peer, because `majority_candidate_locked` (`commit.c:167-172`) already excludes any standby with `in_catch_up != 0` from majority evaluation, and that flag is cleared only by a real WalAck reaching `catchup_derive_target` (`repl.c:713-739`, via `ha_walack_on_ack`) — a separate, already-safe gate keyed off real acks, not the snapshot event. **Design decision: out of this spec's scope to fix** (it predates this spec, applies to any snapshot-install today including plain recovery-driven catch-up, not upgrade-specific) — but flag it as a design.md Risk/Note so a future spec can pick it up (advancing `durable_lsn` on snapshot-install the same way boot-recovery does would close it cleanly, mirroring `persist.c:4338-4353`'s ASSIGN semantics). - -### Discovery: test tooling — three reusable as-is, two confirmed genuine new-build gaps - -Confirmed via direct read (not just trusting the 2026-07-21 update above, which was itself a research note, not a verified read): -- **`snapverify_path`** (`testsuite/snapverify.c`) — confirmed `has_cfgref` is presence-only (`header[16]`), never decodes tag/sha; zero test assertions on it exist today. Needs extending to decode/assert tag+hash for Req 4.2/Req 2 coverage. -- **`snapdiff_paths`** (`testsuite/snapdiff.c`) — confirmed reusable as-is for Req 5.1's determinism check (any two paths, no transport-specific assumptions). -- **`walcompare`/`walcompare_report`** (`testsuite/walcompare.c`) — confirmed zero references to `cfgref_tag`/`cfgref_sha`/config bytes anywhere in the file; compares only `(lsn, term, pcrc)`. Req 4.2 (active-config identity across members) has no assertion tool — needs either extending `walcompare` or a sibling tool. -- **Fault-injection precedent, exact shape confirmed**: `ha_stream_body_read_fault_hook` (`config_repl.c:2104-2106`, wired at `config_repl.c:2183-2193`) and `xce_snapshot_chunk_hook` (`config_repl.c:2847`, wired at `config_repl.c:2937-2949`) are both `#ifdef XCE_TESTING`-gated, `NULL`-by-default, file-scope function-pointer globals, `extern`-declared and set/reset around the driving call in the test file. This is the exact shape the two NEW deterministic hooks (Req 7.2's marker-apply→forced-checkpoint-durable window, Req 7.2/5.4's leadership-change-mid-distribution window) should mirror — a hook inside the new `apply_committed_entry` `WAL_CONFIG_MARKER` case (or wherever the cross-member install-wait loop lives) taking identifying args (marker lsn/term) so a test can force-fail/force-delay at that exact boundary instead of relying on chaotic-soak timing luck. -- **Test fixture pattern, exact shape confirmed**: `test_apply_committed_entry.c` (bare in-memory `cluster_t`, no disk, direct `apply_committed_entry()` calls, spy hooks counting `HOOK_AFTER_EVENT` fires) and `test_replay_apply_seam.c` (real disk-backed `persist_t`, two-phase write-then-recover, proving replay routes through the same seam) are the two patterns new host-frame-restore and marker-apply tests should extend — both already have a `tace_encode_worker`/`trr_encode_worker` local-capnp-encode helper (mirroring `persist.c`'s file-static `capnp_encode_worker`) that a new host-admin-frame test can copy the idiom from. - -### Design Synthesis (per `rules/design-synthesis.md`) - -**1. Generalization**: The host-admin-state frame (Req 1.6/1.9/1.10) and the queue-admin-state frame (existing, now migrated to capnp per Req 1.9) are the same underlying shape — `(name, status)` with independently-extensible fields. Rather than two hand-written near-duplicate collect/restore function pairs, design one generic `collect_admin_state_frames(table, status_mask)`/`restore_admin_state_frame(table, name, status, create_if_missing_fn)` pair parameterized over which table (`workermgr`/`qmgr`) and which create-if-missing behavior (hosts: `makeWorker`-based placeholder; queues: none, always pre-provisioned) applies — the two concrete restore functions (`restore_host_state`, `restore_queue_state`) become thin instantiations, not independent implementations. This directly follows Design Invariant 1 (classify config- vs join-provisioned) by construction: the generic function takes the classification as a parameter rather than assuming one. - -**2. Build vs. Adopt**: Every major piece here is "adopt existing internal mechanism," not "build new" — the snapshot transport (`snapshot_transfer_s`/`ha_handle_snapshot_chunk`), the checkpoint/CAS machinery (`persist_checkpoint`/`checkpoint_in_progress`), the fence-and-retry pattern (`config_serve_fence`), the placeholder-entity pattern (`lost_and_found`'s `makeWorker()` precedent), and the fault-injection hook shape (`ha_stream_body_read_fault_hook`) are all reused verbatim rather than reimplemented. The only genuinely NEW machinery is: the `WAL_CONFIG_MARKER` case in `apply_committed_entry`, the new fence field + verify function (structurally mirroring `config_serve_fence` but a distinct field), the config-bytes-embed in the snapshot header/entity payload, the `HostAdminState`/`QueueAdminState` capnp structs, and the two new test hooks. No external library or new dependency is involved anywhere in this spec. - -**3. Simplification**: Considered whether the new fence needs to be a SEPARATE atomic from `config_serve_fence` — confirmed yes (the two conditions are logically independent: hash-mismatch vs materialization-incomplete, and Req 3.4 needs to distinguish "keep serving under last-verified config" from "no verified config exists yet," which requires knowing WHICH fence is active). Considered whether `apply_wal_entry` needs a matching `WAL_CONFIG_MARKER` case for symmetry with `apply_committed_entry` — confirmed no, it has zero live callers, touching it would be pure speculative-symmetry work with no runtime effect (Design Invariant would call this exactly the "convenient but unjustified" mistake to avoid). - -### Risks carried into design.md -- The narrow window between `schedule_checkpoint`'s CAS release (`checkpoint_task_fn`, `persist.c:2222`) and `persist_checkpoint`'s own re-acquire (`persist.c:2265`) allows a second concurrent trigger to slip in and start its own `persist_checkpoint()` call — pre-existing, not introduced by this spec, but the new marker-triggered call site adds a second caller into this same race window and should be noted. -- `durable_lsn` not advancing on snapshot-install (above) — pre-existing, out of scope, flagged for a future spec. - ---- - -## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28) - -### 1. Current State Investigation - -All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles: - -| Spec | Phase | Scope | -|---|---|---| -| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` | -| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller | -| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing | - -**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.** - -**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."* - -### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker - -`admin-config-tool/spec.json`'s `blocked_on` array still lists: -> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation" - -This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity. - -- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists. -- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note). - -### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477 - -`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this. - -- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle). -- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked. - -### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration - -`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here. - -### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence - -Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways. - -### Summary Table - -| # | Finding | Classification | Effort | Risk | Action | -|---|---|---|---|---|---| -| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation | -| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable | -| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None | -| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None | - -### Research Needed (carried forward) -- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only. - -**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap. diff --git a/.kiro/specs/config-upgrade-snapshot/spec.json b/.kiro/specs/config-upgrade-snapshot/spec.json deleted file mode 100644 index e118f82a..00000000 --- a/.kiro/specs/config-upgrade-snapshot/spec.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "feature_name": "config-upgrade-snapshot", - "created_at": "2026-06-21T11:30:00Z", - "updated_at": "2026-08-15T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-08-15T00:00:00Z", - "note": "Corrected stale phase metadata (was 'tasks-generated' though every task 1-8.9 was already [x] complete). Task 8.5 reproduced bug-857 on the pre-change behavior and confirmed it no longer occurs; task 8.6's live chaotic multi-kill soak (2026-07-29) confirmed zero materialized-state/config divergence after a config upgrade. bug-857 closed as a result.", - "known_limitations": [ - "bug-1471 / bug-1472-remainder: a host admin-closed before the final checkpoint can produce host-frame byte divergence (WAL_WORKER_CTRL apply path, worker.c/persist.c) -- explicitly out of this spec's own materialization boundary per its research.md/tasks.md notes, never converted into a task here. No existing spec currently owns fixing it; flagged for a new task/spec." - ] - } -} diff --git a/.kiro/specs/config-upgrade-snapshot/tasks.md b/.kiro/specs/config-upgrade-snapshot/tasks.md deleted file mode 100644 index 10756be4..00000000 --- a/.kiro/specs/config-upgrade-snapshot/tasks.md +++ /dev/null @@ -1,197 +0,0 @@ -# Implementation Plan - -- [x] 1. Define HostAdminState and QueueAdminState Cap'n Proto structs and bump the snapshot format version - - Add `HostAdminState { name; status }` and `QueueAdminState { name; status }` messages to the controller schema, independently extensible for future admin-state fields - - Bump the on-disk checkpoint format version so any binary reading a checkpoint under the old queue-frame layout hard-rejects it rather than misinterpreting the bytes - - Observable: schema compiles, generates the new message types, and a checkpoint file written under the new version is rejected by code still expecting the old version - - _Requirements: 1.9, 5.7_ - -- [ ] 2. Core: snapshot entity and configuration payload — collection and write side -- [x] 2.1 Collect host admin-state frames into the checkpoint - - Walk the live host table the same way existing job/queue frame collection does, encoding only administratively-set status bits (open/closed/locked/tagged) per host as a HostAdminState frame - - Exclude every worker-reported, dynamically-changing field (resource values, liveness timestamps, non-admin status bits) from the encoded frame - - Observable: a checkpoint taken while a host is administratively closed contains exactly one HostAdminState frame for that host with only the admin bits set, and none of its dynamic fields - - _Requirements: 1.6, 1.7, 1.9_ - -- [x] 2.2 Migrate queue admin-state frames to the new wire format - - Replace the queue frame's fixed raw byte layout with QueueAdminState encoding, preserving the same name+status content - - Observable: a checkpoint's queue section decodes via the new message type, and a checkpoint written under the prior raw layout is rejected fail-loud rather than silently reinterpreted - - _Requirements: 1.9, 5.7_ - -- [x] 2.3 Embed the active configuration bytes into every checkpoint - - Read the locally-held active configuration bytes directly from where they are already durably saved on disk, and append them as a new checkpoint section after the host frames - - Ensure this section is populated on every checkpoint (not only a marker-triggered one), so a member joining or catching up after the marker-triggered snapshot has rotated away still receives configuration and state together - - Keep configuration bytes out of WAL entry payloads entirely — only the checkpoint file carries them - - Observable: any checkpoint file, marker-triggered or periodic, contains a configuration-bytes section whose content hash matches the currently active configuration; two independent materializations at the same marker boundary and configuration produce byte-identical checkpoint files - - _Requirements: 2.1, 2.3, 2.4, 5.1_ - -- [x] 3. (P) Restore a host admin-state frame safely regardless of whether the host has ever joined this member - - Look up the host by name in the live host table; if present, OR the restored admin-status bits into its current status - - If absent, create a minimal placeholder entry (unjoined, dead-until-connected) carrying only the restored admin-status bits, using the same host-creation mechanism the existing synthetic sentinel host already uses without requiring a live join - - Ensure the placeholder is valid enough that the host's real, later join is treated as an ordinary reconnect rather than a first-time join, so its existing admin-bit-preserving behavior applies unchanged - - Guard against a retried restore attempt re-creating or conflicting with a placeholder an earlier, partially-failed attempt already created — find first, create only if truly missing - - Observable: restoring a host admin-state frame for a host with no local record creates exactly one placeholder entry carrying the correct admin bits; restoring it again (simulating a retry) does not create a duplicate or error; when that host later joins for real, its admin status (e.g. closed) is still set afterward - - _Requirements: 1.10, 1.11, 5.8_ - - _Boundary: Host Restore (worker.c)_ - -- [x] 4. Decode host and queue admin-state frames on checkpoint load - - Add host-frame decoding to the checkpoint load path, dispatching each decoded frame to the host restore capability - - Update queue-frame decoding to read the new QueueAdminState encoding instead of the old raw layout, without changing the existing queue-state-restore behavior itself - - Preserve the existing restore discipline: every frame in the checkpoint restores successfully before any position is advanced; any restore failure leaves prior state untouched - - Observable: loading a checkpoint containing host and queue admin-state frames restores every host and queue's admin status correctly regardless of restore order between entity types - - _Requirements: 1.10_ - - _Depends: 2.1, 2.2, 3_ - -- [x] 5. Materialize the upgrade snapshot when a configuration-change marker is applied - - Add explicit handling for the configuration-change marker in the single, unified entity-apply dispatch point (the same seam every member's apply already routes through — primary, standby, and restart replay alike), replacing its current no-op treatment - - Trigger a forced checkpoint at that exact point, so materialization happens immediately and deterministically at the marker's boundary rather than waiting for a periodic trigger - - Ensure the forced checkpoint shares the existing single-in-flight checkpoint guard, so a concurrent periodic checkpoint and this one can never both proceed at once - - Treat a "checkpoint already in progress" outcome (e.g. a second configuration marker committing while an earlier one's checkpoint is still running) as an expected, self-resolving condition, not a failure to retry specially - - Ensure this trigger only materializes state — it must not itself change how entities are interpreted or re-homed under the new configuration - - Observable: applying a configuration-change marker produces a checkpoint at that marker's exact position, on both the primary and a standby, without requiring a controller restart; a second marker committing while the first's checkpoint is still running does not corrupt or lose either checkpoint - - _Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3_ - - _Depends: 2.1, 2.2, 2.3_ - -- [ ] 6. Core: activation fence gating serving and promotion on materialization -- [x] 6.1 (P) Add a materialization-and-hash-verification fence independent of the existing configuration-byte fence - - Introduce a new fence condition distinct from the existing configuration-byte-mismatch fence, since "bytes don't match" and "materialization for a matching configuration isn't done yet" are different conditions that must be distinguishable - - Verify by reading the current committed configuration-change marker and this member's own materialized-and-verified state for it, always re-reading the live, currently-active configuration bytes on disk on every verification attempt rather than trusting a previously-embedded or cached copy - - Set the fence and log the actionable mismatch loudly on failure; clear it on success; never terminate the process on this condition - - Observable: a member whose materialization for the current marker is incomplete or mismatched shows the fence set and a loud log entry naming the marker; once materialization completes and bytes match, the fence clears without any process restart - - _Requirements: 1.8, 3.2, 3.4_ - - _Boundary: Activation Fence (ha.c/ha.h)_ - -- [x] 6.2 Gate primary request-serving and promotion on the new fence - - Extend the primary request-dispatch gate with this new fence check, composed alongside the existing configuration-byte fence rather than merged into it, admitting replication and read-only operations while fenced and refusing state-changing client requests - - Extend the promotion gate the same way, so a member whose materialization is incomplete cannot become primary and immediately violate the serving gate the moment it wins an election - - Introduce no new promotion-eligibility mechanism for WAL/log completeness — this fence is about materialization completeness only, layered on top of the existing Raft log-freshness election gate, not replacing or duplicating it - - Wire a non-blocking, self-triggering retry so the fence re-verifies and clears automatically once conditions allow, without any operator action - - Observable: a fenced primary continues answering read-only and status requests but rejects a state-changing client request with a not-ready response; a fenced member that wins an election does not begin serving under the new configuration until the fence clears on its own - - _Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5_ - - _Depends: 6.1_ - -- [x] 7. Remove the per-node restart/replay fallback and replace it with the marker-triggered materialization - - Delete the configuration-upgrade handler's restart-request step now that materialization is driven entirely by the marker-apply trigger, decoupled from that request-handling code path - - Add a deterministic test seam that can force a leadership change to land exactly between a configuration-change marker's commit and every member completing its snapshot install, for later regression use - - Observable: pushing a configuration upgrade no longer restarts any controller process, and every member reaches the new configuration through the marker-apply trigger alone - - _Requirements: 1.2, 1.3, 1.4, 2.2, 5.4, 6.3_ - - _Depends: 5_ - -- [ ] 8. Validation: regression coverage for every new mechanism -- [x] 8.1 (P) Unit-test the marker-apply trigger and its interaction with WAL retention - - Assert the configuration-marker case fires a checkpoint exactly once per apply and is safe under a forced retry - - Assert the trigger does not itself re-home entities - - Add a deterministic fault-injection seam at the marker-apply-to-checkpoint-durable boundary, mirroring the existing fault-hook pattern already used elsewhere in this codebase, for later chaotic-scenario use - - Assert a snapshot encoded in an unrecognized format version (simulating a rolling binary upgrade or the prior raw queue-frame layout) is rejected fail-loud, never partially installed - - Assert that when a standby is genuinely mid-catch-up (has not yet acknowledged the marker's position) at the moment the marker-triggered checkpoint's WAL trim runs, that standby's still-needed WAL segments are not deleted - - Observable: the new test suite passes, demonstrating exactly-once checkpoint firing, safe retry, format-mismatch rejection, and trim safety under the new trigger path - - _Requirements: 1.1, 1.5, 1.8, 5.2, 5.3, 5.6, 5.7, 6.3, 7.3, 7.4_ - - _Depends: 5_ - - _Boundary: Marker Apply Trigger (test_upgrade_snapshot_apply.c)_ - -- [x] 8.2 (P) Unit-test host placeholder restore and rejoin preservation - - Assert restoring a host admin-state frame for a never-joined host creates a placeholder carrying the correct admin bits - - Assert a retried restore after a simulated partial failure finds and reuses the existing placeholder rather than conflicting with it - - Assert that a host's real first join after a placeholder was created preserves the admin status through the ordinary reconnect path - - Observable: the new test suite passes, demonstrating create-if-missing correctness, idempotent retry, and admin-status survival through a real join - - _Requirements: 1.10, 1.11, 5.8, 7.1_ - - _Depends: 3_ - - _Boundary: Host Restore (test_upgrade_snapshot_restore.c)_ - -- [x] 8.3 (P) Unit-test the activation fence - - Assert the fence sets on incomplete materialization or a configuration-hash mismatch and clears on successful verification - - Assert the request-dispatch gate's allowlist behavior while fenced, and that promotion is refused while fenced - - Observable: the new test suite passes, demonstrating fence set/clear transitions and correct gating of both serving and promotion - - _Requirements: 3.2, 3.3, 3.4, 4.3, 5.5_ - - _Depends: 6.2_ - - _Boundary: Activation Fence (test_upgrade_activation_fence.c)_ - -- [x] 8.4 (P) Build a cross-member active-configuration comparator and extend checkpoint verification tooling - - Build a new offline, file-based tool that compares each member's active configuration content hash, recomputed fresh from each member's own on-disk configuration file, mirroring the existing checkpoint-comparison tool's shape - - Extend the existing checkpoint-structure verification tool to decode and assert the embedded configuration reference's tag and hash (not merely report their presence), and to report host-frame count and configuration-bytes-section presence - - Observable: the comparator correctly reports two members with matching configuration as identical and two with differing configuration as divergent; the extended verification tool reports a concrete tag/hash value and section counts for a real checkpoint file - - _Requirements: 4.2_ - - _Depends: 2.3_ - - _Boundary: Regression Tooling (configcompare, snapverify)_ - -- [x] 8.5 Reproduce bug-857 on the pre-change behavior and confirm it no longer occurs - - Run the original materialized-state-divergence repro scenario against the pre-change code path and confirm the divergence occurs - - Run the same scenario against the post-change behavior and confirm every member's materialized state converges identically - - Observable: the same repro scenario demonstrably fails on the old behavior and passes on the new one - - _Requirements: 6.2_ - - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ - -- [x] 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover - - Drive a configuration upgrade while repeatedly killing and restarting the primary, and assert every member's materialized state and active configuration are identical afterward - - Assert a host's administratively-set status specifically survives the same scenario, not merely inferred from the aggregate state check - - Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members - - _Requirements: 4.1, 4.3, 6.1, 6.4_ - - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ - - **Unblocked 2026-07-29 — bug-1479 fully fixed and live-verified.** `run-ha-soak.sh --config-upgrade` now reports a clean PASS through all 3 `force_elect_node` rounds (see Implementation Notes bullet "Task 8.6 — bug-1479 fully fixed 2026-07-29" below for the full account). Prior blockers bug-1472/bug-1477/bug-1479 all fixed and verified in sequence; no further blocker remains. - -- [x] 8.7 Inject a crash and a leadership change at the two critical upgrade windows - - Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart - - Force a leadership change between a configuration-change marker's commit and every member completing its snapshot install, and assert the new leader continues driving materialization to completion without any special resume logic - - Observable: both fault-injection scenarios complete with the post-upgrade consistency invariant and the crash-safety invariant intact - - _Requirements: 5.2, 5.4, 5.5, 6.3_ - - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ - -- [x] 8.8 Exercise a second configuration upgrade committing before the first's installation completes everywhere - - Commit a second configuration-change marker while the first upgrade's snapshot is still installing on at least one member, and assert the earlier upgrade's in-flight state is cleanly superseded rather than left partially applied or conflated with the newer one - - Observable: after both markers settle, every member reflects only the newer configuration and state, with no trace of a partially-applied intermediate state - - _Requirements: 4.4, 7.2_ - - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ - -- [x] 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions - - Repeat the mid-catch-up WAL-trim assertion from the unit-level fixture, but under the full chaotic multi-kill soak rather than an isolated fixture - - Observable: across a full chaotic-soak run with configuration upgrades in flight, no standby ever loses a WAL segment it still needed for catch-up - - _Requirements: 7.4_ - - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ - -## Implementation Notes - -- **Task 8.9 unblocked and completed (2026-07-28):** bug-1474 (the blocker below) was fixed and verified upstream. Re-running `run-ha-soak.sh --config-upgrade-wal-trim` on top of that fix surfaced a SECOND, unrelated bug: **bug-1478**, a duplicated `target = max(committed_lsn, last_written)` replication-target formula independently re-derived in two places (`ha.c`'s heartbeat/stateReport dispatch and `config_repl.c`'s `catchup_derive_target`, used by the ACK-driven catch-up continuation path) — once config-upgrade-snapshot's own marker-triggered checkpoint (task 5) advanced `last_written` one LSN past a genuinely-already-committed `committed_lsn` (the checkpoint's own consumed, never-replicated LSN slot), the unconditional `max()` permanently wedged every standby's catch-up target at an unreachable value. Root-caused and fixed via `ha_compute_repl_target_lsn(committed_lsn, last_written)` (declared `ha.h`, defined `ha.c`) — a pure function used by BOTH call sites, restoring the `last_written` fallback to ONLY the genuine `committed_lsn==0` bootstrap case. Fixing the second call site exposed a stale test expectation (three pre-existing tests in `test_convergence_verification.c` hardcoded the OLD buggy formula's output as their expected value, which — combined with that test file's harness not running teardown on an assertion failure — produced a live-thread/global-state corruption hang, not a defect in the fix itself); updated those tests' fixture values to decouple them from the now-fixed edge case (already covered by dedicated pure-function tests in `test_ha_role.c`). Bug-1478 is a pre-existing HA-replication defect (commit `06135954f`, ~2 weeks before this spec) that task 5 merely exposed — same routing class as bug-1471/1472/1474/1477 — but was narrowly enough scoped to fix directly rather than block on. GREEN: full `controller_unit_test` suite = 1040 tests, 0 failures, run clean multiple times across both fix iterations. Live-verified: `run-ha-soak.sh --config-upgrade-wal-trim` reports a clean PASS (18 jobs, zero job loss, `ctrl-1` reconverged 3-of-3 using the preserved floor segment for real incremental catch-up). See `.wolf/buglog.json` bug-1474 and bug-1478 for full forensic detail. - -- **Task 8.9 (live soak, genuinely requires chaos — unlike 8.5/8.7/8.8's unit-level redirections): infra complete, task BLOCKED on a new pre-existing bug (bug-1474).** Unlike every other 8.x subtask, this one's own bullet ("under the full chaotic multi-kill soak rather than an isolated fixture") makes the point of the task proving the mid-catch-up WAL-trim guard (`repl_trim_lsn`/`trim_wal_segments`, bug-1147) holds under REAL timing uncertainty, not just deterministic fixture control — so it could not be redirected to a cheaper unit-level proof the way 8.5/8.7/8.8 were. Built `run_config_upgrade_wal_trim_round()` + `--config-upgrade-wal-trim` in `run-ha-soak.sh` (deploys with small `--wal-segment-bytes`, kills a non-primary TARGET, forces ≥2 real segment rotations past its floor on survivors, submits a live config upgrade to fire the marker-triggered checkpoint's trim, then directly inspects survivors' on-disk `wal/` directory BEFORE restarting TARGET — mirrors the unit fixture's own seg-1-gone/seg-2-preserved/seg-3-preserved assertions against real files instead of a tmpdir fixture). Ran twice against independent live 3-node clusters: **both runs failed** — the floor segment protecting the killed target was trimmed both times. Root-caused (not guessed) to **bug-1474**, a general HA connection-management race entirely unrelated to config-marker triggering: killing a standby frees its replicator slot and sets the retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an already-connected peer as "missing" (comparing indices under two different conventions — relative `votes_idx` vs. absolute `peer_idx`, per the reviewer's independent finding), spawns a redundant reconnect, and `repl_add_standby`'s first-empty-slot allocation reuses the just-freed slot, `memset`-clobbering the guard before trim ever runs. Reviewer independently re-derived this from `repl.c:74-133`/`ha.c:9219-9254` plus raw preserved logs (not the implementer's quoted excerpts) and confirmed via `git blame` the responsible code predates this spec by weeks and is reachable identically under the OLD periodic-checkpoint trigger — genuinely out of this spec's boundary, same class of finding as bug-1471/1472. Reviewer also confirmed no scoped adjustment to the round's chaos could dodge the race without suppressing the live heartbeat behavior Req 7.4 requires testing under, so BLOCKED (not a redesign) is the correct call. **Spec-wide status after this task**: every implementable task (1 through 8.5, 8.7, 8.8) is `[x]`; the only two remaining open items (8.6, 8.9) are both blocked on real, independently-verified, out-of-spec-boundary HA defects this spec's own testing effort discovered (bug-1472, bug-1474) — neither blocks any OTHER task in this spec, and neither requires further config-upgrade-snapshot work once fixed upstream. - -- **Task 8.8 (unit-level, no live soak needed — deliberately excludes 8.6):** proves Req 4.4/7.2 (a second marker committing before the first's install completes must cleanly supersede it, never conflate) via two independently-built `t42`/`t43`-family fixtures in `test_config_repl.c`: `c_control` commits+applies+checkpoints marker A then marker B normally; `c_raced` holds the shared `checkpoint_in_progress` CAS across *both* markers' commit+apply (inlining `t43_commit_marker`'s 4-step body rather than changing its signature), so both markers' forced-checkpoint attempts hit `-EBUSY` and skip — genuinely "still installing." Mid-race assertions (captured before releasing the CAS) prove `mc->cluster_config` already reflects marker B while `cfgref_present` is still false and the fence is set; post-resolve assertions (after releasing the CAS, calling `persist_checkpoint()`, then an explicit `upgrade_materialization_verify()`) prove the settled state reflects only B. Discovered mid-implementation that whole-file `snapdiff_paths` byte-identity between the two fixtures' final checkpoints does NOT hold — `c_control`'s marker-A checkpoint physically consumes one extra WAL LSN (`persist_write_checkpoint_marker`, persist.c:1684) before marker B is appended, so B lands at a different absolute LSN in each fixture, and `write_snapshot`'s sentinel header embeds `checkpoint_lsn` at byte offset 8. Adapted to a field-aware comparison (full `memcmp` from byte 16 onward) plus `snapdiff_paths` diagnostic corroboration that offset 8 is the *only* divergence — a more precise proof than blind byte-identity, reviewer independently re-derived and confirmed this against `write_snapshot`'s actual layout rather than trusting the claim. One RED→GREEN cycle (`bug-1473`, test-construction gap not a production bug): the bare `persist_checkpoint()` resolve call doesn't itself trigger `upgrade_materialization_verify()` in a delaymgr-less unit fixture — production's self-healing (`schedule_upgrade_materialization_retry`, fired from `ha.c:2541-2542` on every commit-advance) only runs on a live cluster; the fixture needs the manual call, mirroring the existing 6.1-c fence tests' pattern. Reviewer's only findings were non-blocking style notes (could have used the public `upgrade_materialization_retry_fn` wrapper instead of calling `upgrade_materialization_verify` directly — functionally identical here since the fence was already independently confirmed set). Suite: 1027 → 1028 tests, 0 failures. - -- **Task 8.7 (both scenarios, unit-level, no live soak needed):** design.md's Testing Strategy was explicit this task should use deterministic fault injection rather than chaotic-soak timing probability, and both hooks needed already existed from earlier tasks (`persist_marker_checkpoint_fault_hook`, task 8.1; `xce_upgrade_marker_committed_hook`, task 7) — task 8.7 was purely additive test work, zero production code touched. **Scenario 1** (`test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization`, `test_upgrade_snapshot_apply.c`) builds on the existing seam test (which only proved the fault hook fires and gates the checkpoint call) by clearing the fault, tearing down without deleting the on-disk WAL, and booting a genuinely fresh `persist_t`/`cluster_t` via `persist_recover()` — a real second-process boot-replay simulation, deliberately not a same-process re-call (which the test's own comment argues would be a weaker, trivially-passing proof). **Scenario 2** (`test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain`, `test_config_repl.c`) proves Req 5.4 by building a promoted member and a non-promoted control member from byte-identical committed-but-unapplied markers, draining each via the respective real production path (`persist_apply_pump_run_sync`, which production promotion — `raft_cb_become_leader`, `ha.c` — calls unconditionally right after the role flip; vs. plain `persist_apply_pump_run` for the control), and asserting the resulting checkpoints are byte-identical via `snapdiff_paths` (task 8.4's tool). Reviewer independently verified (not trusted) both design claims by reading `ha.c` directly (`upgrade_materialization_fenced` only refuses an *already*-fenced member; the sync-drain call is unconditional and unguarded by role-specific logic) and specifically interrogated whether this was a tautology (calling the same underlying function twice) — concluded it genuinely is not, since the real variable under test is role state at drain time plus the production entry point, and the test separately asserts promotion alone has zero effect on the fence. Also confirmed via direct comparison against the existing 6.2/8.3 fence tests that this is new *continuation* coverage (fence unset → promotion succeeds → drain completes → config activates), not a duplicate of the existing *gating* coverage (fence set → promotion refused, never reaching the drain). - -- **Task 8.6 progress (infrastructure + two real production bugs found and fixed; task NOT YET complete — one more bug blocks a clean live-soak PASS):** building this task's live 3-node chaotic-upgrade soak required substantial new infrastructure that never existed anywhere in this codebase: a client-side send path for `OP_upgradeConfig` (`xce_upgradeconfig_req_raw`/`xce_upgradeconfig`/`xcecli_upgradeconfig` — the auto-generated capnp encoder for this request type is a literal no-op for its `Data` fields, worked around via a hand-rolled bypass mirroring the existing `xce_replconfigpush_req_raw` idiom), a `cmd_hostctrl`/`cmd_upgradeconfig` CLI pair in `test.c`, and a new `--config-upgrade` mode in `run-ha-soak.sh` (+ `SKILL.md` docs). Building and running this surfaced two real, independently-confirmed production defects: - - **bug-1468** (`cluster.c`): `myreq_handler`'s `OPCode_upgradeConfig` dispatch guard was inverted by an unrelated commit (`fd93997f`, 2026-05-31) — it rejected the only role (primary) that could ever legitimately reach it, making config upgrades unreachable from any network client at all. Fixed: restored correct polarity. - - **bug-1469** (`persist.c`): `apply_committed_entry`'s `WAL_CONFIG_MARKER` case never called `config_activate_committed()` — so the node that *originates* a config upgrade never refreshed its own in-memory config/queue state as part of committing it (only receiving standbys did, via a separate out-of-band push path). If that node was killed before some later, independent event happened to refresh it, its checkpoint froze with stale config content — a genuine, **live-reproduced** instance of bug-857's originally-hypothesized failure mode (task 8.5 could only prove a different, unit-level mechanism; this is the first real empirical case). Fixed: added `config_activate_committed(c)` inside the marker-apply case, positioned *before* the marker-triggered checkpoint write (checkpoint content is built from live runtime tables that only reflect the reload after activation runs, not from `myconf` directly) and guarded against `c->myconf == NULL` (a real crash risk hit in an existing test fixture). Fixing this also exposed and required fixing a genuine race in a pre-existing test (`test_commit_advance_retry_clears_push_before_commit_fence`) that the new activation side-effect made observable for the first time — narrowed the race window, did not weaken any assertion. Reviewer independently re-derived the lock-safety claim from scratch (traced every mutex site in `config_repl.c`, confirmed none reachable from `config_activate_committed`) and the thread-class claim (`apply_committed_entry` runs on a `VTASK_LIVE_BLOCK` pool worker, confirmed via `persist_apply_pump_kick`) rather than trusting the implementer's account — both held. - - **Still open, blocking a clean PASS — bug-1471**: a separate, much narrower 1-byte checkpoint divergence in a host-admin-status frame (not a config-activation issue), found independently by two concurrent sessions working this task and root-cause-not-yet-isolated. Also noted but deliberately **not** triggered by the current soak scenario: a deeper, genuinely separate gap where no live mechanism exists anywhere in this codebase to add a *new* queue to an already-running member's runtime `qmgr` without a restart (`config_activate_committed`/`load_primary_config` refresh `myconf->cluster_config` but nothing syncs that into the live `qmgr`) — the soak's upgrade-target config change was deliberately scoped to modifying an *existing* queue's field rather than adding one, specifically to test what Requirements 4.1-4.3/6.1/6.4 actually ask for without tripping over this separate, deeper architectural gap. Both are flagged in `.wolf/buglog.json` for whoever next picks up bug-1471 (or a dedicated follow-up for the queue-creation gap) — task 8.6's own checkbox stays unchecked until the soak reports a clean PASS. - - **Process note**: this task was interrupted by the same monthly-spend-limit condition hit earlier in this session (task 6.1), across three separate dispatch/resume cycles. Each resumption correctly checked live `git status`/`diff` before acting rather than blindly redoing work, and the final state was independently re-verified end-to-end (build + full 1025-test suite + file-integrity spot check) before being reviewed and committed — no corruption or duplicated work resulted, but it's worth flagging that this class of interruption can leave multiple agent instances transiently active in the same worktree. - -- **Task 8.6 re-run 2026-07-29 (bug-1477 fix reconfirmed; NEW blocker bug-1479 reached, still open):** rebuilt `xce_controller`/`xce_test`/`controller_unit_test` fresh against the working tree's bug-1479 fix attempt (`ha_raft_on_repl_nack`/`repl_get_catchup_join_lsn` in `ha.c`/`ha.h`/`repl.c`/`repl.h`, 1048/1048 `controller_unit_test` green) and re-ran `run-ha-soak.sh --config-upgrade` end to end (deploy preserved, not deleted, at `~/tmp/xce_ha_bug1479_verify`). The chaotic 3-round primary rotation, the live config-upgrade commit, and the checkpoint-identity/config-identity/host-admin-status-survival checks ALL PASSED cleanly a second time (confirms bug-1477 stays fixed). The run then failed in this task's own dedicated end-of-round `force_elect_node` loop (the freshness-fence maneuver bullet 3 of task 8.6's Requirement 6.4 verification): `force-elect ctrl-1` no-op'd correctly, but `force-elect ctrl-2` hard-failed — after killing spoiler `ctrl-3`, `submit_jobs` against `{ctrl-1,ctrl-2}` got 0/3 acked across all 5 retry attempts. Root-caused (log-correlated across both nodes, not guessed): this is **bug-1479 still not fully fixed** — the fix's new primary-side NACK-routing code is correct but unreachable, because the standby's own AppendEntries-reject path (`ha_apply_wal_catchup_entry`/`ha_apply_term_entry_via_library`, `ha.c` ~3230-3232/3320-3321/3373-3374) sends no reply at all on reject, and `ha_send_wal_ack_reply` (the only `WalAckReply` sender in the controller) hardcodes `success=1` — there is no production path that ever sends a NACK. See `.wolf/buglog.json` bug-1479 (occurrence 2, updated 2026-07-29) for the full live-log evidence trail. This is a pre-existing HA-replication defect, same family as bug-1466/1471/1472/1474/1475/1477/1478, out of THIS spec's boundary (task 8.6 only exercises the `force_elect_node` mechanism that exposes it, same relationship the task already had to bug-1477/bug-1472) — task 8.6's checkbox stays unchecked until it is fixed and a fresh `run-ha-soak.sh --config-upgrade` reports a clean PASS through all 3 `force_elect_node` rounds. New follow-up spec created to track the fix plus a broader systematic check for siblings of this same gap: `.kiro/specs/raft-adapter-conformance/` (a `raft.tla` action-by-action conformance pass over the controller's Raft adapter — `ha.c`/`repl.c`/`config_repl.c` — distinct in scope from the existing `raft-sim-conformance` spec, which validates only `src/lib/raft/raft_sim.c`, the in-process library harness, and explicitly excludes this controller-adapter wire code). - -- **Task 8.6 — bug-1479 fully fixed 2026-07-29 (task now `[x]`, clean live PASS):** closing the gap identified in the re-run above needed TWO more compounding fixes beyond the primary-side NACK routing, each verified with its own dedicated live soak re-run because the first two looked complete after a clean unit-test pass yet still failed the identical live scenario for a NEW reason each time. **Fix A (standby-side NACK sending):** `ha_apply_term_entry_via_library`'s logOk-reject path (the one live-reachable reject site; the other two in `ha_apply_wal_catchup_entry` are documented dormant/unreachable with real senders) now calls a new `ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success)` with `success=0` instead of silently dropping the reject — `ha_send_wal_ack_reply` had hardcoded `success=1` unconditionally, the ONLY `WalAckReply` sender anywhere in the controller, which made the primary's `ha_raft_on_repl_nack` unreachable dead code no matter how correct it was. New `ha_wal_ack_send_hook` (ha.h, XCE_TESTING) lets a unit test observe the NACK without a live endpoint (`test_bug1479_ae_reject_sends_nack_not_silence`, `test_config_repl.c`). Live re-run (`~/tmp/xce_ha_bug1479_verify_v2`) confirmed NACKs now genuinely reach the primary (`ctrl-1.log`: "WAL NACK from standby ... success=0 — marking catch-up") — but the SAME force-elect ctrl-2 scenario still failed, because of gap B. **Fix B (catch-up stream ceiling):** `ha_stream_wal_to_standby`'s catch-up range was hard-bounded at `committed_lsn`, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) — a self-referential deadlock invisible from either side's code in isolation. New pure function `ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn)` (ha.c/ha.h), wired into `ha_dispatch_repl_action_for_standby`, widens the STREAM_WAL ceiling (and the `ha_evaluate_repl_action` convergence check gating it) to the leader's own log tail instead of just `committed_lsn` — matching `raft.tla`'s real AppendEntries semantics (`leaderCommit` is advisory to the follower, never a send-cap) — while re-using bug-1478's exact guard (never widen onto a trailing local-only checkpoint's own unstreamable LSN slot) so this fix did not reopen that one. 4 new pure-function tests in `test_ha_role.c` (`test_compute_repl_stream_ceiling_lsn_*`). Both fixes RED-confirmed individually (each reverted to its pre-fix behavior in turn, re-run, confirmed the predicted failure, restored) before being combined. **Final verification (`~/tmp/xce_ha_bug1479_verify_v3`, not deleted):** `controller_unit_test` 1053/1053 PASS twice clean; live `run-ha-soak.sh --config-upgrade` PASSED end-to-end — all 3 `force_elect_node` rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7, never previously reached), final verdict "PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members." `.kiro/specs/raft-adapter-conformance/`'s concrete seed defect (missing NACK reply) is now fixed here; that spec's broader `raft.tla` action-by-action conformance audit remains valid, separate future work (fix B's own committed_lsn-vs-log-tail conflation is exactly the kind of second sibling gap that audit is meant to catch systematically). - -- **Task 8.5 redirection (design.md's original mechanism was wrong; this is not the same as a review rejection):** the original hypothesis for bug-857's divergence — a job whose queue is removed by a config upgrade gets re-homed to `lost_and_found` on some members but not others, based on per-node restart timing — turned out to be **aspirational, never-implemented code**. Exhaustive grep across `job.c`/`queue.c`/`worker.c`/`config_repl.c` at both the pre-fix baseline commit (`e0c5e8b7`) and current HEAD confirmed: `create_lost_and_found` only ever creates the sentinel queue/host objects (and its entire body is `#ifndef XCE_TESTING`-gated); `restore_job`/`restore_job_array` never consult `qmgr`; `restore_queue_state` explicitly *skips* rather than re-homes when a queue is missing. `.wolf/buglog.json`'s bug-857 entry itself flags this: "identified via TLA+ gap analysis... NOT empirically reproduced yet." design.md's Requirements Traceability table assigned 6.2 to a live "HA soak new scenario" — also written against the same since-disproven hypothesis. Redirected (orchestrator decision, not a task-local implementer call) to a real, code-backed mechanism instead: `schedule_checkpoint`'s pre-existing, unmodified-by-this-spec WAL-record-count threshold trigger is genuinely decoupled from the config-marker `(term,lsn)` boundary, so two members' independently-timed checkpoints can capture different job/queue sets — the actual, general-purpose truth behind bug-857's claim ("no snapshot transfers an identical materialized state"). This is testable entirely within `controller_unit_test`, no live multi-process soak or old-commit worktree needed, since the threshold-trigger mechanism is still live, unmodified production code coexisting with the new marker-trigger at HEAD. Landed as `test_bug857_regression.c` (2 tests: divergence under threshold-only triggering vs. convergence under the real marker-triggered path, the latter under deliberately differing per-member processing pace to make the convergence claim non-trivial). Reviewer independently re-derived and confirmed the mechanism claim from first principles (not just re-running the tests) before approving — see `git blame`/review transcript for the full verification chain (schedule_checkpoint predates this spec via `git merge-base --is-ancestor`, zero lines of its body touched by tasks 1-7, `persist_apply_pump_run`'s strictly-sequential dispatch loop explains why the marker-triggered checkpoint's convergence isn't a timing coincidence). design.md's Requirements Traceability table (row 6.2) and Testing Strategy section were both corrected to match. **Heads-up for tasks 8.6-8.9**: they still legitimately need live multi-process HA-soak-level testing for Requirement 6.1 (chaotic multi-kill + config upgrade) — that infrastructure (a live config-upgrade CLI trigger doesn't exist anywhere in this codebase yet; confirmed via grep of `test.c`/`src/frontend`/`src/tools`) was NOT built by this task and will need to be built by whichever of 8.6-8.9 first needs to drive a live `OP_upgradeConfig` RPC against a running cluster. - -- **Task 8.4 round-1 rejection (real gap, caught by review):** the first pass built `configcompare_paths()` (correct, well-tested against `persist.c`/`config_repl.c`'s real `workdir/name/conf/current`+`conf/cluster.bin.<hex>` resolution convention) and linked `configcompare.c` into both `xce_test_src` and `controller_unit_test_src` in meson, but never registered it as a `cmd_configcompare` subcommand in `testsuite/test.c` — unlike its siblings `walcompare`/`snapverify`/`snapdiff`, each of which has a `cmd_*` wrapper in BOTH `top_commands[]` and `client_commands[]`. Left as delivered, the tool was a tested library function with no shell entry point — unusable by task 8.6/8.9's later HA soak scenarios, which design.md explicitly says will invoke `configcompare` directly against each node's data directory. Fixed with a 25-line, `test.c`-only remediation mirroring `cmd_snapdiff`'s exact shape; verified end-to-end via `echo 'configcompare <dir1> <dir2>' | xce_test` (checked `$?` for both IDENTICAL/exit-0 and DIVERGENT/exit-1 cases, not just printed text). **Heads-up for any future task in this spec (or elsewhere) that adds a new offline testsuite tool in this family**: linking the `.c` file into the meson source lists is NOT sufficient — it must also get a `cmd_*` wrapper + `top_commands[]`/`client_commands[]` entries in `test.c`, or it silently has no way to be invoked outside the unit test binary. - -- **Task 8.3: no new tests needed** — the 10 tests already in `test_config_repl.c` from tasks 6.1/6.2 fully satisfy both bullets and Requirements 3.2/3.3/3.4/4.3/5.5 (verified independently by implementer and reviewer; zero-diff task). The two "fence sets" tests genuinely toggle different halves of `upgrade_materialization_verify`'s condition (materialized vs. bytes_verified), not duplicates. Retry-trampoline automatic-firing (delaymgr actually running) is untested, matching the exact precedent depth of `config_activate_retry_fn`/`config_schedule_activate_retry` (also untested at that depth anywhere in this codebase) — judged out of scope, not a gap. - -- **Task 8.2: no new tests needed** — task 3's `test_upgrade_snapshot_restore.c` already satisfies all 3 bullets and Requirements 1.10/1.11/5.8/7.1 (verified independently by both implementer and reviewer; zero-diff task). Requirement 7.1's full scenario is compositionally covered across two files (`test_upgrade_snapshot_restore.c`'s real-rejoin test + `test_persist_snapshot_entities.c`'s `test_snapent_load_snapshot_restores_hosts_and_queues_end_to_end`'s real checkpoint-decode test), not by one continuous test — judged adequate since `restore_host_state` is a single invocation-site-independent function. design.md's Requirements Traceability table (row 7.1) under-credits this and should eventually be corrected to cite both files, not just `test_upgrade_snapshot_restore.c` — docs-only, not a testing gap. - -- **Task 6.2 round-1 rejection (real bug, caught by review):** the first implementation wired the fence into both gates + a retry trampoline correctly, but `upgrade_materialization_fence` had no production path that could ever SET it from its zero-initialized default — every setter call site was gated on the fence already being set (closed loop). Fixed with two unconditional call sites: `persist.c`'s `apply_committed_entry` `WAL_CONFIG_MARKER` case now unconditionally calls `upgrade_materialization_verify(c)` right after `persist_checkpoint_marker_triggered` returns (regardless of its rc), and `myapp.c`'s boot sequence unconditionally calls it right after `config_activate_committed` (Req 5.2, crash/restart resume). New cross-module direction: `persist.c` now externs an `ha.c` function (mirrors the existing `persist.c → worker.c` extern pattern, no circular include). Lock-reentrancy re-verified safe (neither call chain touches `apply_pump_lock`). - -- **Task 6.1 process note:** the Claude account's monthly API spend limit was hit mid-task (implementer subagent cut off after landing `ha.c`/`ha.h` but before tests/build). The orchestrator finished this task directly (no subagent implementer/reviewer pair) rather than stopping: verified the partial diff, wrote 4 tests into `test_config_repl.c` (reusing existing `t42_/t43_` fixtures rather than creating `test_upgrade_activation_fence.c` — that file is task 8.3's own boundary per design.md, not 6.1's), built, ran the full suite (1001 Tests 0 Failures 0 Ignored), and self-reviewed the diff in lieu of an independent reviewer subagent. Flagging this so a future session/reviewer knows this task's diff did not go through the normal two-subagent review gate. - -- **Task 5 correction to design.md:** design.md's "Marker Apply Trigger" section says "Call `persist_checkpoint()` directly" — this is unsafe as written. `persist_apply_pump_run` holds `apply_pump_lock` across the whole dispatch loop (including the `apply_committed_entry` call), and `persist_checkpoint()` itself re-acquires that same non-recursive mutex — a real self-deadlock on the live pump path (primary AND standby online-apply). Separately, `last_applied_lsn`/replay position only advance AFTER `apply_committed_entry` returns at both call sites, so pinning `checkpoint_lsn` from `last_applied_lsn` inside the marker case would mislabel the checkpoint with the previous entry's LSN, not the marker's own. Fixed via `persist_checkpoint_write_locked(p, checkpoint_lsn)` (no self-lock, explicit LSN param) + `persist_checkpoint_marker_triggered(p, lsn)`, called from `apply_committed_entry`'s new `WAL_CONFIG_MARKER` case with the marker's own `lsn` parameter. design.md should be updated to reflect this; not done as part of task 5 (docs-only, deferred). -- **Task 5 → ha-term-safety/raft-log-replication finding (bug-1466, NOT this spec's to fix):** `persist_write_checkpoint_marker` advances `p->current_lsn` directly, bypassing `ha_primary_log_append`/`raft_log_append`, breaking `ha_primary_log_append`'s own documented `p->current_lsn == s->log_len + 1` invariant (ha.c:4054-4060) once any checkpoint interleaves with primary appends. Pre-existing (periodic checkpoint already did this); task 5 just made the interleaving common instead of rare, surfacing it as real test flakiness for the first time. `ha->raft.log_len` feeds real RequestVote freshness comparisons (ha.c:258-261, 401-406) — NOT advisory-only as bug-1466's original text claimed; severity corrected in `.wolf/buglog.json`. Out of this spec's boundary (Raft log/term safety is `ha-term-safety`/raft-log-replication's territory) — flagging for whoever next owns that area. - -- **Task 3 → task 4 heads-up:** `restore_host_state(cluster_t*, const char*, uint32_t)` (worker.c) is standalone and NOT yet wired into `load_snapshot`. Task 4 dispatches decoded `HostAdminState` frames to it. `restore_host_state` runs on the live vtask pool (called from `ha_handle_snapshot_chunk` → `persist_load_snapshot`), not a quiesced phase — a concurrent real first-time join for the same host is architecturally possible; verified safe by convergent key_find/makeWorker semantics (see worker.c comment above `restore_host_state`'s key_find call), not by absence of concurrency. Task 4 should confirm this contract still holds once wired in. - -- **Task 2.3 → task 6.1 heads-up:** `read_active_config_bytes` (persist.c) is the fully-inlined (no config_repl.c call) pattern for reading live config bytes off disk fresh each time — task 6.1's activation-fence verify function needs the identical "always re-read live bytes, never trust a cached/embedded copy" discipline (design.md is explicit about this). Reuse/mirror `read_active_config_bytes` rather than re-deriving the config_repl.c-avoidance reasoning from scratch. - -- **Task 2.2 → task 4 MUST re-enable — DONE (task 4):** `test_snapent_load_snapshot_restores_entities_before_advancing_position` is un-ignored and passes for real now that `load_snapshot`'s queue decode reads `QueueAdminState` capnp. -- `test_snapent_checkpoint_queue_status_roundtrip`'s comments (~line 1019) are stale (still reference "format_version=1" / "task 3.2 not yet implemented" from an earlier spec pass) — harmless, not a task 2.2 defect, but worth cleaning up whenever that test is next touched. - -- **Task 2.1 → task 3 heads-up:** host-frame collection is unconditional (every host in `c->workermgr` gets a frame, status masked to admin bits, may be 0) — confirmed correct against requirements.md/design.md by both implementer and reviewer. Consequence for task 3 (`restore_host_state`): a checkpoint may contain many zero-admin-status host frames on large clusters. Not required to special-case this, but consider whether `restore_host_state` should skip placeholder creation when `admin_status == 0` on a miss, purely as a cost optimization — current design.md contract doesn't require it either way. - -- **Build environment (bug-918 recurrence 4, fixed in this worktree):** `nix develop` was hard-broken — `xbuild`'s `capnpc` package pins its `src` to the MUTABLE `main` ref of `shen390s/c-capnproto` with a hardcoded sha256, so it hash-mismatches every time that branch moves upstream (recurred 3x before in this same worktree per `.wolf/buglog.json`). Fixed durably by overriding `capnpc`'s `src` in `src/nixSupport/xce.nix` to a `pkgs.fetchFromGitHub` pinned at an immutable commit (`rev`+`sha256`) instead of tracking `main`. Any future task in this spec that needs a fresh `nix develop` should not hit this again; if it does, the fix pattern is the same (re-pin to the then-current upstream commit). -- **Test-runner gotcha:** running `controller_unit_test` with default ASan settings hangs indefinitely in this sandbox — LeakSanitizer's stop-the-world thread scan never completes (confirmed: 50+ min wall clock, ~5s CPU, `futex_do_wait`). Always run with `ASAN_OPTIONS=detect_leaks=0` (matches the project's own meson test registration). Do not wrap the binary in `stdbuf` or anything else that adds its own `LD_PRELOAD` — conflicts with ASan's preload requirement and the binary refuses to start. -- **After any `SNAPSHOT_FORMAT_VERSION` bump:** grep the whole `testsuite/` tree for other hardcoded copies of the old format-version literal before considering the bump done — `test_persist_snapshot_entities.c` (a different, already-shipped spec's test file) had its own independent `SNAPENT_TEST_FORMAT_VERSION` constant that silently went stale after task 1's bump and broke 3 unrelated tests until caught. diff --git a/.kiro/specs/controller-status/design.md b/.kiro/specs/controller-status/design.md deleted file mode 100644 index 244e2cb6..00000000 --- a/.kiro/specs/controller-status/design.md +++ /dev/null @@ -1,313 +0,0 @@ -# Design Document — controller-status (`ctrlstatus`) - -## Overview - -**Purpose:** add a single, cheap, read-only client→controller query — `ctrlstatus`, an XCE analog of LSF `lsid` — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name). - -**Users:** operators inspecting a running cluster, and test/E2E authors orchestrating failover (including `wait-primary`, which switches its probe to this query). - -**Impact:** today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing **opcode-only read** pattern of `clusterconf`, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on **any** node by adding it to the HA dispatch-gate allow-lists. - -### Goals -- A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4). -- Answerable on **any** node — a standby reports a hint view (the primary it knows); an **authoritative** answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3). -- Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3). -- A `ctrlstatus` test-client command (8.x) and a `wait-primary` probe switched to an authoritative (primary-only) status query (9.x). - -### Non-Goals -- No operator-facing CLI/REST `lsid` surface (Go tier) — later pass. -- No change to HA election, replication, or commit behavior — the query only **reads** existing state. -- No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it. - -## Boundary Commitments - -### This Spec Owns -- A new Cap'n Proto opcode `ctrlstatus` and its reply payload (`CtrlStatus`, `CtrlPeer`). -- The controller handler `handle_ctrlstatus` that snapshots local HA/config state into the reply. -- The **dispatch-gate allow-list entries** that let `ctrlstatus` answer in every controller role (incl. no-quorum). -- The libxce client API (`xce_ctrlstatus_req` / `xce_ctrlstatus` / `xcecli_ctrlstatus` + free helper). -- The `ctrlstatus` test-client command and the `wait-primary` probe switch. - -### Out of Boundary -- The Go CLI/REST `lsid` surface (later spec). -- Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is **produced or maintained** — this spec only reads it. -- The existing `clusterconf` opcode and its resource/queue/worker payload (left unchanged). - -### Allowed Dependencies -- The Cap'n Proto schema + build-time codegen (`src/schema/controller.capnp` → generated `controller.capnp.h/.c`). -- The existing request/reply plumbing (`init_reply_by_req`, `xce_reply`, opcode-only `*_req` senders) and client failover wrapper (`XCECLI_FAILOVER_CALL`, `is_write=false`). -- Read access to existing HA state (`ha_state_t`), config (`myconf_t`: `committed_config_tag`, `config_sha256`, `peers_cfg`), and persist accessors (`persist_recovery_unrecoverable`, `persist_durable_lsn`). - -### Revalidation Triggers -- Any change to the `CtrlStatus`/`CtrlPeer` reply shape or the `ctrlstatus` opcode ordinal → re-check libxce decode + the test command + any future consumer. -- Any change to the dispatch-gate role allow-lists → re-check that `ctrlstatus` still answers in every role and that no other opcode's gating widened. -- A change to the meaning of `committed_config_tag` / `config_sha256` → re-check the version/checksum fields. - -## Architecture - -### Existing Architecture Analysis -- **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.** -- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:3802`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation). -- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:4525`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch. -- **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`). - -### Pattern & dependency direction -Extend-in-place along the existing request path. Dependency direction (unchanged): `schema → libxce (request/reply/client) → controller handler`; the test client depends on libxce. No upward dependencies introduced. - -```mermaid -graph LR - TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus] - Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false] - Failover --> Gate[ha_dispatch_gate allow_list all roles] - Gate --> Handler[handle_ctrlstatus snapshots local state] - Handler --> HAState[ha_state_t and myconf_t] - Handler --> Reply[xce_reply CtrlStatus] -``` - -## File Structure Plan - -### Modified Files -- `src/schema/controller.capnp` — add `OPCode ctrlstatus @40`; new `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")` and `struct CtrlPeer $C.mapname("xce_ctrlpeer_t")`; reply-union member `ctrlstatus @17: CtrlStatus`. (Build regenerates `controller.capnp.h/.c` → `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, `free_CtrlStatus`.) -- `src/controller/reply.c` — add `handle_ctrlstatus` (snapshot local HA/config state into a `CtrlStatus` reply; small peer list; `xce_reply`). Mirrors `handle_clusterconf`/`ha_nop_reply_state_report`. -- `src/controller/cluster.c` — add `case OPCode_ctrlstatus: rc = handle_ctrlstatus(...)` to the opcode switch. -- `src/controller/ha.c` — add `OPCode_ctrlstatus` to the `ha_dispatch_gate` allow-lists for `CTRL_PRIMARY` (incl. the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT`. **(the requirements-critical change)** -- `src/lib/request.c` + `src/include/request.h` — add `xce_ctrlstatus_req` (opcode-only sender, clone of `xce_clusterconf_req`). -- `src/lib/xceapi.c` — add `xce_ctrlstatus` (send req, `cbuf_get` status, `MOVE_PTR` reply union). -- `src/lib/client.c` + `src/include/client.h` — add `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, `is_write=false`), the authoritative `xcecli_ctrlstatus_primary` (follows redirect / rotates the master list until `responderRole == PRIMARY`), and the `xcecli_free_ctrlstatus` macro. -- `src/controller/testsuite/test.c` — add `cmd_ctrlstatus` + table registration (uses `xcecli_ctrlstatus`, shows the responder's view); switch `cmd_wait_primary`'s probe (currently `joblist_query(...)` at the deadline-loop call site) to `xcecli_ctrlstatus_primary` (authoritative, primary-only success). - -### New Files -- None (capnp C types are generated; all handlers/APIs slot into existing TUs). - -## System Flows - -### Status query: any-node hint then authoritative-from-primary -```mermaid -sequenceDiagram - participant C as Client - participant S as Standby node - participant P as Primary node - C->>S: ctrlstatus request - S-->>C: reply responderRole STANDBY plus known primary hint - Note over C,S: Standby answer is a non authoritative hint view - C->>P: re-query targeting the reported primary - P-->>C: authoritative reply responderRole PRIMARY -``` -The dispatch gate allow-lists `ctrlstatus` in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An **authoritative** answer comes only from the node whose `responderRole == PRIMARY`. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's `primaryName` to an address via the reply's `peers` list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain `ctrlstatus` command may display the first (hint) reply with its "answered by" framing; `wait-primary` and any authoritative use require the primary's reply. -Gate decision: `ctrlstatus` is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it). - -### wait-primary probe (Req 9) -`wait-primary` keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from `joblist_query` to `xcecli_ctrlstatus`. Success predicate: the status reply reports a **non-`"unknown"`** current primary. On success it prints the served primary's identity. (Rationale + caveat below.) - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | -|-------------|---------|------------|------------| -| 1.1, 1.2, 1.4 | Single-call read-only focused status | `handle_ctrlstatus`, `CtrlStatus` schema | opcode `ctrlstatus`, reply `CtrlStatus` | -| 1.3 | "unknown" for undeterminable fields | `handle_ctrlstatus` | empty Text/sentinel mapping | -| 2.1, 2.2, 2.3 | Current primary identity (unknown when unelected; standby's known primary) | `handle_ctrlstatus` (reads `ha->primary_host` under `ha->lock`) | `CtrlStatus.primaryName` | -| 3.1, 3.2, 3.3 | Answered-by node + role, distinct from primary | `handle_ctrlstatus` | `CtrlStatus.responderName`, `responderRole` | -| 4.1, 4.2 | Configured controllers + count | `handle_ctrlstatus` (reads `peers_cfg`) | `CtrlStatus.peers`, `peerCount` | -| 5.1, 5.2, 5.3 | Config version + content checksum (loaded pair + committed) | `handle_ctrlstatus` (reads loaded `config_version_tag`/`config_sha256` + `committed_config_tag`) | `CtrlStatus.configVersion`, `configChecksum`, `committedConfigVersion` | -| 6.1–6.4 | Term, commit/apply, fenced, cluster name | `handle_ctrlstatus` | `CtrlStatus.term`, `committedLsn`, `appliedLsn`, `fenced`, `clusterName` | -| 7.1, 7.2, 7.3 | Failover-aware any-node delivery | `xcecli_ctrlstatus`, dispatch-gate allow-lists | `XCECLI_FAILOVER_CALL` (is_write=false), gate entries | -| 8.1, 8.2, 8.3 | Test-client `ctrlstatus` command | `cmd_ctrlstatus` | client command + exit code | -| 9.1–9.4 | wait-primary uses an authoritative (primary-only) status probe | `cmd_wait_primary` probe swap, `xcecli_ctrlstatus_primary` | redirect-to-primary, `responderRole == PRIMARY` success | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| `CtrlStatus`/`CtrlPeer` schema | Schema | Reply payload for the status query | 2–6 | capnp codegen (P0) | API (reply) | -| `handle_ctrlstatus` | Controller | Snapshot local HA/config state into the reply | 1–6 | `ha_state_t`, `myconf_t`, persist accessors (P0) | Service | -| dispatch-gate allow-lists | Controller HA | Permit `ctrlstatus` in every role | 7 | `ha_dispatch_gate` (P0) | Service | -| `xcecli_ctrlstatus` / `xcecli_ctrlstatus_primary` | libxce | Failover-aware read (any-node) + authoritative reach-primary read + free | 1, 7, 9 | failover wrapper, redirect hint (P0) | Service | -| `cmd_ctrlstatus` | Test client | Print the status (responder view) | 8 | `xcecli_ctrlstatus` (P0) | State | -| `cmd_wait_primary` probe | Test client | Authoritative serving-primary probe | 9 | `xcecli_ctrlstatus_primary` (P0) | State | - -### Schema — reply payload (Cap'n Proto) -```capnp -struct CtrlPeer $C.mapname("xce_ctrlpeer_t") { - name @0: Text; - host @1: Text; - port @2: UInt32; -} - -struct CtrlStatus $C.mapname("xce_ctrlstatus_t") { - clusterName @0: Text; - primaryName @1: Text; # empty ⇒ "unknown" (no elected/known primary) - responderName @2: Text; # the node that answered - responderRole @3: UInt8; # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY - term @4: UInt64; - # Loaded config (consistent pair: version is the 8-byte prefix of the checksum) - configVersion @5: UInt64; # loaded config_version_tag (8-byte SHA-256 prefix) - configChecksum @6: Data; # loaded config_sha256 (full 32 bytes) - # Committed config version (may differ from loaded mid config-change) - committedConfigVersion @7: UInt64; # committed_config_tag - committedLsn @8: UInt64; - appliedLsn @9: UInt64; - fenced @10: Bool; # in_catch_up OR recovery_unrecoverable - peers @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count"); - peerCount @12: UInt32; -} -``` -- **Config fields (Req 5):** `configVersion`/`configChecksum` are the **loaded** config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). `committedConfigVersion` is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree. -- New OPCode ordinal: `ctrlstatus @40` (next free after `preVoteReply @39`). New reply-union member: `ctrlstatus @17` (next free after `walackreplymsg @16`). Field ordinals are append-only — never reuse/reorder. - -#### `handle_ctrlstatus` — Service contract -| Field | Detail | -|-------|--------| -| Intent | Build a `CtrlStatus` from the responding node's local HA/config state and reply | -| Requirements | 1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4 | - -**Responsibilities & Constraints** -- Read-only: must not modify any HA, persist, or config state (1.2). -- Field sourcing: `clusterName`/`responderName` from `c->myconf`/own peerinfo; `responderRole`, `term`, `appliedLsn`, `committedLsn`, `committedConfigVersion` (`committed_config_tag`) via `atomic_load_explicit(..., acquire)`; `configVersion` from the loaded `config_version_tag` and `configChecksum` from the loaded `config_sha256[32]` (a self-consistent pair — Req 5.3); `primaryName`/peers as below; `fenced = in_catch_up || persist_recovery_unrecoverable(...)`. -- **Locking:** `primary_host`/`primary_port` are guarded by `ha->lock` — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config. -- **"unknown" mapping (1.3, 2.2, 2.3):** if `primary_host[0] == '\0'` (no known primary), emit an empty `primaryName`, which the client renders as `"unknown"`. -- **Peers (4.x):** iterate `c->myconf->peers_cfg.candidates[0..count-1]` → one `CtrlPeer{name,host,port}` each; set `peerCount`. -- Answerable in every role; returns the local view even when fenced/no-quorum. - -**Dependencies** -- Inbound: `cluster.c` opcode switch — routes the request (P0). -- Outbound: `ha_state_t`, `myconf_t`, persist accessors — read-only snapshot (P0). -- Outbound: `init_reply_by_req`, `xce_reply` — reply encode/send (P0). - -#### Dispatch-gate allow-lists — Service contract -- Add `OPCode_ctrlstatus` to the allow-list in `ha_dispatch_gate` for `CTRL_PRIMARY` (including the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT` (7.1, 7.2). -- Constraint: the change is scoped strictly to `ctrlstatus`; no other opcode's gating may widen. `ctrlstatus` never requires quorum or a committed read. - -#### libxce client API — Service contract -```c -/* opcode-only request sender (clone of xce_clusterconf_req) */ -int xce_ctrlstatus_req(endpoint_t* ep); -/* send + await reply; on success MOVE_PTR the reply union out */ -int xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status); -/* failover-aware read (is_write=false): returns whatever node answers (may be a - * standby hint view). Use for the operator command. */ -int xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status); -/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the - * first answer is from a non-primary, follow the redirect — target the reported - * primaryName via the reply's peers list, else rotate the configured master list - * — until the primary answers or the failover budget is exhausted. */ -int xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status); -#define xcecli_free_ctrlstatus(s) xce_free_ctrlstatus(s) -``` -- Preconditions: valid client context. -- Postconditions: on success `*status` owns a decoded `CtrlStatus` (caller frees); on failure returns a typed negative (7.3) and `*status` untouched. For `xcecli_ctrlstatus_primary`, success additionally guarantees `responderRole == PRIMARY` (an authoritative answer). -- Invariants: `is_write=false` — retryable/rotatable across the configured controllers. - -#### `cmd_ctrlstatus` (test client) — State contract -- Calls `xcecli_ctrlstatus`; on success prints the fields in human-readable form (8.1) — `Primary`, `answered by <responderName> (<role>)` distinct from `Primary:` (3.3), peers + count, `configVersion` and `committedConfigVersion`, `configChecksum` as hex, term, commit/apply, fenced, cluster name; renders empty `primaryName` as `unknown`. Returns 0 on success, `<0` on failure with the error printed (8.2, 8.3). Frees the reply. - -#### `cmd_wait_primary` probe swap — State contract -- Replace the single `joblist_query(...)` probe call in the deadline loop with `xcecli_ctrlstatus_primary` (the authoritative variant). -- **Success predicate (9.1, 9.2):** the probe obtains an authoritative reply with `responderRole == PRIMARY` (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses. -- **Decision (resolves review Issue 1):** require the answer to come **from the primary**, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale `primary_host` after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "**serving** primary." The any-node hint view remains available for the operator `ctrlstatus` command via `xcecli_ctrlstatus`. -- **Preserved (9.4):** the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change. - -## Data Models - -The only new data is the reply payload above. `CtrlStatus` is a flat value object (no persistence, no aggregate); `CtrlPeer` is a bounded value list (≤ configured controller count, typically 3). `configChecksum` is the loaded config's 32-byte SHA-256 and `configVersion` is its own 8-byte prefix (a self-consistent pair); `committedConfigVersion` is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical `configVersion` + `configChecksum`; differing content differs in both (5.3). A node mid config-change may show `configVersion != committedConfigVersion`. - -## Error Handling - -- **No reachable controller (7.3):** `xcecli_ctrlstatus` returns the failover wrapper's typed negative (e.g. no-master/timeout); `cmd_ctrlstatus` prints it and returns non-zero (8.3); `wait-primary` keeps polling until its deadline (9.3). -- **Undeterminable field (1.3):** emit the defined "unknown"/empty value, never uninitialized data. -- **Standby with no known primary (2.3):** `primaryName` empty → "unknown"; the call still succeeds (the status itself is valid). -- **Read-only guarantee (1.2):** the handler takes only read locks/atomic loads; no state mutation on any path. - -## Testing Strategy - -### Unit / handler -- `handle_ctrlstatus` on a PRIMARY returns `primaryName==self`, `responderRole==PRIMARY`, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x). -- `handle_ctrlstatus` on a STANDBY with a known primary returns that primary and `responderRole==STANDBY`; with no known primary returns empty `primaryName` (2.2, 2.3). -- `fenced` reflects `in_catch_up`/`recovery_unrecoverable` (6.3); the handler mutates no state (1.2). -- `configVersion`/`configChecksum` are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; `committedConfigVersion` is exposed separately (5.1, 5.2, 5.3). - -### Integration -- Dispatch-gate: a `ctrlstatus` request is answered (not `-XCE_ERR_STANDBY`/`-XCE_ERR_NOT_READY`) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2). -- libxce round-trip: `xcecli_ctrlstatus` decodes all fields; `is_write=false` rotates across configured controllers; no-reachable-controller returns a typed failure (7.3). -- Authoritative read: `xcecli_ctrlstatus_primary`, when the first answer is from a standby, follows the redirect to the primary and returns a reply with `responderRole == PRIMARY`; it never returns a standby/hint reply as the authoritative result (9.1, 9.2). - -### E2E (live `/xce-deploy-ha`) -- `ctrlstatus` against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with `answered by <standby>` (2.3, 8.1). -- `wait-primary` (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the **new primary itself** answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the `ha-commit-durability` Phase 6 liveness/>60s-election investigation. - -## Security Considerations - -`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode. - ---- - -## Design Delta — Requirements 10 & 11 (2026-06-20) - -> **Merge note (as-built correction).** The handler shipped as `ctrlstatus_fill` + `handle_ctrlstatus` in **`src/controller/ctrlstatus.c`** (not `reply.c`); `configChecksum` is carried as **Text** lowercase-hex (the `Data` capnp type maps to the project's structured `xce_data_t`); `primaryName` is **role-derived** (primary → self, else empty) after E2E fix `ade0081` — `ha->primary_host` is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record. - -### What changed and why -Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` — an apparent commit/apply inversion — because `ctrlstatus_fill` reads the **primary-only** markers `persist.committed_lsn` (`ctrlstatus.c:130`) and `ha->committed_config_tag` (`:109`), which stay `0` on a node that never led. And there was no responder timestamp to line a captured status up against `ctrl-N.log`. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp. - -### Boundary Commitments (delta) -- **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`. -- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only. (Note: superseded by **ha-heartbeat-committed-propagation** — that spec implemented exactly this, adding `ha_consume_heartbeat_committed`/`ha_observe_heartbeat_committed`, invoked from the primary's heartbeat in `cluster.c`, which now refreshes `last_known_committed_lsn`/`last_known_committed_config_tag` on every heartbeat, term-guarded and monotonic. See that spec's design.md.) -- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:183`) and `ha->last_known_committed_config_tag` (`ha.h:185`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering. (Note: `last_known_committed_config_tag` is now also `_Atomic uint64_t` — migrated by ha-heartbeat-committed-propagation from the plain `uint64_t` this design was written against.) -- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, `ha_observe_heartbeat_committed` on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.) - -### Locked design decisions (from research items) -1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:5319` (`ha_vote_self_freshness`), `ha.c:6732`.) -2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2). (Note: superseded by **ha-heartbeat-committed-propagation** — `last_known_committed_lsn`/`_config_tag` are now also refreshed on every heartbeat via `ha_observe_heartbeat_committed`, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.) -3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.) -4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line. - -### Research item carried to implementation -- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` was non-atomic at design time (written under lock in `ha_convergence.c`); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by **ha-heartbeat-committed-propagation**, which migrated the field to `_Atomic uint64_t` at `ha.h:185`, written via `atomic_store_explicit` at `ha_convergence.c:368` and `ha.c:9011`; the as-built `ctrlstatus_fill` correctly reads it with `atomic_load_explicit`, not a bare aligned load.) - -### Schema delta (Cap'n Proto) -Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder): -```capnp - responderTimeMs @13: UInt64 $C.mapname("responder_time_ms"); # CLOCK_REALTIME epoch millis at snapshot (Req 11.1) -``` -No change to `committedLsn @8` / `committedConfigVersion @7` wire shape — Req 10 changes only the **value source**, not the field. - -### `ctrlstatus_fill` delta — Service contract (`src/controller/ctrlstatus.c`) -- **Req 10 (committed sourcing):** replace the unconditional reads at `ctrlstatus.c:108–110,130` with the role-branch above: - - `responderRole == CTRL_PRIMARY` → `committedLsn = persist.committed_lsn`, `committedConfigVersion = ha->committed_config_tag` (atomic acquire loads, unchanged). - - else → `committedLsn = ha->last_known_committed_lsn` (atomic acquire), `committedConfigVersion = ha->last_known_committed_config_tag` (plain aligned load). - - `appliedLsn` (`ha->applied_lsn`) is unchanged — it is already correct in every role. -- **Req 11 (timestamp):** stamp `st->responder_time_ms` from `clock_gettime(CLOCK_REALTIME)` (`tv_sec*1000 + tv_nsec/1e6`) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment. -- **Read-only invariant (10.4, 1.2) preserved:** all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation. - -### `cmd_ctrlstatus` delta — State contract (`src/controller/testsuite/test.c`) -- Print one added line rendering `responder_time_ms` as local `YYYY-MM-DD HH:MM:SS,mmm` (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print. - -### File Structure Plan (delta) -**Modified files:** -- `src/schema/controller.capnp` — add `responderTimeMs @13: UInt64` to `CtrlStatus` (regen `controller.capnp.h/.c` → `xce_ctrlstatus_t.responder_time_ms`). -- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: role-branch committed sourcing (Req 10) + `CLOCK_REALTIME` timestamp stamp (Req 11). -- `src/controller/testsuite/test.c` — `cmd_ctrlstatus`: render the responder timestamp line (Req 11.3). -- `src/controller/testsuite/test_ctrlstatus.c` — add/extend unit tests (below). - -No new files (the capnp field is generated; all logic slots into existing TUs). - -### Requirements Traceability (delta) -| Requirement | Summary | Components | Interfaces | -|-------------|---------|------------|------------| -| 10.1 | Off-primary reports *learned* committed LSN + config version | `ctrlstatus_fill` role-branch | `CtrlStatus.committedLsn`, `committedConfigVersion` (value source = `ha->last_known_committed_*`) | -| 10.2 | No learned value yet → 0/unknown | `ctrlstatus_fill` | `last_known_committed_lsn == 0` (correct unknown, per 1.3) | -| 10.3 | No spurious applied>committed inversion | `ctrlstatus_fill` | `appliedLsn` vs role-correct `committedLsn` | -| 10.4 | Read-only reporting; no production/replication change | `ctrlstatus_fill`; out-of-boundary routing | atomic/aligned loads only | -| 11.1 | Responder wall-clock at sample time | `ctrlstatus_fill` (`CLOCK_REALTIME`) | `CtrlStatus.responderTimeMs` | -| 11.2 | Same clock/format as logs | `cmd_ctrlstatus` render (xcelog.h form) | local `YYYY-MM-DD HH:MM:SS,mmm` | -| 11.3 | Test client prints the timestamp | `cmd_ctrlstatus` | added print line | - -### Testing Strategy (delta) -**Unit (`test_ctrlstatus.c`, TDD red→green):** -- STANDBY with `last_known_committed_lsn = 36`, `persist.committed_lsn = 0`, `applied_lsn = 38` → reply reports `committedLsn == 36` (not 0) and `appliedLsn == 38` (no inversion below the learned value) (10.1, 10.3). -- STANDBY pre-join (`last_known_committed_lsn == 0`) → reply reports `committedLsn == 0` / `committedConfigVersion == 0` (correct unknown) (10.2). -- PRIMARY → reply reports `committedLsn == persist.committed_lsn` and `committedConfigVersion == ha->committed_config_tag` (authoritative own markers unchanged) (10.1 primary branch). -- `committedConfigVersion` off-primary equals `last_known_committed_config_tag` (symmetry with the LSN) (10.1). -- `responder_time_ms` is non-zero and within a tolerance of a test-captured `CLOCK_REALTIME` sample; the handler still mutates no state (11.1, 1.2/10.4). - -**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility. - -**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.) diff --git a/.kiro/specs/controller-status/requirements.md b/.kiro/specs/controller-status/requirements.md deleted file mode 100644 index fe370a58..00000000 --- a/.kiro/specs/controller-status/requirements.md +++ /dev/null @@ -1,117 +0,0 @@ -# Requirements Document - -## Introduction - -This feature adds a **controller status query** (an XCE analog of LSF's `lsid`): a single, cheap request that returns basic cluster/controller identity and health — which node is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and a content checksum, and a few HA health/progress fields. Today no such query exists: a client must infer "is there a serving primary" indirectly (for example, by listing jobs), and there is no single call that reports who the primary is, the configured controller topology, or the committed configuration version. This makes operational visibility and failover-aware test orchestration awkward. - -The status query is served by **any** controller node — a standby reports who the primary is from the state it already tracks — and rides the existing client failover behavior, so a caller can get status without knowing which node is primary. The test client also gains a command that prints this status, and its `wait-primary` probe switches to this query so that waiting for a primary is cheaper and additionally reports which node became the serving primary. - -## Boundary Context - -- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe. -- **Out of scope**: an operator-facing CLI/REST status surface (a later pass); any change to HA election, replication, or commit behavior (this query only **reads** existing controller state); presentation/output formats beyond the query's structured reply and the test client's human-readable print. -- **Adjacent expectations**: relies on the existing client master-rotation/failover already used by other read operations; consumes the controller's existing HA state (current primary, role, term, committed configuration version, commit/apply progress, fenced state) without changing how that state is produced or maintained. The status query is a **focused identity/health view** and is intentionally distinct from the existing full cluster-configuration listing (which enumerates resources, queues, and workers). - -## Requirements - -### Requirement 1: Single-call controller status query - -**Objective:** As an operator or test author, I want to retrieve basic controller and cluster status in a single request, so that I can see cluster identity and health without inferring it from unrelated operations. - -#### Acceptance Criteria -1. When a client issues a controller status query, the Controller shall return, in a single request/response exchange, a response containing the status fields defined in Requirements 2 through 6. -2. The Controller shall treat the status query as read-only and shall not modify any cluster or controller state as a result of answering it. -3. If the responding controller cannot determine the value of a status field, then the Controller shall return a well-defined "unknown" value for that field rather than omitting it or returning uninitialized data. -4. The status response shall be limited to controller/cluster identity and health fields and shall not include the full resource, queue, or worker inventory. - -### Requirement 2: Current primary identity - -**Objective:** As an operator, I want the status response to identify the current primary, so that I know which node is authoritative. - -#### Acceptance Criteria -1. While a primary is currently elected and serving, when a client issues a status query, the Controller shall report the current primary's identity in the response. -2. If no primary has currently been elected, then the Controller shall report the primary identity as "unknown" rather than a stale or guessed value. -3. When the status query is answered by a standby node that knows the current primary, the Controller shall report that primary's identity; if the standby does not currently know a primary, the Controller shall report the primary identity as "unknown". - -### Requirement 3: Responding node identity and role - -**Objective:** As an operator, I want to know which node answered and its role, so that I can interpret the response in context without confusing it with the authoritative cluster view. - -#### Acceptance Criteria -1. When a client issues a status query, the Controller shall report the identity (name) of the node that answered the query. -2. When a client issues a status query, the Controller shall report the answering node's current role as one of: primary, standby, electing, or initializing. -3. The responding node's role shall be presented as a property of the answering node (for example, "answered by <node> (<role>)") and shall be distinct from the authoritative current-primary identity of Requirement 2, so that a transient answering-node role (such as electing or initializing) is not misread as the cluster's primary state. - -### Requirement 4: Configured controller topology - -**Objective:** As an operator, I want the configured controllers, so that I can see the cluster's intended membership and size. - -#### Acceptance Criteria -1. When a client issues a status query, the Controller shall report the list of configured controllers, each identified by its name and network address. -2. When a client issues a status query, the Controller shall report the total count of configured controllers. - -### Requirement 5: Committed configuration version and checksum - -**Objective:** As an operator, I want the configuration version and a content checksum, so that I can confirm which configuration the cluster has committed and detect content divergence between nodes even when their version numbers match. - -#### Acceptance Criteria -1. When a client issues a status query, the Controller shall report the committed configuration version currently in effect on the responding node. -2. When a client issues a status query, the Controller shall report a checksum (CRC) of the committed configuration content currently in effect on the responding node. -3. The reported configuration checksum shall be derived from the configuration content such that two nodes with identical committed configuration content report the same checksum, and two nodes with differing content report different checksums (enabling detection of same-version content divergence). - -### Requirement 6: HA health and progress fields - -**Objective:** As an operator, I want basic HA health and progress, so that I can assess election state and catch-up. - -#### Acceptance Criteria -1. When a client issues a status query, the Controller shall report the current term (election epoch). -2. When a client issues a status query, the Controller shall report its commit progress position and its apply progress position. -3. While the responding node is in a fenced or non-promotable state, when a client issues a status query, the Controller shall report that fenced state in the response. -4. When a client issues a status query, the Controller shall report the cluster name. - -### Requirement 7: Failover-aware, any-node delivery - -**Objective:** As a client, I want the status query to reach a reachable controller automatically, so that I can get status even across failover or when I do not know which node is primary. - -#### Acceptance Criteria -1. When a client issues a status query, the XCE client shall attempt the configured controllers using the same failover/rotation behavior used by other read operations until one responds or the attempts are exhausted. -2. When any configured controller, whether primary or standby, is reachable, the XCE client shall return that controller's status response. -3. If no configured controller is reachable within the client's failover budget, then the XCE client shall report a typed failure rather than reporting success with empty status. - -### Requirement 8: Test-client status command - -**Objective:** As a test author, I want a client command that prints the controller status, so that I can inspect cluster identity and health from a script. - -#### Acceptance Criteria -1. When the status command is issued in the test client, the test client shall print, in human-readable form, the current primary identity, the answering node's identity and role (as "answered by"), the configured controllers and their count, the committed configuration version, the committed configuration checksum, the current term, the commit/apply progress, the fenced state, and the cluster name. -2. When the underlying status query succeeds, the test client command shall report success (zero status). -3. If the underlying status query fails, then the test client command shall report failure (non-zero status) and shall print the error. - -### Requirement 9: wait-primary uses the status query - -**Objective:** As a test author, I want `wait-primary` to probe via the status query, so that the wait is cheaper and reports which node became the serving primary. - -#### Acceptance Criteria -1. When `wait-primary` probes the cluster, the wait-primary command shall use the controller status query, rather than a job-listing operation, to determine whether a serving primary is present. -2. When the status query reports a serving primary within the timeout, the wait-primary command shall complete successfully and shall report the identity of the serving primary. -3. If no serving primary is reported within the timeout, then the wait-primary command shall fail. -4. The change to the wait-primary probe shall preserve its existing timeout-argument handling and its success/failure exit semantics, adding only the served-primary identity to its success output. - -### Requirement 10: Off-primary committed-state fidelity - -**Objective:** As an operator, I want a standby's reported committed positions to reflect what it has actually learned from the primary, so that the status does not show a misleading committed value of zero while the node has clearly applied replicated entries. - -#### Acceptance Criteria -1. While the responding node is not the current primary and has learned a committed position from the primary, when a client issues a status query, the Controller shall report the committed configuration version and commit progress position that it has learned from the primary, rather than a node-local committed value that remains unset on a node that has never served as primary. -2. If the responding node is not the current primary and has not yet learned any committed position from a primary, then the Controller shall report the committed configuration version and commit progress position as the well-defined "unknown"/zero value (consistent with Requirement 1.3), rather than a stale or guessed value. -3. While the responding node is actively applying replicated entries, the Controller shall source the reported commit progress position and apply progress position consistently, so that the reported commit position is not spuriously lower than the position the node has actually committed or learned — avoiding an apparent "applied greater than committed" inversion that does not reflect the node's real state. -4. The committed-state fidelity in this requirement shall be a read-only reporting concern only and shall not change how committed positions are produced, replicated, or persisted. - -### Requirement 11: Responder snapshot timestamp for log correlation - -**Objective:** As an operator or test author, I want each status response to carry the responding node's clock time at the moment it sampled its state, so that I can line a captured status up against that node's logs. - -#### Acceptance Criteria -1. When a client issues a status query, the Controller shall include in the response the responding node's wall-clock time at the moment the status fields were sampled. -2. The reported timestamp shall be expressed on the same clock and in a form that an operator can directly compare against the responding controller's log timestamps. -3. When the status command is issued in the test client, the test client shall print the responder snapshot timestamp in human-readable form alongside the other status fields. diff --git a/.kiro/specs/controller-status/research.md b/.kiro/specs/controller-status/research.md deleted file mode 100644 index 14b5ab0d..00000000 --- a/.kiro/specs/controller-status/research.md +++ /dev/null @@ -1,153 +0,0 @@ -# Gap Analysis — controller-status (ctrlstatus) - -Analyzed the requirements against the existing codebase. The existing `clusterconf` opcode is a clean **opcode-only read** to model the plumbing on, and **all the status data already exists in-process**. The single requirements-critical new piece is the HA **dispatch-gate allow-list** (so the op answers on a standby/electing/no-quorum node). - -## Summary - -- **Recommended approach: a NEW dedicated lean opcode** (`ctrlstatus`), NOT an extension of `clusterconf`. This decisively resolves the spec's open "extend clusterconf vs new op" question: `clusterconf` is **primary-only when serving** (absent from every standby allow-list) and returns the heavy resource/queue/worker dump — extending it would still be primary-only (fails Req 7 any-node, Req 2.3 standby-reports-primary) and would carry the inventory (fails Req 1.4 focused view). -- **Config CRC: already exists, reuse.** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob (`myconf.h:31`, `config_repl.c:120-123`); full digest cached in `config_sha256[32]` (`myconf.h:32`) and round-trip-verified on load (`config_repl.c:652`). Requirement 5's version + checksum are both directly readable — no new computation. -- **All other status data exists**: role/term/applied_lsn/committed_config_tag are `_Atomic` (lock-free reads); `primary_host`/`primary_port` need `ha->lock`; peers via `c->myconf->peers_cfg.candidates[]`; fenced via `in_catch_up` + `persist_recovery_unrecoverable()`. -- **The one real gap**: `ha_dispatch_gate` (`ha.c:2059`) is an allow-list that defaults to `-XCE_ERR_STANDBY` (standby), `-XCE_ERR_NOT_READY` (init/electing), and gates the primary no-quorum branch. `OPCode_ctrlstatus` must be explicitly allow-listed in STANDBY + INIT + ELECTING + PRIMARY-no-quorum so a status read always answers — including when fenced/no-quorum, which is exactly when an operator queries it. -- **Generated codegen helps**: capnp structs are generated at build (`src/lib/meson.build`), so adding the schema struct + rebuild auto-produces `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, and `free_CtrlStatus`. - -## Requirement → Asset Map (gaps tagged) - -| Req | Need | Existing asset | Gap | -|-----|------|----------------|-----| -| 1 single-call read | opcode-only read + lean reply | `clusterconf` opcode-only pattern (`request.c:1734` `nomsg`); no-alloc reply model `ha_nop_reply_state_report` (`ha.c:1784`) | **Missing**: new `CtrlStatus` struct + `handle_ctrlstatus` (clone the no-alloc model, not the heap-xlist clusterconf one) | -| 2 primary identity (`unknown`) | current primary name | `ha->primary_host`/`primary_port` (`ha.h:149`) | **Constraint**: read under `ha->lock`; map empty→`"unknown"` | -| 3 answered-by node + role | own name + role | `c->myconf->name` (`myconf.h:29`); `ha->role` atomic (`ha.h:146`) | Reuse (lock-free) | -| 4 peers + count | configured controllers | `c->myconf->peers_cfg.candidates[]` + `candidates_count` (`cluster.h:32-38`) | Reuse, **no gap** | -| 5 config version + CRC | version + content checksum | `config_version_tag` (8-byte SHA-256 prefix) + `config_sha256[32]` (`myconf.h:31-32`); `committed_config_tag` (`ha.h:173`) | Reuse, **no gap** — expose, don't compute | -| 6 term/commit/apply/fenced/name | HA health | `ha->term`, `ha->applied_lsn` atomics; `persist.committed_lsn` atomic (no accessor — read directly); `in_catch_up` + `persist_recovery_unrecoverable()` (`persist.h:432`); cluster name | Reuse | -| 7 failover-aware any-node | retryable read on any node | `XCECLI_FAILOVER_CALL(is_write=false)` (`client.c:315`) | **Missing (CRITICAL)**: dispatch-gate allow-list entries for STANDBY/INIT/ELECTING/PRIMARY-no-quorum (`ha.c:2059`) | -| 8 test-client command | `ctrlstatus` print command | `cmd_*` table + display patterns in `test.c` | **Missing**: `cmd_ctrlstatus` (small) | -| 9 wait-primary uses it | switch the probe | `cmd_wait_primary` probe at `test.c:1161` | **Missing**: one-line probe swap + the new client fn | - -## Implementation Approach Options - -### Option A — Extend `clusterconf` with HA-status fields -Add primary/role/term/config-version fields to `ClusterConf` + `handle_clusterconf`. -- ✅ Fewest new symbols. -- ❌ **Fails Req 7 / 2.3**: `clusterconf` is not standby-allow-listed → primary-only when serving; a status op must answer on any node. ❌ **Fails Req 1.4**: still returns the resource/queue/worker inventory. ❌ Bloats a heap-xlist handler. -- **Rejected.** - -### Option B — New dedicated lean opcode `ctrlstatus` (RECOMMENDED) -New `OPCode_ctrlstatus @40`, lean `struct CtrlStatus`, `handle_ctrlstatus` on the no-alloc `state_report` model, **dispatch-gate allow-listed on all roles**, libxce `xce_ctrlstatus`/`xcecli_ctrlstatus` (is_write=false), `cmd_ctrlstatus`, wait-primary probe swap. -- ✅ Lean focused view (Req 1.4), any-node answerable (Req 7/2.3), no heap alloc, clean separation from `clusterconf`. -- ✅ Clones the proven opcode-only read plumbing; capnp regen auto-generates the C types. -- ❌ Touches ~9 files across schema/controller/libxce/client + the HA dispatch gate. -- **Recommended.** - -### Option C — Hybrid (reuse plumbing pattern, new op) -This is Option B in practice: reuse the `clusterconf` *plumbing pattern* (request/reply/failover/free) and the `state_report` *handler pattern*, but a brand-new opcode + struct + dispatch-gate work. No separate value beyond B. - -## Effort & Risk - -- **Effort: M (3–7 days).** All data exists and the clone pattern is proven (much is mechanical: schema + regen + req/api/cli/free + handler + command). The non-mechanical parts are the **dispatch-gate allow-listing** (HA-critical path, needs care + tests) and per-layer tests (capnp round-trip, handler-on-standby, client failover, wait-primary swap). -- **Risk: Low–Medium.** Low: no new infrastructure, content CRC already exists, read-only/no HA-behavior change. Medium: the dispatch gate is on the request-handling critical path — a wrong allow-list entry could let a standby serve something it shouldn't or fail to answer; must be scoped strictly to this read-only status op and covered by tests proving it answers on standby/electing/no-quorum **without** widening any other opcode's gating. - -## Research Items to Carry into Design - -1. **Reply field set & types**: expose `committed_config_tag` (8-byte version) as the version AND the full `config_sha256[32]` (hex) as the checksum? The tag is already a content-hash prefix — design should decide whether the full 32-byte digest is also returned for stronger same-version divergence detection (Req 5.3). -2. **Dispatch-gate semantics (key decision)**: a status read should answer **regardless of quorum/serving/fence** (that's when you most want it). Confirm `ctrlstatus` bypasses the quorum/serving gate in all role branches, and define precisely what each role returns (e.g. an ELECTING node returns role=electing, primary=unknown). -3. **Handler location**: `reply.c` (next to `handle_clusterconf`) vs `ha.c` (next to `ha_nop_reply_state_report`, the no-alloc model it should follow). Lean toward the `state_report` model. -4. **`"unknown"` representation** in the capnp reply (empty Text vs explicit sentinel) and how the client/test command renders it. -5. **wait-primary "serving primary present" definition** from the status reply: primary identity != unknown — and/or the responding node reports role=primary? Define the exact success predicate so the probe is unambiguous, preserving Req 9.4 (timeout/exit semantics unchanged). -6. **`committed_lsn` accessor**: none exists (`persist.committed_lsn` is read directly elsewhere). Design may add a thin `persist_committed_lsn()` for the handler, or read the atomic directly — minor. - -## Files in the clone map (for design/tasks) - -- `src/schema/controller.capnp` — `OPCode ctrlstatus @40`; `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")`; reply member `ctrlstatus @17`. (regenerates `controller.capnp.h/.c`) -- `src/controller/cluster.c:335` — `case OPCode_ctrlstatus → handle_ctrlstatus`. -- `handle_ctrlstatus` — new (model: `ha_nop_reply_state_report`, `ha.c:1784`; reply send via `xce_reply`, `init_reply_by_req`). -- `src/controller/ha.c:2059` — **add `OPCode_ctrlstatus` to STANDBY + INIT + ELECTING + PRIMARY-no-quorum allow-lists (the critical gap).** -- `src/lib/request.c` — `xce_ctrlstatus_req` (opcode-only, clone `request.c:1734`); `src/include/request.h` decl. -- `src/lib/xceapi.c:400` — `xce_ctrlstatus` (`MOVE_PTR` reply union). -- `src/lib/client.c:315` — `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, is_write=false); `src/include/client.h:55,90` decl + free macro. -- `src/controller/testsuite/test.c` — `cmd_ctrlstatus` command + registration; swap `cmd_wait_primary` probe at `test.c:1161`. - ---- - -## Design Synthesis (decisions resolved for design.md) - -The 6 carried research items are resolved as: -1. **Reply field set**: expose BOTH `configVersion` (the 8-byte `committed_config_tag`) AND `configChecksum` (the full 32-byte `config_sha256`) — version for quick compare, full digest for strong same-version divergence detection (Req 5.3). -2. **Dispatch-gate semantics**: `ctrlstatus` is allow-listed in ALL role branches (PRIMARY incl. no-quorum, STANDBY, ELECTING, INIT) and bypasses the quorum/serving/fence gate — a status read must answer precisely when the cluster is degraded. Each role returns its local view (PRIMARY→self; STANDBY→its known primary or unknown; ELECTING/INIT→unknown). -3. **Handler placement & shape**: `handle_ctrlstatus` in `reply.c` next to `handle_clusterconf`, following the no-alloc `ha_nop_reply_state_report` model; only the small bounded peer list allocates. `primary_host`/`port` read under `ha->lock`; everything else atomic/immutable. -4. **"unknown" representation**: empty `Text primaryName` on the wire; the client renders it as `"unknown"` (Req 1.3/2.2/2.3). -5. **wait-primary predicate**: success = reply reports a non-"unknown" primary (NOT "responder==PRIMARY", which could starve if a standby answers first under failover rotation). Accepted caveat: a brief standby-stale-primary window; still strictly more informative than the old joblist probe. Preserves all existing wait-primary timeout/exit semantics (Req 9.4). -6. **`committed_lsn` access**: read the persist atomic directly (no accessor exists); optionally add a thin `persist_committed_lsn()` — minor, implementer's choice. - -**Build-vs-adopt**: ADOPT the `clusterconf` opcode-only plumbing pattern + the `state_report` no-alloc handler model; BUILD only the new schema structs, handler body, dispatch-gate entries, and the thin client/test wiring. No new infrastructure; config checksum reused (not built). - ---- - -## Design-review updates (2026-06-19, /kiro-validate-design) - -Two review issues resolved with the user; design.md updated: - -1. **wait-primary predicate → AUTHORITATIVE from the primary (user decision).** A standby's reply is a non-authoritative HINT (it may name a just-failed primary from stale `primary_host`). The op still answers on any node (Req 2.3/7), but for an authoritative answer the client follows the redirect — maps the reply's `primaryName` to an address via the reply's `peers` list, else rotates the configured master list — until a reply with `responderRole == PRIMARY`. New `xcecli_ctrlstatus_primary` provides this; `wait-primary` uses it (success = the primary itself answered), eliminating the stale-standby false-success (the same false-positive class removed from the durability soak). The plain `xcecli_ctrlstatus` (any-node hint view) backs the operator `ctrlstatus` command. -2. **Config fields → loaded pair + committed tag (user decision).** Report `configVersion`+`configChecksum` from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND `committedConfigVersion` (`committed_config_tag`) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing. - -Also fixed: mermaid render issues — removed `/`, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram). - ---- - -## Gap Analysis — Delta for Requirements 10 & 11 (2026-06-20) - -Reopened after a live HA-deploy E2E finding: a standby answering `ctrlstatus` showed -`committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` (an apparent -applied-greater-than-committed inversion), and there was no way to correlate a captured -status against the per-node logs. Scope of this delta = Req 10 (off-primary committed -fidelity) + Req 11 (responder snapshot timestamp). Reqs 1–9 are implemented and unchanged. - -### Requirement-to-Asset Map - -| Need (Req) | Existing asset | Gap | -|---|---|---| -| 10.1 standby reports *learned* committed LSN | `ctrlstatus_fill` reads `c->persist.committed_lsn` (`ctrlstatus.c:130`) — a **primary-only** marker, 0 on a node that never led | **Constraint**: wrong source field; the learned value `ha->last_known_committed_lsn` (`ha.h:170`) exists but is unused here | -| 10.1 standby reports *learned* committed config tag | `ctrlstatus_fill` reads `ha->committed_config_tag` (`ctrlstatus.c:109`) — also primary-only | **Constraint**: learned `ha->last_known_committed_config_tag` (`ha.h:172`) exists but is unused here | -| 10.1 freshness of the learned value | `last_known_committed_lsn` written only at `cluster_apply_ctrlhello_reply` (`cluster.c:291`, `ha_convergence.c:375`) and an election safe-commit path (`ha.c:925`) | **Constraint (key)**: refreshed only at **CtrlHello join / RDHUP-failover handshake**, NOT per heartbeat. The `OP_nop` heartbeat (`cluster.c:300`) carries no committed payload; the richer `HeartbeatMsg{committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is **defined but never consumed** on receive → the learned value can LAG between joins | -| 10.2 genuinely-unknown → 0/unknown | existing `NULLSTR`/0 conventions (Req 1.3) | none — `last_known_committed_lsn` is 0 before first learn, which is the correct "unknown" | -| 11.1 responder wall-clock at sample time | none on `CtrlStatus` (schema `controller.capnp:496`, fields @0–@12; next free ordinal **@13**) | **Missing**: new field + a `CLOCK_REALTIME` stamp in `ctrlstatus_fill`. Note `ha_on_heartbeat_received` uses `CLOCK_MONOTONIC` — **not** reusable for log correlation | -| 11.2 same clock/format as logs | logger header `src/include/xcelog.h` produces `[YYYY-MM-DD HH:MM:SS,mmm]` | **Research Needed**: confirm the exact formatter/helper in `xcelog.h` to reuse (or match) for byte-comparable output | -| 11.3 test client prints timestamp | `cmd_ctrlstatus` in `src/controller/testsuite/test.c` (C `xce_test` client; print block ~line 1297) — the "client" is xce_test, not Go | **Missing**: one print line; libxce decode is codegen from schema (auto after regen) | - -### Implementation Approach Options - -**Req 10 — Option A (Extend `ctrlstatus_fill`) — RECOMMENDED.** When `responderRole != CTRL_PRIMARY`, -source `committedLsn` from `ha->last_known_committed_lsn` and `committedConfigVersion` from -`ha->last_known_committed_config_tag`; keep the primary's own `persist.committed_lsn` / -`committed_config_tag` when it IS primary (those are the authoritative live values). Read-only, -satisfies Req 10.4, mirrors the existing `last_known_*` usage in the RDHUP/freshness paths -(`ha.c:2570`, `ha.c:4126`). A simpler variant — report `max(persist.committed_lsn, -last_known_committed_lsn)` uniformly (no role branch) — also kills the 0-inversion; design picks. -*Trade-off*: ✅ tiny, in-boundary, fixes the inversion. ❌ value can lag between joins (see below). - -**Req 10 — out-of-boundary (do NOT pull into this spec).** Making the off-primary committed -value *live-fresh* requires consuming `HeartbeatMsg.committedLsn` (or extending `OP_nop`) and -storing it into `last_known_committed_lsn` on **every** heartbeat — a heartbeat/replication -protocol change. That violates Req 10.4 ("shall not change how committed positions are produced, -replicated, or persisted") and belongs to **ha-heartbeat-liveness** / **ha-repl-hardening**. -controller-status should report the honest best-known value and the design must state the lag caveat. - -**Req 11 — Option A (Extend) — RECOMMENDED.** Add `responderTime @13: UInt64` (epoch millis) to -`CtrlStatus`; stamp it in `ctrlstatus_fill` from `CLOCK_REALTIME`; render in `cmd_ctrlstatus` -formatted via the `xcelog.h` convention so it lines up with `ctrl-N.log`. *Design decision*: -numeric epoch-millis (client formats — locale-safe, clean) vs pre-formatted `Text` (zero client -work, mirrors the `configChecksum`-as-Text precedent). Recommend numeric epoch-millis. -*Trade-off*: ✅ one field + one stamp + one print line. ❌ adds a schema ordinal (codegen regen). - -### Effort & Risk - -- **Req 10**: **S** (≤1 day) — one function's field sourcing + TDD (standby-known-primary asserts learned value; pre-learn asserts 0). **Risk Low** — read-only, no HA-state change. -- **Req 11**: **S** (1–2 days) — schema field + `ctrlstatus_fill` stamp + `cmd_ctrlstatus` print + codegen regen + round-trip decode test. **Risk Low**. -- **Combined**: **S**. The one **Medium-risk temptation to refuse**: expanding into heartbeat committed-propagation — keep it out of this spec. - -### Research Items to Carry to Design - -1. Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h`; reuse it (or document the chosen format) so Req 11's value is byte-comparable to log lines. -2. Decide Req 11 wire type: `UInt64` epoch-millis (recommended) vs `Text`. -3. Decide Req 10 sourcing: role-branch (`persist.*` on primary, `last_known_*` off-primary) vs uniform `max()`; keep LSN and config-tag symmetric. -4. State the **staleness caveat** in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to **ha-heartbeat-liveness / ha-repl-hardening** as out-of-boundary. diff --git a/.kiro/specs/controller-status/spec.json b/.kiro/specs/controller-status/spec.json deleted file mode 100644 index 2ce7c61e..00000000 --- a/.kiro/specs/controller-status/spec.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "feature_name": "controller-status", - "created_at": "2026-06-19T00:00:00Z", - "updated_at": "2026-07-17T11:15:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation": { - "completed": true, - "tasks_done": "1.1-5.1 (all)", - "notes": "ctrlstatus opcode + handler + all-roles dispatch gate + libxce any-node & authoritative reads + ctrlstatus command + wait-primary swap, all reviewed and committed; E2E-verified live (primary self-reports, standbys answer with answered-by, config SHA-256 checksum identical across nodes, authoritative wait-primary names the primary). E2E found+fixed the primaryName-from-vestigial-primary_host bug. Out-of-boundary upstream finding: libxce failover returns EINVAL on a dead-first-master (affects joblist equally) -> route to libxce-failover." - }, - "delta": { - "reopened_at": "2026-06-20T00:00:00Z", - "reason": "Live E2E finding (2026-06-20): ctrlstatus answered by a standby reports committedLsn=0 / committedConfigVersion=0 while appliedLsn=38 — it reads the primary-only raw markers (persist.committed_lsn, ha->committed_config_tag) instead of the standby's learned committed position (ha->last_known_committed_lsn / last_known_committed_config_tag). Plus an observability gap: no responder timestamp to correlate a captured status against ctrl-N.log.", - "new_requirements": "10 (off-primary committed-state fidelity), 11 (responder snapshot timestamp for log correlation)", - "needs": "design + tasks regen for Req 10-11 only; prior tasks 1.1-5.1 remain implemented", - "impl_status": "6.1-6.4 all implemented + reviewed + committed and complete. 6.4 (live-cluster E2E), previously blocked on the sandbox overlayfs fdatasync issue (bug-795), was VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy per tasks.md's own Implementation Notes -- this field was stale until this validation pass corrected it." - }, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-17T11:15:00Z", - "note": "First post-implementation validation run -- all tasks (1.1-6.4) were already [x] with no prior validate_impl record, and this delta block's impl_status field had gone stale (still said 6.4 BLOCKED despite tasks.md recording it VERIFIED since 2026-07-02). Full suite 9/9 green (one transient ASAN failure during the run was traced to a concurrent second session competing for the same build directory, not a regression -- reproduced clean 3x in isolation), 36/36 requirement acceptance criteria mapped, cross-task wire contract verified field-for-field end to end (schema -> typemap -> handler -> dispatch -> libxce client -> CLI), two live smoke runs (a 3-round sequential HA soak plus a targeted live 3-node deploy exercising Req 10/11 directly) both PASS.", - "known_limitations": [ - "A live observation during smoke testing: querying ctrlstatus on a standby within <1s of a fresh submit can show appliedLsn > committedLsn. This is the documented, in-boundary staleness caveat from design.md's Locked Decision #2 (last_known_committed_lsn refreshes only at CtrlHello join/RDHUP, not per-heartbeat) -- explicitly anticipated and accepted at design time, not a regression of the original zero-value bug this spec's delta phase fixed." - ] - } -} diff --git a/.kiro/specs/controller-status/tasks.md b/.kiro/specs/controller-status/tasks.md deleted file mode 100644 index 4a896e39..00000000 --- a/.kiro/specs/controller-status/tasks.md +++ /dev/null @@ -1,108 +0,0 @@ -# Implementation Plan - -> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor. -> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2). - -- [x] 1. Foundation: status reply schema + codegen -- [x] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema - - Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count. - - Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory. - - Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean. - - _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_ - -- [x] 2. Controller: status handler + any-role dispatch -- [x] 2.1 Implement the controller status handler - - Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role. - - Route the new opcode to this handler in the controller request dispatch switch. - - TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state. - - Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds. - - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_ - - _Boundary: handle_ctrlstatus_ -- [x] 2.2 Allow the status query through the HA dispatch gate in every role - - Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode. - - TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed. - - Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode. - - _Requirements: 7.1, 7.2_ - - _Boundary: ha_dispatch_gate_ - - _Depends: 2.1_ - -- [x] 3. libxce client API -- [x] 3.1 Add the failover-aware any-node status call - - Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success. - - TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure. - - Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable. - - _Requirements: 1.1, 7.1, 7.2, 7.3_ - - _Boundary: libxce client_ - - _Depends: 1.1, 2.1, 2.2_ -- [x] 3.2 Add the authoritative reach-the-primary status call - - Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted. - - TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result). - - Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves. - - _Requirements: 2.3, 9.1, 9.2_ - - _Boundary: libxce client_ - - _Depends: 3.1_ - -- [x] 4. Test client command + wait-primary integration -- [x] 4.1 Add the `ctrlstatus` client command - - Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query. - - Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure. - - _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_ - - _Boundary: cmd_ctrlstatus_ - - _Depends: 3.1_ -- [x] 4.2 Switch wait-primary to the authoritative status probe - - Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics. - - Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer. - - _Requirements: 9.1, 9.2, 9.3, 9.4_ - - _Boundary: cmd_wait_primary_ - - _Depends: 3.2_ - -- [x] 5. Validation (live cluster) -- [x] 5.1 End-to-end status + authoritative wait-primary on a live cluster - - Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves. - - Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks. - - _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_ - - _Depends: 4.1, 4.2_ - -- [x] 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11) - > Sequential, single request path (schema → typemap → `ctrlstatus_fill` → render → E2E); no `(P)`. Read-only reporting only — the live-fresh-committed heartbeat change is OUT of boundary (route to ha-heartbeat-liveness / ha-repl-hardening). -- [x] 6.1 Add the responder snapshot timestamp field to the schema + hand-authored typemap - - Append a responder snapshot-time field to the status payload at the next free ordinal (after the peer count), carrying the responder's wall-clock at sample time as epoch milliseconds (append-only — never reorder existing field ordinals). Add the matching field to the hand-authored `xce_ctrlstatus_t` typedef in `src/include/typemap.h` and verify it field-for-field against the regenerated codec (per the 1.1/2.1 learning: `$C.mapname` types are NOT auto-generated here). - - Observable: the build regenerates `controller.capnp.h/.c` clean, the typemap matches the codec, and a codec round-trip carries the new field's value unchanged. - - _Requirements: 11.1_ - - _Boundary: schema + typemap_ -- [x] 6.2 Source off-primary committed values + stamp the timestamp in `ctrlstatus_fill` - - TDD (red first, in `test_ctrlstatus.c`): a STANDBY whose learned committed position is non-zero while its local committed marker is zero and it has applied entries reports the LEARNED committed LSN and committed config version (not zero) and the real applied position (no applied-greater-than-committed inversion); a STANDBY that has not yet learned any committed position reports committed LSN and committed config version as zero/unknown; a PRIMARY reports its own authoritative local committed LSN and committed config tag (unchanged); and the responder snapshot timestamp is non-zero and within tolerance of a test-captured wall-clock sample. Assert the handler still mutates no state. - - Change `ctrlstatus_fill` (`src/controller/ctrlstatus.c`): role-branch the committed sourcing — when the responder is primary, keep its own committed LSN / committed config tag; otherwise report the learned committed LSN / learned committed config tag. Stamp the responder snapshot timestamp from the wall clock (`CLOCK_REALTIME`, NOT the monotonic clock) at snapshot time. All additions are atomic/aligned loads + a clock read — no state mutation. - - Observable: the new unit tests go red → green; the standby case reports the learned non-zero committed value instead of zero; the no-mutation assertion holds. - - _Requirements: 10.1, 10.2, 10.3, 10.4, 11.1_ - - _Boundary: ctrlstatus_fill_ - - _Depends: 6.1_ -- [x] 6.3 Render the responder snapshot timestamp in the `ctrlstatus` client command - - Add one printed line to `cmd_ctrlstatus` (`src/controller/testsuite/test.c`) rendering the responder snapshot timestamp as a local `YYYY-MM-DD HH:MM:SS,mmm` string in the same form the controller logger uses, so it can be compared directly against `ctrl-N.log`. Confirm and reuse (or match) the exact wall-clock formatter in `src/include/xcelog.h`. The committed lines need no rendering change — task 6.2 only changed their values. - - Observable: piping `ctrlstatus` through the client prints a responder-timestamp line in the log timestamp format alongside the other status fields. - - _Requirements: 11.2, 11.3_ - - _Boundary: cmd_ctrlstatus_ - - _Depends: 6.1, 6.2_ -- [x] 6.4 End-to-end validation on a live cluster (off-primary committed + log correlation) <!-- VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy — see Implementation Notes --> - - Run against a live 3-node cluster: submit jobs so the primary advances its committed position, then query a STANDBY. Assert the standby's reported committed LSN is non-zero (tracking the primary within the documented join-refresh caveat) rather than zero while it has applied entries, and that the printed responder timestamp matches that standby's own `ctrl-N.log` line for the same query. - - Observable: the live run shows a non-zero off-primary committed LSN (no zero-while-applied inversion) and a responder timestamp that lines up with the standby's log; manual environment prerequisite (live cluster), same status as the other live-E2E task (5.1). - - _Requirements: 10.1, 10.3, 11.2, 11.3_ - - _Depends: 6.2, 6.3_ - -## Implementation Notes (5.1 E2E outcome) -- 5.1 ran live on a 3-node `/xce-deploy-ha`-style cluster (ext2/3 fs). PASSED in-scope: `ctrlstatus` shows the primary self-reporting (`Primary: ctrl-1` on the primary), standbys honestly report `Primary: unknown` with `answered by ctrl-2 (standby)` (the dispatch-gate lets standbys answer — Req 7); peers + count correct; `configChecksum` = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); `configVersion`==`committedConfigVersion`; authoritative `wait-primary` names the primary in <1s on a serveable cluster (Req 9.1/9.2). -- 5.1 FOUND + FIXED a real 2.1 bug: `ha->primary_host` is VESTIGIAL (never written anywhere in the controller), and `voted_for_name` is the last-vote (often self), so neither names the current primary off-primary. Corrected: PRIMARY self-reports its own name; STANDBY/ELECTING/INIT report empty → "unknown" (the authoritative client path reaches the real primary by rotation). Commit ade0081. -- 5.1 UPSTREAM FINDING (out of boundary, NOT controller-status): with a DEAD node FIRST in the masters list, the libxce client failover returns `EINVAL` instead of advancing to the live nodes — so the authoritative `wait-primary` (and ANY client op) can't reach the primary while a dead node leads the list. CONFIRMED pre-existing: plain `joblist` fails IDENTICALLY (`xce_joblist: Invalid argument`) in the same condition; with a live-only masters list `wait-primary` succeeds in 46ms. This spec consumes failover unchanged (per Boundary/Adjacent expectations) → route to libxce-failover, do NOT patch here. The durability soak avoided it by restarting killed nodes (no dead-first-master during waits). - -## Implementation Notes -- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`. -- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened. -- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary. -- 1.1 LEARNING (affects later tasks): the `$C.mapname(...)` capnp types are NOT auto-generated in this schema — the generated `controller.capnp.c` references HAND-AUTHORED C typedefs in `src/include/typemap.h` (same convention as `xce_clusterconf_t`). So `xce_ctrlstatus_t`/`xce_ctrlpeer_t` (and the `xce_reply_t.data.ctrlstatus` union member) live in `typemap.h`, verified field-for-field against the regenerated codec. `OPCode_ctrlstatus = 40`, `RequestReply_data_ctrlstatus = 12`. `configChecksum` is `xce_data_t*`. The handler/libxce tasks build/encode against these typemap.h types. -- 2.1 LEARNING + SCHEMA AMENDMENT: capnp `Data` in THIS schema maps to the project's STRUCTURED `xce_data_t` union (table-switched), NOT raw bytes — `encode_Data_ptr` can't serialize a SHA-256 blob. So `configChecksum @6` was changed `Data → Text` (a 64-char lowercase-hex string of the full 32-byte `config_sha256`); typemap member is `char*`; freed in `ctrlstatus_free_fields`. Ordinal @6 unchanged, no released decoder yet → safe. Handler lives in NEW `src/controller/ctrlstatus.c` (`ctrlstatus_fill`/`ctrlstatus_free_fields`/`handle_ctrlstatus`), proto in `src/controller/include/cluster.h`, routed at `cluster.c` opcode switch, test `src/controller/testsuite/test_ctrlstatus.c` (registered in `controller_test.c`, built via `controller/meson.build`). `primary_host` read under `ha->lock`; peer name/host strings are BORROWED from myconf (not freed). The libxce client (3.x) decodes `configChecksum` as Text. -- 3.1 LEARNING: the ctx-side reply union `xce_ctx_t.reply.u` (in `src/include/xcectx.h`) is SEPARATE from `xce_reply_t.data` (typemap.h) — the decode path needs BOTH a `ctrlstatus` member there AND a `case OPCode_ctrlstatus` in `xcectx.c` handle_reply (mirror clusterconf). Free: clusterconf's `xce_free_clusterconf` macro is a LATENT-BROKEN alias (undefined symbol, never expanded); use the REAL generated free — `xce_free_ctrlstatus(s) → free_CtrlStatus(s)`, `xcecli_free_ctrlstatus(s) → xce_free_ctrlstatus(s)` (any TU expanding it must include the generated `controller.capnp.h`). `xcecli_ctrlstatus` is is_write=false; NULL ctx → `-EINVAL`. -- 3.2: `xcecli_ctrlstatus_primary` = bounded loop (cap `failover.retry.max_retries`) reusing `xcecli_ctrlstatus` + `xce_ctx_capture_redirect_hint` (map `primaryName`→addr via `peers[]`, else master rotation); authoritative iff `responderRole == CTRL_PRIMARY` (== 0; `ctrl_role_t` PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 in `ha.h` — REVALIDATION TRIGGER: a role-enum reorder breaks the pinned wire ordinal `XCECLI_CTRLSTATUS_ROLE_PRIMARY=0`). On exhaustion → `-E_XCE_NOT_READY`, `*status` untouched (never a hint). Pure helpers `xcecli_ctrlstatus_is_authoritative`/`_primary_addr` unit-tested; live standby→primary redirect is 5.1. -- 6.1: `responderTimeMs @13: UInt64` appended to `CtrlStatus` (after `peerCount @12`, append-only) + matching `uint64_t responder_time_ms` in the hand-authored `xce_ctrlstatus_t` (`src/include/typemap.h`); codec wire offset 48, round-trips. Commit 65ddafe. -- 6.2: `ctrlstatus_fill` role-branches committed sourcing — PRIMARY → own `persist.committed_lsn`/`committed_config_tag` (atomic acquire); else → `ha->last_known_committed_lsn` (atomic acquire) + `ha->last_known_committed_config_tag` (plain aligned read, NO `ha->lock` per design — `ha.h:172` is non-atomic, written under lock at `ha_convergence.c:377`). `appliedLsn` unchanged. Timestamp stamped from `CLOCK_REALTIME` (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): `last_known_committed_lsn` refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6. -- 6.3 LEARNING: the controller log timestamp formatter is NOT in `xcelog.h` (an 85-line thin wrapper that just `#include "tlog.h"`); the `YYYY-MM-DD HH:MM:SS,mmm` prefix is assembled inline in vendored tinylog (`tlog.c` `_tlog_root_default_format` ~393/398, via `my_tlog_format` in `src/lib/utils.c:221/228`) — no caller-facing helper, so it was REPLICATED byte-equivalently (`localtime_r` + `strftime "%Y-%m-%d %H:%M:%S"` + `,%03u` of `ms%1000`) in a new header-only `src/controller/testsuite/ctrlstatus_fmt.h` (`ctrlstatus_format_time_ms`), shared by `test.c` (`cmd_ctrlstatus` prints `responderTime:` after `fenced`) and `test_ctrlstatus.c` (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to `ctrl-N.log` under the same TZ. -- 6.4 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy` via `src/tools/deploy-ha.sh`, submitted jobs against the full masters list, then ran `ctrlstatus` pointed directly at standby ctrl-2 (`set masters localhost:9992`). Result: `answered by ctrl-2 (standby)`, `committedLsn: 4` / `appliedLsn: 4` — non-zero, matching, no inversion (the exact 2026-06-20 bug symptom does not reproduce). `responderTime: 2026-07-02 14:18:19,689` is a live, current wall-clock value consistent with the session timeline. NOTE on the log-correlation half of the Observable: `ctrlstatus_fill` (ctrlstatus.c) has no logging call for the query itself (pure snapshot-fill, no INFO/TRACE line), so there is no literal `ctrl-N.log` line to string-match against — verified the weaker-but-sufficient form instead (responderTime is live/plausible, not frozen/stale/wrong-TZ). PASS. Found and logged (unrelated, low-priority): bug-1400 (xce_test cmd_jobsub SEGVs on malformed positional args instead of required getopt flags — test-client input validation gap, not a production bug) and bug-1401 (ctrlstatus reply decode leaks ~120 bytes per call in `decode_CtrlStatus_ptr`, ASAN-only visibility, ~103 allocs/12KB observed over a dozen calls in this session). diff --git a/.kiro/specs/dirty-content-fingerprint/design.md b/.kiro/specs/dirty-content-fingerprint/design.md deleted file mode 100644 index d530eb93..00000000 --- a/.kiro/specs/dirty-content-fingerprint/design.md +++ /dev/null @@ -1,237 +0,0 @@ -# Design Document - -## Overview -**Purpose**: This feature gives every XCE component's `--version` output the ability to distinguish two builds made from the same commit but with different uncommitted changes — replacing the existing plain `-dirty` boolean suffix (which cannot tell two dirty builds apart) with a short, content-derived fingerprint. - -**Users**: Developers and operators iterating on a dev-loop build (`xce-build.sh`) who rebuild repeatedly while editing, and need to tell which specific edit produced the binary in front of them, without falling back to manual file-timestamp or diff inspection. - -**Impact**: Extends the already-shipped `build-time-versioning` feature's commit_id/build_time mechanism with a third, optional value. `commit_id`, `build_time`, and the startup-log output are all left completely untouched — same computation, same value, same call sites. Only the `--version`-composition call site changes: when a fingerprint is available, it strips `commit_id`'s existing `-dirty` suffix (if present) and appends the fingerprint instead, producing a clean `<commit_id>-<hash>` shape without redundant markers; when no fingerprint is available, `--version` falls back to showing `commit_id` exactly as computed today (see `research.md`'s "Leave commit_id's computation fully untouched" decision). - -### Goals -- A dirty build's `--version` output distinguishes itself from a different dirty build of the same commit, whenever their uncommitted content differs. -- Two builds with identical uncommitted content produce the identical fingerprint, reproducibly. -- A clean build's `--version` output is completely unchanged (just the commit identifier, as today). -- All six components (controller, scheduler, worker, apiserver, xcecli, xceadmin) get this capability. -- The daemon startup log line is untouched by this feature. - -### Non-Goals -- Showing which files changed or their diff content anywhere in `--version` output (fingerprint only, not a diff viewer). -- Extending the startup log line to include the fingerprint (explicitly out of scope per Requirement 3). -- Closing the Nix production path's untracked-file blind spot (documented, accepted limitation — see `research.md`). -- Any change to `commit_id`'s computation, value, or its existing `-dirty` suffix, or to `build_time`'s computation — genuinely unaffected by this feature everywhere. The `-dirty`-stripping needed for a clean `--version` format is display-only logic inside the `--version`-composition call, not a change to `commit_id` itself. - -## Boundary Commitments - -### This Spec Owns -- Computation of a new `dirty_fingerprint` value at both existing computation sites (`xce-build.sh`, `flake.nix`). -- Threading that value through the existing single-resolved-value-pair meson-option mechanism as a third option. -- The `--version`-only composition logic that combines `commit_id` and `dirty_fingerprint` for display (including the `-dirty`-stripping rule, applied only at this one call site — see `research.md`). - -### Out of Boundary -- The startup log line's content or format (unchanged; still shows only `commit_id`/`build_time`, per the original feature, via its completely untouched composition call site). -- The original feature's `commit_id`/`build_time` computation, values, or the `-dirty` suffix `commit_id` itself carries — genuinely unmodified everywhere, per the corrected design decision in `research.md`. -- Any mechanism for an operator to retrieve or view the actual uncommitted diff/file list (out of scope per requirements' Boundary Context). -- Making the Nix path's fingerprint reflect unstaged untracked files (documented limitation, not solved here). - -### Allowed Dependencies -- The original `build-time-versioning` feature's meson option mechanism (`src/meson.options`, `src/meson.build`'s `add_project_arguments()`), extended with one new option, not replaced. -- `xce-build.sh`'s existing `commit_id`/`build_time` computation block and `MESON_OPTS` construction (task 2.1 of the original feature). -- `xce.nix`/`xcecli.nix`'s existing `preConfigure` build-phase hook (task 2.2 of the original feature). -- Each component's existing version-composition function/mechanism (`xce_compose_version()` for C, the Go ldflags vars, `_build_info`/`cli.py` for Python) — extended with one new optional field, not replaced. - -### Revalidation Triggers -- `xce_compose_version()`'s signature or call sites change shape (would affect how the new field is threaded through). -- The original feature's `commit_id`/`build_time` meson options are renamed or restructured. -- `xce-build.sh`'s `MESON_OPTS` construction pattern changes. -- `self.narHash`'s semantics change in a future Nix version (unlikely, but the design's Nix-path assumption depends on it). - -## Architecture - -### Existing Architecture Analysis -- The original feature established: two computation sites (`xce-build.sh` for dev-loop, `xce.nix`/`xcecli.nix` for production) each resolve `commit_id`/`build_time` into meson options; six consumption sites each read those options once and never recompute. This feature adds a third value to that exact same shape. -- Each of the six components already has exactly one place where its version string is composed for `--version` and, separately (for the four daemons), one place where the startup log's version fields are logged — both currently reading the identical `commit_id`/`build_time` values. This feature's `--version`-only scope means the `--version` composition call site gains the new field while the startup-log composition call site does not — the first deliberate divergence between those two call sites in this feature family. - -### Architecture Pattern & Boundary Map -Same two computation sites, one additional resolved value, fed only into each component's `--version` composition call — never into its startup-log call. - -```mermaid -graph TB - DevScript[xce-build.sh git diff plus untracked files sha256] - NixPkg[flake.nix eval time self.narHash truncated] - MesonOpt[meson dirty_fingerprint option] - VersionCompose[per-component --version composition] - LogCompose[per-component startup log composition unchanged] - - DevScript --> MesonOpt - NixPkg --> MesonOpt - MesonOpt --> VersionCompose - MesonOpt -.not consumed.-> LogCompose -``` - -**Architecture Integration**: -- Selected pattern: extend the existing single-resolved-value-pair fan-out with a third, optional value that only one of each component's two existing consumption call sites reads. -- Domain/feature boundaries: computation (2 sites, unchanged locations) stays separated from consumption; the new divergence is which consumption call site (per component) reads the new value, not a new computation or plumbing layer. -- Existing patterns preserved: meson option → C define / Go ldflags / Python generated module, identical to the original feature. -- New components rationale: no new components — every change extends an existing computation site or an existing composition call site with one additional field/argument. -- Steering compliance: no new external dependency; TDD required for every new behavior (per `testing.md`). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Dev-loop computation | `git diff HEAD` + `git ls-files --others --exclude-standard`, `sha256sum` (coreutils, already used by `xce-build.sh`) | Computes the fingerprint from tracked-modification diffs and untracked-file content | `xce-build.sh` | -| Nix production computation | `self.narHash` (Nix flake evaluator, eval-time) | Content hash of the dirty tree's tracked-file state, truncated the same way | `xce.nix`, `xcecli.nix`; empirically confirmed reachable and content-sensitive (see `research.md`) | -| C build | Meson preprocessor define (existing mechanism, one new define) | Carries `dirty_fingerprint` into `xce_compose_version()`'s `--version`-composition call only | `src/meson.build` | -| Go build | `-ldflags -X` (existing mechanism, one new var) | Injects `main.DirtyFingerprint`, read only by each binary's `--version`/`-version` handling | `go_build`, Makefile | -| Python build | Meson `custom_target` (existing mechanism, one new field) | `_build_info.py` gains a third field, read only by `cli.py`'s `click.version_option()` | `meson-build.sh` | - -## File Structure Plan - -### Modified Files -- `src/meson.options` — add a `dirty_fingerprint` string option (default `''`), sibling to the existing `commit_id`/`build_time`. -- `src/meson.build` — add `-DVERSION_DIRTY_FINGERPRINT=` to the existing `add_project_arguments()` call. -- `src/include/utils.h`, `src/lib/utils.c` — extend `xce_compose_version()` (or add a sibling composition function used only by `--version` call sites) to optionally append `-<dirty_fingerprint>` when non-empty; leave the existing signature's startup-log call sites unaffected (they simply don't pass the new field, or call the original unchanged function). -- `src/controller/main.c`, `src/scheduler/src/main.c`, `src/controller/testsuite/test.c`, `src/worker/src/main.c` — `--version`/version-flag handling call sites gain the new field; startup-log call sites are explicitly left unchanged (one-line, clearly-commented divergence — see `research.md`'s Design Decision). -- `src/tools/xce-build.sh` — compute `dirty_fingerprint` (git diff + untracked file content, sha256 truncated, empty when clean) and pass it via a new `-Ddirty_fingerprint=` entry in `MESON_OPTS`, alongside the existing `commit_id`/`build_time` entries. -- `src/flake.nix` — the existing eval-time `let` block gains `myDirtyFingerprint` (derived from `self.narHash`, truncated, empty on a clean tree), computed alongside `myCommitId` and passed into `nixSupport/default.nix` as a new argument, the same way `myCommitId` already is. -- `src/nixSupport/default.nix` — accepts and forwards `myDirtyFingerprint` into each `callPackage ./xce.nix`/`./xcecli.nix` call via `inherit`, alongside the existing `myCommitId`. -- `src/nixSupport/xce.nix`, `src/nixSupport/xcecli.nix` — accept the new `myDirtyFingerprint` argument and append `"-Ddirty_fingerprint=" + myDirtyFingerprint` to the existing `mesonFlags` list (eval-time, alongside `-Dcommit_id=`) — not through the `preConfigure`/`mesonFlagsArray` mechanism, which stays reserved for `build_time`. -- `src/tools/go_build` — read `XCE_DIRTY_FINGERPRINT` (empty-safe) and append it to the existing `-ldflags -X` construction as `main.DirtyFingerprint`. -- `src/frontend/apiserver/meson.build`, `src/frontend/xcecli/meson.build` — pass the new env var into `go_build`'s env, same pattern as `XCE_GIT_COMMIT`/`XCE_BUILD_TIME`. -- `src/frontend/Makefile` — compute and pass the same `-ldflags -X main.DirtyFingerprint=...` for `build-server`/`build-cli`'s dev-loop path. -- `src/frontend/apiserver/cmd/main.go` — declare `var DirtyFingerprint = ""`; `-version` flag's composed string appends it when non-empty; startup log's `logger.Info(...)` call is left unchanged (no new zap field). -- `src/frontend/xcecli/cmd/main.go` — declare the same var; the version string passed to `router.Execute(cm, version)` (which sets `rootCmd.Version`) appends it when non-empty. -- `src/xceadmin/meson.build` — pass one new positional arg (or env var) to `meson-build.sh`. -- `src/xceadmin/meson-build.sh` — `_build_info.py` gains a third field, `DIRTY_FINGERPRINT` (empty-string fallback, matching the existing per-field `unknown`-vs-empty convention used for the other two fields — see Implementation Notes). -- `src/xceadmin/src/xceadmin/_build_info.py` (committed fallback stub) — gains `DIRTY_FINGERPRINT = ""` alongside the existing two `"unknown"` fields. -- `src/xceadmin/src/xceadmin/cli.py` — `_compose_version()` appends the fingerprint when non-empty. - -> No new files. Every change extends an existing computation site, an existing meson option chain, or an existing composition call site. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1, 1.2, 1.3 | Content-derived indicator present when dirty, deterministic, distinguishes different uncommitted content | `xce-build.sh` (git diff + untracked files), nix `self.narHash` | `dirty_fingerprint` meson option, per-component `--version` composition | Computation → meson → `--version` composition | -| 1.4 | Excludes ignored files | `xce-build.sh`'s `git ls-files --others --exclude-standard` (respects `.gitignore` natively); nix's `self.narHash` (only ever sees files the flake's git input would include, which already excludes ignored files) | n/a | n/a | -| 2.1 | Clean tree shows commit id alone | All 6 `--version` composition call sites | `dirty_fingerprint` empty string → no suffix appended | n/a | -| 3.1, 3.2 | `--version`-only scope; startup log unchanged | Each daemon's two separate composition call sites | Startup-log call site never receives `dirty_fingerprint` | n/a | -| 4.1 | Applies to all 6 components | All 6 consumption sites | Same as original feature's Requirement 1.1/1.2 fan-out, one new field | n/a | -| 5.1, 5.2 | Graceful degradation when content can't be determined | Every `--version` composition call site (falls back to `commit_id` as-is when `dirty_fingerprint` is empty) | `dirty_fingerprint` empty on computation failure; `commit_id`'s own pre-existing, untouched `-dirty` suffix supplies the "still indicate modified" signal for free | n/a | - -## Components and Interfaces - -### C Daemons (controller, scheduler, worker) - -#### Version composition, `--version`-only extension -| Field | Detail | -|-------|--------| -| Intent | Append a content-derived fingerprint to `--version`'s commit id when the tree is dirty, without touching the startup log's version fields | -| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1 | - -**Responsibilities & Constraints** -- `commit_id` and its existing `-dirty` suffix logic (from the original `build-time-versioning` feature) are **not modified** — identical computation, identical value, at both computation sites. -- `xce_compose_version()` (or a new sibling function reusing its internals) gains an optional `dirty_fingerprint` parameter, consumed **only by the `--version` call path**: when `dirty_fingerprint` is non-empty, the composition strips a trailing `-dirty` from `commit_id` (if present) and appends `-<dirty_fingerprint>` instead, producing `<commit_id>-<hash>`; when `dirty_fingerprint` is empty, the composition uses `commit_id` completely as-is — which already correctly shows either the bare hash (clean tree) or `<hash>-dirty` (tree is dirty but the fingerprint computation failed, Requirement 5.2's fallback, satisfied by the pre-existing mechanism with no new sentinel value needed). -- Each `main.c`'s `--version`/`argp_program_version` call site passes the new field; the corresponding startup-log INFO call site is **not touched at all** — it keeps calling the original, unmodified composition path exactly as the original feature shipped it, so it never even references `dirty_fingerprint`. -- Worker's hand-added `--version` case (from the original feature) gets the same treatment as controller/scheduler. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: extends `xce_compose_version()` — check during implementation whether adding a parameter to the existing function is cleaner than adding a second function; either is acceptable as long as the startup-log call site's behavior is provably unchanged (byte-for-byte) by a regression test. -- Validation: (1) a build with `dirty_fingerprint` empty and a clean tree must produce `--version` output byte-identical to the original feature's already-shipped clean-tree behavior (`<commit_id>` alone); (2) a build with `dirty_fingerprint` empty and a dirty tree (fingerprint computation failed) must produce `--version` output byte-identical to the original feature's already-shipped dirty-tree behavior (`<commit_id>-dirty`) — Requirement 5.2, satisfied without new code beyond "don't touch commit_id when dirty_fingerprint is empty"; (3) a build with a real, non-empty `dirty_fingerprint` must produce `<bare-commit-hash>-<dirty_fingerprint>`, with the `-dirty` suffix stripped, not doubled. -- Risks: (1) accidentally threading the new field into the startup-log call site would silently violate Requirement 3.2 — must be caught by a test asserting the startup log line is unchanged from the original feature's already-shipped format; (2) the `-dirty`-stripping logic must anchor to the literal trailing suffix (not a blind substring removal) — covered by a dedicated unit test. - -### Go Binaries (apiserver, xcecli) - -#### ldflags fingerprint injection, `--version`/`-version`-only -| Field | Detail | -|-------|--------| -| Intent | Same `--version`-only extension for the Go toolchain | -| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1 | - -**Responsibilities & Constraints** -- `go_build` reads `XCE_DIRTY_FINGERPRINT` (empty-safe) and appends `-X main.DirtyFingerprint=$v` to its existing `-ldflags` construction. `GitCommit` itself (and whatever `-dirty` suffix it already carries) is completely untouched. -- `apiserver/cmd/main.go`: the `-version` flag's composition applies the same strip-and-append rule as the C daemons — if `DirtyFingerprint` is non-empty, strip a trailing `-dirty` from `GitCommit` and append `-DirtyFingerprint`; if empty, print `GitCommit` as-is (already correctly showing `-dirty` or not). The existing `logger.Info("********** Initialize Logger done!")` structured-field call is left completely unchanged (no new zap field, `GitCommit` passed exactly as before) — this is the explicit Requirement 3.2 boundary for this component. -- `xcecli/cmd/main.go`: the version string passed into `router.Execute(cm, version)` applies the same strip-and-append rule; xcecli has no startup log (already established as out of scope by the original feature), so there is no divergence to manage here. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: reuses `go_build`'s existing env-var pattern, one more variable. -- Validation: apiserver's startup log must be provably unchanged (byte-for-byte against the original feature's shipped format) even when `DirtyFingerprint` is non-empty. -- Risks: none identified beyond the same one already flagged for C — accidental leakage of the new field into the startup-log call site. - -### Python (xceadmin) - -#### Generated build-info module, third field -| Field | Detail | -|-------|--------| -| Intent | Same `--version`-only extension for xceadmin, which has no startup log at all (one-shot CLI, already out of scope per the original feature) | -| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1, 4.1 | - -**Responsibilities & Constraints** -- `meson-build.sh` writes `DIRTY_FINGERPRINT` (empty-string fallback) into `_build_info.py` alongside the existing `GIT_COMMIT`/`BUILD_TIME` fields. `GIT_COMMIT` itself (which already carries a `-dirty` suffix on a dirty tree, inherited from the shared `commit_id` meson option) is completely untouched. -- The committed fallback stub (`src/xceadmin/src/xceadmin/_build_info.py`) gains `DIRTY_FINGERPRINT = ""` — an editable install (`pip install -e .`) shows no fingerprint suffix, consistent with the stub's existing all-`unknown`/empty convention. -- `cli.py`'s `_compose_version()` applies the same strip-and-append rule as the other components: if `DIRTY_FINGERPRINT` is non-empty, strip a trailing `-dirty` from `GIT_COMMIT` and append `-DIRTY_FINGERPRINT`; if empty, use `GIT_COMMIT` as-is. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: one more field in the same generated-module mechanism the original feature already built (task 5.1/5.2). -- Validation: editable install must still show no fingerprint (empty field, no import error); wheel install (built dirty) must show a real, non-empty fingerprint. -- Risks: none identified — xceadmin already has no startup log to accidentally leak into. - -### Build Orchestration - -#### Dev-loop fingerprint computation -| Field | Detail | -|-------|--------| -| Intent | Compute a content-based fingerprint of the uncommitted tree, with zero working-tree mutation | -| Requirements | 1.1, 1.2, 1.3, 1.4, 5.1, 5.2 | - -**Responsibilities & Constraints** -- `commit_id` and its existing `-dirty` suffix (`xce-build.sh` task 2.1 of the original feature) are **not touched** by this computation — this task only adds a new, independent `dirty_fingerprint` computation alongside it. -- Computes `dirty_fingerprint` as one of two states: **empty** when the tree is clean, *or* when the tree is dirty but the hash computation itself fails (in the latter case, `commit_id`'s own pre-existing, untouched `-dirty` suffix already signals "modified" once `--version` falls back to using it as-is — Requirement 5.2, satisfied with no new sentinel value); a **real hash** (`sha256sum` truncated to a short hex prefix) over the concatenation of `git diff HEAD`'s output plus the raw content of each file `git ls-files --others --exclude-standard` lists, in stable sorted order, when the tree is dirty and the computation succeeds. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: added to the same `xce-build.sh` block that already computes `commit_id`/`build_time` (original feature's task 2.1), no restructuring. -- Validation: the exact scenario this feature exists for — edit a file, build, note the fingerprint; edit further, rebuild, confirm a different fingerprint; revert to the first edit's exact content, rebuild, confirm the fingerprint matches the first build's again. -- Risks: `git diff HEAD` output size for a very large uncommitted change set is unbounded in principle — acceptable for this codebase's realistic dev-loop usage (source-code-sized diffs), not a concern requiring truncation of the *input*, only the *output* hash. - -#### Nix production fingerprint computation -| Field | Detail | -|-------|--------| -| Intent | Same capability for the Nix production path, within its sandboxed, no-git constraint | -| Requirements | 1.1, 1.2, 1.3, 5.1 | - -**Responsibilities & Constraints** -- `myCommitId` and its existing `self.dirtyShortRev`-derived `-dirty` suffix (original feature) are **not touched** by this computation. -- `myDirtyFingerprint` is derived from `self.narHash` at **eval time**, in `flake.nix`'s existing `let` block, alongside `myCommitId` — **not** via a `preConfigure` build-phase hook. Unlike `build_time` (which needed a build-phase hook specifically because `builtins.currentTime` breaks pure-eval), `self.narHash` is a legitimate flake self-introspection field with no such trap — empirically confirmed reachable and stable under plain `nix build`/`nix eval` (no `--impure`) in `research.md`. It is passed into `xce.nix`/`xcecli.nix` (via `nixSupport/default.nix`) as a new function argument, mirroring `myCommitId`'s existing parameter, and appended to each derivation's `mesonFlags` list directly (`"-Ddirty_fingerprint=" + myDirtyFingerprint`), the exact same way `-Dcommit_id=` already is — not through `mesonFlagsArray`/`preConfigure`, which stays reserved for genuinely build-phase-only values like `build_time`. -- Resolves to the same two states as the dev-loop path: **empty** when the tree is clean, checked the same way `myCommitId` already branches on `hasAttr "shortRev"` (a clean tree only has `self.shortRev`, not `self.dirtyShortRev`); otherwise a **real hash**, truncated to match the dev-loop path's format. -- **Known, accepted limitation** (see `research.md`): does not reflect brand-new untracked files unless staged. Must be documented as a code comment at this `let`-binding, not silently left as a surprise. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -**Implementation Notes** -- Integration: extends `flake.nix`'s existing eval-time `myCommitId` computation with one sibling value, passed through to `xce.nix`/`xcecli.nix` as one more `mesonFlags` entry — no new build-phase mechanism, no change to the existing `build_time` `preConfigure` hook. -- Validation: `nix build .#xcecore_debug` (no `--impure`, the documented canonical command) must still succeed and produce a binary whose `--version` fingerprint changes when a tracked file in the flake's source is edited and the flake is re-evaluated. -- Risks: low — `research.md` already empirically confirmed `self.narHash` is reachable and content-sensitive without `--impure`; the implementer should still re-confirm live as part of RED-phase validation rather than merely trusting this document, since the earlier `build_time`/`builtins.currentTime` mistake in the original feature came from exactly that kind of unverified assumption. - -## Testing Strategy - -### Unit Tests -- C: `test_utils.c` cases asserting `xce_compose_version()`'s `--version`-path composition: (1) `dirty_fingerprint` non-empty + `commit_id` ending in `-dirty` → strips the suffix, appends the fingerprint (`<hash>-<fingerprint>`, not `<hash>-dirty-<fingerprint>`); (2) `dirty_fingerprint` empty + clean `commit_id` → `<hash>` alone, unchanged; (3) `dirty_fingerprint` empty + `commit_id` ending in `-dirty` (fingerprint computation failed, tree still dirty) → `<hash>-dirty`, Requirement 5.2's fallback, produced by simply not touching `commit_id` — no special-casing needed in the composition function beyond the empty check. -- Shell: a unit-level check of `xce-build.sh`'s fingerprint-computation logic confirms it emits a real hash when the tree is dirty and the computation succeeds, and confirms it emits empty (not a fabricated value) both when the tree is clean and when the hash-computation step is made to fail (simulated) on a dirty tree — the latter case relying on `commit_id`'s own separate, untouched `-dirty` computation to still convey "modified" once composed. -- Go: a test asserting `-version`/`--version` output includes `DirtyFingerprint` when set, and that apiserver's startup-log-composition path is unaffected by its presence. -- Python: a `pytest` case asserting `xceadmin --version` includes the fingerprint field when `_build_info.DIRTY_FINGERPRINT` is non-empty, and omits any suffix when empty (both the committed-stub and generated-module cases, mirroring the original feature's task 5.2 test pattern). - -### Integration Tests -- `xce-build.sh`-driven dev-loop build (any C daemon): edit a tracked file, build, capture `--version`'s fingerprint; edit again with different content, rebuild, confirm the fingerprint differs; revert to the first edit's exact content, rebuild, confirm the fingerprint matches the first capture exactly. This is the direct regression test for Requirement 1's core capability. -- Same daemon, confirm the startup log line is untouched (byte-identical to the original `build-time-versioning` feature's already-shipped format) regardless of whether the tree is dirty. -- A clean-tree build (`git stash` the dev tree, or run in a freshly-cloned worktree) confirms `--version` shows the commit id alone, no suffix — the direct regression test for Requirement 2. - -### E2E / Consistency -- For the Nix path: confirm `nix build .#xcecore_debug` (no `--impure`) succeeds and the resulting binary's `--version` fingerprint changes when a tracked source file is edited before evaluation — the direct regression test for the `self.narHash`-based mechanism actually working end-to-end, not just at the `nix eval` probe level this design's research was based on. diff --git a/.kiro/specs/dirty-content-fingerprint/requirements.md b/.kiro/specs/dirty-content-fingerprint/requirements.md deleted file mode 100644 index 2292b0af..00000000 --- a/.kiro/specs/dirty-content-fingerprint/requirements.md +++ /dev/null @@ -1,56 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Dirty-content fingerprint for --version output: extends the build-time-versioning feature (already shipped) so the version string can distinguish between different uncommitted-change states at build time, not just a boolean "-dirty" flag. Today, xce-build.sh computes commit_id as `git rev-parse --short HEAD` with a plain "-dirty" suffix when the tree has uncommitted changes -- this means two different builds made from the same dirty commit (edit file, rebuild, edit further, rebuild again) are indistinguishable from their version string alone, since the suffix never changes as long as *some* uncommitted change exists. - -New behavior: when the source tree is clean, --version continues to show just `<commit_id>` (no change). When the tree is dirty, --version shows `<commit_id>-<hash>` where `<hash>` is a short (e.g. 8 hex char) content-based digest computed from exactly what `git add -A && git commit` would capture -- i.e. `git diff HEAD`'s content for modified/staged tracked files, plus the full content of untracked files. Any actual content change (even to the same set of files) must change the hash; an unchanged uncommitted state must produce the same hash across repeated builds. This is a *stricter* version of the same requirement the original feature already implements ("mark version info as modified when uncommitted changes exist"), not a new requirement. - -**Superseded during design** (kept here for input-history fidelity, not as authoritative intent -- see `research.md`'s "Leave commit_id's computation fully untouched" decision for the resolved approach): an earlier framing of this input proposed that "the literal '-dirty' string is retired in favor of the hash's mere presence." That approach was rejected after design tracing revealed it would silently break the startup log's existing modified-tree signal (the startup log reads the same `commit_id` value `--version` does). The resolved design instead leaves `commit_id`'s `-dirty` suffix completely untouched everywhere, and strips it only inside the `--version`-composition call, only when a fingerprint is actually available. - -Scope: --version output only, for all six components (controller, scheduler, worker, apiserver, xcecli, xceadmin) -- the existing startup log line (controller/scheduler/worker/apiserver) is explicitly OUT of scope for this enhancement and keeps its current behavior/format unchanged; this is a deliberate, scoped exception to the original feature's guarantee that `--version` and the startup log never disagree, analogous to how xcecli's startup log was already excluded in the original spec for being a one-shot CLI. - -Open technical question requiring investigation during design (light discovery, extension feature): the project's dev-loop build path (xce-build.sh) has real `git` available and can straightforwardly compute this hash via `git diff HEAD` + untracked file contents. The production Nix build path (xce.nix/xcecli.nix) runs in a sandboxed derivation without `git` on PATH by design (this repo deliberately keeps git out of the nix devShell buildInputs) and currently gets its "-dirty" marker for free from Nix's own eval-time `self.dirtyShortRev` introspection, which is NOT a content hash (it can't distinguish two different dirty states either). Design must investigate whether Nix's flake evaluator exposes an equivalent eval-time content hash (e.g. `self.narHash`) that could serve this purpose without shelling out to git inside the sandbox -- verify this empirically rather than assuming. If no such eval-time-safe content hash exists for the sandboxed path, document that as a real constraint and decide a reasonable fallback -- do not invent a mechanism that would require adding git as a nix sandbox dependency, since that was an explicit prior architectural decision in the original spec's research. - -Hash algorithm/length is an implementation detail for design to decide (short, stable, no external dependency beyond what's already available in each toolchain/environment). - -## Boundary Context (Optional) -- **In scope**: computing and displaying, in `--version` output only, a content-derived indicator that distinguishes different uncommitted-change states, for controller, scheduler, worker, apiserver, xcecli, and xceadmin; graceful degradation when that indicator cannot be determined in a given build environment. -- **Out of scope**: any change to the startup log format of controller, scheduler, worker, or apiserver (unchanged by this feature); the underlying commit identifier / build timestamp mechanism already shipped by the build-time-versioning feature (extended, not replaced); exposing the names of changed files or diff content anywhere in `--version` output (this feature only makes two builds comparable for content-equality, it does not reveal what changed); a tool for comparing or explaining the difference between two builds; retrofitting this indicator into already-built/deployed binaries. -- **Adjacent expectations**: this feature depends on the already-shipped build-time-versioning feature's commit identifier and build timestamp display; it is additive to that mechanism. An operator wanting to know exactly what changed between two builds is still expected to inspect the source tree themselves (e.g. via `git diff`); this feature only tells them whether two builds' uncommitted content was identical or different, it does not show the content itself. - -## Requirements - -### Requirement 1: Content-Based Distinguishability of Uncommitted State -**Objective:** As an operator comparing two builds made from the same base commit at different points during active development, I want their version identifiers to differ whenever the uncommitted content differs, so that I can tell apart two builds that a plain "modified" marker would otherwise show as identical. - -#### Acceptance Criteria -1. When the source tree contains uncommitted changes at build time, the system shall include, alongside the build's commit identifier, a content-derived indicator reflecting the actual content of those uncommitted changes. -2. If two builds are produced from the same commit but with different uncommitted content, then the system shall produce different content-derived indicators for the two builds. -3. If two builds are produced from the same commit with identical uncommitted content, then the system shall produce the identical content-derived indicator for both builds. -4. The system shall derive the content-derived indicator only from files that would be included by committing all current changes (tracked-file modifications and untracked files), excluding any file the source tree's ignore rules exclude from commits. - -### Requirement 2: Clean-Tree Behavior Unchanged -**Objective:** As an operator inspecting a build produced from an exact, unmodified commit, I want its version identifier to remain exactly the commit identifier with no additional marker, so a clean build stays as concise as it is today and is never confused with a modified one. - -#### Acceptance Criteria -1. While the source tree contains no uncommitted changes at build time, the system shall present the version identifier as the commit identifier alone, with no content-derived indicator or other modified-state marker appended. - -### Requirement 3: Scope — Version Query Surface Only -**Objective:** As an operator, I want the content-derived indicator available when I explicitly query a component's version, without changing the routine startup log output I already rely on. - -#### Acceptance Criteria -1. When an operator queries a component's version via its version-query command, the system shall include the content-derived indicator described in Requirement 1 for that build. -2. The system shall present the commit identifier and build timestamp in the startup log of controller, scheduler, worker, and apiserver without the content-derived indicator, leaving the existing startup log format unchanged. - -### Requirement 4: Applies to Every Version-Displaying Component -**Objective:** As an operator, I want this distinguishing capability available consistently across every component that already displays version information, so I do not need to remember which components support it. - -#### Acceptance Criteria -1. The system shall provide the content-derived indicator described in Requirement 1 for controller, scheduler, worker, apiserver, xcecli, and xceadmin. - -### Requirement 5: Graceful Degradation When Content Cannot Be Determined -**Objective:** As a developer running the build tooling in an environment where the uncommitted content cannot be reliably inspected, I want the build to still succeed and clearly indicate the limitation, so that I never see a fabricated or misleading indicator. - -#### Acceptance Criteria -1. If the build tooling cannot determine the content of uncommitted changes at build time, then the system shall omit the content-derived indicator rather than fabricate one. -2. If the build tooling cannot determine the content of uncommitted changes at build time while the source tree does have uncommitted changes, then the system shall still indicate that the version information reflects a modified tree, so the operator is not misled into believing the build is clean. diff --git a/.kiro/specs/dirty-content-fingerprint/research.md b/.kiro/specs/dirty-content-fingerprint/research.md deleted file mode 100644 index 5bcf74e8..00000000 --- a/.kiro/specs/dirty-content-fingerprint/research.md +++ /dev/null @@ -1,79 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `dirty-content-fingerprint` -- **Discovery Scope**: Extension (builds directly on the shipped `build-time-versioning` feature's commit_id/build_time mechanism) -- **Key Findings**: - - Nix's flake evaluator exposes `self.narHash` at eval-time, reachable from inside `xce.nix`/`xcecli.nix` exactly like the already-used `self.shortRev`/`self.dirtyShortRev` — empirically confirmed content-sensitive (different tracked-file content produces a different hash; identical content reproduces the identical hash) and requires no `git` on the sandboxed build's PATH, respecting the same architectural constraint the original feature's research already established. - - `self.narHash` has a real, empirically-confirmed blind spot: it reflects modifications to already-**tracked** files, but does **not** pick up brand-new **untracked** files unless they are `git add`-staged first. This differs from the dev-loop path's planned `git diff HEAD` + untracked-file-content approach, which naturally includes unstaged untracked files. - - The dev-loop path (`xce-build.sh`) has real `git` available (established precedent from the original feature's task 2.1) and can compute an untracked-file-inclusive fingerprint directly via `git diff HEAD` plus `git ls-files --others --exclude-standard` (the exact enumeration `git add -A` would use, already respecting `.gitignore`), with zero working-tree mutation. - - `commit_id`'s existing computation (and its `-dirty` suffix) must stay completely untouched by this feature -- an earlier design draft that retired `-dirty` at the computation site was caught and corrected after tracing its blast radius onto the startup log, which reads the same `commit_id` value and would have silently lost its only modified-tree signal. The `-dirty`-stripping needed for a clean `--version` format instead happens only inside the `--version`-composition call, only when a fingerprint is actually available (see Design Decisions). - -## Research Log - -### Does Nix expose an eval-time content hash of the dirty tree, usable inside the sandboxed build without `git` on PATH? -- **Context**: The original build-time-versioning feature's nix path (`xce.nix`/`xcecli.nix`) gets its `-dirty` marker for free from `self.dirtyShortRev`, which is a boolean-ish suffix, not a content hash — it can't distinguish two different dirty states from each other, the same limitation this feature exists to fix for the dev-loop path. Needed to determine, empirically, whether an equivalent content-sensitive value exists for the nix path without shelling out to git inside the sandbox (which would violate the original feature's explicit "no git as a nix sandbox dependency" architectural decision). -- **Sources Consulted**: Live empirical testing against this repo's actual `src/flake.nix`, via `nix flake metadata ./src --json` and a temporary `debugNarHash = self.narHash;` output added to and then reverted from `src/flake.nix`. -- **Findings**: - 1. `nix flake metadata ./src --json`'s `locked.narHash` field is present even for a dirty git-type flake input, and is reachable from inside the flake's own expressions as `self.narHash` (confirmed via `nix eval ./src#debugNarHash --raw`, temporarily added then reverted). - 2. Editing a tracked file in the flake's directory (`src/flake.nix` itself, via a temporary comment line) changed `narHash` from `sha256-50twgW+...` to `sha256-FEEP3A...`; a second, differently-worded edit changed it again to `sha256-CdD8g1...`; reverting the file to its exact original committed content (via `git checkout --`) reproduced the exact original `narHash` (modulo unrelated background tree changes between captures -- see caveat below). This confirms `narHash` is a genuine, deterministic content hash of tracked-file state, not merely a "some change exists" boolean. - 3. Creating a new, unstaged, untracked file (`src/README.md`) inside the flake's directory did **not** change `narHash` at all -- even after further editing that same untracked file's content. Only after `git add`-staging it (still uncommitted) did `narHash` change. This is a real, confirmed blind spot: unstaged untracked files are invisible to Nix's dirty-tree evaluation. - 4. Caveat on finding #2's revert check: the final "revert to baseline" capture did not exactly match the very first baseline capture, because unrelated `.wolf/*` bookkeeping files (tracked, modified by this session's OpenWolf hook automation between captures) also feed into the same repo-wide `narHash`. This is expected -- `narHash` reflects the *entire* dirty tree's tracked-file state, not just the file under test -- and does not undermine the core finding (content-determinism), it only means two captures separated by real intervening changes elsewhere in the tree will legitimately differ, exactly as intended. -- **Implications**: `self.narHash` is a viable, eval-time-safe basis for a nix-path content fingerprint, with one documented, accepted limitation: it will not reflect a build made dirty *only* by adding new untracked files that were never staged. Given the nix production path is not the path implicated in the original motivating incident (that was `xce-build.sh scheduler debug`), and given a workaround requiring `git add` or `git` on the sandbox's PATH was already ruled out by the original feature's architectural decision, this asymmetry is accepted and documented rather than engineered around. - -### Untracked-file enumeration for the dev-loop path, without mutating the working tree -- **Context**: The dev-loop path (`xce-build.sh`) needs to include untracked files' content in its fingerprint (unlike the nix path, which has the staging blind spot above) while making zero changes to the developer's working tree (no `git add`, no `git stash`). -- **Findings**: `git ls-files --others --exclude-standard` lists exactly the untracked files that are not excluded by `.gitignore`/`.git/info/exclude` -- the same set `git add -A` would pick up -- without touching the index or working tree. Combined with `git diff HEAD` (which already covers all tracked modifications, staged or not, relative to `HEAD`), concatenating `git diff HEAD`'s output with the raw content of each file `git ls-files --others --exclude-standard` lists (in a stable, sorted order) and hashing the result gives a fingerprint matching Requirement 1's AC4 exactly ("only from files that would be included by committing all current changes... excluding ignored files"). -- **Implications**: No new git subcommands beyond what's already used elsewhere in this codebase (`git diff`, `git status --porcelain` equivalents); no working-tree mutation; deterministic given a fixed input order. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Single shared meson option (matches original feature's pattern) | `xce-build.sh`/nix each compute their own fingerprint value, both feed a new `dirty_fingerprint` meson option, consumed once by C/Go/Python exactly like `commit_id`/`build_time` already are | Reuses the exact, already-proven single-resolved-value-pair architecture; zero new plumbing pattern to learn | Two computation sites, asymmetric capability (nix misses unstaged untracked files) | Selected -- see Decision below | -| Compute fingerprint independently per consumption site | Each of the 6 components computes its own fingerprint at its own build step | No new meson option | Directly reintroduces the exact multi-computation-site drift risk the original feature's architecture was designed to avoid | Rejected | - -## Design Decisions - -### Decision: Extend the existing single-resolved-value-pair architecture with a third value -- **Context**: The original feature already established "one resolved value pair (commit_id, build_time), two computation sites, six consumption sites reading the same values, never recomputing." This feature adds a third value (the content fingerprint) that is conceptually parallel to `commit_id` (both describe "what source produced this build") but is only ever non-empty when the tree is dirty. -- **Alternatives Considered**: - 1. Fold the fingerprint directly into `commit_id`'s own string (e.g., `xce-build.sh` composes `<hash>-<content_hash>` and passes that whole string as `commit_id`) -- rejected: conflates two independently-meaningful values into one opaque string, makes Requirement 2 (clean tree = commit id alone, byte-identical to today) awkward to guarantee at the composition site instead of trivially true by construction. - 2. A wholly separate new meson option (`dirty_fingerprint`), computed alongside `commit_id`/`build_time` by the same two computation sites, consumed only by the composition step that already builds the final version string for `--version` (not by the startup-log composition path) -- selected. -- **Selected Approach**: Add `dirty_fingerprint` as a new meson string option (default `''`), computed by `xce-build.sh` (git diff + untracked file content, sha256 truncated) and by `xce.nix`/`xcecli.nix` (`self.narHash`, truncated the same way, empty when the tree is clean since there is nothing to distinguish). The existing C `xce_compose_version()`/Go ldflags/Python `_build_info` consumption sites gain a fourth field, used only by the `--version`-composition call site of each component, not by the startup-log composition call site -- this is the concrete mechanism satisfying Requirement 3's "--version only, startup log unchanged" boundary, and requires each of the 6 components' `main`/`cli.py` to call the composition function twice with different arguments (once for `--version`'s output, once for the startup log's, as today) rather than sharing one fully-identical string as the original feature's Requirement 2.2 did for `commit_id`/`build_time`. -- **Rationale**: Keeps `commit_id` exactly as-is (Requirement 2 falls out for free -- a clean tree's `dirty_fingerprint` is simply empty, so nothing is appended), reuses the proven two-computation-sites/single-value-pair shape, and cleanly isolates the `--version`-only scope to a single new field rather than a startup-log/version-string fork. -- **Trade-offs**: The two composition call sites (startup log vs. `--version`) per component, previously guaranteed identical by construction (Requirement 2.2 of the original feature), now diverge by design for this one new field -- acceptable and explicitly scoped by this feature's Requirement 3, but must be clearly commented in each component's source so a future change doesn't "fix" the divergence by accident. -- **Follow-up**: Verify during implementation that omitting `dirty_fingerprint` from the startup-log composition call is a one-line, obviously-intentional diff (not something that looks like an oversight) in each of controller/scheduler/worker/apiserver's `main.c`/`main.go`. - -### Decision: Leave commit_id's computation fully untouched; strip its -dirty suffix only inside --version's composition, only when a fingerprint is available -- **Context**: The already-shipped `build-time-versioning` feature's `commit_id` computation unconditionally appends a literal `-dirty` suffix when the tree has uncommitted changes (`xce-build.sh` task 2.1, `flake.nix`'s `self.dirtyShortRev`), and **both** the `--version` output and the startup log read that same `commit_id` value today. An earlier draft of this decision proposed retiring `-dirty` from `commit_id`'s own computation entirely -- but tracing that change's full blast radius revealed a real regression: since the startup log reads the identical `commit_id` value, removing `-dirty` at the computation site would silently strip the startup log's only modified-tree signal too, even though this feature's Requirement 3 explicitly requires the startup log to be left unchanged. That draft under-scoped its own blast radius by only considering the `--version` surface. -- **Alternatives Considered**: - 1. Retire `-dirty` from `commit_id`'s computation entirely (both computation sites) -- rejected after tracing the blast radius: silently regresses the startup log's existing modified-tree signal, a real behavior change to a surface this feature must leave untouched. - 2. Leave `commit_id`'s `-dirty` suffix untouched everywhere, append `dirty_fingerprint` after it unconditionally when non-empty -- rejected: produces the redundant `<hash>-dirty-<fingerprint>` shape when a fingerprint is available, not the plain `<commit_id>-<hash>` shape the requirements were written around (the user's own stated preference during requirements gathering: "just commit id - hash of changed file contents, if no file has been changed just commit id is enough"). - 3. Leave `commit_id`'s computation **completely unmodified** everywhere (both computation sites, and therefore the startup log too); perform the `-dirty`-stripping **only inside the `--version`-composition call**, and **only when `dirty_fingerprint` is non-empty** -- **selected**. -- **Selected Approach**: `commit_id` is never touched by this feature -- identical computation, identical value, at both computation sites, for both consumption surfaces. `dirty_fingerprint` is computed independently (empty when clean or when the tree is dirty but the hash computation fails; a real content hash when the tree is dirty and computable). The `--version`-only composition call applies this rule: if `dirty_fingerprint` is non-empty, strip a trailing `-dirty` from `commit_id` (if present) and append `-<dirty_fingerprint>` instead; if `dirty_fingerprint` is empty, use `commit_id` completely as-is -- which, for a genuinely dirty tree where only the fingerprint computation failed, still correctly shows `<hash>-dirty` (Requirement 5.2's fallback, satisfied automatically by the pre-existing, unmodified `-dirty` mechanism, with zero new fallback marker/sentinel value needed). -- **Rationale**: Zero risk to the startup log (its composition call path never changes, doesn't even know `dirty_fingerprint` exists) and zero risk to any of the original feature's already-shipped tests that assert `commit_id`'s `-dirty` behavior (none of that behavior changes) -- Requirement 5.2 falls out for free from a mechanism that already exists, rather than needing a new literal sentinel string. Strictly simpler than the rejected alternatives. -- **Trade-offs**: The stripping logic lives in exactly one place (the `--version` composition function), so it must be implemented carefully (string-suffix check, not a blind substring removal, in case a commit hash itself could theoretically end in characters resembling "dirty" -- extremely unlikely with hex hashes, but the implementation should anchor the check to the literal `-dirty` suffix position, not just presence anywhere in the string). -- **Follow-up**: None of the original feature's already-shipped tests need updating, since `commit_id`'s behavior is provably unchanged -- implementation should add a regression assertion confirming this explicitly (e.g., `test-build-version-regression.sh`'s existing `-dirty` assertions continue to pass completely unmodified). - -### Decision: Nix path accepts the untracked-file blind spot rather than working around it -- **Context**: Empirically confirmed (see Research Log) that `self.narHash` does not reflect new untracked files unless staged, while the dev-loop path's `git diff` + `git ls-files --others --exclude-standard` approach does. -- **Alternatives Considered**: - 1. Have the nix `preConfigure` build-phase hook shell out to `git add -A --dry-run`-style enumeration or otherwise touch git state to force-include untracked files -- rejected: `git` is deliberately not a nix devShell/build dependency in this repo (original feature's explicit architectural decision), and any workaround here would reintroduce exactly that dependency. - 2. Accept the asymmetry, document it clearly, and let the nix path's fingerprint simply not change for untracked-only dirty deltas -- selected. -- **Selected Approach**: `flake.nix`'s existing eval-time `let` block (which already computes `myCommitId` via `self.shortRev`/`self.dirtyShortRev`) also computes `myDirtyFingerprint` from `self.narHash` (truncated), passed into `xce.nix`/`xcecli.nix` the same way `myCommitId` already is -- eval-time, via `mesonFlags`, not through the build-phase `preConfigure`/`mesonFlagsArray` mechanism reserved for `build_time`. No git-plumbing is added to the nix path. -- **Rationale**: The motivating incident for the entire `build-time-versioning` feature family was a dev-loop (`xce-build.sh`) build; the nix production path was never implicated and is not this feature's primary target. A silent, undocumented gap would be a defect; a clearly documented, deliberately scoped one is an acceptable trade-off consistent with the original feature's own precedent (research.md's "two independent computation sites... risk mitigated by identical shape, not identical computation"). -- **Trade-offs**: An operator diagnosing two nix-built dirty artifacts that differ *only* by an unstaged untracked file addition will see identical `dirty_fingerprint` values despite the builds genuinely differing -- must be documented in design.md and in the nix path's own code comments, mirroring how the original feature documented its own `builtins.currentTime` pure-eval correction inline. -- **Follow-up**: If this gap ever proves operationally significant (e.g., a real incident traced to it), a future spec could revisit having `xce-build.sh` itself pre-stage untracked files in a way that doesn't require nix to shell out to git -- out of scope here. - -## Risks & Mitigations -- The `--version`-only `-dirty`-stripping logic (see Decision above) must anchor its check to the literal trailing `-dirty` suffix, not a blind substring removal, to avoid ever mis-stripping a commit hash that coincidentally contains similar characters -- mitigated by keeping the strip logic in exactly one place (the shared composition function) with a dedicated unit test for the boundary case. -- Two computation sites for the same new value (`xce-build.sh` git-based, nix `self.narHash`-based) could, in principle, produce different fingerprints for what a human would call "the same change" -- mitigated the same way the original feature's `commit_id`/`build_time` already are: identical *shape* (both are short hex digests, both empty on a clean tree), not identical *computation*, and each is only ever compared against a build from the *same* computation site in practice (an operator compares two dev-loop builds against each other, or two nix builds against each other, not across the two). -- The nix path's untracked-file blind spot (documented above) could mislead an operator into thinking two nix-built dirty artifacts are content-identical when they differ only by an added-but-unstaged file -- mitigated by explicit documentation in design.md, a code comment at the `myDirtyFingerprint` eval-time `let`-binding in `flake.nix`, and (per Requirement 5) the general graceful-degradation framing that this indicator is a best-effort distinguishing aid, not a cryptographic guarantee of the full working-tree state. -- Computing `git diff HEAD` plus untracked-file content on every dev-loop build adds a small amount of build-time cost -- mitigated by scope: this repo's working trees are source-code sized (not data-file sized), and the original feature's own `xce-build.sh` already runs `git rev-parse`/`git status --porcelain` on every invocation with no observed performance concern. - -## References -- `src/flake.nix:12-32` -- existing `myCommitId` eval-time computation this feature's nix-path decision extends, and the empirical `debugNarHash` probe site (added and reverted during this research). -- `src/flake.nix:12-32` -- existing eval-time `myCommitId` computation (original feature) that this feature's `myDirtyFingerprint` extends with the same mechanism, not the sibling `preConfigure` build-phase `date -u` hook in `src/nixSupport/xce.nix`/`xcecli.nix` (original feature's task 2.2), which stays reserved for `build_time` and is otherwise untouched. -- `src/tools/xce-build.sh:80-119` -- existing `commit_id`/`build_time` computation block (original feature's task 2.1) that this feature's dev-loop fingerprint computation extends. -- `.kiro/specs/build-time-versioning/research.md` -- the original feature's own research log, in particular its "One resolved value pair, fed from two independent computation sites" decision, which this feature's architecture directly continues. diff --git a/.kiro/specs/dirty-content-fingerprint/spec.json b/.kiro/specs/dirty-content-fingerprint/spec.json deleted file mode 100644 index 50662e88..00000000 --- a/.kiro/specs/dirty-content-fingerprint/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "dirty-content-fingerprint", - "created_at": "2026-08-21T02:44:43Z", - "updated_at": "2026-08-24T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/dirty-content-fingerprint/tasks.md b/.kiro/specs/dirty-content-fingerprint/tasks.md deleted file mode 100644 index f8b1c76b..00000000 --- a/.kiro/specs/dirty-content-fingerprint/tasks.md +++ /dev/null @@ -1,119 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: dirty_fingerprint meson option and C-wide define - - `dirty_fingerprint` meson option (string, default `''`) added in `src/meson.options`, byte-for-byte matching `commit_id`/`build_time`'s style/type/yield; `add_project_arguments()` extended with `-DVERSION_DIRTY_FINGERPRINT=` - - Observable confirmed live (independently re-verified by review, fresh scratch build): a bare `meson setup` with zero `-D` options configures cleanly (73 targets); all 542 C compile commands contain the new define (empty string), matching the existing two defines' counts exactly; no consumption code touched yet - - _Requirements: 1.1, 5.1_ - -- [x] 2. Foundation: compute real fingerprint values at both build entry points -- [x] 2.1 (P) xce-build.sh computes the dev-loop fingerprint, independently of commit_id - - `dirty_fingerprint` computed via `git diff HEAD` + `git ls-files -z --others --exclude-standard | LC_ALL=C sort -z` content, `sha256sum` truncated to 8 hex chars; empty on a clean tree or on computation failure (3 independently-verified graceful-degradation paths: mktemp failure, sha256sum failure, git diff failure -- none abort the script despite `set -euo pipefail`); zero working-tree/index mutation (read-only git invocations only); `commit_id`'s own computation left byte-for-byte untouched - - Observable confirmed live (independently re-verified by review, real builds with HEAD pinned to control for concurrent background commits): editing a tracked file and rebuilding produces a fingerprint; different content produces a different fingerprint; reverting to the first edit's exact bytes reproduces the first fingerprint exactly; a `.gitignore`-matched `*.bak` file leaves the fingerprint unchanged; `commit_id`'s `-dirty` suffix stayed intact across every build; every profile (asan/debug/release) carries the option; the original feature's unmodified `test-build-version-regression.sh` passes twice in full (asan+debug) - - _Requirements: 1.1, 1.2, 1.3, 1.4, 5.1_ - - _Boundary: xce-build.sh_ - -- [x] 2.2 (P) Nix flake computes the fingerprint via self.narHash at eval time, independently of myCommitId - - `myDirtyFingerprint` computed as `builtins.hashString "sha256" self.narHash` truncated to 8 hex chars (`self.narHash` is Nix's SRI base64 format, not hex -- rehashing was confirmed necessary, not optional), gated on the same `hasAttr "shortRev"`-style clean/dirty branch `myCommitId` already uses; threaded through `nixSupport/default.nix` into `xce.nix`/`xcecli.nix`'s eval-time `mesonFlags` list as `-Ddirty_fingerprint=`, alongside the untouched `-Dcommit_id=`; `build_time`'s separate `preConfigure`/`mesonFlagsArray` mechanism left completely undisturbed - - Observable confirmed live (independently re-verified by review, including a full re-run of both `nix build ./src#xcecore_debug` and `./src#xcecli`, no `--impure`): `nix eval` resolves a real, non-empty, content-sensitive `-Ddirty_fingerprint=` value matching an independent Python cross-check; editing a tracked file changes the resolved value; `-Dcommit_id=` unaffected throughout; `nix log` confirms both values reach the real `mesonConfigurePhase` invocation. Full artifact completion is blocked in this sandbox by a pre-existing, unrelated network limitation (an autoscaler Go module fetch needing `goproxy.cn`, confirmed recurring identically on two disjoint Go module sets) -- not a regression from this task, and orthogonal to the eval/meson-configuration behavior this task's Observable actually requires - - _Requirements: 1.1, 1.2, 1.3, 5.1_ - - _Boundary: Nix packaging_ - -- [x] 3. Core: C daemon --version fingerprint display -- [x] 3.1 Extend the shared version-composition mechanism with an optional fingerprint field, without touching commit_id - - New sibling function `xce_compose_version_ex()` added alongside the completely untouched `xce_compose_version()` (zero diff lines in its body or its 3 pre-existing tests); anchors the `-dirty` strip via a true tail-slice `strcmp` (not `strstr`), so it cannot false-match a `-dirty` substring elsewhere; correctly appends `-<fingerprint>` with exactly one dash even when the commit field has no `-dirty` to strip; delegates via a genuine `return xce_compose_version(...)` call (not a re-implementation) when the fingerprint is empty/NULL, structurally guaranteeing byte-identical output, proven by a dedicated equivalence regression test - - No `main.c` call sites were touched by this task (grep-confirmed all still call the original, untouched `xce_compose_version()`) -- that's tasks 3.2/3.3's job - - Observable confirmed live (independently re-verified by review, full lib suite on both debug and ASAN profiles): 301 tests, 0 failures, 0 ignored on both profiles; all 3 required cases present with byte-exact assertions, plus 2 meaningful additional regression/defensive tests; buffer handling confirmed snprintf-bounded with no overflow risk - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 5.2_ - - _Boundary: utils.h, utils.c_ - -- [x] 3.2 (P) Controller and scheduler: fingerprint in --version, startup log unaffected - - `argp_program_version` for controller/scheduler now composed via a genuinely separate buffer/call to `xce_compose_version_ex()` (reading `STR(VERSION_DIRTY_FINGERPRINT)`, matching the existing commit_id/build_time macro pattern); the startup-log call site's `xce_compose_version()` call, arguments, and buffer are byte-for-byte unchanged (independently verified by review via diff + grep); `controller/testsuite/test.c` (no startup log, confirmed via grep) got a direct in-place swap; all three files carry explicit code comments marking the divergence as deliberate - - Observable confirmed live (independently re-verified by review, real builds + live daemon starts): `xce_controller --version`/`xce_test --version`/`xce_scheduler --version` all show `<commit>-<fingerprint>`, cross-checked against `compile_commands.json`; live controller and scheduler startup logs show the old `<commit>-dirty` format unchanged, fingerprint never present, build_time matching `--version` exactly - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1_ - - _Boundary: controller/main.c, scheduler/main.c, controller/testsuite/test.c_ - - _Depends: 3.1_ - -- [x] 3.3 (P) Worker: fingerprint in --version, startup log unaffected - - `OPT_VERSION`'s case now composes via a genuinely separate `version_display_buf`/`xce_compose_version_ex()` call, byte-for-byte mirroring task 3.2's already-approved controller/scheduler pattern (independently confirmed by review via direct diff comparison against commit `fd62e26`); both `XCECD_LOG` startup-log call sites (`--peers` and no-`--peers`) left completely untouched, same `xce_compose_version()` call, same `version_buf` - - Observable confirmed live (independently re-verified by review, real build + live daemon runs in both branches): `xcecd --version` shows `<commit>-<fingerprint>` matching `compile_commands.json` exactly, `-dirty` correctly stripped not doubled; both startup log branches show the old fingerprint-less `<commit>-dirty` format unchanged - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1_ - - _Boundary: worker/main.c_ - - _Depends: 3.1_ - -- [x] 4. Core: Go binary --version/-version fingerprint display -- [x] 4.1 go_build injects the fingerprint via ldflags - - `go_build` reads `XCE_DIRTY_FINGERPRINT` (empty-safe) and appends `-X main.DirtyFingerprint=...` to the existing `-ldflags` construction; falls back to plain empty string (deliberately NOT `"unknown"`, unlike `GitCommit`/`BuildTime`), matching `dirty_fingerprint`'s convention everywhere else in this feature; the existing `GitCommit`/`BuildTime` `unknown`-fallback logic is completely untouched - - Observable confirmed live (independently re-verified by review, real scratch Go builds traced via `bash -x`): fingerprint unset -> `-X main.DirtyFingerprint=` empty, binary confirms empty (not fabricated); fingerprint set -> flag carries the exact value; all three vars unset -> `GitCommit`/`BuildTime` still fall back to `unknown`, `DirtyFingerprint` stays empty, the two fallback conventions coexist without interference - - _Requirements: 1.1, 5.1_ - - _Boundary: go_build script_ - -- [x] 4.2 (P) apiserver and xcecli meson builds pass the resolved value through - - Both `apiserver/meson.build` and `xcecli/meson.build` add `'XCE_DIRTY_FINGERPRINT=' + get_option('dirty_fingerprint')`, unquoted, identically positioned to the existing `XCE_GIT_COMMIT`/`XCE_BUILD_TIME` entries - - Observable confirmed live (independently re-verified by review with a distinct test value, fresh configure, and real `ninja` builds of both binaries): the value reaches `go build`'s real invocation and shows up in `go version -m`'s recorded `-ldflags` build setting for both apiserver and xcecli; matches the C-side `-DVERSION_DIRTY_FINGERPRINT=` byte-for-byte from the same configure; degrades to empty identically to the two pre-existing vars when omitted - - _Requirements: 1.1, 4.1_ - - _Boundary: apiserver/meson.build, xcecli/meson.build_ - - _Depends: 4.1_ - -- [x] 4.3 (P) Makefile dev-loop path computes the same fingerprint, independently of GIT_COMMIT - - `GIT_COMMIT`/`GIT_STATUS`/`BUILD_TIME` (and `GIT_COMMIT`'s own `-dirty` suffix) confirmed byte-for-byte untouched throughout two rounds of adversarial remediation - - `DIRTY_FINGERPRINT` computed independently, mirroring task 2.1's algorithm exactly (cross-checked byte-identical on the real dirty tree); failure-detection required two remediation rounds to get right -- round 1 fixed a `{ git diff HEAD; ...; } >file` grouped command silently swallowing a `git diff HEAD` failure (only the last command's exit status was checked); round 2 found the SAME class of bug one step further down (`git ls-files | sort | xargs cat`'s pipe also swallowing an `ls-files` failure) and fixed it by pulling `git ls-files` out of the pipe entirely into its own directly-redirected, individually-checked step -- deliberately choosing this over a `set -o pipefail;` prefix (which empirically also worked) because this repo's own `build-deployment.md` documents a past incident of code silently relying on `/bin/sh` happening to be bash with no `SHELL:=bash` pin - - One narrow, explicitly-disclosed residual gap remains and was assessed as acceptable: a `sort`-only failure (not `cat`) in the trailing `sort -z | xargs -0 -r cat --` pipe can still be masked -- independently confirmed present, low-risk, and accurately scoped, not grounds for further rejection - - Observable confirmed live (independently re-verified across 3 review rounds, real builds + two independently-reproduced failure-injection wrappers with control tests proving each wrapper genuinely triggers the old bug): `make -n build-cli`'s fingerprint matches a standalone reference computation byte-for-byte; both the `git diff`-fails and `git ls-files`-fails scenarios correctly degrade to empty, not a fabricated hash; `GIT_COMMIT`'s `-dirty` behavior unaffected throughout - - _Requirements: 1.1, 1.2, 1.3, 1.4, 5.1_ - - _Boundary: Makefile_ - - _Depends: 4.1_ - -- [x] 4.4 (P) apiserver: fingerprint in -version, startup log unaffected - - `DirtyFingerprint` package var (default `""`) + `composeVersionDisplay()` helper doing the strip-and-append via a genuine anchored `strings.TrimSuffix` (not a blind substring replace, independently confirmed via a self-constructed mutation test); `GitCommit`'s declaration and the `logger.Info` startup-log call site (including its zap fields) confirmed byte-for-byte untouched - - Observable confirmed live (independently re-verified by review, real build across all 5 scenarios + a live daemon-start log capture): `-version` shows `<hash>-<fingerprint>` never `<hash>-dirty-<fingerprint>`; the startup log shows the raw, un-fingerprinted `GitCommit` (still `-dirty`-suffixed when applicable) on the same binary whose `-version` output differs; `go.sum`/`go.mod` untouched - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1_ - - _Boundary: apiserver/cmd/main.go_ - - _Depends: 4.1_ - -- [x] 4.5 (P) xcecli: fingerprint in --version - - `DirtyFingerprint` var + `composeVersionDisplay()` helper, a verbatim logical duplicate of apiserver's just-approved implementation (task 4.4); applied at the sole consumption point (the version string fed into `router.Execute(cm, version)`); `router.go` confirmed untouched (independently re-verified by the parent controller after the review, not just the reviewer, given a mid-review incident where the reviewer's own tooling briefly discarded the uncommitted diff before catching and byte-for-byte restoring it) - - Observable confirmed live (independently re-verified by review, real builds across all 3 scenarios + full xcecli module test suite): `--version`/`-v` shows `<hash>-<fingerprint>` on dirty+fingerprint, `<hash>` alone on clean, `<hash>-dirty` unchanged on dirty+degraded-fingerprint; all exit 0; no regressions in any other xcecli package - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 4.1_ - - _Boundary: xcecli/cmd/main.go_ - - _Depends: 4.1_ - -- [x] 5. Core: xceadmin fingerprint display -- [x] 5.1 (P) meson-build.sh generates the fingerprint field in the build-info module - - `dirty_fingerprint` threaded through as an 8th positional arg (`$8`), correctly numbered and consistent between `meson.build` and `meson-build.sh`; `_build_info.py` gains `DIRTY_FINGERPRINT` with an independent empty-string fallback (not `"unknown"`); `GIT_COMMIT`/`BUILD_TIME`'s existing generation and fallback confirmed byte-for-byte untouched - - Observable confirmed live (independently re-verified by review, full real build+test+wheel cycles across 3 scenarios): dirty tree -> real, non-empty `DIRTY_FINGERPRINT` present identically in the mirrored tree, the build/lib copy, an installed venv, and the packaged wheel, cross-checked byte-for-byte against that same build's `compile_commands.json`; explicit-empty fingerprint -> genuinely `""` while `GIT_COMMIT`/`BUILD_TIME` pass through real values; all-empty -> `GIT_COMMIT`/`BUILD_TIME` still fall back to `"unknown"` while `DIRTY_FINGERPRINT` stays `""`; 261 passed/3 skipped in all three scenarios - - _Requirements: 1.1, 1.2, 1.3, 1.4, 5.1_ - - _Boundary: xceadmin/meson.build, xceadmin/meson-build.sh_ - - _Depends: 1_ - -- [x] 5.2 (P) Committed fallback stub and CLI fingerprint wiring - - `_build_info.py`'s fallback stub gains `DIRTY_FINGERPRINT = ""`; `cli.py`'s new `_compose_version_display()` applies the strip-and-append rule via a genuine anchored `removesuffix("-dirty")` (independently confirmed via a second, differently-constructed mutation test); both fallback paths (`_build_info.py`'s module stub and `cli.py`'s separate `ImportError` fallback class) correctly carry the field - - A pre-existing test's hard-coded literal-`GIT_COMMIT` assertion was updated to compute its expected value via the production `_compose_version_display()` function instead -- independently assessed by review as a legitimate strengthening (the old assertion's premise no longer holds once a fingerprint is present), not a weakened/gutted check; the 4 new dedicated unit tests independently guard the transform itself with hard-coded literal assertions - - Observable confirmed live (independently re-verified by review, both install paths via a full real `/xce-build xceadmin` cycle): editable install shows `0.1.0 (unknown, unknown)`, no import error; wheel built from the dirty tree shows `<hash>-<fingerprint>`, never `<hash>-dirty-<fingerprint>`; full suite 267 passed/3 skipped (261 baseline + 6 new) - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 4.1_ - - _Boundary: xceadmin/src/xceadmin_ - - _Depends: 5.1_ - -- [x] 6. Integration: cross-toolchain consistency and regression verification -- [x] 6.1 Dev-loop distinguishability and clean-tree regression test - - New script `src/tools/test-dirty-fingerprint-distinguishability.sh` uses a genuine isolated `git worktree add --detach` (unique `mktemp -u` path, `trap cleanup EXIT` registered before creation, safe no-op if creation never happened, `rm -rf`+`prune` fallback) to sidestep this actively-shared repo's ambient concurrent-editing noise -- all file edits confined strictly inside the isolated worktree, the real repo's own working tree never touched - - Drives a real `xce-build.sh scheduler` cycle through the 4 required states via genuine builds; fails loudly with a clear message on any mismatch - - Observable confirmed live (independently re-verified by review across two full live runs, including a deliberately sabotaged copy forcing edit-B to reproduce edit-A's content): clean/no-suffix, dirty-A, dirty-B-differs-from-A, reverted-to-A-matches-A all held with real fingerprints; sabotaged run correctly failed with a clear message and exit 1, proving genuine (not just synthetic) discriminating power; `git worktree list` and the real repo's working tree confirmed byte-identical to baseline before and after every run, both by implementer and reviewer - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1_ - - _Depends: 2.1, 3.2_ - -- [x] 6.2 Startup-log-unchanged consistency test - - New script `src/tools/test-dirty-fingerprint-startup-log-unchanged.sh` reuses task 6.1's already-approved isolated-git-worktree technique; builds scheduler and apiserver for real on a genuinely dirty tree, starts both daemons, captures `--version`/`-version` (expect fingerprint) and each startup log (expect old fingerprint-free format) - - Apiserver's injected fingerprint value is reused from scheduler's own extracted value (same dirty tree, same content) rather than independently recomputed -- independently assessed by review as legitimate, not tautological: apiserver has no independent fingerprint-computation code of its own in production (only ever receives an externally-injected value via `-ldflags`), the script still exercises apiserver's own real, unmocked `composeVersionDisplay()`/`logger.Info` code paths on a genuinely running compiled binary, and a round-trip `api_fp == sched_fp` check would catch a real strip-and-append bug - - Observable confirmed live (independently re-verified by review, fresh full run + a separately-authored 4-case discrimination probe outside the script's own demo block): both daemons' `--version`/`-version` show real fingerprints while their startup logs show the unchanged, fingerprint-free old format; no leftover processes, worktrees, or build dirs after either run - - _Requirements: 3.1, 3.2_ - - _Depends: 3.2, 4.4_ - -- [x] 6.3 Nix production path end-to-end fingerprint verification - - New script `src/tools/test-dirty-fingerprint-nix-production-path.sh` (reusing task 6.1's isolated-worktree technique) edits `flake.nix`, runs `nix build ./src#xcecore_debug -L --keep-going` (no `--impure`) twice with different content, and cross-checks `nix eval`'s eval-time `myDirtyFingerprint` value against the real build log's `mesonConfigurePhase flags:` line for each - - This sandbox's `nix build` dispatches by default to an unreachable-in-practice remote builder (`ssh://rshen@172.31.109.18`, confirmed genuinely expected behavior, not a misconfiguration) making a directly-runnable binary impractical to retrieve; the eval+build-log cross-check is not a weaker substitute -- independently confirmed structurally sound by review, since `self.narHash`'s value is computed at flake-evaluation time and baked as a literal string into the derivation before any build phase runs, so eval-time and build-time values cannot diverge by construction, not just by empirical coincidence. Forcing `--builders ''` (local-only) was investigated per a live suggestion and does reach real local `.c` compilation with a direct log match, but still can't produce a fully linked binary in this sandbox (a second, unrelated network dependency), confirming no stronger approach is currently available; recorded as a learning for future sessions - - Observable confirmed live (independently re-verified by review, real `nix build` invocations including a live-witnessed remote-builder connection attempt and a genuine local-only run): two builds with different tracked-file edits produce different fingerprints in both eval and build-log evidence; reverting to the first edit's exact content reproduces the first fingerprint exactly; `--impure` never used; no `git` dependency in the build sandbox; `flake.nix`/`nixSupport/` and the worktree list confirmed byte-identical to baseline after every run - - _Requirements: 1.1, 1.2, 1.3, 5.1_ - - _Depends: 2.2_ - -## Implementation Notes -- **Task 4.3**: a `{ cmd1; cmd2; } >file` or `cmd1 | cmd2 | cmd3` shell construct's exit status (as checked by `if ! ... ; then ...; fi`) only reflects the LAST command's exit status, not any earlier command's -- this silently swallows an earlier failure and can produce fabricated output from partial/incomplete data instead of the intended graceful-degradation path. Took two rejected review rounds to fully generalize (first fixed for `git diff HEAD`, then found again one step later for `git ls-files | sort | xargs`). When writing any multi-stage shell pipeline whose failure must be individually detectable, either enable `pipefail` explicitly (verify the actual invoking shell first -- `$(shell ...)` in Make does NOT inherit a Makefile's `SHELL`/`.SHELLFLAGS` settings) or restructure each stage to its own directly-redirected, individually-`$?`-checked step (the safer default absent a `SHELL:=bash`-style pin, per this repo's own documented `/bin/sh`-is-not-guaranteed-bash incident in `build-deployment.md`). diff --git a/.kiro/specs/election-iomgr-refactor/design.md b/.kiro/specs/election-iomgr-refactor/design.md deleted file mode 100644 index 3857b305..00000000 --- a/.kiro/specs/election-iomgr-refactor/design.md +++ /dev/null @@ -1,539 +0,0 @@ -# Design Document: election-iomgr-refactor - -## Overview - -The XCE controller was originally fully event-driven: `cluster_init` → `cluster_start` → -`cluster_run` (shutdown-check loop). The HA additions broke this by inserting blocking calls -before `cluster_run`: `ha_wait_for_heartbeat_or_timeout` and `am_i_primary_quorum` block the -main thread, causing SSL context confusion, capnp UAF, and mixed-I/O races. - -This design restores full event-driven operation with no blocking calls after phase1. -`myapp_run` does setup then immediately enters `cluster_run`. All election I/O flows through -iomgr callbacks. When the election reaches a terminal state, `ha_elect_evaluate_and_signal` -calls `cluster_init_phase2` inline from vtask context (phase2 is already thin — managers were -moved to phase1 in a prior refactor), then atomically sets `ha->role`. The dispatch gate -(`ha_dispatch_gate`) opens automatically: the next client request sees PRIMARY or STANDBY and -is handled normally. No semaphores, no wbuf signals, no changes to `cluster_run`. - -### Goals -- Restore event-driven model: `myapp_run` → `cluster_run` with zero blocking between them -- All election I/O through one epoll event loop (iomgr) -- `ha_elect_evaluate_and_signal` calls thin `cluster_init_phase2` from vtask context; sets role atomically -- `cluster_run` unchanged — still just the shutdown-check loop -- `myreq_handler` / `ha_dispatch_gate` is the sole FSM dispatcher — no new state machinery needed -- Role-based vote gate: PRIMARY and STANDBY-with-live-primary deny votes - -### Non-Goals -- Changes to `cluster_run` or `wbuf` signal values -- Incoming vote RPC handling (already correct in cluster.c) -- Heartbeat watchdog thread (existing, unchanged) -- Config replication, RDHUP paths (unchanged) -- Async DNS resolution (deferred) -- TLS certificate verification for vote connections - -## Boundary Commitments - -### This Spec Owns -- `vote_request_thread` + `am_i_primary_quorum` blocking loop → replaced by `ha_start_election_async()` -- `on_connected` callback added to `netio.c` -- `client_ssl_ctx` in `ha_state_t` + `myapp_run` -- `vote_reply_msg_handler` for iomgr-native reply handling (Note: this handler was later generalized into `peer_ctrl_msg_handler`, a single state-aware dispatcher that also routes repl acks and PreVote replies — superseded by ha-prevote, see that spec's design.md) -- Election timeout watchdog thread -- `ha_elect_evaluate_and_signal` extended to call `cluster_init_phase2` then set role -- `handle_request_vote` extended with role-based deny gate (Note: `handle_request_vote` is now `#ifdef XCE_TESTING`-only; production vote dispatch is `cluster.c`'s `case OPCode_requestVote:` calling `raft_on_vote_request()` — superseded by raft-adapter-conformance, see that spec's design.md) -- `myapp_run` restructured: phase1 + SSL + `ha_start_election_async` + `cluster_run` directly - -### Out of Boundary -- `cluster_run` internals (unchanged) -- `case OPCode_requestVote:` in cluster.c (already correct, no changes) -- `ha_wait_for_heartbeat_or_timeout` (existing; now called internally by `ha_start_election_async`) -- Config replication SSL - -### Allowed Dependencies -- `netio.c` / `endpoint_t` (add `on_connected` callback) -- `ha_elect_ctx_t` (existing; mutex guards concurrent access) -- `iomgr_t` / `iomgr_monitor` (existing) -- `xce_request()` / `endpoint_enq()` (existing) -- `handle_request_vote_reply()` (existing) -- `cluster_init_phase2()` (called from vtask context after election; must be vtask-safe) - -### Revalidation Triggers -- Changes to `endpoint_t` structure -- Changes to `cluster_init_phase2` preconditions (anything it accesses that may not be ready) -- Changes to `ha_dispatch_gate` CTRL_ELECTING allowlist - -## Architecture - -### Before: Blocking startup -```mermaid -graph TD - myapp_run --> cluster_init_phase1 - cluster_init_phase1 --> ha_wait_hb_or_timeout - ha_wait_hb_or_timeout -->|blocks 2s| timeout - timeout --> am_i_primary_quorum - am_i_primary_quorum -->|pthreads block on sem| result - result --> cluster_init_phase2 - cluster_init_phase2 --> cluster_start - cluster_start --> cluster_run -``` - -### After: Fully event-driven -```mermaid -graph TD - myapp_run --> cluster_init_phase1 - cluster_init_phase1 --> ssl_setup - ssl_setup --> cluster_register_listener - cluster_register_listener --> iomgr_vtask_start - iomgr_vtask_start --> ha_start_election_async - ha_start_election_async -->|returns immediately| cluster_run - - iomgr_worker -->|EPOLLOUT handshake| on_vote_connected - on_vote_connected --> xce_request_vote - iomgr_worker -->|EPOLLIN reply| vtask_vote_reply_msg_handler - vtask_vote_reply_msg_handler --> handle_request_vote_reply - handle_request_vote_reply --> ha_elect_evaluate_and_signal - ha_elect_evaluate_and_signal --> cluster_init_phase2_thin - cluster_init_phase2_thin --> set_ha_role_atomic - set_ha_role_atomic -->|gate opens| serve_traffic - - election_watchdog -->|timeout| ha_elect_evaluate_and_signal - heartbeat_watchdog -->|hb timeout or received| ha_elect_evaluate_and_signal -``` - -### Key Decisions - -- **`ha_start_election_async()` is fully non-blocking**: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately. (Note: peer vote-endpoint connection and vote tallying are no longer driven directly by this function — they are now owned by the Raft library (`src/lib/raft/raft.c`): `ha_start_election_async()` calls `raft_start_election()`, which fires `raft_cb_send_vote_request` per peer and `raft_cb_become_leader`/`raft_cb_become_follower` on a terminal result. Superseded by raft-log-replication and raft-commit-safety, see those specs' design.md.) -- **Two promotion paths in `ha_elect_evaluate_and_signal`**: - - *Initial startup* (`ha->phase2_done == 0`): calls `cluster_init_phase2` (loads config data, inits structures), sets `ha->phase2_done = 1`. Phase2 runs exactly once per process lifetime — config upgrade always does a full restart. - - *RDHUP failover* (`ha->phase2_done == 1`): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly. (Note: see CTRL_PRIMARY_COMMITTING note below — this now enters CTRL_PRIMARY directly, no COMMITTING state.) -- **`is_first_boot` = `ha->phase2_done == 0`**: Not `!c->running` (which is 0 until `cluster_run` sets it, making all elections look like first boot). `phase2_done` is a dedicated bool in `ha_state_t`. (Still accurate — verified current in `ha.c`.) -- **`CTRL_PRIMARY_COMMITTING`**: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary). **(Note: superseded by ha-single-leader-role, see that spec's design.md — `CTRL_PRIMARY_COMMITTING` was subsequently removed. `ctrl_role_t` now has only 4 values (`CTRL_PRIMARY`, `CTRL_STANDBY`, `CTRL_INIT`, `CTRL_ELECTING`); a winning candidate enters `CTRL_PRIMARY` directly via `raft_cb_become_leader`/`ha_elect_evaluate_and_signal`, with no transitional committing state and no committing-timeout watchdog. Job-traffic admission during startup is now gated by `ha_has_write_quorum()` inside the `CTRL_PRIMARY` case of `ha_dispatch_gate`, not by a separate FSM state. Every other reference to CTRL_PRIMARY_COMMITTING in this document — the sequence diagrams, the `ha_repl_ack_received()`/`ha_dispatch_gate` COMMITTING-case code samples, the Error Handling and Testing Strategy sections below — describes this now-removed mechanism.)** -- **`cluster_run` unchanged**: Still just `cbuf_get(&wbuf)` for stop signal. -- **`delaymgr_start` and `persist_start` move to `myapp_run`**: Currently called inside `cluster_start()`. In the new design `cluster_start` is not called from `myapp_run`. These two calls move explicitly into `myapp_run` before `ha_start_election_async`. -- **`bootstrap_config_if_needed` removed entirely**: Replaced by the freshness gate in voting + primary replication push. No pre-election config probe needed. -- **`ha_send_ctrlhello_to_all_standbys`**: Iterates `ha->standby_endpoints` (populated when standbys connect and send `OPCode_hello`), sends `CtrlHelloRequest` to each. This triggers the existing catch-up replication in `handle_ctrlhello`. After catch-up, standby sends `OPCode_replCatchUpDone` → `ha_repl_ack_received`. -- **`committing_timeout_ms`**: Added to `ha_config_t` with default `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000` (10s). Covers the case where standbys are down at startup (single-node degraded mode). -- **Timeout mechanism — `delay_task_ms` + cancelled flag**: delaymgr is enhanced with one new API: - - `delay_task_ms(mgr, delay_ms, runcount, fn, arg)` — millisecond-granularity one-shot task using `clock_gettime(CLOCK_MONOTONIC)`. Precision bounded by `mgr->precision_ms`. - - **Cancellation via state flag**: each context struct (`election_timeout_ctx_t`, `committing_timeout_ctx_t`) carries an `_Atomic int cancelled` field. Setting it to 1 makes the callback a no-op when it fires. This is O(1), lock-free, and race-free — even if the worker dequeued the task and is mid-execution, the flag check at callback entry handles it correctly. No ordq scan or ID tracking needed. - ```c - typedef struct { - _Atomic int cancelled; - ha_state_t *ha; - cluster_t *c; - ha_elect_ctx_t *ctx; - } election_timeout_ctx_t; - - static int election_timeout_fn(void *arg) { - election_timeout_ctx_t *t = arg; - if (atomic_load(&t->cancelled)) return 0; - ha_elect_evaluate_and_signal(t->ha, t->ctx); - return 0; - } - - /* Schedule: */ - delay_task_ms(&c->delaymgr, ha->config.election_timeout_ms, 1, election_timeout_fn, ctx); - - /* "Cancel" when quorum already reached: */ - atomic_store(&ctx->cancelled, 1); - ``` - The existing heartbeat watchdog keeps its dedicated `sem_timedwait` pthread (precise rearm on every heartbeat — not suitable for delaymgr's polling model). - -## File Structure Plan - -``` -src/lib/ -├── netio.c # Add on_connected callback to endpoint_t; invoke in io_handler -├── delay.c # Add delay_task_ms() — ms-granularity using clock_gettime(CLOCK_MONOTONIC) -│ # Cancellation via caller-owned _Atomic int cancelled flag (no API needed) -└── delay.h # Expose delay_task_ms() - -src/controller/ -├── ha.h # Add to ha_state_t: client_ssl_ctx, phase2_done, repl_ack_count, n_peers -│ # Add to ha_config_t: committing_timeout_ms (default 10000ms) -│ # Add CTRL_PRIMARY_COMMITTING to ctrl_role_t enum -│ # (Note: repl_ack_count, committing_timeout_ms, and CTRL_PRIMARY_COMMITTING -│ # do not exist in current ha.h — superseded by ha-single-leader-role, -│ # see that spec's design.md. client_ssl_ctx, phase2_done, n_peers verified current.) -│ # Declare: ha_start_election_async(), ha_election_cleanup(), -│ # ha_repl_ack_received(), ha_arm/disarm_committing_timeout(), -│ # ha_send_ctrlhello_to_all_standbys() -│ # (Note: ha_repl_ack_received/ha_arm/disarm_committing_timeout do not -│ # exist — superseded by ha-single-leader-role. ha_send_ctrlhello_to_all_standbys -│ # verified current, called from raft_cb_become_leader.) -├── ha.c # Remove: vote_request_thread, vote_thread_ctx_t -│ # Add: vote_ep_ctx_t, on_vote_connected(), vote_reply_msg_handler() -│ # (Note: vote_reply_msg_handler was generalized into peer_ctrl_msg_handler — -│ # superseded by ha-prevote, see that spec's design.md) -│ # Add: ha_start_election_async(), election_timeout_watchdog -│ # Extend ha_elect_evaluate_and_signal: COMMITTING path + replication push -│ # Extend handle_request_vote: role-based deny gate -│ # (Note: handle_request_vote is now #ifdef XCE_TESTING-only — superseded -│ # by raft-adapter-conformance, see that spec's design.md) -│ # Add: ha_repl_ack_received(), ha_arm/disarm_committing_timeout() -│ # Extend ha_dispatch_gate: CTRL_PRIMARY_COMMITTING case -│ # (Note: none of these three exist — superseded by ha-single-leader-role; -│ # see the CTRL_PRIMARY_COMMITTING note in Key Decisions above) -├── cluster.c # cluster_init_phase2: vtask-safe (managers already in phase1) -│ # ha_send_ctrlhello_to_all_standbys() or reuse handle_ctrlhello path -│ # cluster_run: UNCHANGED -└── myapp.c # Restructured: phase1 + SSL + delaymgr/persist_start + ha_start_election_async + cluster_run - # Remove: ha_wait_for_heartbeat_or_timeout, am_i_primary_quorum, - # for(;;) retry loop, cluster_init_phase2, cluster_start calls, - # bootstrap_config_if_needed (replaced by freshness gate + primary push) -``` - -### Modified Files -- `src/lib/netio.c` — Add `on_connected` to `endpoint_t`; fire in `io_handler` after `ssl_handshake_done=1` -- `src/lib/delay.c` — Add `delay_task_ms()` using `clock_gettime(CLOCK_MONOTONIC)` for ms precision; precision still bounded by `mgr->precision_ms` -- `src/lib/delay.h` — Expose `delay_task_ms()` -- `src/controller/ha.h` — Add `client_ssl_ctx`, `phase2_done`, `repl_ack_count`, `CTRL_PRIMARY_COMMITTING`; declare new functions -- `src/controller/ha.c` — Main changes; `ha_dispatch_gate` extended for COMMITTING; uses `delay_task_ms`/`delay_task_cancel` for watchdogs -- `src/controller/cluster.c` — `ha_send_ctrlhello_to_all_standbys` helper; `OPCode_replCatchUpDone` calls `ha_repl_ack_received`; `cluster_run` unchanged -- `src/controller/myapp.c` — Restructured: non-blocking after phase1; `delaymgr_start`, `persist_start` moved here; `bootstrap_config_if_needed` removed - -## System Flows - -### 3-node startup — ctrl-a wins (with COMMITTING phase) -```mermaid -sequenceDiagram - participant Main as myapp_run - participant Async as ha_start_election_async - participant iomgr as iomgr Worker - participant vtask as vtaskmgr Worker - participant PeerB as ctrl-b endpoint - participant PeerC as ctrl-c endpoint - - Main->>Main: cluster_init_phase1 + SSL + register_listener + delaymgr_start + persist_start - Main->>Async: ha_start_election_async (non-blocking) - Async->>PeerB: endpoint_connect(client_ssl_ctx) - Async->>PeerC: endpoint_connect(client_ssl_ctx) - Async->>iomgr: iomgr_monitor(peer_eps) - Async-->>Main: returns immediately - - Main->>Main: cluster_run() — wbuf shutdown loop - - iomgr->>PeerB: handshake → on_vote_connected → OP_requestVote - iomgr->>PeerC: handshake → on_vote_connected → OP_requestVote - iomgr->>vtask: vote replies → quorum reached - vtask->>vtask: ha_elect_evaluate_and_signal → cluster_init_phase2 (first boot) - vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING - vtask->>PeerB: push_config_to_standbys + ha_send_ctrlhello (triggers WAL catch-up) - vtask->>PeerC: push_config_to_standbys + ha_send_ctrlhello - - iomgr->>vtask: OPCode_replCatchUpDone from ctrl-b → ha_repl_ack_received (ack=1) - iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=2 ≥ quorum-1) - vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY - Note over vtask: dispatch gate fully open — serves job traffic -``` - -### Primary crash — heartbeat timeout triggers promotion (RDHUP failover) -```mermaid -sequenceDiagram - participant HBWatch as Heartbeat Watchdog on ctrl-b - participant vtask as vtaskmgr Worker - participant iomgr as iomgr on ctrl-b - participant PeerC as ctrl-c endpoint - - Note over HBWatch: heartbeat_timeout_ms — primary silent - HBWatch->>vtask: ha_on_primary_ctrl_fd_rdhup → ha->role = CTRL_ELECTING - HBWatch->>iomgr: ha_start_election_async (retry — is_first_boot=false) - iomgr->>PeerC: vote exchange → ctrl-b wins - vtask->>vtask: ha_elect_evaluate_and_signal (is_first_boot=false → skip phase2) - vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING - vtask->>PeerC: ha_send_ctrlhello (WAL already current — catch-up is fast) - iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=1 ≥ quorum-1) - vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY - Note over vtask: ctrl-b is new primary — state already synchronized via prior replication -``` - -### Incoming request during election (NOT_READY) -```mermaid -sequenceDiagram - participant Client - participant iomgr as iomgr Worker - participant vtask as vtaskmgr Worker - - Client->>iomgr: TCP connect + OP_jobsubmit - iomgr->>vtask: myreq_handler - vtask->>vtask: ha_dispatch_gate(CTRL_ELECTING, OPCode_jobsubmit) → XCE_ERR_NOT_READY - vtask->>Client: reply status = XCE_ERR_NOT_READY (client retries) -``` - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1 | No pthreads for vote I/O | `ha_start_election_async` | 3-node startup | -| 1.2 | Vote RPC via iomgr path | `on_vote_connected`, `xce_request` | 3-node startup | -| 1.3 | Non-responding peer → deny | election watchdog → evaluate | — | -| 2.1 | Async TCP+TLS via iomgr | `endpoint_connect`, `iomgr_monitor` | 3-node startup | -| 2.2 | TLS client context | `client_ssl_ctx` in `ha_state_t` | 3-node startup | -| 2.3 | Send vote after handshake | `on_vote_connected` callback | 3-node startup | -| 2.4 | Connect/TLS failure → deny | `vote_ep_err_handler` | — | -| 3.1 | Reply via iomgr packet | `vote_reply_msg_handler`§ | 3-node startup | -| 3.2 | No capnp UAF | vtask handler reads before free | 3-node startup | -| 3.3 | Extract reply fields | `vote_reply_msg_handler`§ decodes | 3-node startup | -| 3.4 | Term fencing on reply | `handle_request_vote_reply` (existing) | — | -| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING† | `ha_elect_evaluate_and_signal` → committing state† | 3-node startup | -| 4.2 | Freshness gate guarantees ground-truth | `handle_request_vote` config_tag check‡ | — | -| 4.3 | Cold start: only highest config_tag wins | freshness gate + role gate combined | — | -| 4.4–4.7 | Retry, timeout, heartbeat abort | watchdog, re-arm, heartbeat watchdog | Primary crash | -| 5.1 | ELECTING gate: NOT_READY for non-election | `ha_dispatch_gate` (existing + COMMITTING)† | NOT_READY flow | -| 5.2 | COMMITTING: only repl acks allowed | `ha_dispatch_gate` new COMMITTING case† | 3-node startup | -| 5.3 | PRIMARY: full traffic | `ha_dispatch_gate` (existing) | — | -| 5.4–5.5 | STANDBY gate, term persistence | `ha_dispatch_gate`, `ha_persist_term` (existing) | — | -| 6.1 | PRIMARY denies all votes | `handle_request_vote` role gate‡ | — | -| 6.2 | STANDBY+fresh hb denies votes | `handle_request_vote` heartbeat check‡ | — | -| 6.3 | HB timeout → ELECTING | heartbeat watchdog fires RDHUP path | Primary crash | -| 6.4 | INIT/ELECTING evaluates Raft gates | `handle_request_vote` existing gates‡ | — | -| 6.5 | Primary crash → new primary | HB timeout → re-election → COMMITTING → PRIMARY† | Primary crash | -| 7.1 | Winner pushes config to standbys | `push_config_to_standbys()` in committing path | 3-node startup | -| 7.2 | Winner replicates WAL to standbys | `handle_ctrlhello()` catch-up (existing) | 3-node startup | -| 7.3 | config_tag=0 standby gets config via push | no independent fetch; primary pushes | 3-node startup | -| 7.4 | Ack tracked per standby | `ha_state_t.repl_ack_count`† | 3-node startup | -| 8.1 | COMMITTING blocks job traffic | `ha_dispatch_gate` COMMITTING case† | 3-node startup | -| 8.2 | Majority ack → CTRL_PRIMARY | `ha_repl_ack_received()` threshold check† | 3-node startup | -| 8.3 | CTRL_PRIMARY opens full traffic | `ha_dispatch_gate` PRIMARY case (existing) | 3-node startup | -| 8.4 | Ack timeout → degraded open | timeout watchdog in committing state† | — | -| 9.1–9.4 | Regressions | Single-node, ground-truth invariant, RDHUP hooks-only, priority stagger | — | - -† Rows marked † describe the `CTRL_PRIMARY_COMMITTING` mechanism, removed by ha-single-leader-role — -see the Key Decisions note above (a winner now enters `CTRL_PRIMARY` directly; job-traffic admission -during startup is gated by `ha_has_write_quorum()` inside `ha_dispatch_gate`'s `CTRL_PRIMARY` case). -‡ Rows marked ‡ cite `handle_request_vote`, now `#ifdef XCE_TESTING`-only; production vote-request -handling is `cluster.c`'s `case OPCode_requestVote:` → `raft_on_vote_request()` (superseded by -raft-adapter-conformance, see that spec's design.md). -§ Rows marked § cite `vote_reply_msg_handler`, generalized into `peer_ctrl_msg_handler` (superseded -by ha-prevote, see that spec's design.md). - -## Components and Interfaces - -### Summary - -| Component | File | Intent | Req Coverage | -|-----------|------|--------|--------------| -| `endpoint_t.on_connected` | netio.c | Async handshake notification | 2.1, 2.3 | -| `ha_state_t.client_ssl_ctx` | ha.h/myapp.c | TLS client context | 2.2 | -| `vote_ep_ctx_t` | ha.c | Per-peer context for callbacks | 1.1, 2.1–2.4 | -| `on_vote_connected()` | ha.c | Send vote RPC after handshake | 2.3, 3.1 | -| `vote_reply_msg_handler()`§ | ha.c | Decode reply; update tally | 3.1–3.4 | -| `ha_start_election_async()`¶ | ha.c | Non-blocking: connect peers + arm watchdog | 1.1–1.3, 4.1–4.3 | -| Election timeout watchdog | ha.c | Fire evaluate if deadline expires | 1.3, 4.2, 4.3 | -| `handle_request_vote` role gate‡ | ha.c | Deny votes from PRIMARY/healthy-STANDBY | 6.1–6.4 | -| `ha_elect_evaluate_and_signal` extended | ha.c | Enter COMMITTING; push config+WAL; set role† | 4.1, 7.1–7.4 | -| `CTRL_PRIMARY_COMMITTING` state† | ha.h + ha_dispatch_gate | Block job traffic; allow repl acks | 5.2, 8.1 | -| `ha_repl_ack_received()`† | ha.c | Count standby acks; transition to PRIMARY on quorum | 8.2–8.3 | -| `ha_state_t.repl_ack_count`† | ha.h | Atomic ack counter per election round | 7.4, 8.2 | - -† = superseded by ha-single-leader-role (CTRL_PRIMARY_COMMITTING removed; see Key Decisions note above). -‡ = `handle_request_vote` is now test-only; production path is `raft_on_vote_request()` (raft-adapter-conformance). -§ = renamed/generalized to `peer_ctrl_msg_handler` (ha-prevote). -¶ = peer connect + vote tally now delegated to `raft_start_election()`/`raft_on_vote_reply()` (raft-log-replication, raft-commit-safety). - -### Key Interfaces - -#### `ha_start_election_async()` - -**(Note: superseded by raft-log-replication and raft-commit-safety, see those specs' design.md. This -function no longer connects vote endpoints or tallies votes directly — it delegates to -`raft_start_election()`, which fires `raft_cb_send_vote_request` per peer and, on a terminal -result, `raft_cb_become_leader`/`raft_cb_become_follower`. The signature and non-blocking contract -below remain accurate.)** - -```c -/* Non-blocking. Arms heartbeat watchdog + election watchdog. - * Connects outgoing vote endpoints. On terminal state: - * ha_elect_evaluate_and_signal() calls cluster_init_phase2() then sets ha->role. */ -void ha_start_election_async(ha_state_t *ha, - cluster_t *c, - int n_peers, - xce_candidate_t **peers, - uint64_t my_config_tag, - uint64_t my_lsn, - const char *my_name); -``` - -#### `ha_elect_evaluate_and_signal` (extended) - -**(Note: superseded by ha-single-leader-role, see that spec's design.md. The COMMITTING transition -below — `CTRL_PRIMARY_COMMITTING`, `repl_ack_count`, `ha_arm_committing_timeout` — no longer exists; -a winning candidate is set to `CTRL_PRIMARY` directly. The `is_first_boot`/`phase2_done` gating and -the STANDBY branch below remain accurate and current.)** - -```c -/* After election determines role: */ -int is_first_boot = !ha->phase2_done; /* phase2_done set after first cluster_init_phase2 call */ - -if (result == CTRL_PRIMARY) { - if (is_first_boot) { - cluster_init_phase2(c); /* load config+data; runs once per process lifetime */ - ha->phase2_done = 1; - } - /* Enter COMMITTING: enable hooks but block job traffic until majority ack (Req 8.1) */ - atomic_store(&ha->role, CTRL_PRIMARY_COMMITTING, memory_order_release); - ha->repl_ack_count = 0; /* reset ack counter for this round */ - /* Push config and replicate WAL to standbys (Req 7.1, 7.2) */ - push_config_to_standbys(c, ...); - ha_send_ctrlhello_to_all_standbys(c); /* triggers catch-up replication */ - /* Arm committing timeout (degraded-mode fallback, Req 8.4) */ - ha_arm_committing_timeout(ha, c); -} else { - if (is_first_boot) { - cluster_init_phase2(c); /* init structures; config arrives via primary push */ - ha->phase2_done = 1; - } - atomic_store(&ha->role, CTRL_STANDBY, memory_order_release); - /* re-arm: heartbeat watchdog fires ha_on_primary_ctrl_fd_rdhup on next timeout */ -} -``` - -#### `ha_repl_ack_received()` — called when a standby acks config+WAL catch-up - -**(Note: this function does not exist in current code — superseded by ha-single-leader-role, see -that spec's design.md. Standby ack counting toward promotion was removed along with -`CTRL_PRIMARY_COMMITTING`; job-traffic admission is now `ha_has_write_quorum()`'s live count of -connected standbys, evaluated per-request inside `ha_dispatch_gate`, not a one-time per-round tally.)** - -```c -void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) { - int acks = atomic_fetch_add(&ha->repl_ack_count, 1) + 1; - int quorum = (ha->n_peers + 2) / 2; /* majority of total cluster */ - if (acks >= quorum - 1) { /* -1 because primary counts itself */ - ha_disarm_committing_timeout(ha); - ha_promote_to_primary(c); /* sets role=CTRL_PRIMARY, enables hooks */ - } -} -``` - -Called from the existing `handle_ctrlhello` reply path (CtrlHelloReply ack from standby after catch-up) and from `push_config_to_standbys` ack handler. - -#### `handle_request_vote` role gate (extended) - -**(Note: superseded by raft-adapter-conformance, see that spec's design.md. `handle_request_vote` -is now `#ifdef XCE_TESTING`-only, kept as a test harness entry point; production vote-request -dispatch is `cluster.c`'s `case OPCode_requestVote:`, which applies its own freshness gate and then -calls `raft_on_vote_request()` — the PRIMARY-role deny is now `on_vote_eligible`/`RAFT_LEADER` -inside the raft library, not this snippet. `CTRL_PRIMARY_COMMITTING` in the snippet below also no -longer exists — see the Key Decisions note above.)** - -```c -ctrl_role_t my_role = atomic_load_explicit(&ha->role, memory_order_acquire); -if (my_role == CTRL_PRIMARY || my_role == CTRL_PRIMARY_COMMITTING) - return 0; /* deny — I am primary (or committing); don't grant votes to challengers */ -if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) - return 0; /* deny — my primary is alive */ -/* CTRL_INIT, CTRL_ELECTING, or STANDBY without live primary: evaluate Raft gates below */ -``` - -#### `ha_heartbeat_is_fresh()` - -**(Fixed: signature and field name were stale — corrected to match current `ha.c`. The function is -non-static (exported) and reads `ha->last_activity`, a whole-second `time_t`, not a -`struct timespec last_heartbeat` — the design's original nsec-precision sketch was implemented as -whole-second precision, and it now null-guards `last_activity == 0` for "no primary message seen yet".)** - -```c -int ha_heartbeat_is_fresh(ha_state_t *ha) { - time_t last = ha->last_activity; /* time_t load is atomic on LP64 */ - if (last == 0) - return 0; /* no primary message seen yet */ - struct timespec now; - clock_gettime(CLOCK_MONOTONIC, &now); - long ms = (long)(now.tv_sec - last) * 1000L; - return ms < (long)ha->config.heartbeat_timeout_ms; -} -``` - -#### `myapp_run` (restructured — no blocking) -```c -int myapp_run(myconf_t *myconf) { - cluster_t c; memset(&c, 0, sizeof(c)); c.myconf = myconf; - cluster_init_phase1(&c); /* tunables, managers, endpoints zarray */ - initialize_openssl(); - c.ha.ssl_ctx = create_ssl_context(myconf->certfile, myconf->keyfile); - c.ha.client_ssl_ctx = SSL_CTX_new(TLS_client_method()); - SSL_CTX_set_verify(c.ha.client_ssl_ctx, SSL_VERIFY_NONE, NULL); - iomgr_init(&c.iomgr); iomgr_set_ssl_ctx(&c.iomgr, c.ha.ssl_ctx); - vtaskmgr_init(&c.vtaskmgr, nvcpus); - iomgr_start(&c.iomgr); vtaskmgr_start(&c.vtaskmgr); - cluster_register_listener(&c); /* listener in epoll before election */ - delaymgr_init(&c.delaymgr); delaymgr_start(&c.delaymgr); /* moved from cluster_start */ - persist_init(&c.persist, ...); persist_start(&c.persist); /* moved from cluster_start */ - load_primary_config(&c); /* pre-load config_tag for freshness gate */ - /* bootstrap_config_if_needed() REMOVED — freshness gate + primary push replaces it */ - /* Build peer array from myconf->peers_cfg ... */ - ha_start_election_async(&c.ha, &c, n_peers, peer_ptrs, my_tag, my_lsn, myconf->name); - return cluster_run(&c); /* unchanged — just waits for stop signal */ -} -``` - -**Note**: `vote_ep_ctx_t` carries `cluster_t *c` so that `ha_elect_evaluate_and_signal` -(called from vtask context via `vote_reply_msg_handler`§) has access to `cluster_t` for -`cluster_init_phase2`, `push_config_to_standbys`, and `ha_send_ctrlhello_to_all_standbys`. -§ (renamed/generalized to `peer_ctrl_msg_handler` — superseded by ha-prevote.) - -#### `ha_dispatch_gate` — CTRL_PRIMARY_COMMITTING case - -**(Note: superseded by ha-single-leader-role, see that spec's design.md. This `case` does not exist — -`ha_dispatch_gate`'s switch has only `CTRL_PRIMARY`, `CTRL_INIT`, `CTRL_ELECTING`, `CTRL_STANDBY`. -The write-admission gating this case describes now lives inside the `CTRL_PRIMARY` case itself, via -`ha_has_write_quorum()`: state-changing ops are refused with `XCE_ERR_NOT_READY` when the live -connected-standby count is below majority, while the same replication/read-only opcode allowlist -below — plus fact-bearing ops (`OPCode_notify`/`update`/`setpreason`/`clrpreason`) — stays admitted.)** - -```c -case CTRL_PRIMARY_COMMITTING: - /* Allow replication acks (config push ack, CtrlHelloReply, WAL ack) so - * ha_repl_ack_received() can count them toward majority. - * Deny job dispatch and queue/resource management until majority ack. */ - switch (opcode) { - case OPCode_heartbeat: - case OPCode_replConfig: /* config push ack from standby */ - case OPCode_replConfigPush: - case OPCode_upgradeConfigAck: - case OPCode_hello: /* CtrlHelloReply ack from standby after catch-up */ - case OPCode_walAck: - case OPCode_replEntry: - case OPCode_replAck: - case OPCode_replCatchUpDone: /* standby signals catch-up complete → count ack */ - return 0; - default: - return -XCE_ERR_NOT_READY; - } -``` - -`OPCode_replCatchUpDone` handler in `cluster.c` calls `ha_repl_ack_received(ha, c)` to -increment the ack counter and conditionally transition to `CTRL_PRIMARY`. -(Note: `ha_repl_ack_received` does not exist — see notes above; superseded by ha-single-leader-role.) - -## Error Handling - -- **Connect failure**: endpoint skipped; peer absent from election. After watchdog timeout, election evaluates with available votes only. -- **SSL handshake failure**: `vote_ep_err_handler` calls `handle_request_vote_reply(grant=0)`. -- **`cluster_init_phase2` failure from vtask**: logs ERROR; posts stop to wbuf → clean shutdown. -- **Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING**: `ha_dispatch_gate` returns `XCE_ERR_NOT_READY`; client retries. (Note: `CTRL_PRIMARY_COMMITTING` no longer exists — superseded by ha-single-leader-role; the equivalent current behavior is `ha_dispatch_gate`'s `CTRL_PRIMARY` case refusing state-changing ops via `ha_has_write_quorum()` until majority standbys are connected.) -- **Committing timeout (Req 8.4)**: if majority ack not received within `committing_timeout_ms`, watchdog fires `ha_promote_to_primary` anyway (degraded mode) with a WARN log. Prevents deadlock when standbys are unavailable at startup. -- **Vote re-arm on STANDBY**: `ha_elect_evaluate_and_signal` re-arms `ha_start_election_async`. Heartbeat watchdog fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING → re-election. - -## Testing Strategy - -### Unit Tests -1. `ha_start_election_async` returns in < 5 ms without blocking -2. `on_vote_connected` encodes and sends correct `OP_requestVote` -3. `vote_reply_msg_handler` reads capnp fields before free — ASAN clean -4. `handle_request_vote` role gate: PRIMARY→deny; STANDBY+fresh→deny; STANDBY+stale→evaluate; ELECTING→evaluate -5. `ha_elect_evaluate_and_signal` enters COMMITTING then PRIMARY after majority ack -6. Election watchdog fires within `election_timeout_ms + 200ms` when no replies -7. `ha_repl_ack_received`: majority acks → `CTRL_PRIMARY`; below majority → stays COMMITTING -8. Committing timeout watchdog: promotes to PRIMARY (degraded) if acks don't arrive - -### Integration Tests -9. 3-node startup: ctrl-a wins election → CTRL_PRIMARY_COMMITTING → pushes config to ctrl-b/c → receives replCatchUpDone × 2 → CTRL_PRIMARY → serves jobs -10. ctrl-b/c start with config_tag=0; receive config via primary push (no independent fetch); confirm via replCatchUpDone -11. Primary crash (SIGKILL): ctrl-b heartbeat timeout → re-election → COMMITTING → ack from ctrl-c → PRIMARY; serves traffic from last WAL state -12. Client during COMMITTING receives `XCE_ERR_NOT_READY` and retries successfully after gate opens - -### Regression Tests -13. Single-node self-elects; no standbys → committing timeout fires → degraded PRIMARY immediately -14. ASAN clean: no UAF, no stack-use-after-return in election or committing path diff --git a/.kiro/specs/election-iomgr-refactor/requirements.md b/.kiro/specs/election-iomgr-refactor/requirements.md deleted file mode 100644 index da64e6f2..00000000 --- a/.kiro/specs/election-iomgr-refactor/requirements.md +++ /dev/null @@ -1,206 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE controller operates a 3-node HA cluster where one node is elected PRIMARY before -serving jobs. The current election mechanism mixes blocking per-peer pthreads with the iomgr -epoll event loop, causing SSL context confusion, capnp use-after-free, and race conditions. - -This feature replaces the blocking pthread-based election with a fully async iomgr-driven -implementation, and formalizes the complete HA protocol: election by config freshness, -replication push after primary determined, and availability gate after majority ack. - -## Boundary Context - -- **In scope**: Outgoing vote RPC transport; SSL context lifecycle for vote connections; - election quorum; role-based vote grant/deny; config+state replication push to standbys - after election; availability gate (majority ack before serving traffic). -- **Out of scope**: Incoming vote RPC handling (already correct in cluster.c); - dynamic cluster membership changes (candidate list is static); config schema changes. -- **Adjacent expectations**: iomgr started and listener registered before election begins. - The FSM state machine and `ha_dispatch_gate` remain the authority on opcode routing. - -## Requirements - -### 1. No Blocking Threads for Outgoing Vote RPCs - -**Objective:** As the cluster operator, I want the controller election to complete reliably -without SSL errors or memory corruption, so that the cluster reaches PRIMARY state on every -startup. - -#### Acceptance Criteria - -1. The controller shall not create pthreads whose sole purpose is to send a vote request and - block waiting for a reply during election. -2. When the controller initiates an election round, the controller shall send `OP_requestVote` - to each peer using the same I/O mechanism used for all other outgoing RPCs. -3. If a peer does not respond within the election probe timeout, the controller shall treat - that peer as a denied vote without stalling the event loop. - ---- - -### 2. Async Outgoing Vote Connection via iomgr - -**Objective:** As the controller process, I want outgoing vote connections managed by the -iomgr event loop, so that SSL handshake and data transfer do not block other I/O. - -#### Acceptance Criteria - -1. When the controller sends a vote request to a peer, the controller shall establish the TCP - connection and complete the TLS handshake without blocking the main event loop thread. -2. The controller shall use a TLS client context (not a server context) for outgoing vote - connections. -3. When the TLS handshake with a peer completes, the controller shall send the - `OP_requestVote` packet through the established connection. -4. If the TCP connection or TLS handshake to a peer fails, the controller shall count that - peer as a denied vote and continue the election with the remaining peers. - ---- - -### 3. Vote Reply Handling Through iomgr Dispatch - -**Objective:** As the controller process, I want vote replies to arrive as normal iomgr -packets, so that the capnp segment lifetime is managed consistently with all other RPCs. - -#### Acceptance Criteria - -1. When a peer sends a vote reply, the controller shall decode it through the standard iomgr - packet-receive path rather than a raw blocking `read`. -2. The controller shall not access capnp-decoded fields after the capnp segment for that - packet has been freed. -3. When a vote reply is received, the controller shall extract the peer's `term`, - `vote_granted`, `peer_config_tag`, and `peer_last_log_lsn` and update the election tally. -4. If the peer's term in a vote reply is greater than the candidate's current term, the - controller shall update its own term before recording the vote result. - ---- - -### 4. Election Quorum, Ground-Truth, Timeout, and Retry - -**Objective:** As the cluster operator, I want the election to converge to a single PRIMARY -that holds the ground-truth cluster state, so that no data is lost and the cluster becomes -available even if nodes start at different times. - -#### Acceptance Criteria - -1. When vote replies from a majority of candidate nodes (≥ ⌈(n+1)/2⌉) grant the vote, the - controller shall transition to CTRL_PRIMARY_COMMITTING (pre-availability state). -2. The controller shall grant a vote to a candidate only if the candidate's `config_tag` is - greater than or equal to the voter's own `config_tag`. This freshness gate guarantees that - the elected primary holds the ground-truth configuration — no post-election config fetch - is ever needed by the winner. -3. On cold start, only the node holding the highest `config_tag` can win election. All other - nodes (config_tag=0 on first boot) will lose the freshness gate and cannot become primary. -4. When all peers have replied but no majority granted the vote, the controller shall retry - after the next heartbeat timeout. -5. When no vote replies arrive within the election probe timeout, the controller shall - complete the round as a split vote and retry. -6. While in CTRL_ELECTING state, if a heartbeat is received from a node that has become - PRIMARY, the controller shall transition to CTRL_STANDBY and abandon the election. -7. The controller shall retry elections indefinitely until it wins a quorum or detects an - existing primary via heartbeat. - ---- - -### 5. FSM State Consistency - -**Objective:** As the controller process, I want the FSM state to accurately reflect the -controller's role, so that the dispatch gate correctly accepts or rejects incoming RPCs. - -#### Acceptance Criteria - -1. While in CTRL_ELECTING state, the controller shall accept `OP_requestVote` and - `OP_probeConfigTag` and shall reply with `XCE_ERR_NOT_READY` to all other opcodes. -2. While in CTRL_PRIMARY_COMMITTING state, the controller shall accept replication-related - opcodes (config push ack, WAL ack) and shall reply with `XCE_ERR_NOT_READY` to job - dispatch and queue management opcodes. -3. When the controller transitions to CTRL_PRIMARY (fully available), it shall begin accepting - job dispatch and queue management opcodes. -4. When the controller transitions to CTRL_STANDBY, it shall begin accepting replication and - heartbeat opcodes. -5. The controller shall maintain a monotonically increasing term counter across election rounds - and persist it to disk so the term survives a process restart. - ---- - -### 6. Vote-Granting Rules Based on Voter Role - -**Objective:** As the cluster operator, I want votes granted only when the voter has no live -primary, so that a healthy primary is never displaced, and a crashed primary triggers a new -election only after the heartbeat timeout expires. - -#### Acceptance Criteria - -1. While in CTRL_PRIMARY state, the controller shall deny all incoming `OP_requestVote` - requests regardless of the candidate's term or config_tag. -2. While in CTRL_STANDBY state with a live primary connection (heartbeat received within - `heartbeat_timeout_ms`), the controller shall deny all incoming `OP_requestVote` requests. -3. When in CTRL_STANDBY state and the primary heartbeat has not been received within - `heartbeat_timeout_ms`, the controller shall transition to CTRL_ELECTING and begin - granting votes to eligible candidates. -4. While in CTRL_INIT or CTRL_ELECTING state, the controller shall evaluate incoming - `OP_requestVote` requests using the term, voted_for, and freshness gates (config_tag, - last_log_lsn) without any role-based denial. -5. When a primary crashes (stops sending heartbeats), the surviving standbys shall detect - the loss within `heartbeat_timeout_ms`, transition to CTRL_ELECTING, and elect a new - primary from among themselves using the last committed state. - ---- - -### 7. Post-Election Config and State Replication - -**Objective:** As the cluster operator, I want the elected primary to immediately propagate -its ground-truth configuration and cluster state to all standbys, so that standbys have -up-to-date state and can become primary on failover without any post-election config fetch. - -#### Acceptance Criteria - -1. When the controller wins election and enters CTRL_PRIMARY_COMMITTING, it shall push the - cluster configuration to all peer controllers that have a lower `config_tag`. -2. When the controller wins election, it shall replicate its WAL (committed job/queue/worker - state) to all standbys via the existing catch-up replication mechanism. -3. When a new standby starts with `config_tag=0` and loses the election, it shall receive the - cluster configuration via the primary's replication push — it shall NOT attempt an - independent config fetch before the primary pushes it. -4. The controller shall confirm config and state receipt from each standby via acknowledgement - before counting that standby toward the majority ack threshold. - ---- - -### 8. Primary Availability Gate - -**Objective:** As the cluster operator, I want the primary to begin serving job traffic only -after the majority of controllers have confirmed they hold the replicated state, so that -failover can always promote a standby with current data. - -#### Acceptance Criteria - -1. After winning election, the controller shall enter CTRL_PRIMARY_COMMITTING and shall not - accept job dispatch or queue management requests until the availability gate opens. -2. When a majority of peer controllers (≥ ⌈(n+1)/2⌉ − 1 peers, since primary counts itself) - have acknowledged receipt of the current config and committed WAL state, the controller - shall transition from CTRL_PRIMARY_COMMITTING to CTRL_PRIMARY. -3. When CTRL_PRIMARY state is reached, the controller shall begin accepting all client - opcodes including job dispatch, queue control, and resource management. -4. If the majority ack threshold cannot be reached within a configurable timeout, the - controller shall log a warning and open anyway (degraded mode) so the cluster does not - deadlock when standbys are unavailable at startup. - ---- - -### 9. No Regression in Existing Behaviors - -**Objective:** As the cluster operator, I want all existing behaviors preserved after the -refactor, so that nodes that previously worked correctly continue to do so. - -#### Acceptance Criteria - -1. When a single-node cluster starts (candidate count = 1), the controller shall self-elect - to CTRL_PRIMARY without contacting any peers. -2. The elected primary shall never need to fetch config from another node — the vote - freshness gate guarantees it already holds the highest config_tag. -3. When the primary's control channel connection drops (RDHUP), the controller shall trigger a - new election. The winning standby shall promote via `ha_promote_to_primary` (hooks only — - its state is already current via continuous replication). -4. The controller shall prioritize election by node priority index: lower-index nodes start - campaigning first, reducing split votes on simultaneous startup. diff --git a/.kiro/specs/election-iomgr-refactor/research.md b/.kiro/specs/election-iomgr-refactor/research.md deleted file mode 100644 index 22d81b7c..00000000 --- a/.kiro/specs/election-iomgr-refactor/research.md +++ /dev/null @@ -1,518 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `election-iomgr-refactor` -- **Discovery Scope**: Complex Integration -- **Key Findings**: - - `endpoint_connect()` already sets SSL client mode and triggers epoll-driven async handshake — almost exactly what outgoing vote connections need. Gap: blocking DNS + no "handshake done" callback. - - The main thread can safely block on `sem_timedwait(&ctx->done)` while iomgr runs in its own thread. This is identical to the heartbeat watchdog pattern already in ha.c. - - Incoming vote replies via iomgr (`myreq_handler → case OPCode_requestVote:`) already work. We only need to add `case OPCode_reply:` with `data.requestvotereplymsg` to handle the vote reply on the receiver side, and a new `handle_vote_reply_from_ep()` function for outgoing endpoints. - -## Research Log - -### iomgr Outgoing Connection Support -- **Context**: Can iomgr manage outgoing vote connections instead of ad-hoc pthread raw socket I/O? -- **Sources**: netio.c lines 654–678, 432–460, 962–990; ha.c lines 1388–1430; config_repl.c connect_ssl_raw -- **Findings**: - - `endpoint_connect()` does blocking DNS (gethostbyname in socket_connect) then sets SSL client state and calls handle_ssl_handshake() once. The rest is epoll-driven. - - `handle_ssl_handshake()` uses SSL_do_handshake() (non-blocking); retries on WANT_READ/WANT_WRITE via subsequent epoll events. - - `endpoint_calc_flags()` arms EPOLLOUT during handshake phase automatically. - - `iomgr_monitor()` uses EPOLL_CTL_ADD with EPOLLIN only; endpoint_update_epoll() adjusts flags dynamically. - - Gap: no "handshake completed" callback on endpoint_t. Solution: add `on_connected` callback invoked by io_handler after ssl_handshake_done transitions from 0 to 1. -- **Implications**: Minimal netio.c change (add callback field + invocation); no new epoll machinery needed. - -### Election Timeout Mechanism -- **Context**: How to implement per-round election timeout without blocking the event loop? -- **Sources**: ha.c ha_arm_heartbeat_timeout_vtask (line ~1658), am_i_primary_quorum sem_wait (line 1339) -- **Findings**: - - Main thread blocks on `sem_wait(&ctx->done)` while iomgr + vote-reply threads run — this is safe. - - Replace `sem_wait` with `sem_timedwait` using `election_timeout_ms` deadline. On ETIMEDOUT, set ctx->result=0 and proceed to evaluate. - - This mirrors the heartbeat watchdog pattern exactly. No extra thread needed. -- **Implications**: Two-line change to am_i_primary_quorum; no new timer infrastructure required. - -### Vote Reply Routing -- **Context**: Who calls handle_request_vote_reply() in the new design? -- **Sources**: cluster.c myreq_handler, ha.c handle_request_vote_reply -- **Findings**: - - Outgoing vote endpoints will receive `OPCode_reply` packets (since the server sends via xce_reply → OPCode_reply). - - The outgoing endpoint needs a `msg_handler` that decodes the RequestReply, extracts the requestvotereplymsg, and calls handle_request_vote_reply(). - - This handler runs in vtaskmgr context (from io_handler → vtask_handle_msg), which is safe since ha_elect_ctx_t operations are mutex-protected. -- **Implications**: New `vote_reply_msg_handler()` function registered on each outgoing endpoint. No new threading. - -### SSL Context for Outgoing Vote Connections -- **Context**: The bug was server-method SSL context used for SSL_connect. -- **Findings**: `endpoint_connect()` calls `SSL_new(iomgr->ssl_ctx)` then `SSL_set_connect_state()`. Since iomgr->ssl_ctx was created with TLS_server_method(), SSL_set_connect_state() on a server context is undefined behavior. -- **Selected approach**: Create a dedicated `client_ssl_ctx` (TLS_client_method, SSL_VERIFY_NONE) stored in ha_state_t. Pass it to endpoint_connect via `endpoint_set_ssl_ctx(ep, ha->client_ssl_ctx)` before connecting. -- **Rationale**: Separating server and client contexts is the correct OpenSSL pattern. The client context does not need a certificate — vote RPC security comes from the term/config_tag payload, not TLS cert chain. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks | -|--------|-------------|-----------|-------| -| Keep pthreads | Fix SSL context + UAF bugs only | Minimal code change | Same architectural complexity; new bugs likely | -| Full iomgr-native | Outgoing vote via endpoint_connect + on_connected callback + reply via msg_handler | One I/O loop, no UAF possible, matches existing patterns | Requires netio.c and ha.c changes; more complex | -| Hybrid: iomgr connect, pthread reply | Use iomgr for connect/handshake, pthread to wait for reply | Partial improvement | Still two I/O paths; UAF still possible | - -**Selected**: Full iomgr-native. The minimal netio.c change (add on_connected callback) enables a clean design with no mixed-I/O issues. - -## Design Decisions - -### Decision: on_connected callback in endpoint_t -- **Context**: Need to know when SSL handshake completes on outgoing vote connection so we can send the vote RPC. -- **Alternatives**: (1) Poll ssl_handshake_done in main thread — requires busy-wait. (2) Add callback — clean, one extra field. -- **Selected**: Add `void (*on_connected)(endpoint_t *ep, void *ctx)` to endpoint_t. Set ctx pointer via endpoint_set_conn_userdata(). -- **Rationale**: Mirrors msg_handler and err_handler patterns already on endpoint_t. - -### Decision: sem_timedwait for election timeout -- **Context**: Need election round to time out if no replies arrive. -- **Alternatives**: (1) Delaymgr — 200ms polling jitter, designed for recurring tasks. (2) Separate watchdog thread. (3) sem_timedwait inline in am_i_primary_quorum. -- **Selected**: sem_timedwait with election_timeout_ms deadline. -- **Rationale**: Identical to heartbeat watchdog pattern; no extra thread; precise CLOCK_REALTIME deadline. - -### Decision: client_ssl_ctx stored in ha_state_t -- **Context**: Need TLS_client_method context for outgoing connections; iomgr uses TLS_server_method. -- **Selected**: Create client_ssl_ctx in myapp_run alongside ssl_ctx; store in ha_state_t; set on outgoing endpoints before connect. -- **Rationale**: Clear ownership, no per-connection allocation, reused across election rounds. - -## Risks & Mitigations -- **DNS blocking in endpoint_connect()**: socket_connect uses gethostbyname. For loopback/localhost addresses (test environment), this resolves instantly. For production, replace with getaddrinfo + async resolver if needed — deferred. -- **on_connected callback races**: The callback fires in iomgr's io_worker thread. ha_elect_ctx_t already uses a mutex. All handle_request_vote_reply calls are mutex-protected — safe. -- **outgoing endpoint lifetime**: The outgoing endpoint must outlive the election round. Endpoints are stored in a local array in am_i_primary_quorum and cleaned up after sem_timedwait returns. - ---- - -# Gap Analysis (Updated) - -> Analysis date: 2026-06-01. Phase: tasks-generated / ready_for_implementation. -> This section audits the **current implementation state** vs. the 9-requirement spec -> to determine what remains before the spec can be marked complete. - -## Analysis Summary - -- **Scope**: election-iomgr-refactor — replace blocking pthread election with full iomgr-native async path; add COMMITTING gate and majority-ack availability gate. -- **Implementation status**: **~90% complete**. All 15 core implementation tasks (1.1–5.3) are confirmed done. Integration tests 6.1–6.3 are in meson.build and test files exist. Only tasks 6.4 and 6.5 remain. -- **Remaining work**: (1) RDHUP failover integration test (`test_ha_rdhup_failover.c` or similar); (2) full regression + ASAN verification pass. -- **Recommendation**: The chosen approach (full iomgr-native, raft-library backed) is already implemented. No architecture changes needed. Proceed to task 6.4 directly. -- **Effort remaining**: S (1–2 days). Risk: Low. - -## Requirement-to-Asset Map - -| Requirement | Asset | Status | -|-------------|-------|--------| -| Req 1: No blocking threads for vote RPCs | `ha_start_election_async` in ha.c; old `am_i_primary_quorum` / `vote_request_thread` removed | ✅ Done | -| Req 2: Async outgoing vote via iomgr | `on_vote_connected`, `vote_ep_ctx_t`, `endpoint_set_on_connected` (netio.c:782), `client_ssl_ctx` in ha_state_t | ✅ Done | -| Req 3: Vote reply via iomgr dispatch | `vote_reply_msg_handler`, `peer_ctrl_msg_handler`, capnp fields read before `xce_free_request` | ✅ Done | -| Req 4: Quorum, ground-truth, timeout, retry | `ha_elect_evaluate_and_signal` (quorum math), `raft_on_vote_request` (term/voted_for), `election_timeout_fn`, raft retry callbacks | ✅ Done | -| Req 5: FSM state consistency | `ha_dispatch_gate` extended for CTRL_PRIMARY_COMMITTING; CTRL_ELECTING gate | ✅ Done | -| Req 6: Role-based vote denial | `handle_request_vote` role gate; `ha_heartbeat_is_fresh`; RDHUP → `ha_on_primary_ctrl_fd_rdhup` | ✅ Done | -| Req 7: Post-election config + WAL replication | `push_config_to_standbys`, `ha_send_ctrlhello_to_all_standbys` in cluster.c | ✅ Done | -| Req 8: Availability gate | `ha_repl_ack_received` majority check; `committing_timeout_fn` degraded fallback | ✅ Done | -| Req 9: No regressions | `cluster_init` legacy wrapper preserved; single-node shortcircuit in `ha_start_election_async` | ✅ Done (pending 6.5 verification) | - -## Gaps and Constraints - -### Gap 1: Task 6.4 — RDHUP failover integration test (MISSING) -- **What's needed**: A Unity test that verifies: SIGKILL primary → standby detects heartbeat timeout → CTRL_ELECTING → CTRL_COMMITTING → CTRL_PRIMARY via `ha_rdhup_election_hook` + hook infrastructure. -- **Pattern to follow**: `test_p2_integration.c` uses in-process persist + no live network; `test_ha_failover.c` (already exists, 0 bytes based on meson but check) or `test_ha_connection_reuse.c` for the RDHUP hook pattern. -- **Where to add**: New file `testsuite/test_ha_rdhup_integration.c`; add to meson.build alongside `test_ha_failover.c`. -- **Key hooks to use**: `ha_rdhup_election_hook` (defined in ha.h under `XCE_TESTING`), `ha_phase2_hook` (already used in `test_ha_election.c`), `ha_push_config_hook`, `ha_send_ctrlhello_hook`. -- **Complexity**: Low — all test seams already exist. This is wiring hooks together, not new functionality. - -### Gap 2: Task 6.5 — Regression + ASAN clean (PENDING CI) -- **What's needed**: Run `meson test --suite controller` under `ASAN_OPTIONS=detect_leaks=0`; verify count ≥ baseline; no new ASAN errors. -- **Status**: Cannot confirm without running the build. The test files exist and are in meson.build. The main risk is that `test_ha_rdhup_integration.c` (6.4) is the final piece before 6.5 is runnable. -- **Action**: Run `/xce-build controller` after 6.4 is written. - -### Constraint: `ha_wait_for_heartbeat_or_timeout` still compiled -- **Observation**: `ha_wait_for_heartbeat_or_timeout` and `am_i_primary` still exist in ha.c (not deleted — the old implementation is preserved for `ha_dispatch_gate` and other legacy paths). -- **Impact**: Not a blocking issue. The function is no longer called from myapp_run. It's dead code but not harmful. May be removed in cleanup pass. -- **Research Needed**: Grep callsites before deleting to ensure nothing else calls it. - -## Current State Investigation - -### Existing Assets (relevant to remaining work) - -**Test infrastructure already in place:** -- `ha_rdhup_election_hook` — declared in ha.h under `XCE_TESTING`; used in `ha_on_primary_ctrl_fd_rdhup` to inject mock election -- `ha_phase2_hook` — used in `test_ha_election.c` to count phase2 invocations without real cluster -- `ha_push_config_hook`, `ha_send_ctrlhello_hook` — used in evaluate_committing tests -- `ha_tcp_connect_hook` — for injecting socketpair fds instead of real TCP - -**Test files for 6.4 reference:** -- `test_ha_election.c` (985 lines) — shows how to set up ha_state_t + ha_elect_ctx_t + hooks for unit testing -- `test_ha_evaluate_committing.c` (444 lines) — shows COMMITTING path test pattern -- `test_ha_failover.c` — exists in meson.build (line 112); may already cover parts of 6.4 - -**Key flow for task 6.4 test:** -1. Create `ha_state_t` with CTRL_STANDBY + live heartbeat timestamp -2. Set `ha_rdhup_election_hook` to a stub that immediately calls `ha_elect_evaluate_and_signal` with 1-node PRIMARY result -3. Set `ha_phase2_hook` to no-op (phase2_done=1 simulating post-first-election standby) -4. Call `ha_on_primary_ctrl_fd_rdhup(ha, NULL)` — should: set CTRL_ELECTING, call hook, evaluate → COMMITTING → PRIMARY -5. Assert final role = CTRL_PRIMARY - -## Implementation Approach Options - -### Option A: Write minimal 6.4 test using existing hooks (Recommended) -- **Files to create**: `testsuite/test_ha_rdhup_integration.c` (estimate 150–200 lines) -- **Files to modify**: `meson.build` — add new test file -- **Why**: All test seams exist; pattern is established in test_ha_election.c and test_ha_evaluate_committing.c. No production code changes needed. -- **Trade-offs**: ✅ Fast; ✅ ASAN-clean by design (hooks avoid real network); ❌ In-process only (not a true 3-process integration test — but matches existing test philosophy) - -### Option B: Extend test_ha_failover.c if already started -- **Files to modify**: `testsuite/test_ha_failover.c` if it exists and has content -- **Why**: Avoids new file; keeps related tests together -- **Trade-offs**: ✅ Co-locates failover tests; ❌ Need to verify file has content first - -### Option C: Live 3-process integration test via deploy-ha.sh -- **Scope**: Shell script test using `deploy-ha.sh` + log parsing -- **Why**: Most realistic; validates actual process behavior -- **Trade-offs**: ✅ Full end-to-end; ❌ Slow; ❌ Non-deterministic timing; ❌ Not part of `meson test` suite — better as a separate CI job - -## Effort and Risk - -| Item | Effort | Risk | Justification | -|------|--------|------|---------------| -| Task 6.4 (RDHUP test) | S (half day) | Low | All test seams exist; pattern established | -| Task 6.5 (regression) | S (1 hour CI run) | Low | Code already in place; just needs build + test run | -| Remove dead code (ha_wait_for_heartbeat_or_timeout) | S | Low | Optional cleanup; no impact on correctness | - -## Recommendations for Design Phase - -This spec is already past the design phase and into implementation. The recommendations are: - -1. **Proceed directly to task 6.4**: Write `test_ha_rdhup_integration.c` using Option A. The `ha_rdhup_election_hook` + `ha_phase2_hook` + `ha_push_config_hook` seams cover all assertions needed without live network I/O. - -2. **Check test_ha_failover.c content first**: If it already has RDHUP tests (it's in meson.build), extend rather than create a new file. - -3. **Run `/xce-build controller` for 6.5**: After 6.4 is written, run the full test suite to confirm no regressions. Watch for ASAN errors in the new `vote_ep_ctx_t` allocation/free path. - -4. **Deferred research item — DNS blocking**: `endpoint_connect` uses blocking `gethostbyname`. For localhost test targets this is instant. For production multi-host clusters, this becomes a delaymgr task concern. Mark as a follow-up for the ha-peer-connection-reuse spec. - ---- - -# Cross-Spec Gap Audit (2026-06-03) - -> Triggered by: `/kiro-validate-gap ha-config-bootstrap election-iomgr-refactor, check am_i_primary() still be used?` - -## Analysis Summary - -- This spec is **100% complete** (phase: `implemented`). No gaps remain. -- `am_i_primary()` is **dead code** — not called from any production path. See ha-config-bootstrap/research.md for full analysis. -- The stale comment at cluster.c:1015 (`/* Determine initial role based on am_i_primary() result set by myapp_run */`) should be updated in a cleanup pass to reference `ha_elect_evaluate_and_signal`. -- `ha_wait_for_heartbeat_or_timeout` is also dead code (confirmed: removed from myapp_run per task 5.1/5.3); its function body in ha.c is unreachable from production entry points. - -## Deferred Cleanup Items - -| Item | File | Action | -|------|------|--------| -| `am_i_primary()` function body | ha.c:869 | Delete; also delete 3 test cases in test_ha_role.c | -| `start_heartbeat_listener()` / `stop_heartbeat_listener()` stubs | ha.c:951-960 | Delete (no-op stubs, no callers) | -| Stale comment about `am_i_primary` | cluster.c:1015 | Update to reference `ha_elect_evaluate_and_signal` | -| `ha_wait_for_heartbeat_or_timeout` | ha.c | Delete (removed from myapp_run, no other callers) | - -None of these affect correctness or test pass rate. Safe to clean up in a single dedicated commit. - ---- - -# Gap Analysis — Completion Audit (2026-06-02) - -> Analysis date: 2026-06-02. Phase: `implemented`. All tasks `[x]`. -> Purpose: confirm full coverage of the 9 requirements against the live codebase -> before closing the spec. - -## Analysis Summary - -- **Scope**: election-iomgr-refactor — async iomgr election, COMMITTING gate, majority-ack availability gate, RDHUP failover wiring. -- **Implementation status**: **100% complete**. All 15 implementation tasks (1.1–5.3) and all 5 validation tasks (6.1–6.5) are marked done and verified in the live codebase. -- **No remaining gaps**: The previous gap (tasks 6.4, 6.5) was closed by `test_ha_rdhup_integration.c` (6 test cases, all wired into `controller_test.c`) and the full test suite build. -- **Deferred item still open**: DNS blocking in `endpoint_connect` (`gethostbyname`) — non-blocking for localhost; tracked as a follow-up for `ha-peer-connection-reuse` spec. -- **Effort remaining**: Zero. The spec can be marked closed. - -## Requirement-to-Asset Map — Final State - -| Requirement | Key Assets | Verified | -|-------------|-----------|---------| -| Req 1: No blocking vote threads | `ha_start_election_async`, `raft_start_election`; `vote_request_thread` / `am_i_primary_quorum` removed from `myapp.c` | ✅ | -| Req 2: Async outgoing vote via iomgr | `endpoint_set_on_connected` (netio.c:782), `on_connected` callback (netio.c:441), `vote_ep_ctx_t`, `client_ssl_ctx` in `ha_state_t` | ✅ | -| Req 3: Vote reply via iomgr dispatch | `vote_reply_msg_handler`; capnp fields read before `xce_free_request`; `vote_reply_dispatch` / `raft_on_vote_reply` | ✅ | -| Req 4: Quorum, ground-truth, timeout, retry | `ha_elect_evaluate_and_signal` quorum math; `raft_election_timeout_fn`; `raft_on_vote_request` term/freshness gate; raft retry via `raft_on_vote_reply` | ✅ | -| Req 5: FSM state consistency | `ha_dispatch_gate` — `CTRL_PRIMARY_COMMITTING` case allows replication ops, denies job ops; `CTRL_ELECTING` case; `CTRL_STANDBY` case | ✅ | -| Req 6: Role-based vote denial | `handle_request_vote` role gate; `ha_heartbeat_is_fresh`; RDHUP → `ha_on_primary_ctrl_fd_rdhup` → `ha_start_election_async` | ✅ | -| Req 7: Post-election config + WAL replication | `push_config_to_standbys` (ha.c:420-425); `ha_send_ctrlhello_to_all_standbys` (ha.c:435); `ha_start_primary_heartbeat` (ha.c:441) | ✅ | -| Req 8: Availability gate | `ha_repl_ack_received` majority check; `committing_timeout_fn` degraded fallback; `ha_promote_to_primary` sets `CTRL_PRIMARY` | ✅ | -| Req 9: No regressions | Single-node shortcircuit in `raft_start_election`; `cluster_init` legacy wrapper; `test_ha_rdhup_integration.c` 6 RDHUP scenarios all PASS | ✅ | - -## Validation Coverage - -| Task | Test File | Cases | Status | -|------|-----------|-------|--------| -| 6.1 Election + committing timeouts | `test_ha_election.c`, `test_ha_evaluate_committing.c` | 30+ | ✅ | -| 6.2 FSM gate + vote denial | `test_ha_vote_gate.c`, `test_ha_role.c` | 15+ | ✅ | -| 6.3 3-node startup integration | `test_ha_rdhup_integration.c` (hook-based) | 6 | ✅ | -| 6.4 RDHUP failover integration | `test_ha_rdhup_integration.c` | 6 | ✅ | -| 6.5 Regression + ASAN | Full `meson test --suite controller` | 230+ | ✅ | - -## Constraints / Deferred Items - -| Item | Status | Tracking | -|------|--------|----------| -| `gethostbyname` blocking in `endpoint_connect` | Deferred — safe for localhost clusters | `ha-peer-connection-reuse` spec | -| `ha_wait_for_heartbeat_or_timeout` dead code | Still compiled; no callers in production path | Low priority cleanup | -| `am_i_primary` still compiled (legacy) | Not called from `myapp.c`; kept for unit-test compatibility | Can be removed in cleanup pass | - -## Conclusion - -The `election-iomgr-refactor` spec is **fully implemented and verified**. No blocking issues remain. The spec phase is `implemented` and can be closed. The one deferred item (DNS blocking) is tracked under `ha-peer-connection-reuse`. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Status: CLOSED — No remaining gaps - -All 15 implementation tasks (1.1–5.3) and all 5 validation tasks (6.1–6.5) confirmed done. The spec is in `implemented` phase. Deferred cleanup items (dead code: `am_i_primary`, `ha_wait_for_heartbeat_or_timeout`, `start_heartbeat_listener`/`stop_heartbeat_listener` stubs) are cosmetic and tracked as tech debt. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -> Spec phase: `implemented`. This pass audits requirements-vs-CURRENT-codebase drift, -> analyzing the working tree **as it is on disk** — including uncommitted changes to -> `ha.c`, `ha.h`, `myapp.c`, and `src/lib/raft/raft.c` (post `ha-repl-hardening` / -> "finalize configuration replication" work). - -## Analysis Summary - -- **Implementation status**: 8 of 9 requirements **Implemented** in the current tree; Req 9 (no regression) is **Partial** — the working tree has **2 failing controller unit tests** introduced by the uncommitted `ha.c` vote-path changes (verified by building and running `controller_unit_test`). -- **Architecture drift since the spec was written (benign)**: the election core was extracted into a reusable raft library (`src/lib/raft/raft.c`), promotion now uses heartbeat-reply **majority convergence** (`ha_check_majority_convergence`) as the primary mechanism with `ha_repl_ack_received` retained, the committing timeout default moved 10s → 60s, and raft init moved from per-election (`ha_start_election_async`) to one-time startup (`ha_init_raft`, uncommitted). All of these still satisfy the requirement text. -- **Real gaps**: (1) two unit tests contradict the new vote-deny semantics; (2) semantic divergence between the legacy `handle_request_vote` path (ha.c) and `raft_on_vote_request` (raft.c) on duplicate-vote handling; (3) the new `myapp_run` skip-election "self" sentinel does not match production self-vote naming; (4) stale comments + known dead code. -- **Effort to close**: S (≤1 day). **Risk**: Medium (election-safety semantics must be decided deliberately, not just patched to make tests green). - -## Requirement-to-Asset Map - -| Req | Summary | Current Assets (on-disk) | Status | -|-----|---------|--------------------------|--------| -| 1.1–1.3 | No blocking vote pthreads; vote RPC via standard I/O; probe timeout = deny | `ha_start_election_async` (ha.c:3030) delegates to `raft_start_election`; `vote_request_thread`/`am_i_primary` deleted (tombstones ha.c:885–887, ha.h:211); timeout via `raft_cb_schedule_timeout` → `delay_task_ms` | Implemented | -| 2.1–2.4 | Async TCP+TLS via iomgr; TLS client ctx; send vote after handshake; connect failure = deny | `endpoint_set_on_connected` (netio.c:782, fired at netio.c:441–443 with fire-once semantics); `ha->client_ssl_ctx` applied in vote-ep setup (ha.c:160–167); `on_vote_connected` (ha.c:2291); `vote_ep_err_handler` (ha.c:2945+) | Implemented | -| 3.1–3.4 | Reply via iomgr dispatch; no capnp UAF; field extraction; term fencing | `vote_reply_msg_handler` + role-routing `peer_ctrl_msg_handler` (ha.c:2565–2579); fields read before free; term fencing in `raft_on_vote_reply` (raft.c) | Implemented | -| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING | `raft_cb_become_leader` → role store (ha.c:505–507) | Implemented | -| 4.2–4.3 | Freshness gate = ground truth; cold start only highest config_tag wins | `raft_cb_vote_eligible` app-policy gate invoked from cluster.c:331–373 (`raft_on_vote_request` step 5); tag=0 node refuses to campaign (ha.c:3133–3145) | Implemented | -| 4.4–4.5, 4.7 | Split-vote/no-reply retry; retry indefinitely | `raft_on_election_timeout` (raft.c:250), `raft_on_leader_lost` (raft.c:268) with priority-scaled backoff; heartbeat watchdog re-arms (myapp.c skip-election path now arms `ha_arm_heartbeat_timeout_vtask` — uncommitted fix) | Implemented | -| 4.6 | ELECTING + heartbeat from new PRIMARY → STANDBY | Achieved via an **equivalent mechanism**, not literal heartbeat: `ha_dispatch_gate` denies heartbeat/ctrlhello in CTRL_ELECTING; the candidate instead becomes STANDBY via vote-grant → `raft_cb_become_follower` (ha.c:596–659), and the winner's ctrlhello is retried each heartbeat tick once the loser is STANDBY | Implemented (mechanism drift vs. requirement wording — document, don't change) | -| 5.1–5.4 | Dispatch-gate per role | `ha_dispatch_gate` (ha.c:1412): ELECTING allows requestVote+probeConfigTag only; COMMITTING allowlist (ha.c:1436–1454); STANDBY allowlist; PRIMARY full. Note: CTRL_INIT now also admits OPCode_heartbeat (deliberate extension for INIT→STANDBY) | Implemented | -| 5.5 | Monotonic persisted term | `ha_persist_term`/`ha_restore_term` (ha.c:1647/1694); `raft_cb_persist` (ha.c:677) persists before grant; `ha_init_raft` (uncommitted, ha.c:2984) seeds raft from restored term **once** at startup — fixes term reset across re-elections | Implemented | -| 6.1 | PRIMARY/COMMITTING deny all votes | raft.c role gate (step 3, equal-term challenger) + ha.c:1838 legacy gate + term-ordered demote-before-grant (ha.c:1805–1825) | Implemented | -| 6.2 | STANDBY + live primary denies votes | **Drift**: ha.c:1840 (uncommitted) now requires `primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha)`. This matches the requirement text ("live primary **connection**") *more* literally than before, but `test_standby_fresh_heartbeat_denies_vote_request` (test_ha_vote_gate.c) was not updated and **FAILS** (fd defaults to −1 in `ha_init`) | Partial — code OK per requirement, test broken | -| 6.3, 6.5 | HB timeout → ELECTING → new primary | heartbeat watchdog → `ha_on_primary_ctrl_fd_rdhup` (ha.c:3183) → `ha_start_election_async`; verified by `test_ha_rdhup_integration.c` (still registered, meson.build:120) | Implemented | -| 6.4 | INIT/ELECTING evaluate term/voted_for/freshness gates | Production path: `raft_on_vote_request` (raft.c:106+). **Drift**: uncommitted raft.c change adds **idempotent re-grant** to the *same* candidate (raft.c step 4), while uncommitted ha.c change makes the legacy `handle_request_vote` **candidate-agnostic deny** (any second vote in term denied, ha.c:1851). The two vote paths now disagree; `test_vote_granted_to_same_candidate_same_term` (test_ha_election.c:432) **FAILS** | Partial — semantic divergence + failing test | -| 7.1–7.3 | Winner pushes config + WAL; tag=0 standby never fetches | `push_config_to_standbys` + `ha_send_ctrlhello_to_all_standbys` (cluster.c:1376) in COMMITTING entry (ha.c:544–559); heartbeat-driven repl decision loop (HA_REPL_PUSH_CONFIG / WAL / snapshot, ha.c:1288+); `fetch_config_from_primary` removed from production (myapp.c:289) | Implemented | -| 7.4 | Per-standby ack before counting toward majority | Evolved beyond the spec: per-standby `standby_converged[]` tracked from heartbeat `stateReport` replies; plus `OPCode_replCatchUpDone` → `ha_repl_ack_received` (ha.c:2485) | Implemented | -| 8.1–8.3 | COMMITTING blocks jobs; majority ack → PRIMARY | gate case (ha.c:1436); `ha_check_majority_convergence` (ha.c:1244, primary mechanism) and `ha_repl_ack_received` (ha.c:2923, quorum −1 math) → `ha_promote_to_primary` | Implemented | -| 8.4 | Configurable ack timeout → degraded open | `committing_timeout_fn` (ha.c:2875+) with converged/pending WARN detail; `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 60000` (ha.h:131) — design said 10000, requirement only demands "configurable" | Implemented (design-doc drift only) | -| 9.1 | Single-node self-elect, no peer contact | `raft_start_election` n_peers==1 shortcircuit (raft.c:83); ctx freed inline (ha.c:3152–3157); zero-peer guard (ha.c:1981) | Implemented | -| 9.2 | Winner never fetches config | Freshness gate + push model; no fetch call sites remain (stale comment at ha.c:3136 still mentions the removed `fetch_config_from_primary`) | Implemented | -| 9.3 | RDHUP → re-election; promotion is hooks-only | `ha_on_primary_ctrl_fd_rdhup` + `phase2_done` skip; `ha_promote_to_primary` | Implemented | -| 9.4 | Priority stagger | `election_priority_step_ms` stagger in myapp.c:225–231 and watchdog offset ha.c:1597; raft `my_priority` backoff (raft.c:273) | Implemented | -| 9.* | "All existing behaviors preserved" | **2 unit tests fail in the working tree** (see below); everything else green in the run performed for this audit | **Partial** | - -### Verified test failures (working tree, `controller_unit_test`, ASAN build) - -``` -test_vote_granted_to_same_candidate_same_term:FAIL: Expected 1 Was 0 -test_standby_fresh_heartbeat_denies_vote_request:FAIL: Expected 0 Was 1 (Req 6.2) -``` - -Both are caused by the uncommitted `handle_request_vote` changes in ha.c: -1. ha.c:1840 — deny now additionally requires `primary_ctrl_fd >= 0` (test sets fresh heartbeat but fd = −1 from `ha_init`). -2. ha.c:1851 — duplicate-vote deny is now term-scoped and candidate-agnostic (denies same-candidate retransmission), whereas raft.c (the production vote path via cluster.c:373) deliberately keeps the idempotent same-candidate re-grant. - -### Additional drift findings (no test failure yet) - -- **`myapp_run` skip-election "self" sentinel is ineffective in production** (myapp.c, uncommitted): the check `strcmp(voted_for_name, "self") != 0` assumes self-votes are recorded as `"self"`, but `raft_cb_persist` (ha.c:700–713) stores the node's **real candidate name** on a self-vote; `"self"` is only the test-mode/out-of-range placeholder. A restarted node that voted for itself would be classified "leader known" and skip its own election. Mitigated by the newly-armed heartbeat watchdog (it will fire and trigger a fresh election after `heartbeat_timeout_ms`), so the cost is delayed availability, not a stuck node — but the sentinel should also compare against `c->myconf->name`. -- **Stale comments**: ha.c ~826–849 still describe per-election raft re-init ("overwritten by `ha_raft_reinit()` called from `ha_start_election_async`") which the uncommitted `ha_init_raft` refactor removed; ha.c:3136 references the deleted `fetch_config_from_primary`. -- **No test coverage for `ha_init_raft`**: the new one-time-init invariant (raft state must survive across election cycles; double-init would reset term/voted_for) has no dedicated unit test; no testsuite file references `ha_init_raft`. -- **Known dead code still compiled** (carried from prior audits): `ha_wait_for_heartbeat_or_timeout` (ha.c:1587, ha.h:551). - -## Implementation Approach Options - -Since the spec is implemented, options address only the remaining gap closure. - -### Option A: Reconcile in place (extend existing) — Recommended -- Decide the intended duplicate-vote semantics **once** and align all three artifacts: - - Standard Raft (and raft.tla referenced in `src/controller/CLAUDE.md`) permits idempotent re-grant to the same candidate; raft.c already implements this. Recommended: restore the `strcmp(..., candidate_id) != 0` condition in `handle_request_vote` (ha.c:1851) so the legacy/test path matches raft.c, keeping `test_vote_granted_to_same_candidate_same_term` valid. -- Update `test_standby_fresh_heartbeat_denies_vote_request` to set `ha.primary_ctrl_fd = <valid fd>` (the code change matches Req 6.2's "live primary connection" wording and should stay); add a sibling case asserting STANDBY + fresh heartbeat + fd = −1 → **grant** (no live connection). -- Fix the myapp.c sentinel: treat `voted_for_name == c->myconf->name` the same as `"self"`. -- Clean stale comments (ha.c:826–849, ha.c:3136); optionally delete `ha_wait_for_heartbeat_or_timeout`. -- Add a small `ha_init_raft` unit test (init once, run two election cycles, assert term monotonic / voted_for survives). -- Trade-offs: ✅ smallest diff, keeps both vote paths consistent; ❌ keeps the duplicated legacy vote path alive. - -### Option B: Retire the legacy vote path (new/removed components) -- Delete `handle_request_vote` from ha.c entirely; port its tests to drive `raft_on_vote_request` + `raft_cb_vote_eligible` (the actual production gate via cluster.c). Single source of truth for vote semantics. -- Trade-offs: ✅ eliminates the dual-path divergence class permanently; ❌ larger test-port effort (test_ha_election.c, test_ha_vote_gate.c re-plumb), M-size, riskier right before closing the hardening branch. - -### Option C: Hybrid -- Apply Option A now (unblock the test suite and the branch), file the Option B consolidation as a cleanup spec item alongside the existing dead-code list. -- Trade-offs: ✅ immediate green suite + tracked debt; ❌ divergence risk persists until the cleanup lands. - -## Effort and Risk - -| Gap | Effort | Risk | Justification | -|-----|--------|------|---------------| -| Reconcile duplicate-vote semantics (ha.c ↔ raft.c) + fix test | S | Medium | One-line condition either way, but it is election-safety semantics — verify against raft.tla checklist before choosing | -| Update Req 6.2 test for `primary_ctrl_fd` gate (+ new fd=−1 grant case) | S | Low | Code already matches requirement text; test-only change | -| myapp.c "self" sentinel fix | S | Medium | Touches startup skip-election logic; needs a 3-node restart scenario to validate (deploy-ha) | -| `ha_init_raft` unit test | S | Low | Established Unity patterns in test_ha_election.c | -| Stale comments + dead-code removal | S | Low | Cosmetic; callers already verified absent | -| Option B legacy-path retirement (if chosen) | M | Medium | Test re-plumb across 2 suites; no production behavior change intended | - -## Recommendations / Next Steps - -1. **Do not close the branch with the suite red.** Fix the two failing tests via Option A (preferred) before committing the working-tree changes; the current diff would fail `/xce-build controller`. -2. **Preferred approach: Option C** — Option A reconciliation now, Option B (single vote path) queued with the existing dead-code cleanup list. -3. **Keep**: `ha_init_raft` one-time init, the `primary_ctrl_fd >= 0` deny refinement, the term-scoped `voted_for` check in myapp.c, and the skip-election watchdog arming — all are correctness improvements consistent with the requirements. -4. **Requirement-text note**: Req 4.6's "heartbeat received while ELECTING → STANDBY" is satisfied by an equivalent mechanism (vote-grant → follower + primary ctrlhello retry). Record this as accepted drift rather than reworking the dispatch gate. - -## Research Needed - -- **Duplicate-vote semantics decision**: confirm against `raft/raft.tla` (Election Safety invariant) that same-candidate re-grant is safe in XCE's persistence model (vote persisted before grant → re-grant is replay-safe). Expected answer: yes; then align ha.c to raft.c. -- **"self" sentinel scenario**: reproduce with deploy-ha — node self-votes (candidate), is killed mid-election, restarts within the same term; verify it does not skip its own election indefinitely. -- **Full regression + ASAN pass**: run `/xce-build controller` after fixes; this audit ran only `controller_unit_test` (2 failures, no ASAN errors observed in that run). - ---- - -## Gap Re-Audit (2026-06-18) - -> Post-implementation audit. Spec phase: `implemented` (all tasks `[x]`). This pass -> re-verifies the 9 requirements against the CURRENT on-disk codebase, which has -> evolved substantially since the 2026-06-09 pass via four downstream specs that -> landed on this branch: `ha-single-leader-role`, `ha-prevote`, -> `ha-convergence-recovery`, and `ha-commit-durability`. -> Evidence gathered via codegraph + grep (no build run this pass). - -### Headline finding - -The async iomgr election mechanism this spec introduced is **intact and in -production** (Req 1–3, 6, 7, 9). However, the spec's **COMMITTING availability-gate -model (Req 4.1, 5.2, 8.1–8.4) was deliberately superseded** by `ha-single-leader-role`: - -- `CTRL_PRIMARY_COMMITTING` enum value **deleted** (commit 93dfc02). The enum in - `ha.h:60–64` now has only `CTRL_PRIMARY / CTRL_STANDBY / CTRL_INIT / CTRL_ELECTING`. -- The dispatch-gate COMMITTING branch **deleted** (71ce41f); `ha_dispatch_gate` - (ha.c:2061) now has 4 role cases, no COMMITTING (ha.c:2067/2110/2127/2145). -- The force-promotion watchdog + ack/convergence machinery **deleted** (a5bf861): - `ha_repl_ack_received`, `ha_check_majority_convergence`, `committing_timeout_fn` - are all absent from the tree (grep: 0 matches). -- Promotion now goes **direct to CTRL_PRIMARY** via a shared sequence - (`ha_promote_to_primary` → hooks + heartbeat + **own-term WAL_NOOP + writable_lsn - gate**, ha.c:2298). The durability guarantee the COMMITTING gate was meant to - provide is now enforced by the own-term commit gate (`ha_arm_own_term_gate`, - ha.c:2259) and the durable-backed vote freshness floor — both owned by - `ha-commit-durability`. -- Test artifact `test_ha_evaluate_committing.c` (cited in earlier audits) is **gone**. - -This is **accepted, intentional drift**, not a regression: a later approved spec -replaced a weaker availability gate with a stronger durability invariant. The -requirement *intent* (a promoted primary holds current state and never serves stale -data) is **better** satisfied now. The requirement *wording* (specific -`CTRL_PRIMARY_COMMITTING` state + majority-ack counter) is obsolete. - -### Status of the two failures flagged on 2026-06-09 - -Both prior-audit failures appear resolved in the current tree (Option A reconciliation -was applied): -- **`primary_ctrl_fd >= 0` deny gate** is now intentional production code - (`handle_request_vote`, ha.c:320: `role == CTRL_STANDBY && ha->primary_ctrl_fd >= 0 - && ha_heartbeat_is_fresh(ha)`), with an explicit comment block (ha.c:312–319) - justifying it against the RDHUP / liveness-watchdog case. Matches Req 6.2's "live - primary **connection**" wording. -- **`myapp_run` "self" sentinel** now also compares the node's real name - (myapp.c:281–283: `voted_for_name != "" && != "self" && != myconf->name`), - closing the self-vote-skips-own-election hole the prior audit raised. -> Caveat: not re-run this pass (no build). If closing the branch, run -> `/xce-build controller` to confirm `controller_unit_test` is green. - -### Requirement-to-Asset Map (current code, tagged) - -| Req | Summary | Current Asset (file:line) | Tag | -|-----|---------|---------------------------|-----| -| 1.1–1.3 | No blocking vote pthreads; vote via standard I/O; probe timeout = deny | `ha_start_election_async` (ha.c:4400) → `raft_start_election` (raft.c:61); timeout via `raft_election_timeout_fn` (ha.c:342) + `delay_task_ms`; `vote_request_thread`/`am_i_primary_quorum` removed | MET | -| 2.1–2.4 | Async TCP+TLS via iomgr; TLS client ctx; vote after handshake; connect fail = deny | `endpoint_set_on_connected` (netio.c) fire-once; `on_vote_connected` → `xce_request`; `client_ssl_ctx` in `ha_state_t`; `vote_ep_err_handler` (ha.c:3937) counts denied vote | MET | -| 3.1–3.4 | Reply via iomgr dispatch; no capnp UAF; field extraction; term fencing | `peer_ctrl_msg_handler` (ha.c:3600) role-routes; `handle_request_vote_reply` (ha.c:3017) reads fields before free; term fencing in `raft_on_vote_reply` (raft.c:338) | MET | -| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING (pre-availability) | **SUPERSEDED**: quorum → direct `CTRL_PRIMARY` via `raft_cb_become_leader` / `ha_promote_to_primary` (ha.c:2298); COMMITTING state deleted | Constraint (intentional drift — ha-single-leader-role) | -| 4.2–4.3 | Freshness gate = ground truth; cold-start only highest config_tag wins | `raft_cb_vote_eligible` → `ha_vote_self_freshness` (ha.c:2550) per-field max(local, learned-committed); durable-backed floor via `ha_candidate_advertise_lsn`/`persist_durable_floor` (ha.c:2587) [ha-commit-durability 4.2]; config-less node refuses to campaign | MET (strengthened) | -| 4.4–4.5, 4.7 | Split-vote / no-reply retry; retry indefinitely | `raft_on_election_timeout` (raft.c:399), `raft_on_leader_lost` (raft.c:417) priority-scaled backoff; heartbeat watchdog re-arm | MET | -| 4.6 | ELECTING + heartbeat from new PRIMARY → STANDBY | Equivalent mechanism (vote-grant → `raft_cb_become_follower`; winner ctrlhello retried), not literal heartbeat→STANDBY. Documented accepted drift since 2026-06-09 | MET (mechanism drift, documented) | -| 5.1 | ELECTING gate: requestVote/probeConfigTag only, else NOT_READY | `ha_dispatch_gate` CTRL_ELECTING case (ha.c:2127); PreVote opcodes added by ha-prevote | MET | -| 5.2 | COMMITTING: repl acks only | **SUPERSEDED**: no COMMITTING state/gate; promotion is atomic to PRIMARY. Durability via own-term gate instead | Constraint (intentional drift) | -| 5.3–5.4 | PRIMARY full traffic; STANDBY repl/heartbeat | `ha_dispatch_gate` CTRL_PRIMARY (ha.c:2067) / CTRL_STANDBY (ha.c:2145) | MET | -| 5.5 | Monotonic persisted term across restart | `ha_persist_term`/`ha_restore_term`; `raft_cb_persist` persists before grant; `ha_init_raft` (ha.c:3991) seeds raft from restored term **once** at startup (myapp.c:242) | MET | -| 6.1 | PRIMARY denies all votes | raft.c role gate + legacy `handle_request_vote` gate (COMMITTING clause now dead/removed — only PRIMARY remains) | MET | -| 6.2 | STANDBY + live primary denies votes | `handle_request_vote` (ha.c:320): `CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh`. Prior failing test reconciled | MET | -| 6.3, 6.5 | HB timeout → ELECTING → new primary | heartbeat watchdog → `ha_on_primary_ctrl_fd_rdhup` → `ha_start_election_async`; `test_ha_rdhup_integration.c` registered (meson.build:123) | MET | -| 6.4 | INIT/ELECTING evaluate term/voted_for/freshness | `raft_on_vote_request` (raft.c:110); dual-path divergence flagged 2026-06-09 — reconcile/retire still tracked (see residual gaps) | MET (with residual dual-path debt) | -| 7.1–7.3 | Winner pushes config + WAL; tag=0 standby never fetches | `push_config_to_standbys` + `ha_send_ctrlhello_to_all_standbys` (cluster.c:1629); heartbeat-driven repl decision loop; `fetch_config_from_primary` removed | MET | -| 7.4 | Per-standby ack before counting toward majority | Re-homed: durable-commit majority via commit engine (`commit_on_ack`/`commit_advance_locked`, commit.c) + `standby_converged[]`; the spec's `repl_ack_count` counter deleted | MET (re-homed to commit engine) | -| 8.1–8.3 | COMMITTING blocks jobs; majority ack → PRIMARY; PRIMARY opens traffic | **SUPERSEDED**: no COMMITTING. Job traffic gated by `ha_dispatch_gate` PRIMARY + own-term `writable_lsn` gate (commit.c:383) — WAIT completion withheld until `committed_lsn >= writable_lsn`. Stronger than the original ack-count gate | Constraint (intentional drift) | -| 8.4 | Configurable ack timeout → degraded open | **SUPERSEDED**: committing timeout deleted; degraded-mode handled by ha-commit-durability's non-promotable fence + bounded catch-up (commit 59d9df2/addf53e) | Constraint (intentional drift) | -| 9.1 | Single-node self-elect, no peer contact | `raft_start_election` n_peers==1 shortcircuit (raft.c:87) | MET | -| 9.2 | Winner never fetches config | Freshness gate + push model; no fetch call sites | MET | -| 9.3 | RDHUP → re-election; promotion hooks-only | `ha_on_primary_ctrl_fd_rdhup` + `phase2_done` skip; `ha_promote_to_primary` shared sequence | MET | -| 9.4 | Priority stagger | `election_priority_step_ms` stagger (myapp.c); raft `my_priority` backoff (raft.c:421) | MET | - -Tally: 9/9 requirements satisfied in **intent**. 6 of 9 requirement groups fully MET -as written; 3 groups (4.1, 5.2, 8.1–8.4) carry **Constraint** tags — their literal -wording is obsolete because the COMMITTING availability gate was intentionally -replaced by a stronger durability gate in a later approved spec. - -### Residual / deferred gaps - -| # | Gap | Type | Notes | -|---|-----|------|-------| -| R1 | Requirements text still describes `CTRL_PRIMARY_COMMITTING` + majority-ack gate (Req 4.1, 5.2, 8.1–8.4) that no longer exist | Doc drift | requirements.md/design.md are now historically inaccurate vs. code. Spec is closed; this is documentation debt, not a code gap. | -| R2 | Dual vote-path divergence (`handle_request_vote` in ha.c vs `raft_on_vote_request` in raft.c) | Constraint / tech debt | Carried from 2026-06-09. Production vote path is raft.c (via cluster.c); ha.c legacy path retained for tests. Option B (retire legacy path) still open. | -| R3 | `ha_init_raft` one-time-init invariant has no dedicated unit test | Missing (test) | grep: no testsuite file drives `ha_init_raft` across two election cycles. Double-init would reset term/voted_for. | -| R4 | Known dead code still compiled: `ha_wait_for_heartbeat_or_timeout`, `am_i_primary`, heartbeat-listener stubs; stale comments referencing deleted `fetch_config_from_primary` (ha.c) | Constraint (cosmetic) | Carried across every prior audit; no correctness impact. | -| R5 | Build/ASAN not re-run this pass | Unknown | Audit was code-graph only. Confirm `controller_unit_test` green before relying on R1/the 2 reconciled tests. | - -### Open-work options (only R1–R3 warrant action; R4 cosmetic, R5 verification) - -**R1 — reconcile requirements text with superseded gate** -- Option A: Add a "Superseded by ha-single-leader-role / ha-commit-durability" banner - to requirements.md §4/§5/§8 (Effort S, Risk Low). *Recommended.* -- Option B: Rewrite Req 4/5/8 to describe the own-term gate model (Effort M, Risk Low) - — higher fidelity but rewrites an already-closed spec. -- Option C: Leave as-is, rely on this research.md note (Effort 0, Risk Low) — readers - of requirements.md alone stay misled. - -**R2 — single vote-path consolidation** (from 2026-06-09 Option B) -- Option A: Keep dual path, document raft.c as authoritative (Effort S, Risk Low). -- Option B: Delete `handle_request_vote` from ha.c, re-plumb its tests onto - `raft_on_vote_request` (Effort M, Risk Medium — election-safety semantics; verify - against raft.tla Election Safety). -- Option C: Hybrid — A now, B as a tracked cleanup-spec item. *Recommended.* - -**R3 — `ha_init_raft` invariant test** -- Option A: Add a unit test (init once, run two election cycles, assert term monotonic - + voted_for survives) using existing Unity patterns in test_ha_election.c - (Effort S, Risk Low). *Recommended.* - -| Open item | Effort | Risk | -|-----------|--------|------| -| R1 doc banner (Option A) | S | Low | -| R2 vote-path consolidation (Option C) | S now / M later | Medium (semantics) | -| R3 ha_init_raft test | S | Low | -| R4 dead-code/stale-comment cleanup | S | Low | -| R5 build + ASAN verification | S | Low | - -### Conclusion - -`election-iomgr-refactor` is **CLOSED** as an implementation: the async iomgr -election it delivered is live and all 9 requirements are satisfied in intent. The -only true open items are documentation/test debt (R1, R3) and pre-existing tech debt -(R2, R4) — none block the feature. The single material caveat is that the -availability-gate requirements (4.1/5.2/8.x) are now historically inaccurate because -a later approved spec replaced the COMMITTING gate with a stronger durability gate; -this should be recorded in requirements.md (R1) so the spec docs stop describing -removed code. diff --git a/.kiro/specs/election-iomgr-refactor/spec.json b/.kiro/specs/election-iomgr-refactor/spec.json deleted file mode 100644 index af4a68d1..00000000 --- a/.kiro/specs/election-iomgr-refactor/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "election-iomgr-refactor", - "created_at": "2026-05-30T12:00:00Z", - "updated_at": "2026-06-01T00:00:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/election-iomgr-refactor/tasks.md b/.kiro/specs/election-iomgr-refactor/tasks.md deleted file mode 100644 index 2030636b..00000000 --- a/.kiro/specs/election-iomgr-refactor/tasks.md +++ /dev/null @@ -1,189 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: delaymgr ms-precision and netio on_connected callback -- [x] 1.1 Add delay_task_ms() to delaymgr for millisecond-granularity one-shot tasks - - Add `struct timespec nextrun_ts` field to `delay_task_t` (alongside existing `time_t nextrun`) - - Add `delay_task_ms(mgr, delay_ms, runcount, fn, arg)` in delay.c using `clock_gettime(CLOCK_MONOTONIC)` for deadline; precision bounded by `mgr->precision_ms` - - Update `delay_worker` to check both `nextrun` (seconds tasks) and `nextrun_ts` (ms tasks) without breaking existing `delay_task` callers - - Expose `delay_task_ms` in delay.h - - Observable: `make test` passes existing delay tests; new unit test verifies a 500ms task fires within ±200ms - - _Requirements: 1.3, 4.4, 8.4_ - - _Boundary: delay.c, delay.h_ - -- [x] 1.2 Add on_connected callback to endpoint_t in netio - - Add `void (*on_connected)(endpoint_t *ep)` field to `endpoint_t` struct - - Add `endpoint_set_on_connected(endpoint_t *ep, void (*cb)(endpoint_t *))` helper function - - In `io_handler`, after `ssl_handshake_done` is set to 1, invoke `ep->on_connected(ep)` then null the pointer (fire-once semantics) - - Observable: unit test creates outgoing endpoint, sets callback, simulates handshake completion; callback fires exactly once - - _Requirements: 2.1, 2.3_ - - _Boundary: netio.c_ - -- [x] 2. Core: iomgr-native outgoing vote connection and reply handling -- [x] 2.1 Add ha_state_t fields and CTRL_PRIMARY_COMMITTING to FSM enum - - Add to `ha_state_t`: `SSL_CTX *client_ssl_ctx`, `int phase2_done`, `_Atomic int repl_ack_count`, `int n_peers` - - Add to `ha_config_t`: `uint32_t committing_timeout_ms` with default `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000` - - Add `CTRL_PRIMARY_COMMITTING` to `ctrl_role_t` enum in ha.h - - Initialize `phase2_done=0`, `repl_ack_count=0` in `ha_init()` - - Observable: project builds without errors; `ha_init` zero-initializes all new fields - - _Requirements: 5.2, 8.1_ - - _Boundary: ha.h_ - -- [x] 2.2 (P) Implement vote_ep_ctx_t and on_vote_connected callback - - Define `vote_ep_ctx_t` struct: `ha_state_t *ha`, `cluster_t *c`, `ha_elect_ctx_t *ctx`, `int peer_idx`, `uint64_t my_term/my_tag/my_lsn`, `const char *my_name` - - Implement `on_vote_connected(endpoint_t *ep)`: cast `ep->data` to `vote_ep_ctx_t`, build `xce_request_t{opcode=OPCode_requestVote, body.requestvotemsg=...}`, call `xce_request(ep, &req)`, set `ep->msg_handler = vote_reply_msg_handler` - - Define `election_timeout_ctx_t`: `_Atomic int cancelled`, `ha_state_t *ha`, `cluster_t *c`, `ha_elect_ctx_t *ctx` - - Observable: unit test with mock endpoint verifies `xce_request` called with correct opcode and fields after `on_vote_connected` fires - - _Requirements: 1.2, 2.3, 3.1_ - - _Boundary: ha.c (vote send path)_ - - _Depends: 1.2, 2.1_ - -- [x] 2.3 (P) Implement vote_reply_msg_handler for iomgr-native reply decoding - - Implement `vote_reply_msg_handler(endpoint_t *ep)`: call `xce_decode(pkt, &req)`, verify `req.opcode == OPCode_reply`, extract `rep->data.requestvotereplymsg`, call `read_RequestVoteReply(&vr_raw, ...)` BEFORE any capnp_free, call `handle_request_vote_reply(ha, ctx, peer_idx, vr_raw.term, vr_raw.voteGranted, ...)` - - Term fencing: if `vr_raw.term > ha->term`, atomically update `ha->term` before recording vote - - Observable: ASAN-clean unit test reads reply fields before free; no heap-use-after-free - - _Requirements: 3.1, 3.2, 3.3, 3.4_ - - _Boundary: ha.c (vote reply path)_ - - _Depends: 2.1_ - -- [x] 2.4 Implement ha_start_election_async() — non-blocking election initiator - - Implement `ha_start_election_async(ha, c, n_peers, peers, my_config_tag, my_lsn, my_name)`: - - Store `n_peers` in `ha->n_peers` - - For each peer: allocate `vote_ep_ctx_t`, init endpoint with `client_ssl_ctx`, set `on_vote_connected` + `vote_ep_err_handler`, call `endpoint_connect` + `iomgr_monitor`; skip peer on connect failure (counted as deny via error handler) - - Allocate `election_timeout_ctx_t` with `cancelled=0`; schedule `delay_task_ms(delaymgr, election_timeout_ms, 1, election_timeout_fn, ctx)` - - Single-node shortcircuit: if n_peers==1, call `ha_elect_evaluate_and_signal` directly with PRIMARY result - - Returns immediately; election result delivered via `ha_elect_evaluate_and_signal` - - Observable: function returns in < 5ms; endpoints registered in iomgr; watchdog scheduled in delaymgr - - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.4, 4.3_ - - _Boundary: ha.c (election orchestration)_ - - _Depends: 1.1, 1.2, 2.2, 2.3_ - -- [x] 3. Core: FSM gate extensions and role-based vote denial -- [x] 3.1 (P) Extend ha_dispatch_gate for CTRL_PRIMARY_COMMITTING state - - Add `case CTRL_PRIMARY_COMMITTING:` to `ha_dispatch_gate` in ha.c: allow `OPCode_heartbeat`, `OPCode_replConfig`, `OPCode_replConfigPush`, `OPCode_upgradeConfigAck`, `OPCode_hello`, `OPCode_walAck`, `OPCode_replEntry`, `OPCode_replAck`, `OPCode_replCatchUpDone`; deny all others with `-XCE_ERR_NOT_READY` - - Observable: unit test sends `OPCode_jobsubmit` in COMMITTING state → returns `XCE_ERR_NOT_READY`; `OPCode_replCatchUpDone` → returns 0 - - _Requirements: 5.1, 5.2, 8.1_ - - _Boundary: ha.c (dispatch gate)_ - - _Depends: 2.1_ - -- [x] 3.2 (P) Extend handle_request_vote with role-based deny gate - - Before existing term/voted_for/freshness checks in `handle_request_vote`: load `ha->role`; return 0 (deny) if `CTRL_PRIMARY` or `CTRL_PRIMARY_COMMITTING`; return 0 (deny) if `CTRL_STANDBY && ha_heartbeat_is_fresh(ha)` - - Implement `ha_heartbeat_is_fresh(ha)`: compare `clock_gettime(CLOCK_MONOTONIC)` against `ha->last_heartbeat`; return 1 if elapsed < `heartbeat_timeout_ms` - - Observable: unit test with mocked roles: PRIMARY→deny, COMMITTING→deny, STANDBY+fresh→deny, STANDBY+stale→evaluate, ELECTING→evaluate - - _Requirements: 6.1, 6.2, 6.4_ - - _Boundary: ha.c (vote gate)_ - - _Depends: 2.1_ - -- [x] 4. Core: committing phase — replication push and availability gate -- [x] 4.1 Extend ha_elect_evaluate_and_signal with COMMITTING path - - Extend `ha_elect_evaluate_and_signal`: after determining role=PRIMARY, check `!ha->phase2_done`; if first boot call `cluster_init_phase2(c)` then set `ha->phase2_done=1`; set `ha->role = CTRL_PRIMARY_COMMITTING`; reset `repl_ack_count=0`; schedule committing timeout watchdog via `delay_task_ms` - - On STANDBY result: if first boot call `cluster_init_phase2(c)` + set `phase2_done=1`; set role=CTRL_STANDBY; election timeout callback sets `cancelled=1` - - Observable: unit test verifies `cluster_init_phase2` called exactly once on first election; second election (RDHUP) skips it; role enters COMMITTING not PRIMARY after winning - - _Requirements: 4.1, 5.2, 9.2_ - - _Boundary: ha.c (election evaluation)_ - - _Depends: 2.4, 3.1_ - -- [x] 4.2 Implement config and WAL replication push to standbys - - In `ha_elect_evaluate_and_signal` COMMITTING path: call `push_config_to_standbys(c, ...)` to push config to standbys with lower `config_tag` - - Implement `ha_send_ctrlhello_to_all_standbys(c)`: iterate `ha->standby_endpoints`, send `CtrlHelloRequest` to each via `xce_request` — triggers existing catch-up replication in `handle_ctrlhello` - - Observable: 3-node startup log shows primary pushing config to ctrl-b/c immediately after COMMITTING; ctrl-b/c logs show config received - - _Requirements: 7.1, 7.2, 7.3_ - - _Boundary: ha.c + cluster.c (replication push)_ - - _Depends: 4.1_ - -- [x] 4.3 Implement ha_repl_ack_received and majority-ack → PRIMARY transition - - Implement `ha_repl_ack_received(ha, c)`: atomically increment `repl_ack_count`; compute quorum `= (ha->n_peers + 2) / 2 - 1`; if `repl_ack_count >= quorum`: set committing timeout `cancelled=1`, call `ha_promote_to_primary(c)` (sets role=CTRL_PRIMARY, enables hooks) - - Wire `ha_repl_ack_received` into `OPCode_replCatchUpDone` handler in cluster.c - - Observable: unit test: 2 of 3 acks arrive → PRIMARY; 1 of 3 acks → stays COMMITTING - - _Requirements: 7.4, 8.2, 8.3_ - - _Boundary: ha.c + cluster.c (ack counting)_ - - _Depends: 4.1_ - -- [x] 4.4 Implement committing timeout watchdog (degraded-mode fallback) - - Define `committing_timeout_ctx_t`: `_Atomic int cancelled`, `ha_state_t *ha`, `cluster_t *c` - - Implement `committing_timeout_fn`: if `!cancelled` and `ha->role == CTRL_PRIMARY_COMMITTING`, log WARN "committing timeout — opening in degraded mode", call `ha_promote_to_primary(c)` - - Schedule via `delay_task_ms(delaymgr, committing_timeout_ms, 1, committing_timeout_fn, ctx)` in the COMMITTING entry path - - Observable: unit test with no standbys: after `committing_timeout_ms`, role transitions to CTRL_PRIMARY with WARN log - - _Requirements: 8.4, 9.1_ - - _Boundary: ha.c (committing watchdog)_ - - _Depends: 1.1, 4.1_ - -- [x] 5. Integration: restructure myapp_run and wire all components -- [x] 5.1 Restructure myapp_run to be fully non-blocking - - Move `delaymgr_init` + `delaymgr_start` and `persist_init` + `persist_start` from `cluster_start` path into `myapp_run` (before `ha_start_election_async`) - - Create `client_ssl_ctx` (TLS_client_method + SSL_VERIFY_NONE) in `myapp_run`; store in `c.ha.client_ssl_ctx` - - Call `load_primary_config` to pre-load config_tag for freshness gate - - Remove: `ha_wait_for_heartbeat_or_timeout`, `am_i_primary_quorum`, `for(;;)` retry loop, `cluster_init_phase2`, `cluster_start`, `bootstrap_config_if_needed` calls - - Call `ha_start_election_async` then `cluster_run` directly — no blocking between them - - Observable: `myapp_run` exits into `cluster_run` within 10ms of startup; no blocking calls visible in strace/callgraph - - _Requirements: 1.1, 9.3, 9.4_ - - _Boundary: myapp.c_ - - _Depends: 2.4, 3.1, 3.2, 4.1, 4.2, 4.3, 4.4_ - -- [x] 5.2 Wire heartbeat-timeout → CTRL_ELECTING → ha_start_election_async for RDHUP failover - - In `ha_on_primary_ctrl_fd_rdhup`: after setting role=CTRL_ELECTING, call `ha_start_election_async` with the cluster's peer list (instead of the current `am_i_primary_quorum` call) - - Verify `ha->phase2_done==1` ensures RDHUP re-election skips `cluster_init_phase2` and goes directly to COMMITTING - - Observable: kill primary with SIGKILL; surviving standby transitions ELECTING→COMMITTING→PRIMARY within `heartbeat_timeout_ms + election_timeout_ms + committing_timeout_ms` - - _Requirements: 6.3, 6.5, 9.3_ - - _Boundary: ha.c (RDHUP path)_ - - _Depends: 5.1_ - -- [x] 5.3 Remove bootstrap_config_if_needed and old election blocking path - - Delete `bootstrap_config_if_needed` call from myapp_run; delete the function body from config_repl.c if no other callers remain - - Delete `ha_wait_for_heartbeat_or_timeout` call from myapp_run (now internal to ha_start_election_async's watchdog) - - Delete `am_i_primary_quorum` blocking pthread loop from ha.c (keep unit-test stub in testsuite) - - Delete `vote_request_thread` and `vote_thread_ctx_t` from ha.c - - Observable: `grep -r "bootstrap_config_if_needed\|vote_request_thread\|vote_thread_ctx_t" src/controller/ --include="*.c"` returns no matches outside of test stubs - - _Requirements: 1.1, 9.2_ - - _Boundary: ha.c, myapp.c, config_repl.c_ - - _Depends: 5.1, 5.2_ - -- [x] 6. Validation: tests, ASAN, and integration verification -- [x] 6.1 Unit tests for election timeout and committing timeout watchdogs - - Test `election_timeout_fn`: fires `ha_elect_evaluate_and_signal` when not cancelled; no-op when `cancelled=1` - - Test `committing_timeout_fn`: promotes to PRIMARY (degraded) when not cancelled; no-op when cancelled - - Test `delay_task_ms` precision: 500ms task fires within ±200ms (delaymgr precision_ms = 50ms for test) - - Observable: all tests pass; no ASAN errors - - _Requirements: 1.3, 4.3, 8.4_ - - _Boundary: ha.c unit tests_ - - _Depends: 1.1, 4.4_ - -- [x] 6.2 Unit tests for FSM gate and role-based vote denial - - Test `ha_dispatch_gate`: COMMITTING allows replCatchUpDone, denies jobsubmit; ELECTING allows requestVote, denies jobsubmit - - Test `handle_request_vote` role gate: all 5 cases (PRIMARY, COMMITTING, STANDBY+fresh, STANDBY+stale, ELECTING) - - Test `ha_repl_ack_received`: quorum=1 of 2 peers acks → PRIMARY; 0 of 2 → stays COMMITTING - - Observable: 15+ new unit test cases pass; `make test` green - - _Requirements: 5.1, 5.2, 6.1, 6.2, 6.4, 8.2, 8.3_ - - _Boundary: ha.c unit tests, cluster.c unit tests_ - - _Depends: 3.1, 3.2, 4.3_ - -- [x] 6.3 Integration test: 3-node startup with full COMMITTING phase - - Start ctrl-a, ctrl-b, ctrl-c within 3 seconds; verify ctrl-a wins election and enters COMMITTING - - Verify ctrl-b/c receive config push (log: "config received from primary"); both send replCatchUpDone - - Verify ctrl-a transitions to CTRL_PRIMARY after 2 acks; verify job dispatch accepted after gate opens - - Verify ctrl-b/c deny vote requests while ctrl-a is COMMITTING or PRIMARY (role gate active) - - Observable: cluster serves a job within 30s of all nodes starting; no XCE_ERR_NOT_READY from primary after COMMITTING exits - - _Requirements: 4.1, 7.1, 7.2, 7.3, 8.1, 8.2, 8.3_ - - _Depends: 5.1, 5.2, 5.3_ - -- [x] 6.4 Integration test: primary crash and standby promotion (RDHUP failover) - - Start 3-node cluster; let ctrl-a become PRIMARY; SIGKILL ctrl-a - - Verify ctrl-b/c detect heartbeat timeout within `heartbeat_timeout_ms`; one wins election → COMMITTING → PRIMARY - - Verify promoted primary serves job traffic from last WAL state (no data loss) - - Verify ASAN-clean run: no heap-use-after-free, no stack-use-after-return - - Observable: cluster operational again within `heartbeat_timeout_ms + election_timeout_ms + committing_timeout_ms + 5s`; ASAN output shows 0 errors - - _Requirements: 6.3, 6.5, 9.3_ - - _Depends: 6.3_ - -- [x] 6.5 Regression: existing unit test suite green; ASAN clean - - Run full `meson test --suite controller` with `ASAN_OPTIONS=detect_leaks=0`; all 230+ existing tests pass - - Verify single-node self-election still works without any network I/O - - Verify no compilation warnings introduced in modified files - - Observable: `meson test --suite controller` exits 0; test count ≥ prior baseline; no new ASAN errors - - _Requirements: 9.1, 9.4_ - - _Depends: 6.4_ - -## Implementation Notes - -- `ha_elect_evaluate_and_signal` is called from vtask context (via vote_reply_msg_handler) and from delaymgr context (via election_timeout_fn). Both are thread-safe because `ha_elect_ctx_t` operations are mutex-protected and `ha->role` is atomic. -- `cluster_init_phase2` must be verified vtask-safe before task 4.1: check that all subsystems it touches (configmgr, jobmgr, etc.) were moved to phase1 in the prior refactor. -- The committing timeout and election timeout contexts must outlive the callbacks. Allocate them on heap inside `ha_start_election_async`; free in `ha_election_cleanup` called after COMMITTING exits. diff --git a/.kiro/specs/eliminate-frontend-makefile/brief.md b/.kiro/specs/eliminate-frontend-makefile/brief.md deleted file mode 100644 index f44d12d8..00000000 --- a/.kiro/specs/eliminate-frontend-makefile/brief.md +++ /dev/null @@ -1,58 +0,0 @@ -# Brief: eliminate-frontend-makefile - -## Problem -`src/frontend/Makefile` and Meson's `tools/go_build` wrapper are two independent build systems for the same two Go binaries (apiserver, xcecli), each computing its own copy of the same version-fingerprint logic (`commit_id`/`build_time`/`dirty_fingerprint`). This is an already-realized, not theoretical, drift risk: the Makefile's independent fingerprint computation needed three adversarial review rounds to fix a shell exit-status bug during `dirty-content-fingerprint` task 4.3 (`roadmap.md:82`), and `build-server` — the Makefile's OWN apiserver build target — is independently confirmed *currently broken* (bug-1597, `.wolf/cerebrum.md:48`): a bare `go build` with no `CGO_CFLAGS`/`CGO_LDFLAGS`, failing outright with `fatal error: client.h: No such file or directory`. `src/docs/deployment.md:243-247` already documents this exact failure as a "build gotcha" and tells developers to use `/xce-build apiserver` instead — i.e., the project's own docs already steer people away from the target that's supposed to be the Makefile's main value proposition. - -## Current State -- `src/frontend/Makefile` (195 lines) has 5 targets: `build-cli`, `build-server`, `test`, `test-backend`, `test-integration` (+ `test-all`). -- `build-cli`/`build-server` are each 2 lines of actual build logic; the surrounding ~130 lines exist entirely to independently re-derive version-fingerprint `-ldflags`, duplicating logic `src/tools/xce-build.sh` already owns. -- Meson already builds both binaries today via `tools/go_build` (`apiserver/meson.build`'s `custom_target`, `xcecli/meson.build`'s equivalent) — confirmed working, lands in the per-profile build dir (e.g. `build.dbg/frontend/apiserver/xce_apiserver`). -- `/xce-build apiserver`/`/xce-build all` (`xce-build.sh`) already build via meson exclusively (`meson compile ... apiserver xcecli`, line 237) — the Makefile's `build-cli`/`build-server` are not in that path at all today. -- `xce-build.sh` DOES still shell out to `cd frontend && make test` for running Go unit tests (lines 240, 287) — this is the one live automation dependency on the Makefile from the project's own canonical build tool. -- No meson `test()` exists anywhere for the CGO-backed backend test suite (`apiserver/meson.build` has a build-only `custom_target`, zero test registrations) — `make test-backend` is the ONLY way to run these tests today. -- One other live script (`run-resource-verify.sh:350`) checks for a binary at the Makefile's hardcoded `/tmp/apiserver` output path and prints a `make build-server`-referencing error if absent. -- `deploy-ha.sh` used to hard-depend on the same `/tmp/apiserver` path; this was just fixed (this session, uncommitted) to prefer the meson build-dir path first, matching the pattern already used for xceadmin/worker/scheduler, falling back to `/tmp/apiserver` only if the meson one is absent. - -## Desired Outcome -`src/frontend/Makefile` is deleted. All Go builds for apiserver/xcecli happen exclusively via Meson (`/xce-build apiserver`/`/xce-build xcecli`/`/xce-build all`). Go test execution (unit + CGO backend + integration) is either wired into meson as real `test()` targets, or reduced to plain, documented `go test`/shell invocations with no Makefile indirection — matching the pattern `xce-build.sh`'s own `xcecli` case (line 249) already uses today for xcecli's tests, bypassing Make entirely. No developer or script depends on a Makefile-specific fixed output path (`/tmp/apiserver`, `/tmp/xce_static`) after this lands. - -## Approach -**Decision confirmed with user (2026-08-27): keep a single build system — Meson survives, `src/frontend/Makefile` is removed entirely, not kept as a thin/compat wrapper.** No alternative approach was considered or offered; this was a direct instruction, not a selection among options. - -**Phased replacement, test-coverage-preserving order** (see Boundary Candidates below for the split). Not a single mechanical deletion — the CGO backend test suite has no meson equivalent today, so that gap must be closed BEFORE the Makefile can be safely removed, or backend CGO regression coverage silently disappears. - -Two sub-approaches were considered for the CGO backend test gap specifically: -1. **Add a meson `test()` wired with the same CGO env `apiserver/meson.build`'s `custom_target` already derives** (cgo_cflags/cgo_ldflags via the same `apiserver_cgo_deps` pkg-config loop, lines 20-38) — single source of truth, no duplicate derivation. **Recommended.** -2. Keep a minimal, build-free shell script (not a Makefile) that just runs `go test ./backend/...` with hand-derived CGO flags — avoids touching `apiserver/meson.build`, but reintroduces exactly the duplicate-derivation risk this whole effort exists to eliminate. - -## Scope -- **In**: deleting `src/frontend/Makefile`; migrating `xce-build.sh`'s 2 `make test` call sites to direct `go test` invocations (mirroring the existing `xcecli` case); adding a meson `test()` (or equivalent) for the CGO backend suite so that coverage isn't lost; migrating `test-integration`'s `XCE_BIN`/`SRV_URL` convention to point at the meson-built binary; updating the ~9 files with live-relevant doc references (`tech.md`, `build-deployment.md`, `testing.md`, `structure.md`, `frontend/CLAUDE.md`, `src/docs/deployment.md`, `xce-deploy-ha/SKILL.md`); updating `run-resource-verify.sh`'s apiserver-binary discovery to the same dual-path pattern `deploy-ha.sh` now uses. -- **Out**: rewriting historical spec docs (`tasks.md`/`research.md`/`design.md` under already-completed specs like `ha-config-bootstrap`, `component-failover-adoption`, etc.) that mention `make test`/`make build-server` as acceptance criteria for work already shipped — those are frozen records, not live automation, and editing them provides no value. Any change to the underlying version-fingerprint computation itself (`xce-build.sh`'s own logic) — this effort only removes the SECOND, duplicate computation site, not the first. - -## Boundary Candidates -- **Seam 1 — Build targets** (`build-cli`/`build-server`): already effectively dead from `/xce-build`'s perspective; lowest risk, smallest diff. Can land independently of the other seams. -- **Seam 2 — Unit test target** (`test`): 2 call sites in `xce-build.sh`, already has a proven direct-`go test` replacement pattern in the same file. Low risk. -- **Seam 3 — CGO backend test target** (`test-backend`): needs a genuinely new meson `test()` registration (no existing equivalent) before the Makefile target can be removed without losing coverage. This is the one seam that's real net-new work, not just deletion — highest-effort seam. -- **Seam 4 — Integration test target** (`test-integration`): depends on Seam 1 (needs a `build-cli` replacement first) plus its own `XCE_BIN` path update. -- **Seam 5 — Doc/script updates**: mechanical once seams 1-4 land; the actual reference list is already enumerated (see this investigation's own findings). - -Given all 5 seams touch the same small `src/frontend/` + `src/tools/xce-build.sh` area, share one dependency graph (Seam 4 depends on Seam 1; nothing else cross-depends), and total scope is well under 20 tasks, this is a **single spec**, not a multi-spec decomposition — Path C. - -## Out of Boundary -- Any change to `apiserver/meson.build`'s existing CGO flag derivation logic itself (only *reusing* it for the new test() target, not modifying it). -- Any change to `xce-build.sh`'s version-fingerprint computation (the "source of truth" side of the duplication, not the Makefile side being removed). -- Rewriting frozen historical spec documents. -- The `set -euo pipefail`-causes-CGO-tests-to-be-structurally-skipped-in-`all` gap noted in `component-failover-adoption/tasks.md:201` — real, adjacent, but a separate `xce-build.sh` control-flow bug, not something this removal effort should silently fix as a side effect. - -## Upstream / Downstream -- **Upstream**: `apiserver/meson.build`'s existing CGO dependency list (`apiserver_cgo_deps`) is the thing Seam 3's new test() target must reuse, not re-derive. -- **Downstream**: `deploy-ha.sh` (already migrated this session), `run-resource-verify.sh` (needs the same dual-path treatment), any future skill/script that currently assumes `/tmp/apiserver`/`/tmp/xce_static` as a fixed discovery path. - -## Existing Spec Touchpoints -- **Extends**: none directly — this is genuinely new spec-shaped work, not an extension. -- **Adjacent**: `dirty-content-fingerprint` (the spec whose task 4.3 review first surfaced the duplication risk this effort resolves — provenance only, not reopened); `build-time-versioning` (owns the version-fingerprint computation `xce-build.sh` is the single source of truth for, post this change). - -## Constraints -- Must not reduce CGO backend test coverage at any point during the migration (Seam 3 must land before Seam 3's Makefile target is removed, not after). -- Nix devShell already bundles the full C/Go/Python toolchain uniformly — there is no "avoid installing the C toolchain" argument for keeping a Make-only path, unlike projects where Go-only contributors might want to skip C dependencies. -- `go test -race` is confirmed unusable in this ASAN-linked CGO environment (independently hit 3 times per `component-failover-adoption/tasks.md:197`) — any new test() wiring must not attempt to add `-race`. diff --git a/.kiro/specs/eliminate-frontend-makefile/spec.json b/.kiro/specs/eliminate-frontend-makefile/spec.json deleted file mode 100644 index c9d50be4..00000000 --- a/.kiro/specs/eliminate-frontend-makefile/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "eliminate-frontend-makefile", - "created_at": "2026-08-27T00:00:00Z", - "updated_at": "2026-08-27T00:00:00Z", - "language": "en", - "phase": "brief", - "description": "Remove src/frontend/Makefile entirely, consolidating Go builds (apiserver, xcecli) and their tests onto Meson's tools/go_build wrapper. Closes a real, already-realized duplicate version-fingerprint-computation risk (roadmap.md backlog item, recorded 2026-08-21). Path C single-scope investigation via /kiro-discovery, 2026-08-27.", - "approvals": { - "requirements": { - "generated": false, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "blocked_on": [] -} diff --git a/.kiro/specs/ha-commit-barrier-rollout/brief.md b/.kiro/specs/ha-commit-barrier-rollout/brief.md deleted file mode 100644 index 51b9ca68..00000000 --- a/.kiro/specs/ha-commit-barrier-rollout/brief.md +++ /dev/null @@ -1,24 +0,0 @@ -# Brief: ha-commit-barrier-rollout - -> Sub-spec 2 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> Authoritative requirements/design/research: `.kiro/specs/ha-consistency-invariants/` (design.md §Write-Path Classification + §Config Barrier + §Idempotency Guards). This brief scopes the slice; do not re-derive decisions. - -## Problem -Only WAL_JOB_SUBMIT goes through a commit barrier. Job state changes, queue control, and worker control apply locally with fire-and-forget replication; job/queue control replies before any durability; config push ignores its majority-ACK result; worker-reported facts are silently lost on rejection (bug-396); `mark_job_*` have no idempotency guards. - -## Desired Outcome -Every state-changing WAL type routed per the design's classification table: request-shaped → WAIT mode with **commit-first-then-reply** (validate → mutate provisional → append → commit_submit → reply); fact-shaped → RETAIN mode (log-is-the-queue, appended + retried until committed, no timeout, never dropped while running+leading). Config upgrade fails to the requester on missed majority. `mark_job_*` transitions are idempotent and ordered. - -## Scope -- **In**: classification table + dispatch in `persist.c` (debug assert on unclassified types); reply-ordering sweep of all control handlers in `job.c` (incl. `do_jobcontrol_op`), queue and worker join/update paths; `handle_jobdispatch` classification (per design Reply-Ordering Contract); RETAIN routing for fact WAL types; `push_config_to_standbys` error propagation + caller failure; `mark_job_*` state-machine guards; integration tests (no success reply observable before commit; fact retention across quorum loss). -- **Out**: the commit engine itself (ha-commit-engine); promotion/COMMITTING (ha-single-leader-role); egress filter and demotion (ha-convergence-recovery); worker protocol changes (worker-rejoin-resync backlog). - -## Requirements (umbrella IDs) -2.1 (all operation types), 2.2, 2.8, 5.1, 5.2, 5.3. - -## Upstream / Downstream -- **Upstream**: ha-commit-engine (WAIT/RETAIN API). -- **Downstream**: ha-single-leader-role (must land after this — barrier before state removal); worker-rejoin-resync (shares idempotency guards). - -## Constraints -Commit-first-then-reply is non-negotiable for request-shaped ops; a missed handler silently violates umbrella Req 2.1 — the sweep must be exhaustive (grep all xce_reply sites in control paths). Crash-loss of uncommitted facts accepted until worker-rejoin-resync lands. diff --git a/.kiro/specs/ha-commit-barrier-rollout/design.md b/.kiro/specs/ha-commit-barrier-rollout/design.md deleted file mode 100644 index aa3e7e26..00000000 --- a/.kiro/specs/ha-commit-barrier-rollout/design.md +++ /dev/null @@ -1,457 +0,0 @@ -# Design: ha-commit-barrier-rollout - -> Sub-spec 2 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **Write-Path Classification**, **Config Barrier**, and **Idempotency Guards** components (`.kiro/specs/ha-consistency-invariants/design.md`), implementing umbrella Migration Strategy phase 2. The umbrella's WAL-type classification table and Reply-Ordering Contract are the contract; this design records their call-site verification (which the umbrella explicitly mandated) and the enforcement mechanism. The commit engine interface (`include/commit.h`, `commit_submit` WAIT/RETAIN) is consumed exactly as shipped by ha-commit-engine — this spec never modifies `commit.c` internals. Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping). - -## Overview - -**Purpose**: This feature extends the majority-commit barrier from job submission to every cluster-state-changing operation: request-shaped operations reply to their requester only after commit (and fail on timeout), fact-shaped operations are appended and retained until committed with no timeout, configuration upgrades fail honestly to the admin tool, and job state transitions become idempotent so retained-fact retry and replay are safe. - -**Users**: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed. - -**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:2198-2224), `do_queuecontrol_op` (queue.c:128-154), `do_hostcontrol_op` (worker.c:796-822), and `handle_worker_join` (join.c:48-149) to commit-first-then-reply [line numbers re-verified against current source — task-evidence line numbers had drifted]; admits fact-bearing opcodes through the dispatch gate during quorum loss (`ha_dispatch_gate`, ha.c:4525, allowlist at ha.c:4644-4648); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:1609-1654) to the `upgradeConfig` requester (`do_upgrade_config_with_bytes`, config_repl.c:1696-1855 — Note: as of ha-term-safety, the requester's commit gate is `config_commit_change`'s WAL_CONFIG_MARKER barrier, which internally calls `push_config_to_standbys`; see Config Barrier section below); adds state-machine guards to `mark_job_*` (job.c: `mark_job_fail` 1830, `mark_job_update` 1929 — Note: `mark_job_start`/`mark_job_finish` were renamed and restructured by raft-apply-decoupling, see Idempotency Guards section below); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:589-700) while hardening its configAck branch. - -### Goals -- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:1412 onward) is the reference implementation (Note: superseded by raft-apply-decoupling — `handle_jobsubmit` no longer "stays unchanged"; it was rearchitected to append via an async commit continuation (`jobsubmit_commit_continuation`/`jobsubmit_apply_complete`) rather than a synchronous append-then-reply, and job-array-single-wal-record added a sibling array-submit path. The commit-first-then-reply *outcome* for the requester is preserved, but the reference shape this design points to has changed; see those specs' design.md) -- Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by `lsn > committed_lsn`, never dropped while running+leading, accepted even during quorum loss -- An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback) -- Config upgrade fails to the requester on missed majority and does not restart into the failed config -- `mark_job_*` idempotent and ordered (duplicate = no-op; terminal→non-terminal rejected) - -### Non-Goals -- Commit engine internals: per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine — consumed as-is) -- Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual) -- Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery) -- Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then) -- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands — confirmed landed: `wal_classify` (persist.c:5796) now classifies WAL_NOOP as RETAIN exactly as pre-declared below) - -## Boundary Commitments - -### This Spec Owns -- The WAL-type → commit-mode classification table, its dispatch at the persistence hook seam, and the unclassified-type guard -- The reply ordering of every request-shaped control handler on the primary (job control, queue control, worker control, worker join/update) and the receipt-ACK verdict for scheduler-origin handlers -- The commit-barrier helper (`persist_commit_barrier`) that handlers use to realize WAIT-mode completion -- Fact admission through the dispatch gate's no-write-quorum path (opcode allowlist only — not quorum accounting) -- Config-change barrier error propagation: `push_config_to_standbys` return contract, configAck majority arithmetic, and the `upgradeConfig` caller's failure path -- `mark_job_*` transition guards -- Retirement of the legacy (engine-less) walAck counting in `repl_wait_majority_ack` - -### Out of Boundary -- `commit.c` / `include/commit.h` — consumed, never modified (interface changes there trigger revalidation here) -- `ha_has_write_quorum` and quorum accounting (ha-commit-engine); the gate's role-based branches other than the PRIMARY no-quorum opcode list -- `committed_lsn` advancement rules, admission cap, membership re-evaluation (commit engine) -- Egress filtering of provisional state to the scheduler (ha-convergence-recovery; dispatch of uncommitted submits is closed there) -- Standby-side apply, WAL wire format, catch-up streaming mechanics (ha-wal-replication) -- Worker/scheduler retry behavior on error replies (client-side, existing semantics) - -### Allowed Dependencies -- `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` and the engine instance embedded in `cluster_t` (ha-commit-engine, P0) — the only commit-evaluation authority -- `persist_wal_async` / `persist_wal_sync_append` append+broadcast primitives and `persist_t.current_lsn` / `committed_lsn` atomics (persist.c, P0) -- `repl_on_config_ack` / `acked_config_lsn` per-standby config tracking (repl.h, P0) — read-only consumption for the config barrier -- Hook dispatch (`xce_run_hooks`, HOOK_AFTER_EVENT seam) — existing pattern, unchanged signatures -- Threading rules per `ha-async-io.md`: commit-barrier waits only on vtask threads; nothing here runs on iomgr -- Dependency direction: handlers (job.c/queue.c/worker.c/join.c) → persist.c barrier/classification → commit engine; persist.c never calls back into handlers - -### Revalidation Triggers -- Any change to `commit.h` signatures or return codes (upstream) → re-verify barrier helper and handler error mapping -- ha-single-leader-role adding WAL_NOOP or removing CTRL_PRIMARY_COMMITTING → classification table gains the WAL_NOOP row (RETAIN, umbrella-fixed); gate COMMITTING branch disappears -- Any new state-changing opcode/WAL type → must be added to the classification table (the debug assert makes this unmissable) and its handler swept for reply ordering -- worker-rejoin-resync landing → re-verify `mark_job_*` guards against its replay ordering -- Changes to `acked_config_lsn` semantics or the upgradeConfigAck payload → re-verify configAck majority arithmetic - -## Architecture - -### Existing Architecture Analysis -- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:6509/6589/6638 — re-verified, was 1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:5769-5789, was 1068-1088) and append. WAL_JOB_SUBMIT is now committed via an explicit call from `handle_jobsubmit` (`persist_submit_job_payloads_and_commit`) strictly before any job object is constructed, never from this hook — the hook's WAL_MODE_WAIT branch is now a pure post-commit no-op (Note: superseded by raft-apply-decoupling task 3.1; the "in-hook sync barrier" description below and elsewhere in this doc no longer matches — see that spec's design.md). Everything else still routes through `wal_hook_append_classified` (append + RETAIN registration), functionally equivalent to the `persist_wal_async` fire-and-forget this doc describes -- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:2198), `do_queuecontrol_op` (queue.c:128), `do_hostcontrol_op` (worker.c:796), `handle_worker_join` (join.c:48) — all violate umbrella 2.1 today [historical framing — this spec fixes it; see Reply-Ordering Sweep]. `handle_jobsubmit` (job.c:1412) is the proven correct shape at the time this spec closed (Note: superseded by raft-apply-decoupling — see Goals section note above) -- Queue/worker/job-state hooks fire while the entity WRITE lock is held; job-submit's hook now fires post-commit outside any entity lock as a pure notification (raft-apply-decoupling task 3.1) — this asymmetry still drives WAIT placement (below), though the job-submit half of the asymmetry is now "no hook work at all" rather than "hook runs lock-free" -- Facts arrive via `do_notify` (cluster.c, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (`ha_dispatch_gate`, ha.c:4525) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing; fixed by this spec, see Fact Gate Admission) -- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:957) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md -- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) calls `repl_wait_majority_ack(configAck)` but only logs the result and returns the push count; the caller ACKed the admin tool with status=0 and restarted regardless [historical framing — this spec fixes it]. The configAck counting (repl.c, was 418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers` [historical — fixed, see Config Barrier] - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph requesters - CLI[client and cli ops] - SCHED[scheduler] - WK[worker] - ADM[admin tool] - end - subgraph controller_primary - GATE[ha dispatch gate<br/>fact allowlist] - H[request handlers<br/>job queue worker join] - F[fact handlers<br/>notify update preason] - HOOKS[persist hooks<br/>classification dispatch] - BAR[persist_commit_barrier] - CB[config barrier<br/>config_repl] - CE[commit engine<br/>consumed] - GUARD[mark_job guards] - end - CLI --> GATE - SCHED --> GATE - WK --> GATE - ADM --> GATE - GATE --> H - GATE --> F - H --> GUARD - F --> GUARD - GUARD --> HOOKS - HOOKS -->|append plus RETAIN register| CE - H -->|before success reply| BAR - BAR -->|commit_submit WAIT| CE - ADM -.upgradeConfig.-> CB - CB -->|configAck majority| CE -``` - -**Architecture Integration** (sliced from umbrella): -- Selected pattern: classification at the single persistence seam + completion at the handler. The seam guarantees no entry type escapes classification (2.1, 2.2); the handler realizes the umbrella Reply-Ordering Contract sequence "validate → mutate (provisional) → WAL append → commit_submit(WAIT) → reply" (1.1) -- WAIT placement decision: WAL_JOB_SUBMIT keeps its in-hook sync barrier (hook runs lock-free, engine-wired upstream). All other request-shaped types complete via `persist_commit_barrier()` called by the handler after entity locks are released — blocking a commit wait under a queue/job/worker WRITE lock would stall readers for up to the commit timeout. Same completion semantics, umbrella-conformant placement (the contract names the handler sequence) -- Existing patterns preserved: hook-based persistence dispatch, provisional in-memory mutation order (umbrella-locked), `errnox` early-return error style, full fan-out broadcast -- Steering compliance: barrier waits only on vtask threads (handlers run on the vtask pool); iomgr untouched; TDD with testsuite/ tests first - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), pthreads, stdatomic | classification, barrier helper, handler restructure | no new dependencies, no .capnp changes | -| Data / Storage | existing WAL (persist.c), wal_types.h enum | classification keyed on `wal_entry_type` + worker detail | no on-disk format change | -| Build / Test | meson, testsuite/ pattern | new `test_commit_barrier_rollout.c` | XCE_TESTING seams | - -## File Structure Plan - -### Directory Structure -``` -src/controller/ -├── persist.c # MODIFIED — classification dispatch, RETAIN registration, -│ # persist_commit_barrier, unclassified guard -├── job.c # MODIFIED — do_jobcontrol_op reply ordering; mark_job_* guards -├── queue.c # MODIFIED — do_queuecontrol_op reply ordering -├── worker.c # MODIFIED — do_hostcontrol_op reply ordering -├── join.c # MODIFIED — handle_worker_join reply ordering -├── ha.c # MODIFIED — dispatch-gate fact allowlist (PRIMARY branch only) -├── config_repl.c # MODIFIED — push_config_to_standbys return contract; caller failure path -├── repl.c # MODIFIED — legacy walAck counting retired; configAck branch hardened -└── testsuite/ - └── test_commit_barrier_rollout.c # NEW — classification, barrier ordering, fact retention, - # config barrier, idempotency guards -src/include/persist.h # MODIFIED — wal_commit_mode_t enum, wal_classify(), - # persist_commit_barrier() declarations -src/include/repl.h # MODIFIED — configAck evaluation reuses the entry_lsn - # parameter (added by ha-commit-engine) as the - # config push tag; no new parameter -``` - -### Modified Files (what and why) -[Line citations below re-verified against current source (was accurate as of this spec's 2026-06-13 close; substantial drift since from file growth — see per-line notes)] -- `src/include/persist.h` — declares `wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }`, `wal_commit_mode_t wal_classify(uint32_t wal_type, int detail)`, `int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)` -- `src/controller/persist.c` — `wal_classify` (persist.c:5796) table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (persist.c:5875 — snapshot `current_lsn`, `commit_submit(WAIT)`). Note: the table has since grown two rows this design never described — `WAL_JOB_ARRAY_SUBMIT` (WAIT, added by job-array-single-wal-record) and `WAL_CONFIG_MARKER` (RETAIN, added by ha-term-safety task 4.2) — see both specs' design.md -- `src/controller/job.c` — `do_jobcontrol_op` (job.c:2198-2224, was 921-934): barrier between `job_ctrl` success and the reply. `mark_job_fail`/`mark_job_update` (job.c:1830/1929, was 696/726): transition guards before mutation. `mark_job_start`/`mark_job_finish` (was job.c:583/638) no longer exist under those names — renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734) with the guard+mutation split across receipt time and apply time (Note: superseded by raft-apply-decoupling, see Idempotency Guards below and that spec's design.md) -- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:128-154, was 119-132): barrier between `queue_ctrl` success and the reply -- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:796-822, was 689-702): barrier between `worker_ctrl` success and the reply -- `src/controller/join.c` — `handle_worker_join` (join.c:48-149, was 33-119): barrier between the AFTER_EVENT hook and the reply at join.c:128-135 (was 107-109); join failure (idx < 0) keeps replying immediately -- `src/controller/ha.c` — `ha_dispatch_gate` (ha.c:4525, was implicitly ~1527) PRIMARY no-quorum allowlist (ha.c:4644-4648, was 1537-1556): adds `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` — confirmed present in current source. The gate has since grown two additional fence layers checked before this allowlist (`config_serve_fenced`, `upgrade_materialization_fenced` — ha-term-safety / config-upgrade-snapshot work); those fences do NOT include the fact opcodes in their own allowlists, so a config-fenced primary rejects facts too — a narrower, config-specific case this design did not anticipate and does not contradict (quorum-loss admission, this spec's concern, is unaffected) -- `src/controller/config_repl.c` — `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1057): on failure send error reply, skip restart. Note: superseded by ha-term-safety task 4.2 — the direct caller of `push_config_to_standbys` is now `config_commit_change` (config_repl.c:535), which `do_upgrade_config_with_bytes` calls instead; the requester's success/failure now gates on a WAL_CONFIG_MARKER consensus record reaching the committed prefix, with `push_config_to_standbys`'s majority-ACK wait as the out-of-band byte-delivery quorum underneath it. See Config Barrier section and ha-term-safety's design.md -- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack` (repl.c:589-700, was 386-462): legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds — confirmed unchanged in current source -- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c` (confirmed present) - -## System Flows - -### Request-shaped control operation (job/queue/worker control, worker join) - -```mermaid -sequenceDiagram - participant R as requester - participant V as vtask thread handler - participant E as entity lock scope - participant P as persist hooks - participant CE as commit engine - R->>V: control request - V->>V: dispatch gate incl write quorum - V->>E: lock, validate, mutate provisional - E->>P: HOOK_AFTER_EVENT append entries - P->>CE: persist_wal_async + register - E->>V: unlock, rc - alt rc != 0 - V->>R: error reply - else rc == 0 - V->>CE: persist_commit_barrier WAIT - alt committed - V->>R: success reply - else timeout or not ready - V->>R: error reply - end - end -``` - -Flow decisions: the barrier waits on a snapshot of `current_lsn` taken after the mutation returns — single-writer appends and contiguous commit advancement guarantee the snapshot covers every entry the operation produced (1.1, 1.4); waiting may also cover unrelated concurrent entries (bounded by the commit timeout, harmless — they must commit anyway); `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` map to the error reply via the existing `errnox` style (1.2, 1.3); no in-memory rollback (provisional model, umbrella-locked — outcome converges per umbrella 2.5, owned downstream). - -### Fact-shaped delivery (worker notify, rusage update, preason, worker dead) -Gate admits the opcode even without write quorum (3.2) → handler applies the transition through the `mark_job_*` guards (5.1-5.3) → hook classifies the WAL type RETAIN → `persist_wal_async` append + broadcast + `commit_submit(RETAIN)` (returns immediately, no timeout) → handler returns (notify: no reply; update/preason: receipt-style reply unchanged, verdict per 2.3/2.4). The fact is pending purely by `lsn > committed_lsn`; catch-up streaming re-delivers the uncommitted suffix and `committed_lsn` advances when ACKs resume (3.1, 3.3, 3.4). No new queue structure exists (log-is-the-queue, umbrella-locked). - -### Configuration upgrade - -> **Note: the diagram and "request restart" step below are superseded by ha-term-safety (task 4.2), see that spec's design.md.** Current source inserts `config_commit_change` (config_repl.c:535, appending a WAL_CONFIG_MARKER) between `do_upgrade_config_with_bytes` and `push_config_to_standbys`, and no restart is requested on success — materialization now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the committed marker. The majority-ACK shape below (push → parallel threads → per-standby ack → majority-of-n_peers) is otherwise still accurate as the mechanism `push_config_to_standbys` uses underneath the marker. - -```mermaid -sequenceDiagram - participant A as admin tool - participant U as do_upgrade_config_with_bytes - participant PS as push_config_to_standbys - participant S as standbys - A->>U: OP_upgradeConfig - U->>U: validate, save to disk, bump generation - U->>PS: push bytes, in-flight tag - PS->>S: parallel push threads - S-->>PS: OP_upgradeConfigAck tag - PS->>PS: majority of n_peers acked tag, self counted - alt majority within timeout - PS-->>U: 0, acked count - U->>U: committed_config_tag advanced - U->>A: success reply - U->>U: request restart - else timeout - PS-->>U: -ETIMEDOUT - U->>A: error reply upgrade failed - U->>U: no restart, tag not committed - end -``` - -Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved, now framed by ha-term-safety as a non-authoritative cache of the marker-derived committed config); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart was originally gated on success (4.4 — closed the roadmap residual "don't restart before the commit barrier passes") but ha-term-safety later removed the restart step entirely in favor of marker-driven materialization on every member — see the supersession note above; config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time). - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1 | success reply only after majority commit | Reply-Ordering Sweep, Commit Barrier Helper | request control | -| 1.2 | commit timeout → error reply | Commit Barrier Helper, Reply-Ordering Sweep | request control | -| 1.3 | not-ready rejection → error reply | Commit Barrier Helper (engine code passthrough) | request control | -| 1.4 | no success-interpretable reply pre-commit | Reply-Ordering Sweep (integration-test assertion) | request control | -| 2.1 | every WAL type classified WAIT/RETAIN | Write-Path Classification | both write flows | -| 2.2 | unclassified type fails fast / fact-fallback | Write-Path Classification (guard) | — | -| 2.3 | dispatch ACK is receipt-only | Reply-Ordering Sweep (handle_jobdispatch verdict) | — | -| 2.4 | exhaustive reply-site verification | Reply-Ordering Sweep (audit) | — | -| 3.1 | facts appended, retried, no timeout | Write-Path Classification (RETAIN routing) | fact delivery | -| 3.2 | facts accepted during quorum loss | Fact Gate Admission | fact delivery | -| 3.3 | uncommitted facts never discarded | Write-Path Classification (RETAIN; log-is-the-queue) | fact delivery | -| 3.4 | facts commit after quorum restore, no redelivery | Write-Path Classification + existing catch-up (verified by test) | fact delivery | -| 4.1 | config confirmed only after majority | Config Barrier | config upgrade | -| 4.2 | config timeout → failed to requester | Config Barrier | config upgrade | -| 4.3 | unacked config never committed | Config Barrier (tag advance in success branch only) | config upgrade | -| 4.4 | failed upgrade does not restart | Config Barrier (caller) | config upgrade | -| 5.1 | duplicate transition is logged no-op | Idempotency Guards | fact delivery | -| 5.2 | terminal→non-terminal rejected with WARN | Idempotency Guards | fact delivery | -| 5.3 | replay-once equivalence | Idempotency Guards | fact delivery | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| Write-Path Classification | controller/persist.c | route every WAL type to its commit mode at the hook seam | 2.1, 2.2, 3.1, 3.3, 3.4 | commit engine (P0), hooks (P0) | Service | -| Commit Barrier Helper | controller/persist.c | handler-callable WAIT completion on the appended watermark | 1.1, 1.2, 1.3 | commit engine (P0) | Service | -| Reply-Ordering Sweep | controller/job.c, queue.c, worker.c, join.c | commit-first-then-reply in every request handler; receipt-ACK verdicts | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 | Commit Barrier Helper (P0) | Service | -| Fact Gate Admission | controller/ha.c | admit fact-bearing opcodes without write quorum | 3.2 | dispatch gate (P0) | Service | -| Config Barrier | controller/config_repl.c, repl.c | per-push config majority with error propagation | 4.1, 4.2, 4.3, 4.4 | repl_on_config_ack tracking (P0) | Service | -| Idempotency Guards | controller/job.c | safe duplicate/replay of job transitions | 5.1, 5.2, 5.3 | — | Service | - -### Write-Path Classification (`persist.c`, `src/include/persist.h`) - -| Field | Detail | -|-------|--------| -| Intent | Single seam where every state-changing WAL append is classified and registered with the engine | -| Requirements | 2.1, 2.2, 3.1, 3.3, 3.4 | - -**Classification table** (umbrella table verified against call sites — verification deviations recorded in research.md §2; this verified table is the implementation contract): - -| WAL type (+ detail) | Verified origin | Shape | Mode | Failure behavior | -|---|---|---|---|---| -| WAL_JOB_SUBMIT | handle_jobsubmit (client) | request | WAIT (existing in-hook sync path) | error to client (reference impl, unchanged) | -| WAL_QUEUE_CTRL | do_queuecontrol_op (client) | request | WAIT (handler barrier) | error to client | -| WAL_WORKER_CTRL, detail JOIN/UPDATE | handle_worker_join, do_hostcontrol_op | request | WAIT (handler barrier) | error reply to worker/client | -| WAL_WORKER_CTRL, detail DEAD | controller-detected (worker.c:565/578) | fact | RETAIN | none — retained until committed | -| WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON | worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier) | fact | RETAIN | none — retained until committed | -| WAL_NOOP | promotion (ha-single-leader-role) | internal | RETAIN | landed and confirmed classified exactly as pre-declared (persist.c:5835-5839) — no longer future | -| WAL_JOB_ARRAY_SUBMIT | handle_jobsubmit_array (client, job-array-single-wal-record) | request | WAIT (handler barrier, single barrier for the whole array) | error to client — row added by job-array-single-wal-record, not in this design's original table (Note: superseded/extended, see that spec's design.md) | -| WAL_CONFIG_MARKER | config_commit_change (ha-term-safety task 4.2) | request-adjacent consensus record | RETAIN | commits through the same barrier as job-state facts; the upgrade requester's own barrier is owned by `do_upgrade_config_with_bytes`, not this row — row added by ha-term-safety, not in this design's original table (Note: superseded/extended, see that spec's design.md and the Config Barrier section below) | -| WAL_CHECKPOINT | local checkpoint marker | local-only | LOCAL | unchanged (never replicated, by design) | - -**Responsibilities & Constraints** -- `wal_classify(wal_type, detail)` is a pure lookup; `detail` disambiguates only WAL_WORKER_CTRL (JOIN/UPDATE vs DEAD) -- Hook dispatch: RETAIN → `persist_wal_async` (append + broadcast, the surviving RETAIN fast path) followed by `commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0)` registration (returns immediately; observability counter). WAIT → WAL_JOB_SUBMIT keeps `persist_wal_sync_append`; other WAIT types append via `persist_wal_async` and completion is owed by the handler barrier (placement decision, Architecture) -- Unclassified guard: under `XCE_TESTING`, `assert(0)` with the offending type logged; in production, `ERROR` log + RETAIN treatment (never dropped, never falsely confirmed) (2.2) -- Fact appends never return commit errors to their callers — `persist_job_hook_fun`'s non-submit return value stays the local-append rc only (3.1) - -**Dependencies** -- Inbound: hook system (HOOK_AFTER_EVENT) — all state-changing writes (P0) -- Outbound: `commit_submit` (P0); `persist_wal_async`/`persist_wal_sync_append` (P0) - -**Contracts**: Service [x] - -##### Service Interface -```c -/* src/include/persist.h */ -typedef enum { - WAL_MODE_WAIT, /* request-shaped: barrier before requester reply */ - WAL_MODE_RETAIN, /* fact-shaped: retained until committed, no timeout */ - WAL_MODE_LOCAL, /* not replicated (WAL_CHECKPOINT) */ -} wal_commit_mode_t; - -/* Pure classification lookup. detail disambiguates WAL_WORKER_CTRL only. - * Unknown type: XCE_TESTING assert; production returns WAL_MODE_RETAIN - * after ERROR log. */ -wal_commit_mode_t wal_classify(uint32_t wal_type, int detail); -``` -- Preconditions: called at the hook seam with the type produced by `detail_to_wal_type` -- Postconditions: total over all `wal_entry_type` values; deterministic -- Invariants: no WAL append path bypasses classification - -### Commit Barrier Helper (`persist.c`, declared in `src/include/persist.h`) - -| Field | Detail | -|-------|--------| -| Intent | Give request handlers a single call realizing commit_submit(WAIT) on everything their mutation appended | -| Requirements | 1.1, 1.2, 1.3 | - -**Responsibilities & Constraints** -- Snapshot `lsn = p->current_lsn` (under `p->lock`, matching `persist_wal_sync_append`'s read at persist.c:1019-1021); if `committed_lsn >= lsn` already, return 0 without blocking -- Delegate to `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout_ms ? timeout_ms : p->config.sync_timeout_ms)`; return its code unmodified (0 / `-ETIMEDOUT` / `-XCE_ERR_NOT_READY`) -- Single-writer + contiguous advancement ⇒ the snapshot covers every entry the calling operation produced; over-waiting on unrelated concurrent entries is accepted (bounded by the timeout; they must commit anyway). `n_peers <= 1` returns 0 immediately (engine self-quorum, standalone behavior preserved) -- Must be called only from vtask threads after entity locks are released; counts against the engine's admission cap (deliberate CP back-pressure) -- Migration completion: with all request paths on the engine (directly or via this helper), the walAck legacy counting in `repl_wait_majority_ack` is removed (upstream marked it "retired by ha-commit-barrier-rollout"); the configAck branch remains the config track's mechanism - -**Contracts**: Service [x] - -##### Service Interface -```c -/* src/include/persist.h */ -/* Block until committed_lsn covers everything appended so far. - * timeout_ms = 0 → p->config.sync_timeout_ms. - * Returns 0 (committed), -ETIMEDOUT (not committed in time — entry - * remains in log; outcome converges per umbrella 2.5, owned downstream), - * -XCE_ERR_NOT_READY (admission cap / own-term gate, retriable). */ -int persist_commit_barrier(persist_t *p, uint32_t timeout_ms); -``` -- Preconditions: commit engine wired (`cluster_init` phase, upstream); vtask thread; no entity locks held -- Postconditions: returns 0 ⇔ `committed_lsn >=` the snapshot watermark -- Invariants: never sends replies, never mutates state - -### Reply-Ordering Sweep (`job.c`, `queue.c`, `worker.c`, `join.c`) - -| Field | Detail | -|-------|--------| -| Intent | Enforce the umbrella Reply-Ordering Contract at every control-path reply site | -| Requirements | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 | - -**Responsibilities & Constraints** -- Restructure pattern (uniform): `rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Confirmed still in place at (line numbers re-verified against current source, drifted from file growth since this spec's 2026-06-13 close — see File Structure Plan §Modified Files for the specific old→new lines): - - `do_jobcontrol_op` (job.c:2198-2224) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately - - `do_queuecontrol_op` (queue.c:128-154) - - `do_hostcontrol_op` (worker.c:796-822) - - `handle_worker_join` (join.c:48-149) — barrier inserted between the AFTER_EVENT hook (join.c:118-120) and the success reply (join.c:137-139); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply -- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:1119, was 459-491) — early reply is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:2330/2358, was 1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c:48/64/88) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only -- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table was committed with the task evidence. The design-time line-number snapshot originally recorded here (job.c/queue.c/worker.c/join.c/opupd.c/cluster.c/config_repl.c/ha.c/opget.c/reply.c/task.c line lists) has drifted wholesale with subsequent file growth across a dozen-plus later specs and is no longer independently meaningful as a citation list; the audit's *coverage claim* (every reply site classified, zero unclassified) is not re-verifiable line-by-line within this pass's scope — re-run the grep sweep against current source before relying on the exhaustiveness claim -- A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4) - -**Contracts**: Service [x] — external reply semantics: success now implies majority durability; new error codes `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary) - -**Implementation Notes** -- Integration: handlers run on vtask threads (cluster.c dispatch switch at cluster.c:325-363) — barrier blocking is steering-compliant -- Validation: integration test drives each restructured opcode against a 3-node-simulated registry with ACKs withheld → error reply, `committed_lsn` unmoved; with ACKs → success reply strictly after commit -- Risks: scheduler/worker callers seeing new timeout errors where they previously saw instant success — deliberate CP behavior (umbrella); retry semantics unchanged on their side - -### Fact Gate Admission (`ha.c`) -Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` to the CTRL_PRIMARY no-write-quorum allowlist in `ha_dispatch_gate` (ha.c:1537-1556) so reported facts reach the RETAIN path during quorum loss (3.2 — closes the bug-396 silent-loss hole; umbrella error strategy: "fact-shaped ops are still appended and retained"). Request-shaped opcodes (jobsub, jobcontrol, queuecontrol, hostcontrol, jobdispatch, join) remain rejected without quorum. The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then). Quorum accounting itself is out of boundary (upstream). - -### Config Barrier (`config_repl.c`, `repl.c` configAck branch) - -| Field | Detail | -|-------|--------| -| Intent | Make the config majority-ACK result authoritative for the upgrade outcome | -| Requirements | 4.1, 4.2, 4.3, 4.4 | - -**Responsibilities & Constraints** - -> **Note: partially superseded by ha-term-safety (task 4.2), see that spec's design.md.** As of this spec's 2026-06-13 close, `do_upgrade_config_with_bytes` called `push_config_to_standbys` directly and its return code alone gated the requester's success/failure reply (described below, still architecturally accurate for `push_config_to_standbys` itself). ha-term-safety inserted a layer: `do_upgrade_config_with_bytes` now calls `config_commit_change` (config_repl.c:535), which appends a `WAL_CONFIG_MARKER` as the authoritative (term,lsn) consensus record and *internally* calls `push_config_to_standbys` as the out-of-band byte-delivery quorum underneath that marker. The requester's success/failure now gates on the marker reaching the committed prefix, not directly on `push_config_to_standbys`'s return. The majority-ACK arithmetic itself (below) is unchanged and still lives in `push_config_to_standbys`/`repl_wait_majority_ack`. - -- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — confirmed unchanged — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (4.3 preserved, now additionally described by ha-term-safety as a "non-authoritative cache" of the marker-derived committed config) -- configAck branch of `repl_wait_majority_ack` (repl.c:616-699, was 418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track") — confirmed unchanged: count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine) -- Caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1056-1094): on non-zero return from `config_commit_change` (was directly from `push_config_to_standbys`, see supersession note above) — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip restart. Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary). Materialization on success now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the marker — no restart is requested or required (a further ha-term-safety change beyond this spec's original "restart is gated on success" framing; see the Flow decisions paragraph below, now stale on this point) -- One in-flight config change at a time (umbrella-locked) — no per-entry registry needed - -**Contracts**: Service [x] — `upgradeConfig` reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart - -**Implementation Notes** -- Integration: `CONF_PUSH_TIMEOUT_MS` remains the config commit timeout -- Validation: integration test withholds configAcks → admin reply is an error, `committed_config_tag` unmoved, no restart request; with acks → success and tag advanced -- Risks: admin tooling that treated any reply as success must observe `status` — existing CLI already surfaces reply status (admin-config-tool backlog item consumes this contract) - -### Idempotency Guards (`job.c`) - -| Field | Detail | -|-------|--------| -| Intent | Make job state transitions safe under duplicate, out-of-order, and replayed application | -| Requirements | 5.1, 5.2, 5.3 | - -**Responsibilities & Constraints** - -> **Note: partially superseded by raft-apply-decoupling, see that spec's design.md.** `mark_job_fail`/`mark_job_update` (job.c:1830/1929) still exist essentially as described below — guard-then-mutate synchronously under the caller-held WRITE lock, used by the client-request path (`job_ctrl`). `mark_job_start`/`mark_job_finish` no longer exist under those names: they were renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734, worker-fact receipt path only) and split into two phases — a receipt-time "peek" of the identical guard (log-only, for the rejected-transition WARN; no mutation) plus a durable-append-only WAL write with no immediate mutation and no AFTER_EVENT hook, followed by the *actual* guarded mutation + hook firing later, deferred to `apply_committed_entry()` (via the extracted `apply_job_start_transition`/`apply_job_finish_transition` cores) once the entry reaches the committed prefix and the apply pump dispatches it. The guard *rules* (duplicate → no-op, backward → rejected+WARN) and their *outcome* are unchanged from what this design specifies below; the *timing* — guard-before-append vs. guard-at-apply-time — is not. - -- Guard placement (as designed; see supersession note above for the current split for START/FINISH): at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire) -- Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers): - - duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1) - - backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return `-EALREADY`-style rejection, state preserved (5.2) - - forward/normal: proceed unchanged; `mark_job_fail`'s existing post-exec special case (job.c:702-716) already tolerates fail-after-finish and is kept as the terminal-state-compatible path -- Replay equivalence (5.3): guards make `restore_job_state`-driven re-application and RETAIN retry converge to single-application state; callers (`on_job_*`, `job_ctrl`) treat the duplicate code as success-no-op -- Shared prerequisite of worker-rejoin-resync (umbrella adjacent expectation) — guards live here, resync consumes them later - -**Contracts**: Service [x] — `mark_job_*` return contract gains the logged no-op/rejection outcomes; all existing callers checked during the sweep - -## Data Models - -### Domain Model (changes only) -- **wal_commit_mode_t** — new enum (persist.h); pure classification metadata, never persisted -- **WAL entries, job_t, queue_t, worker_t** — no format or field changes in this spec (`submit_lsn` tagging is ha-convergence-recovery's egress filter, not here) -- Invariants: every appended state-changing entry is classified; a success reply to a request-shaped operation implies `committed_lsn >=` all entries that operation appended; `committed_config_tag` advances only on per-push config majority; job state never transitions terminal → non-terminal - -## Error Handling - -### Error Strategy -- Request-shaped commit failure: `-ETIMEDOUT` / `-XCE_ERR_NOT_READY` propagate to the requester reply via the existing `errnox` early-return pattern; no in-memory rollback (provisional model, umbrella-locked; convergence owned downstream) -- Fact-shaped paths never surface commit errors to reporters: local append failure (disk) keeps its existing ERROR logging; commit progress is the engine's concern (RETAIN) -- Unclassified WAL type: XCE_TESTING assert (fail fast in CI); production ERROR + RETAIN fallback (2.2 — fail-safe toward retention, never silent confirmation) -- Config failure: explicit error reply via `send_upgrade_error`; no restart; WARN with tag and acked count -- Guard rejections: INFO (duplicate) / WARN (backward) with jobid and both states — log-once per application, no log storms - -### Monitoring -- Retained-uncommitted depth (committed-vs-appended lag) — engine counter, consumed in logs here for fact-backlog visibility -- Per-handler barrier failures WARN with opcode and rc; config barrier WARN already present (config_repl.c:924-926), now coupled to the failed reply - -## Testing Strategy - -### Unit Tests (`testsuite/test_commit_barrier_rollout.c`, XCE_TESTING) -- Classification totality: every `wal_entry_type` value maps to a mode; WAL_WORKER_CTRL detail split (JOIN/UPDATE → WAIT, DEAD → RETAIN); unknown type asserts under XCE_TESTING (2.1, 2.2) -- Barrier helper: returns 0 immediately when `committed_lsn` covers the watermark; `-ETIMEDOUT` with ACKs withheld and `committed_lsn` unmoved; 0 after covering ACK; `n_peers <= 1` immediate success (1.1, 1.2, 1.3) -- Idempotency guards: duplicate FINISH → state unchanged, rc = no-op, no AFTER hook fired (no WAL entry); START on DONE job → rejected, WARN, state preserved; double application of a transition equals single application; fail-after-finish keeps existing PERR behavior (5.1, 5.2, 5.3) -- configAck arithmetic: stale `acked_config_lsn` from a previous tag does not count; zero covering acks with n_peers=3 → timeout; majority with self-count → 0; n_peers=1 → immediate (4.1, 4.3) - -### Integration Tests (testsuite/) -- No-success-reply-before-commit: for each restructured opcode (jobcontrol, queuecontrol, hostcontrol, worker join), with ACKs withheld the reply is an error and arrives only at/after the timeout; with ACKs the success reply is observable only after `committed_lsn` covers the operation's entries (1.1, 1.2, 1.4) -- Fact retention across quorum loss: simulate quorum loss → notify-driven transitions still append (RETAIN, no error), gate admits the fact opcodes; restore ACKs → entries commit without re-delivery (3.1-3.4) -- Dispatch receipt-ACK: handle_jobdispatch still replies immediately; no WAL entry is produced by dispatch processing itself (classification assert clean) (2.3) -- Config barrier: upgradeConfig with withheld configAcks → error reply to requester, `committed_config_tag` unmoved, no restart requested; with acks → success, tag advanced (4.1-4.4) -- Reply-site audit (2.4): scripted grep over all `xce_reply` call sites in control paths, each assigned a category; zero unclassified sites - -### Regression -- Full controller testsuite + ASAN (`/xce-build controller`) green; existing job-submit barrier tests (upstream) unaffected; `test_repl_majority.c` configAck cases updated to the hardened per-push arithmetic - -## Performance & Scalability -- Request-control latency gains one majority RTT (the same cost job submit already pays); fact paths gain no latency (RETAIN returns immediately) -- Concurrent blocked handlers are bounded by the engine's `n_vcpus`-relative admission cap (upstream); excess load degrades to fast NOT_READY rejections — deliberate CP -- Barrier over-wait on concurrent unrelated entries is bounded by `sync_timeout_ms` and only occurs while uncommitted traffic is pending - -## Migration Strategy -This spec is phase 2 of the umbrella's hard-ordered landing ("barrier before state removal"): -1. Classification + RETAIN registration + barrier helper (no behavior change for replies yet) -2. Handler restructure sweep + fact gate admission (reply semantics switch on) -3. Config barrier error propagation; legacy walAck counting retirement; idempotency guards -Each step leaves the system no less consistent than before it; rollback trigger per step: controller testsuite regression or job-control E2E failure. ha-single-leader-role must not land until the sweep (step 2) is complete and audited. diff --git a/.kiro/specs/ha-commit-barrier-rollout/requirements.md b/.kiro/specs/ha-commit-barrier-rollout/requirements.md deleted file mode 100644 index 67fa1eba..00000000 --- a/.kiro/specs/ha-commit-barrier-rollout/requirements.md +++ /dev/null @@ -1,92 +0,0 @@ -# Requirements Document - -> Sub-spec 2 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.1 (all operation types), 2.2, 2.8, 5.1, 5.2, 5.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Locked umbrella decisions (log-is-the-queue for facts, commit-first-then-reply for all request-shaped operations, crash-loss of uncommitted facts accepted until worker-rejoin-resync) are restated, never re-derived. - -## Project Description (Input) - -Only WAL_JOB_SUBMIT goes through a commit barrier today. Job state changes, queue control, and worker control apply locally with fire-and-forget replication; job/queue/worker control handlers reply to their requesters before any durability is established; config push ignores its majority-ACK result and confirms the upgrade to the admin tool regardless; worker-reported facts are silently lost when the dispatch gate rejects them (bug-396 family); and `mark_job_*` transitions have no idempotency guards, so duplicate or replayed application corrupts state. Cluster operators need every state-changing operation routed per the umbrella design's write-path classification table: request-shaped operations confirm to their requester only after majority commit (validate → mutate provisional → append → commit barrier → reply) and fail to the requester on commit timeout; fact-shaped operations are appended and retained until committed with no timeout — never dropped while the primary runs and leads; configuration upgrades fail to the requester on missed majority; and fact transitions are idempotent and ordered so retry/replay is safe. This spec consumes the commit engine delivered by ha-commit-engine (commit_submit WAIT/RETAIN) and never modifies the engine's internals. - -## Introduction - -This spec delivers the barrier-rollout slice of the umbrella contract. After this spec, every cluster-state-changing operation on the primary participates in the majority-commit contract: a requester (client, scheduler, worker, or admin tool) that receives a success reply is guaranteed the operation is durable on a majority of cluster members; a requester whose operation cannot reach majority within the commit timeout receives an error; and reported facts with no requester awaiting confirmation are retained in the log and retried until committed — including across quorum-loss periods. The write path enforces this through an exhaustive classification of WAL entry types: a state-changing entry type that is not classified cannot silently bypass the barrier. Configuration changes follow the same contract on their own track. Job state transitions become idempotent and ordered so that retained-fact retry, replay, and the future worker-rejoin-resync can re-apply them safely. - -## Boundary Context - -- **In scope**: The write-path classification table and its dispatch at the persistence seam (with a fail-fast guard for unclassified types); commit-first-then-reply restructuring of every request-shaped control handler (job control, queue control, worker control, worker join/update) and verification of every control-path reply site; classification verdict for the scheduler dispatch acknowledgment; RETAIN routing for fact-shaped WAL types, including acceptance of reported facts during quorum loss; configuration-change commit-barrier error propagation to the upgrade requester; idempotency and ordering guards for job state transitions; integration tests proving no success reply is observable before commit and that facts survive quorum-loss periods. -- **Out of scope**: The commit engine itself — per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine, consumed as-is); promotion no-op, own-term gate consumers, and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role — must land after this spec); demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery); worker protocol changes of any kind — facts that never reached the controller, or that died with it across failover, are recovered only by worker-rejoin-resync (roadmap backlog, designated backstop); election and vote correctness (raft-library); replication wire format and standby apply mechanics (ha-wal-replication). -- **Adjacent expectations**: Upstream, this spec consumes the `commit_submit` WAIT/RETAIN service interface and the quorum-accounting fixes delivered by ha-commit-engine; changes to that interface require revalidation here. Downstream, ha-single-leader-role depends on this spec having placed the barrier on all write paths before the transitional leader state is removed (hard ordering), and worker-rejoin-resync shares the idempotency guards introduced here. Crash-loss of facts that were appended but not yet committed when the primary process dies is accepted until worker-rejoin-resync lands (umbrella-locked decision). - -## Requirements - -### Requirement 1: Commit-First-Then-Reply for Request-Shaped Operations - -**Objective:** As a cluster operator, I want every operation with a requester awaiting confirmation to be confirmed only after its log entries are durable on a majority of cluster members, so that no acknowledged change can be lost by a single node failure or failover. - -#### Acceptance Criteria - -1. When the primary receives a request-shaped cluster-state-changing operation (job control, queue control, worker control, worker join, or worker update), the primary shall send a success reply to the requester only after every log entry produced by the operation has been acknowledged by a majority of cluster members, counting itself. *(umbrella 2.1)* -2. If majority acknowledgment for a request-shaped operation is not obtained within the configured commit timeout, the primary shall reply to that requester with an error reporting the operation as failed. *(umbrella 2.2)* -3. If the commit barrier rejects a request-shaped operation without waiting (cluster not ready to commit), the primary shall reply to the requester with the corresponding error, never with success. *(umbrella 2.2)* -4. While the log entries produced by a request-shaped operation are not yet committed, the primary shall not emit any reply that the requester can interpret as confirmation of that operation. *(umbrella 2.1)* - -### Requirement 2: Complete Write-Path Classification - -**Objective:** As a cluster operator, I want every state-changing entry type explicitly classified as request-shaped or fact-shaped at the write path, so that no operation type can silently bypass the commit contract. - -#### Acceptance Criteria - -1. The controller shall route every state-changing log entry type through the commit engine under an explicit classification: request-shaped types complete through the commit barrier before requester confirmation, and fact-shaped types are retained until committed. *(umbrella 2.1, 2.8)* -2. If a state-changing log entry type with no classification reaches the write path, the controller shall fail fast at that point in testing builds; in production builds it shall log an error and treat the entry as fact-shaped, never dropping it and never confirming it early. *(umbrella 2.1)* -3. When the scheduler delivers a job dispatch message, the primary's immediate acknowledgment shall constitute only receipt of the message; per-job dispatch outcomes shall continue to be delivered through the existing per-job notifications, and the receipt acknowledgment shall not confirm any state-changing log entry. *(umbrella 2.1)* -4. The controller shall contain no control-path reply site that confirms a request-shaped state-changing operation before its entries are committed; every control-path reply site shall be either a post-commit confirmation or a receipt-only acknowledgment for fact-shaped deliveries. *(umbrella 2.1)* - -### Requirement 3: Fact Retention — Log Is the Queue - -**Objective:** As a cluster operator, I want reported facts that have no requester awaiting confirmation to be retained and retried until committed, so that a quorum-loss window never silently erases what actually happened on the cluster. - -#### Acceptance Criteria - -1. When the primary receives a reported fact with no requester awaiting confirmation (a worker-reported job state transition, worker-reported resource usage, scheduler-reported pending reason, or controller-detected worker death), the primary shall append the fact to its log and continue attempting commitment until the fact's entry is committed, with no commit timeout. *(umbrella 2.8)* -2. While the primary cannot reach a majority of cluster members, when a reported fact arrives, the primary shall accept and append the fact rather than rejecting it. *(umbrella 2.8)* -3. While the primary remains running and holds leadership, the primary shall not discard an uncommitted reported fact. *(umbrella 2.8)* -4. When majority connectivity is restored, the primary shall bring previously appended uncommitted facts to committed state without requiring redelivery from the original reporter. *(umbrella 2.8)* - -### Requirement 4: Configuration Change Commit Barrier - -**Objective:** As a cluster operator, I want configuration changes to follow the same majority-commit contract as other state changes, so that a failover never reverts a configuration upgrade that was reported as successful. - -#### Acceptance Criteria - -1. When the primary distributes a configuration change, the primary shall confirm the change to its requester only after a majority of cluster members have acknowledged it. *(umbrella 5.1)* -2. If majority acknowledgment for a configuration change is not obtained within the configured commit timeout, the primary shall report the configuration change as failed to its requester. *(umbrella 5.2)* -3. If a configuration change has not achieved majority acknowledgment, the primary shall not mark that configuration as committed. *(umbrella 5.3)* -4. If a configuration change was reported as failed to its requester, the primary shall not proceed with the restart that activates the new configuration. *(umbrella 5.2)* - -### Requirement 5: Idempotent and Ordered Fact Application - -**Objective:** As a cluster operator, I want job state transitions to be safe under duplicate and replayed application, so that retained-fact retry, log replay, and future worker rejoin re-synchronization cannot corrupt job state. - -#### Acceptance Criteria - -1. When a job state transition that is already reflected in the job's current state is applied again, the controller shall leave the job's state unchanged and record the duplicate application in its log. *(umbrella 2.8)* -2. If a job state transition would move a job from a terminal state back to a non-terminal state, the controller shall reject the transition, preserve the terminal state, and log a warning. *(umbrella 2.8)* -3. When a fact's log entry is applied more than once through retry or replay, the resulting job state shall equal the state produced by exactly one application. *(umbrella 2.8)* - -## Umbrella Traceability - -| This spec | Umbrella ID | Notes | -|-----------|-------------|-------| -| 1.1, 1.4 | 2.1 | Commit-first-then-reply generalized from job submit to all request-shaped operation types | -| 1.2, 1.3 | 2.2 | Requester-awaiting operations fail on commit timeout / not-ready, never silently degrade | -| 2.1 | 2.1, 2.8 | Classification table is the enforcement mechanism for "all operation types" | -| 2.2 | 2.1 | Fail-fast guard: a new/unclassified WAL type cannot silently bypass the barrier (umbrella revalidation trigger) | -| 2.3 | 2.1 | handle_jobdispatch classification per the umbrella Reply-Ordering Contract (receipt-ACK verdict, task-verified) | -| 2.4 | 2.1 | Exhaustive reply-site sweep; a missed handler silently violates umbrella 2.1 | -| 3.1, 3.3 | 2.8 | Log-is-the-queue: RETAIN mode, no timeout, retry until committed (umbrella-locked decision) | -| 3.2 | 2.8 | Facts accepted across quorum-loss periods (closes the bug-396 silent-loss hole) | -| 3.4 | 2.8 | Commitment resumes from the retained log suffix, not from reporter redelivery | -| 4.1 | 5.1 | Config change confirmed only after majority ACK | -| 4.2, 4.4 | 5.2 | Config change fails to requester on timeout; restart sequencing gated on the barrier (roadmap residual) | -| 4.3 | 5.3 | committed_config_tag advances only on majority ACK | -| 5.1, 5.2, 5.3 | 2.8 | Idempotency guards make RETAIN retry/replay safe; shared prerequisite of worker-rejoin-resync | diff --git a/.kiro/specs/ha-commit-barrier-rollout/research.md b/.kiro/specs/ha-commit-barrier-rollout/research.md deleted file mode 100644 index fc20e9bc..00000000 --- a/.kiro/specs/ha-commit-barrier-rollout/research.md +++ /dev/null @@ -1,170 +0,0 @@ -# Research & Design Decisions: ha-commit-barrier-rollout - -## Summary -- **Feature**: `ha-commit-barrier-rollout` (sub-spec 2 of 4, ha-consistency-invariants umbrella) -- **Discovery Scope**: Extension (light discovery — umbrella research.md is the primary discovery artifact; this pass verified its anchors against the current tree, 2026-06-11) -- **Key Findings**: - - Every reply-before-commit call site named by the umbrella still exists; line anchors re-verified (table below) - - Call-site verification of the umbrella classification table found two deviations the umbrella anticipated ("verified against call sites during implementation; table is the contract"): WAL_JOB_STOP/RESUME are fact-origin only, and WAL_JOB_UPDATE (absent from the umbrella table) is dual-origin - - The configAck branch of `repl_wait_majority_ack` has the same three counting defects the commit engine fixed for walAck (`> 0` counting, `n_active == 0 → success`, `n_active` denominator) - - The dispatch gate rejects `OPCode_notify` (and update/preason) under quorum loss — the bug-396 silent fact-loss hole is gate-level, not just replication-level - -## Verified Code Anchors (2026-06-11, persistent_ha working tree) - -| Concern | Anchor (verified by grep/read) | Status vs umbrella audit | -|---|---|---| -| do_jobcontrol_op replies after mutation, async WAL only | job.c:921-934 (reply at 932) | unchanged | -| handle_jobdispatch early reply | job.c:459-491 (reply at 466-471) | umbrella cited 459-471 — still accurate | -| handle_jobsubmit reference implementation | job.c:495-581 (hook rc checked 563-571, reply 573-578) | umbrella cited 563-578 — still accurate | -| do_queuecontrol_op | queue.c:119-132 (reply at 130); queue_ctrl fires AFTER hook under queue WRITE lock (queue.c:76-112, hook at 107) | new precision: lock-scope drives WAIT placement | -| do_hostcontrol_op | worker.c:689-702 (reply at 700) | new anchor | -| handle_worker_join | join.c:33-119 (AFTER hook 104-105, success reply 107-109) | new anchor | -| do_set_preason / do_clr_preason early replies | job.c:1040-1066 (reply at 1050) / job.c:1068+ (reply at 1078) | scheduler-origin facts — receipt-ACK verdict | -| do_update_op family | opupd.c:48/64/88 reply sites | worker-origin facts — receipt-ACK verdict | -| do_notify (no reply, fire-and-forget) | cluster.c:118-157 | unchanged | -| Hook → WAL type map | persist.c:1068-1088 (`detail_to_wal_type`) | classification dispatch insertion point | -| Job hook: submit sync path / fact async path | persist.c:1093-1169 (sync 1107-1129, async 1169) | unchanged | -| Queue / worker hooks (async) | persist.c:1172-1202 (append 1199) / persist.c:1204-1244 (append 1241) | unchanged | -| persist_wal_sync_append + timeout | persist.c:1003-1049; timeout `p->config.sync_timeout_ms` (DEFAULT_SYNC_TIMEOUT_MS=50, config.h:93) | committed_lsn store at 1035-1036 is removed by ha-commit-engine (upstream) | -| Dispatch gate PRIMARY no-quorum allowlist | ha.c:1527 (`ha_dispatch_gate`), allowlist 1537-1556; notify falls to `default → -XCE_ERR_NOT_READY` | fact admission insertion point | -| push_config_to_standbys ACK result ignored | config_repl.c:866-930 (wait at 916-917, rc logged-only 918-927, returns acked at 929) | umbrella cited 916-928 — still accurate | -| upgradeConfig caller ignores result | config_repl.c:1056-1094 (push at 1057, unconditional success ACK 1062-1073, restart 1091-1093) | umbrella cited 1057 — still accurate | -| configAck counting defects | repl.c:386-462 (`acked_config_lsn > 0` at 422-424, `n_active==0 → success` at 432-435, denominator `n_active` at 437) | mirrors the walAck defects fixed upstream | -| repl_on_config_ack stores reported tag/lsn | repl.c:351-371; `standby_conn_t.acked_config_lsn` (repl.h:25) | enables per-push `>= push_tag` comparison | -| mark_job_start/finish/fail/update unguarded | job.c:583/638/696/726 | unchanged | -| job_ctrl branches | job.c:306-438 — running job forwards to worker (357, no local entry); not-started job mutates via mark_job_update (391/401) / mark_job_fail (415) | classification evidence (deviation §2) | -| dispatch_a_job_ produces no WAL | job.c:180-304 — sends xce_startjob_req, mutates j->job.allocs, fires no hooks | handle_jobdispatch receipt-ACK evidence | -| WAL type enum | src/include/wal_types.h:44-58 (no WAL_NOOP yet) | WAL_NOOP arrives with ha-single-leader-role | -| Control-path xce_reply population | job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c:255-490; config_repl.c:619-1129; ha.c:1035; opget.c:222; reply.c:61; task.c:97 | sweep-task input; re-grep before editing | -| Opcode → handler dispatch | cluster.c:325-363 (jobsub/jobdispatch/notify/update/jobcontrol/queuecontrol/hostcontrol/setpreason/clrpreason/join) | gate allowlist + sweep scoping | - -Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` → 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY; RETAIN returns immediately; n_peers<=1 self-quorum; admission cap n_vcpus-relative; engine embedded in `cluster_t`, reachable from `persist_t` via `p->repl.ce`. `commit.c`/`commit.h` do not exist in the tree yet — ha-commit-engine must be implemented first (hard upstream dependency; tasks assume its interface as specified). - -## Research Log - -### §1 Reply-ordering reality and the WAIT placement question -- **Context**: The umbrella Reply-Ordering Contract fixes the handler sequence (validate → mutate → append → commit_submit(WAIT) → reply). Where should the WAIT physically run — inside the persistence hook (like job submit) or in the handler? -- **Findings**: queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:107 inside lock taken at 76; mark_job_* called under job WRITE lock, e.g. job.c:629-631). Job submit is the exception: its AFTER hook runs against a job list after per-job locks are released (job.c:553), which is why its in-hook sync wait is safe. -- **Implications**: In-hook WAIT for queue/worker/job-control would block entity readers for up to the commit timeout. Handler-level barrier (after locks released, before reply) keeps the same completion semantics and matches the umbrella's handler-sequence wording. Decision below. - -### §2 Classification table call-site verification (umbrella-mandated) -- **Context**: Umbrella table marks WAL_JOB_STOP/RESUME as "request (client job control) → WAIT" and omits WAL_JOB_UPDATE. -- **Findings**: WAL_JOB_STOP/RESUME entries are appended only by `on_job_stop`/`on_job_resume`, reachable solely from `do_notify` (worker-reported stoppedJob/resumedJob facts — cluster.c:131-136). Client job control (`do_jobcontrol_op` → `job_ctrl`) never produces them: for a running job it forwards the signal to the worker (job.c:357) and the resulting transition comes back later as a worker fact; for a not-yet-started job it mutates locally through `mark_job_update` (WAL_JOB_UPDATE) or `mark_job_fail` (WAL_JOB_FAIL). WAL_JOB_UPDATE/WAL_JOB_FAIL are therefore dual-origin (request via job_ctrl, fact via postdone/notify/rusage). -- **Implications**: Classify STOP/RESUME/UPDATE/FAIL as fact/RETAIN at the seam; realize the umbrella's request-shaped guarantee ("error to client") for client job control at the handler barrier in `do_jobcontrol_op`. This satisfies the umbrella's intent through its own prescribed handler sequence — the barrier is a property of the handler, not of the entry type. Not a contradiction: the umbrella explicitly delegated call-site verification, and the failure-column behavior is preserved. - -### §3 handle_jobdispatch classification (umbrella left it open) -- **Findings**: `dispatch_a_job_` (job.c:180-304) sends `xce_startjob_req` to the worker and updates `j->job.allocs` in memory; it fires no hooks and appends no WAL entries. Per-job outcomes return to the scheduler via `notify_dispatch` (job.c:440-457, called at 479-486). The job's externally meaningful state change (RUN) arrives later as a worker fact (WAL_JOB_START). -- **Implications**: Per the umbrella's own criterion ("if its outcomes are fact-shaped … the early reply is an ACK-of-receipt and may stand"), the verdict is **receipt-ACK stands** (this spec's Req 2.3). The orphan-dispatch-of-uncommitted-submit race is closed by the egress watermark filter (ha-convergence-recovery), not here. The classification guard plus a no-WAL-from-dispatch test pin the verdict. - -### §4 Fact loss is gate-level too (bug-396 family) -- **Findings**: `ha_dispatch_gate`'s CTRL_PRIMARY branch rejects `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` with `-XCE_ERR_NOT_READY` whenever write quorum is absent (ha.c:1537-1556 default arm). The worker never reads notify replies (umbrella research §A), so each rejection is a permanently lost fact. -- **Implications**: Umbrella 2.8 ("retain … including across quorum-loss periods") requires admitting these opcodes without write quorum; durability is then owned by RETAIN. Allowlist change is confined to the PRIMARY branch; the COMMITTING branch is left for ha-single-leader-role's wholesale deletion. - -### §5 Config track hardening scope -- **Findings**: configAck counting (repl.c:418-438) shares all three walAck defects; `repl_on_config_ack` already stores the standby's acknowledged tag (repl.c:351-371), so per-push comparison (`>= push_tag`) needs only a parameter, not new state. `committed_config_tag` advance is already success-gated (config_repl.c:918-922). -- **Implications**: Hardening lives in the configAck branch + `push_config_to_standbys` return contract + caller failure path. No engine involvement (umbrella-locked: config is a parallel, simpler track sharing the majority arithmetic rule — one in-flight change at a time). - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Verdict | -|--------|-------------|-----------|---------------------|---------| -| In-hook WAIT for all request types | extend the job-submit sync-append pattern to queue/worker/job-control hooks | one mechanism, no handler edits | blocks commit waits under entity WRITE locks (readers stall up to timeout); job-control hooks fire per-transition, not per-operation | rejected | -| Handler-level barrier (chosen) | hooks append (RETAIN-registered); request handlers call `persist_commit_barrier()` after locks, before reply | matches umbrella handler sequence verbatim; no lock-hold waits; uniform mechanical sweep pattern; covers dual-origin types from request contexts | barrier watermark may over-wait on unrelated concurrent entries (bounded by timeout) | selected | -| Per-entry LSN plumbing (hooks return entry LSNs to handlers) | exact-entry waits | no over-wait | invasive signature changes through the hook system for negligible benefit (over-wait is bounded and rare) | rejected (YAGNI) | - -## Design Decisions - -### Decision: Handler-level commit barrier on the appended watermark -- **Context**: §1 lock-scope finding vs the umbrella WAIT mode for queue/worker/job-control. -- **Selected Approach**: `persist_commit_barrier(p, timeout_ms)` = snapshot `current_lsn`, `commit_submit(WAIT)` on it; called by each request handler between mutation success and reply. -- **Rationale**: Single-writer appends + contiguous commit advancement (engine postcondition) make the watermark cover everything the operation appended; the umbrella's Reply-Ordering Contract literally specifies the handler sequence. -- **Trade-offs**: Possible over-wait on concurrent unrelated uncommitted entries (bounded by `sync_timeout_ms`; those entries must commit anyway). A no-op control operation may wait behind unrelated traffic — accepted (request-shaped ops are gate-rejected during quorum loss, so the wait resolves quickly in healthy clusters). -- **Follow-up**: Integration test pins reply-after-commit ordering per opcode. - -### Decision: Verified classification deviations (STOP/RESUME fact-only; UPDATE dual-origin) -- See Research Log §2. Umbrella conformance preserved through the handler barrier; the verified table in design.md is the implementation contract. Inline umbrella tags kept (2.1/2.8). - -### Decision: Production fallback for unclassified WAL types = RETAIN -- **Context**: Req 2.2 requires fail-fast in testing; production must not crash the primary on a programmer omission. -- **Selected Approach**: XCE_TESTING assert; production ERROR log + RETAIN treatment. -- **Rationale**: Fail-safe direction is retention (never dropped, never falsely confirmed) — a misrouted request-shaped type degrades to a missing barrier caught by the sweep audit/tests, not to data loss. - -### Decision: Failed config upgrade does not restart -- **Context**: Roadmap residual ("don't restart the primary before the commit barrier passes"); umbrella 5.2. -- **Selected Approach**: `do_upgrade_config_with_bytes` gates the success ACK and `restart_requested` on the barrier result; disk save/generation bump remain pre-barrier (existing sequence), reconciled by the existing OP_replConfig anti-entropy (out of boundary). -- **Trade-offs**: A failed upgrade leaves the new config on the primary's disk but uncommitted cluster-wide; explicit failure to the operator makes the state visible. Full save-after-commit resequencing deferred (would touch checkpoint/restart machinery beyond this slice). - -## Risks & Mitigations -- Missed reply site silently violates umbrella 2.1 — exhaustive grep audit task over all `xce_reply` sites + behavioral integration test (no success reply before commit) -- Upstream ha-commit-engine not yet implemented — tasks consume only its published interface; if signatures drift, revalidation trigger fires (boundary section) -- New timeout errors surfacing to scheduler/worker/CLI callers — deliberate CP semantics (umbrella); error codes flow through existing reply envelopes; callers' retry behavior unchanged -- Guard rejections changing `mark_job_*` return codes — sweep checks all callers treat duplicate-as-success-no-op - -## References -- `.kiro/specs/ha-consistency-invariants/{requirements,design,research}.md` — authoritative umbrella contract (esp. design §Write-Path Classification, §Config Barrier, §Idempotency Guards; research locked decisions 1, 2, 4) -- `.kiro/specs/ha-commit-engine/design.md` — consumed engine interface (commit.h verbatim), migration note "retired by ha-commit-barrier-rollout" -- `.kiro/steering/roadmap.md` — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual -- `.kiro/steering/ha-async-io.md`, `testing.md`, `error-handling.md` — threading, TDD, errnox conventions -- Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic - ---- - -## Gap Re-Audit (2026-06-18) - -> **Mode**: POST-IMPLEMENTATION GAP AUDIT. All tasks 1.1–7.2 are `[x]` (phase=implemented). This pass re-verifies every EARS acceptance criterion against the current on-disk controller source (codegraph + grep, `persistent_ha` working tree) rather than against the design intent, and records residual/deferred items the implementation explicitly left open. -> **Verdict up front**: **CLOSED** — 19/19 acceptance criteria MET in code. The only open items are pre-declared, out-of-boundary residuals (bug-448 ref leak, bug-451 RUSAGE OOB, the CTRL_PRIMARY_COMMITTING fact-rejection deferred to ha-single-leader-role, and the WORKER_UPDATE-via-heartbeat WAIT-classification refinement). None block this spec's contract. - -### Requirement-to-Asset Map (verified file:line) - -| Req | Criterion (short) | Asset (verified) | Verdict | -|-----|-------------------|------------------|---------| -| 1.1 | success reply only after majority commit | `persist_commit_barrier` (persist.c:2069) called post-`job_ctrl` in `do_jobcontrol_op` (job.c:1023-1027), post-`queue_ctrl` in `do_queuecontrol_op` (queue.c:144), post-`worker_ctrl` in `do_hostcontrol_op` (worker.c:712-716), post-AFTER_EVENT in `handle_worker_join` (join.c:130) — reply (`xce_reply`) strictly after the barrier in every case | **MET** | -| 1.2 | commit timeout → error reply | barrier returns `-ETIMEDOUT` (persist.c:2105 → `commit_submit(...,COMMIT_MODE_WAIT,timeout)`), `reply.status = rc` in all four handlers (job.c:1029, queue.c, worker.c:718); `timeout_ms==0` borrows `p->config.sync_timeout_ms` (persist.c:2104-2107) | **MET** | -| 1.3 | not-ready rejection → error reply | engine `-XCE_ERR_NOT_READY` passes through barrier unmodified (persist.c:2105 returns engine code); handler maps to `reply.status` | **MET** | -| 1.4 | no success-interpretable reply pre-commit | handlers send exactly one `xce_reply` after the barrier; integration test `test_commit_barrier_rollout.c` pins "no success reply while entries exceed committed_lsn" (task 3.1/7.2) | **MET** | -| 2.1 | every WAL type classified WAIT/RETAIN | `wal_classify` (persist.c:2004-2054) total switch over all 13 `wal_entry_type` values incl. WAL_NOOP row (persist.c:2037, pre-declared); consulted in all 3 hook fns (`persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun`, codegraph callers) | **MET** | -| 2.2 | unclassified type fails fast / fact-fallback | default arm: `ERROR(...)` log + `assert(0)` under `XCE_TESTING` + `return WAL_MODE_RETAIN` production fallback (persist.c:2045-2053) | **MET** | -| 2.3 | dispatch ACK is receipt-only | `handle_jobdispatch`/`dispatch_a_job_` produce no WAL (research §3); `do_set_preason`/`do_clr_preason` reply-first then RETAIN-append (job.c:1150/1178) — receipt verdict pinned by task 3.3 tests | **MET** | -| 2.4 | exhaustive reply-site verification | task 7.1 audit (tasks.md Impl Note 7.1): ZERO request-shaped stragglers; 8 category-A barrier-gated, 6 category-B receipt-ACK, ~29 category-C sites enumerated and committed | **MET** | -| 3.1 | facts appended, retried, no timeout | RETAIN types → `persist_wal_async` + `commit_submit(...,COMMIT_MODE_RETAIN,0)` (returns immediately, no timeout); hook callers get local-append rc only | **MET** | -| 3.2 | facts accepted during quorum loss | `ha_dispatch_gate` CTRL_PRIMARY no-quorum allowlist adds `OPCode_notify/update/setpreason/clrpreason` (ha.c:2099-2102) before the `-XCE_ERR_NOT_READY` default (ha.c:2105) | **MET** | -| 3.3 | uncommitted facts never discarded | log-is-the-queue: pending purely by `lsn > committed_lsn`; no drop path in the RETAIN seam | **MET** | -| 3.4 | facts commit after quorum restore, no redelivery | `committed_lsn` advances over retained suffix via `commit_on_ack`; task 4.3 integration test asserts no re-delivery, no timeout applied | **MET** | -| 4.1 | config confirmed only after majority | `do_upgrade_config_with_bytes` (config_repl.c:1115) calls `push_config_to_standbys` (config_repl.c:941) which returns barrier rc; success ACK (config_repl.c:1133-1142) only when `rc == 0` | **MET** | -| 4.2 | config timeout → failed to requester | `rc != 0` → `send_upgrade_error(ep,req,c,rc,...)` then `return rc` (config_repl.c:1118-1126), forwarding `-ETIMEDOUT` | **MET** | -| 4.3 | unacked config never committed | `committed_config_tag` advance is inside `push_config_to_standbys`'s success branch only (config_repl.c:963-969, atomic_store gated on majority) | **MET** | -| 4.4 | failed upgrade does not restart | failure path returns before `restart_requested` store + `cbuf_put(&c->wbuf)` (config_repl.c:1159-1161); restart unreachable on failure | **MET** | -| 5.1 | duplicate transition is logged no-op | `mark_job_finish` duplicate-on-terminal → `INFO(...)` no-op (job.c:702); `mark_job_update` no-op update → `INFO` skip (job.c:810) — no AFTER hook, no WAL | **MET** | -| 5.2 | terminal→non-terminal rejected with WARN | shared `job_state_is_backward` predicate (job.c:39); `mark_job_start` rejects backward START (job.c:632-633 WARN); `mark_job_update` rejects backward (job.c:803-804 WARN) | **MET** | -| 5.3 | replay-once equivalence | guards make RETAIN-retry / `restore_job_state` replay converge to single-application state; double-apply test (task 6.1) green | **MET** | - -**Score: 19/19 MET.** - -### Supporting-mechanism confirmations -- **Legacy walAck retirement (design goal, Req 1.x)**: `repl_wait_majority_ack` (repl.c:422) is now configAck-only; walAck delegates unconditionally to `commit_submit` (repl.c:657-659 comment + `repl_set_commit_engine` repl.c:660). configAck branch hardened: `acked_config_lsn >= push_tag` (repl.c:500-501), majority `1 + covered > n_peers/2` over `n_peers` (repl.c:511), `n_peers <= 1` immediate (repl.c:470-472), zero-cover multi-member → timeout (repl.c:529). **MET / no residual.** -- **Config-ACK production ingestion (remediation r1 / former CRITICAL F1)**: `repl_on_config_ack` now has live producers — `repl_ack_decode_and_dispatch` decodes `replconfigpushreplymsg` on the promoted-vote ep (ha.c:3403-3549) and a defensive `cluster.c:630-640` OPCode_reply ingestion. Closes the "bricked multi-member config upgrade" hole. **MET.** -- **Barrier-off-iomgr (remediation r1 / former CRITICAL F2)**: `push_config_transport_only` (config_repl.c:879, non-blocking) serves all best-effort ha.c/join.c streaming sites; `push_config_to_standbys` (transport+barrier) has exactly one caller in vtask context (`do_upgrade_config_with_bytes`). **MET.** - -### Residual / Deferred Gaps (all pre-declared out-of-boundary — none block CLOSED) - -| Item | Tag | Where | Status & Rationale | -|------|-----|-------|--------------------| -| bug-448: `job_ctrl` reference leak | **Constraint** | `reference(j)` at job.c:348 on the not-started branch; normal exit never `dereference`s | NOT FIXED. Pre-existing refcount leak surfaced by task 3.2 adversarial review; explicitly out of this spec's boundary (no behavior in the commit contract). Future fix. | -| bug-451: RUSAGE va_list OOB read | **Constraint** | `update_job_rusages` fires `HOOK_DETAIL_JOB_RUSAGE_UPDATE` with `xce_job_rusages_t*` (job.c:1046/1093-1094); `persist_job_hook_fun` default arm reads vararg 2 as `job_t*` → 4-byte OOB read at offset 56, garbage `state` in every WAL_JOB_RUSAGE payload | NOT FIXED. Reviewer-verified byte-precise; declared a boundary item at task 3.3. Production fix candidate = RUSAGE-specific va_list signature in the persist arm. Does not affect any acceptance criterion (jobid lines up; RUSAGE is a fact, retained either way). | -| CTRL_PRIMARY_COMMITTING still rejects facts | **Missing (deferred)** | `ha_dispatch_gate` COMMITTING branch intentionally untouched (design Non-Goals; ha.c) | DEFERRED to ha-single-leader-role, which deletes the transitional state wholesale. During the brief COMMITTING window facts can still be rejected — known residual, hard-ordered downstream. | -| WORKER_UPDATE WAIT-classification too broad | **Constraint** | `mark_worker_back` (worker.c:589) via heartbeat sweep `check_worker_timeout_` (cluster.c:1288) produces a WAIT-classified WAL_WORKER_CTRL/UPDATE with NO `xce_reply` | NOT a 7.1 violation (no reply ⇒ cannot confirm-before-commit; un-barriered WAIT behaves as RETAIN — commits via ACKs). Future-refinement candidate: narrow WORKER_UPDATE WAIT to the request-bearing detail. Carry to ha-single-leader-role / hardening pass. | -| Half-pushed config on disk after failed upgrade | **Constraint** | `do_upgrade_config_with_bytes` saves to disk + bumps generation before the barrier; failure leaves the new config on the primary's disk but cluster-uncommitted | Accepted (design Decision "Failed config upgrade does not restart"). Reconciled by existing OP_replConfig anti-entropy (out of boundary). Operator sees explicit error; tag stays uncommitted. | - -### Integration Risks (forward-looking, not gaps in this spec) -- **ha-single-leader-role hard ordering**: must land AFTER this spec (it removes CTRL_PRIMARY_COMMITTING and adds WAL_NOOP). The classification guard (persist.c:2049) already forces WAL_NOOP to be classified — pre-declared row at persist.c:2037 (RETAIN) means the totality sweep will not trip when the type is wired. Low risk. -- **worker-rejoin-resync**: shares the `mark_job_*` idempotency guards (job.c:39/632/803). Revalidation trigger: re-verify the backward predicate against resync replay ordering when that spec lands. -- **New error codes to clients** (`-ETIMEDOUT`/`-XCE_ERR_NOT_READY` on control ops): deliberate CP behavior; flows through existing reply envelopes; caller retry semantics unchanged. Confirmed by design + reply.status wiring. - -### Open-Work Options (only for the deferrable refinements — the spec itself needs none) - -bug-451 (RUSAGE OOB) is the only residual with a correctness flavor inside this spec's files, so it carries options: -- **Option A — Fix now (RUSAGE-specific va_list arm in `persist_job_hook_fun`)**: read vararg 2 as `xce_job_rusages_t*` for `HOOK_DETAIL_JOB_RUSAGE_UPDATE`. *Effort: S (≈1 file, 1 arm, 1 ASAN test). Risk: Low — isolated to the WAL_JOB_RUSAGE payload encode; no contract change.* -- **Option B — Defer to a dedicated hardening pass / worker-rejoin-resync**: keep the boundary call as recorded. *Effort: 0 now. Risk: Low-Medium — OOB read persists in production (4-byte over-read, no crash observed; garbage state field in a fact payload that is retained regardless).* -- **Option C — Fold into ha-single-leader-role sweep** (which already touches WORKER_UPDATE classification). *Effort: S, batched. Risk: Low, but couples an unrelated correctness fix to a larger spec's timeline.* - -**Recommendation**: Option A for bug-451 (cheap, isolated ASAN-clean fix); leave bug-448 and the COMMITTING/WORKER_UPDATE items as recorded boundary residuals. The spec's own 19/19 contract is CLOSED — no spec-internal open work. diff --git a/.kiro/specs/ha-commit-barrier-rollout/spec.json b/.kiro/specs/ha-commit-barrier-rollout/spec.json deleted file mode 100644 index 7e508828..00000000 --- a/.kiro/specs/ha-commit-barrier-rollout/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "ha-commit-barrier-rollout", - "created_at": "2026-06-11T09:20:51Z", - "updated_at": "2026-06-13T05:30:00-07:00", - "language": "en", - "phase": "implemented", - "umbrella": "ha-consistency-invariants", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "All 16 tasks implemented 2026-06-12/13 via /kiro-impl autonomous mode (each independently reviewed; task 5.2 required one remediation round for two CRITICAL findings). kiro-impl validate gate (task 7.2): GO \u2014 full controller suite 508 PASS / 0 FAIL / 0 sanitizer aborts under ASAN, independently re-censused. Discovered+fixed bug-447 (key_find found-rc leak \u2014 queue/host control silently unpersisted) and bug-396 (worker fact loss on quorum-loss gate rejection). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied. Next: ha-single-leader-role (deps satisfied)." -} \ No newline at end of file diff --git a/.kiro/specs/ha-commit-barrier-rollout/tasks.md b/.kiro/specs/ha-commit-barrier-rollout/tasks.md deleted file mode 100644 index f275fa18..00000000 --- a/.kiro/specs/ha-commit-barrier-rollout/tasks.md +++ /dev/null @@ -1,124 +0,0 @@ -# Implementation Plan: ha-commit-barrier-rollout - -> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. New tests live in `testsuite/test_commit_barrier_rollout.c` plus targeted updates to `test_repl_majority.c` — this shared-file reality, plus job.c being touched by three groups, is why no `(P)` markers are applied: the plan is sequential. -> **Hard upstream prerequisite**: ha-commit-engine must be implemented and green first — every task below consumes `commit_submit(WAIT/RETAIN)` from `include/commit.h` and never modifies `commit.c` internals. -> Verify exact line numbers with grep before editing (research.md §Verified Code Anchors has the 2026-06-11 verified table; files drift). - -- [x] 1. Write-path classification: table, dispatch, RETAIN routing -- [x] 1.1 Write RED tests for classification totality and RETAIN registration - - Create `testsuite/test_commit_barrier_rollout.c` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (meson.build); declare `wal_commit_mode_t` + `wal_classify()` in src/include/persist.h so tests compile - - Tests: every value of `wal_entry_type` (wal_types.h:44-58) maps to a mode — WAL_JOB_SUBMIT/WAL_QUEUE_CTRL → WAIT, WAL_WORKER_CTRL with detail JOIN/UPDATE → WAIT and detail DEAD → RETAIN, all WAL_JOB_* state types → RETAIN, WAL_CHECKPOINT → LOCAL; an unknown type triggers the XCE_TESTING fail-fast path; a fact-type append through `persist_job_hook_fun` registers the entry with the engine in RETAIN mode (observable via the engine's retained/registered accounting against a stub engine) and returns without blocking - - Observable completion: build succeeds with the new header declarations; all new test cases FAIL (red) against the unmodified persist.c - - _Requirements: 2.1, 2.2, 3.1_ - -- [x] 1.2 Implement wal_classify and the hook-seam dispatch to green - - Implement `wal_classify(wal_type, detail)` in persist.c per the design's verified classification table; consult it in `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1093/1172/1204 anchors): RETAIN types → `persist_wal_async` + `commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0)` registration; WAIT types → WAL_JOB_SUBMIT keeps the existing sync path, queue/worker appends stay async (handler barrier owes completion, task 3); LOCAL → unchanged - - Unclassified guard: `assert` under XCE_TESTING with the offending type logged; production ERROR + RETAIN fallback - - Fact appends never propagate commit-progress errors to hook callers (local append rc only) - - Observable completion: all task 1.1 tests PASS; grep shows no bare `persist_wal_async` call in the three hook functions that bypasses classification - - _Requirements: 2.1, 2.2, 3.1_ - -- [x] 2. Commit barrier helper and legacy adapter retirement -- [x] 2.1 Write RED tests for persist_commit_barrier semantics - - Declare `persist_commit_barrier(persist_t *p, uint32_t timeout_ms)` in src/include/persist.h - - Tests (stub engine + replicator, n_peers=3 unless noted): returns 0 immediately when `committed_lsn` already covers `current_lsn`; returns `-ETIMEDOUT` when ACKs are withheld past the timeout and `committed_lsn` stays unmoved; returns 0 once a covering ACK advances the commit index; `timeout_ms = 0` falls back to `p->config.sync_timeout_ms`; n_peers=1 returns 0 immediately (standalone preserved); `-XCE_ERR_NOT_READY` from the engine passes through unmodified - - Observable completion: new cases FAIL (red) — the helper does not exist yet - - _Requirements: 1.1, 1.2, 1.3_ - -- [x] 2.2 Implement the barrier helper and retire the legacy walAck counting - - Implement `persist_commit_barrier` in persist.c: snapshot `current_lsn` under `p->lock` (mirror persist.c:1019-1021), short-circuit when covered, else `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout)`; return engine code unmodified - - repl.c `repl_wait_majority_ack` (repl.c:386-462 anchor): remove the legacy walAck counting branch (upstream marked it "retired by ha-commit-barrier-rollout") — the walAck path delegates to the engine unconditionally; the function's remaining body is the configAck track (hardened in task 5) - - Observable completion: all task 2.1 tests PASS; grep shows no remaining `acked_lsn > 0` counting and no `n_active == 0 → success` short-circuit reachable from any walAck call path - - _Requirements: 1.1, 1.2, 1.3_ - -- [x] 3. Reply-ordering restructure of request-shaped handlers -- [x] 3.1 Write RED integration tests for commit-first-then-reply per opcode - - Tests drive the four handlers directly (vtask-context test harness, stub standby registry): for each of `do_jobcontrol_op` (not-started-job branch, producing WAL_JOB_UPDATE/WAL_JOB_FAIL), `do_queuecontrol_op`, `do_hostcontrol_op`, `handle_worker_join` — with ACKs withheld the requester reply carries an error (`-ETIMEDOUT`) and no success reply is ever observable while the operation's entries exceed `committed_lsn`; with ACKs supplied the success reply is observable only after `committed_lsn` covers the entries; local validation failures still reply immediately with their existing error codes - - Observable completion: new cases FAIL (red) against the current reply-before-commit handlers (anchors: job.c:932, queue.c:130, worker.c:700, join.c:107-109 — re-verify with grep) - - _Requirements: 1.1, 1.2, 1.3, 1.4_ - -- [x] 3.2 Restructure the four handlers to green (incl. bug-447 fix: queue/host control now actually persist+replicate) - - Apply the uniform pattern: `rc = <ctrl fn>; if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)` — in `do_jobcontrol_op` (job.c:921-934), `do_queuecontrol_op` (queue.c:119-132), `do_hostcontrol_op` (worker.c:689-702); in `handle_worker_join` (join.c:33-119) insert the barrier between the AFTER_EVENT hook and the success reply, keep error replies immediate, keep `on_worker_join` after the reply - - Barrier runs after entity locks are released in every handler (the ctrl fns unlock before returning) - - Observable completion: all task 3.1 tests PASS; `handle_jobsubmit` untouched (reference implementation) - - _Requirements: 1.1, 1.2, 1.3, 1.4_ - -- [x] 3.3 Pin the receipt-ACK verdicts with tests - - Tests: `handle_jobdispatch` (job.c:459-491) replies immediately AND dispatch processing appends no WAL entries (classification guard stays silent through a dispatch cycle; `notify_dispatch` outcomes unaffected); `do_set_preason`/`do_clr_preason` and the `do_update_op` rusage path remain receipt-style replies whose subsequent fact appends are RETAIN-registered - - Document the verdict in the test comments with the umbrella Reply-Ordering Contract reference - - Observable completion: verdict tests PASS without modifying those handlers; any future WAL append from dispatch processing trips the classification guard in CI - - _Requirements: 2.3_ - -- [x] 4. Fact admission during quorum loss and retention proof -- [x] 4.1 Write RED tests for the dispatch-gate fact allowlist - - Tests against `ha_dispatch_gate` (ha.c:1527 anchor) with CTRL_PRIMARY and write quorum absent: `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` return 0; `OPCode_jobsub`, `OPCode_jobcontrol`, `OPCode_queuecontrol`, `OPCode_hostcontrol`, `OPCode_jobdispatch`, `OPCode_join` still return `-XCE_ERR_NOT_READY`; with quorum present everything returns 0 as today - - Observable completion: the four fact-opcode cases FAIL (red) against the current allowlist - - _Requirements: 3.2_ - -- [x] 4.2 Implement the allowlist to green - - Add the four fact-bearing opcodes to the CTRL_PRIMARY no-write-quorum switch (ha.c:1537-1556 anchor); CTRL_PRIMARY_COMMITTING branch untouched (deleted wholesale by ha-single-leader-role) - - Observable completion: all task 4.1 tests PASS; existing gate tests for request opcodes unchanged - - _Requirements: 3.2_ - -- [x] 4.3 Write the fact-retention-across-quorum-loss integration test (bug-396 end-to-end evidence complete) - - Scenario (stub registry, n_peers=3): disconnect standbys (quorum lost) → drive worker-fact paths (job start/finish via `do_notify`-level calls) → appends succeed with no error, entries pending (`lsn > committed_lsn`), nothing dropped; reconnect + supply covering ACKs → `committed_lsn` advances over the retained suffix with no re-delivery of the facts; assert no timeout was ever applied to the fact entries - - Observable completion: test PASSES against tasks 1.2 + 4.2 output (if it fails, fix the RETAIN/gate path, not the test); retained-depth observability counter visible in test logs - - _Requirements: 3.1, 3.3, 3.4_ - -- [x] 5. Configuration change commit barrier -- [x] 5.1 Write RED tests for configAck arithmetic and the upgrade failure path - - Unit tests (repl.c configAck branch via `repl_wait_majority_ack`): a standby's `acked_config_lsn` from a previous tag does not count toward the current push; n_peers=3 with zero covering acks → `-ETIMEDOUT` (the `n_active == 0 → success` short-circuit dies on this branch); self + one covering standby (majority of n_peers=3) → 0; n_peers=1 → immediate 0; update `test_repl_majority.c` configAck cases to the per-push arithmetic - - Integration tests (config_repl.c): `push_config_to_standbys` returns `-ETIMEDOUT` with acks withheld and 0 with majority acks, push count via out-param; on failure the `upgradeConfig` requester receives an error reply (send_upgrade_error shape), `committed_config_tag` is unmoved, and `restart_requested` stays clear; on success the existing reply + restart behavior is unchanged - - Observable completion: new cases FAIL (red) against the current logged-only result (config_repl.c:916-929) and unconditional success ACK (config_repl.c:1062-1073) - - _Requirements: 4.1, 4.2, 4.3, 4.4_ - -- [x] 5.2 Implement the config barrier to green (+ remediation r1: wired production config-ACK ingestion, kept barrier off iomgr thread) - - repl.c: configAck branch takes the in-flight push tag (signature change in src/include/repl.h), counts `acked_config_lsn >= push_tag`, majority `1 + count > n_peers / 2` over configured members, `n_peers <= 1` immediate success - - config_repl.c: `push_config_to_standbys` gains `int *acked_out` and returns the barrier result, passing `new_tag` to the wait; `do_upgrade_config_with_bytes` (config_repl.c:1056-1094 anchor) on non-zero return sends `send_upgrade_error`, skips the success ACK and skips the restart request; `committed_config_tag` advance stays success-branch-only - - Observable completion: all task 5.1 tests PASS; a failed upgrade leaves the admin tool with an explicit error and the primary running on the old committed tag - - _Requirements: 4.1, 4.2, 4.3, 4.4_ - -- [x] 6. Idempotency and ordering guards for job transitions -- [x] 6.1 Write RED tests for mark_job_* guards - - Tests: duplicate FINISH on a DONE job → state unchanged, INFO-level no-op, no AFTER_EVENT hook fired (no WAL entry appended); START on a DONE/EXIT job → rejected with WARN, terminal state preserved; UPDATE attempting to clear terminal bits → rejected; double application of the same transition yields state identical to single application (replay equivalence); fail-after-finish keeps the existing PERR-merge behavior (job.c:702-716 anchor); guard rejections return a code all existing callers treat as success-no-op - - Observable completion: new cases FAIL (red) against the unguarded `mark_job_start/finish/fail/update` (job.c:583/638/696/726 anchors) - - _Requirements: 5.1, 5.2, 5.3_ - -- [x] 6.2 Implement the guards to green - - Add transition guards at the top of each `mark_job_*` (under the caller-held job WRITE lock, before BEFORE_EVENT fires): duplicate → log + return no-op; terminal → non-terminal → WARN + reject preserving state; forward transitions unchanged - - Audit all callers (`on_job_*`, `job_ctrl`, `update_job_rusages`) for the new return semantics - - Observable completion: all task 6.1 tests PASS; existing job lifecycle tests green - - _Requirements: 5.1, 5.2, 5.3_ - -- [x] 7. Exhaustive reply-site sweep and final validation -- [x] 7.1 Run the exhaustive xce_reply audit over all control paths - - Grep ALL `xce_reply` call sites under src/controller/ (excluding testsuite/) — research.md lists the design-time population (job.c, queue.c, worker.c, join.c, opupd.c, cluster.c, config_repl.c, ha.c, opget.c, reply.c, task.c); re-run the grep fresh since lines drift - - Classify every site into exactly one category: post-commit confirmation (restructured in task 3 / job submit / config success), receipt-only ACK for fact-shaped delivery (verdicts from task 3.3), or read-only/replication/error reply (no state-changing entries produced) - - Fix any site that confirms a request-shaped state change before commit (a missed handler silently violates umbrella 2.1); commit the audit table (site → category → evidence) in the task completion notes - - Observable completion: zero unclassified or violating reply sites in the recorded audit; any fixes covered by tests added to `test_commit_barrier_rollout.c` - - _Requirements: 2.4, 1.1, 1.4_ - -- [x] 7.2 Full regression and barrier-completeness validation - - Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix regressions from the reply restructure, gate allowlist, configAck hardening, or guards - - End-to-end ordering assertion: across all restructured opcodes in one run, no success reply is observable while the operation's entries exceed `committed_lsn` (integration harness from 3.1 driven as a suite); fact retention test (4.3) and config barrier tests (5.1) green in the same run - - Confirm downstream readiness: classification guard active, sweep audit recorded — the hard precondition for ha-single-leader-role ("barrier before state removal") is satisfied - - Observable completion: full testsuite green under ASAN; audit + ordering evidence recorded - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3_ - -## Implementation Notes - -- (1.1) **commit_submit(COMMIT_MODE_RETAIN) is registration-free** (commit.c:233-239, reviewer-confirmed): commit state derives purely from registry watermarks at ACK time, so RETAIN routing has NO per-entry engine observable. Task 1.2's RETAIN registration call is therefore contract-documentation, not bookkeeping — the observable contract is: non-blocking return, local-append-rc-only propagation, pending via lsn > committed_lsn, ACK-driven advance. Test 8's behavioral pins execute only at GREEN (Unity longjmp at the wal_classify carrier assert). -- (1.1) The unknown-type fail-fast test forks and asserts WTERMSIG==SIGABRT — task 1.2 must implement the XCE_TESTING guard as assert/abort (or the test needs a matching tweak). -- (1.1) wal_entry_type has 13 values (wal_types.h:44-58), no WAL_NOOP yet (lands in ha-single-leader-role, which must add a classification row + sweep entry — the totality sweep will catch it). -- (1.1) Suite baseline is ZERO failures since ha-commit-engine 6.1. detect_leaks=0 remains the harness norm (6 inventoried pre-existing leak groups). -- (1.2) CENSUS METHOD: only ~9 of 48 groups emit Unity summary blocks; the rest report per-test PASS lines only — count distinct Unity results (~465) and grep FAIL lines, never sum summary blocks. commit_submit(RETAIN) early-returns BEFORE taking ce->lock (commit.c:233-239) — registration is lock-free, safe with entity write locks held. wal_hook_append_classified is the single classified-append seam; appended_lsn read = current_lsn - 1 under p->lock (benign race, commented). -- (7.1) AUDIT RESULT (reviewer-confirmed exhaustive, producer-side + opcode-dispatch): **ZERO request-shaped stragglers** — every WAIT-class WAL producer reaches its requester reply only through a commit barrier. 8 category-A barrier-gated sites: handle_jobsubmit success+reject (in-hook sync-majority, job.c:600/609), do_jobcontrol_op (job.c:1023), do_queuecontrol_op (queue.c:152), do_hostcontrol_op (worker.c:721), handle_worker_join (join.c:125), do_upgrade_config success (config_repl.c:1141). 6 category-B receipt-ACKs (RETAIN facts, pinned by 3.3/4.3): jobdispatch, update-resvals, update-rusage, setpreason, clrpreason, repl_config_push standby-ack. Remaining ~29 sites category C (read-only/replication/error/control-plane). All state-changing WAL flows through wal_classify at the 3 hook seams — no handler bypasses with a bare persist_wal_async/sync. - - **Reviewer precision correction**: the audit's "worker_ctrl single-caller" claim is inaccurate — WAL_WORKER_CTRL/WORKER_UPDATE (classified WAIT) is ALSO produced by mark_worker_back (worker.c:589) via the heartbeat liveness sweep check_worker_timeout_ (cluster.c:1288), UN-barriered. NOT a 7.1 violation: that path issues NO xce_reply (controller-detected liveness fact, not a request confirmation), so it cannot confirm-before-commit; an un-barriered WAIT entry there behaves identically to RETAIN (commits later via ACKs). **Future-refinement candidate (not blocking)**: WORKER_UPDATE's WAIT classification could narrow to the request-bearing detail since the sweep path wants RETAIN semantics — carry to ha-single-leader-role or a hardening pass. -- (6.1) For 6.2 (reviewer-confirmed): the "UPDATE clears terminal bits" guard arm (Req 5.2) is UNREACHABLE in production — mark_job_update's 3 callers (job.c:400 SIGCONT clears only PSUSP|USUSP, :410 SIGSTOP ORs PSUSP, :876 postdone ORs PDONE/EXIT) never produce terminal→non-terminal. 6.2 will most naturally share ONE backward-guard predicate across START+UPDATE; the START-on-terminal RED test exercises it, and the forward-update PIN protects the one reachable terminal UPDATE (postdone PDONE-merge) from over-rejection. on_job_* wrappers return 0 unconditionally — 6.2's caller audit must keep them returning 0 (success) on guarded-out transitions; the pins lock this. Guard placement: top of each mark_job_*, under the caller-held WRITE lock, before BEFORE_EVENT. Live anchors drifted: mark_job_start:592, finish:647, fail:705 (PERR-merge :711-721), update:735. -- (5.2) **REJECTED then remediated (r1)** — two CRITICAL findings the first pass missed, both fixed: (F1) the config-ACK signal was never produced in production (repl_on_config_ack had zero callers; the standby's replconfigpushreplymsg reply was dropped on the primary) so the new authoritative barrier would have BRICKED every multi-member config upgrade — this was the unlanded half of ha-config-bootstrap task 8.1. Fixed by a replconfigpushreplymsg decode branch in repl_ack_decode_and_dispatch (ha.c) → repl_on_config_ack on the repl_add_standby-registered promoted-vote ep (production-normal path), + a defensive cluster.c OPCode_reply ingestion for inbound standbys (no-ops against unregistered slots, forward-looking). Schema unchanged (ReplConfigPushReply already carried versionTag). (F2) the 500ms barrier wait reached the iomgr epoll thread via 5 ha.c push sites. Fixed by exporting push_config_transport_only (non-blocking fan-out) for all best-effort ha.c/join.c streaming sites; push_config_to_standbys (transport+barrier) now has exactly ONE caller — do_upgrade_config_with_bytes in vtask context. LESSON: the design's "handlers run on vtask threads" claim was only true for cluster.c-dispatched opcodes, not the ha.c peer-channel push sites the design never analyzed (ha.c was outside its modified-files list). Wire-path test drives real capnp ACK bytes through the production decode branch. -- (3.3) **bug-451 DISCOVERED (reviewer-verified byte-precise, NOT fixed — boundary)**: update_job_rusages fires HOOK_DETAIL_JOB_RUSAGE_UPDATE with (cluster_t*, xce_job_rusages_t*) (job.c:1014-1015) but persist_job_hook_fun's default arm (persist.c:1355-1359) reads vararg 2 as job_t* and dereferences j->job.state at offset 56 — one past the end of the exactly-56-byte xce_job_rusages_t → 4-byte OOB read + garbage state field in every WAL_JOB_RUSAGE payload (jobid lines up by accident). Production fix candidate for task 7.1's sweep or a follow-up: persist arm needs a RUSAGE-specific va_list signature. Census now 488. -- (3.2) bug-447 FIXED via rc normalization inside the three ctrl fns (real errors/misses preserved; reviewer-verified per-fn). Semantics change shipped intentionally: idempotent no-op control actions now return 0 (was leaked 1); queue/host mutations now append WAL + replicate. Join barrier failure → error reply + on_worker_join SKIPPED (coherent reading of design; worker retries via idempotent rejoin; documented at join.c:108-113). PRE-EXISTING ref leak: job_ctrl takes reference(j) (job.c:317) but normal exit never dereferences (bug-448) — out of scope, future fix. -- (3.1) **bug-447 CONFIRMED (reviewer-verified to librtti disassembly)**: key_find returns 1 on found (unnormalized from xkvmgr_find/kvtab_find); queue_ctrl (queue.c:61, gate :106) and worker_ctrl (worker.c:644, gate :678) leak rc=1 → AFTER_EVENT never fires on the found path → **queue/host control silently unpersisted+unreplicated in production today**; job_ctrl appends via mark_job_* hooks but still replies status 1. Fix point = 3.2's handler rc normalization (key_find is in binary librtti, out of repo). RED tests pin it. -- (3.1) Test link gained job.c/queue.c/join.c (test-only meson change) + 8 inert link stubs in test_commit_barrier_reply.c (none on tested paths — reviewer-traced). Reply capture = zeroed stack endpoint ep->outq inspection (test_config_repl.c:2210 precedent; endpoint_update_epoll no-ops when iomgr NULL). RED-window orphaned fsync threads self-resolve at GREEN (reviewer finding 5). -- (2.2) Legacy walAck counting RETIRED: walAck track = entry_lsn==0 → 0 (LSN-0 interim) else unconditional commit_submit (ce==NULL → -EINVAL, fail-loud). configAck track byte-equivalent (task 5's surface intact, incl. its n_active==0 shortcut and acked_config_lsn>0 counting — both die at task 5). persist_commit_barrier: ce==NULL → 0 (vacuous, documented). test_repl_unit's two lsn=0 walAck callers are now vacuous for quorum counting (INFO for task 5/7.2). CENSUS = 472 (465 at 1.2 + 7 from 2.1). -- (2.1) BARRIER WATERMARK RESOLVED (reviewer-judged): design's literal "Snapshot lsn = p->current_lsn" is loose phrasing contradicted by its own anchors — the coherent contract is watermark = current_lsn - 1 (last appended; persist.h:71 invariant committed_lsn <= current_lsn - 1 makes the literal reading unsatisfiable). Tests pin current_lsn - 1 with off-by-one killing power in BOTH directions; task 2.2 must implement that. Resolved contract recorded in persist.h doc comment. diff --git a/.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md b/.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md deleted file mode 100644 index 2e48a29f..00000000 --- a/.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md +++ /dev/null @@ -1,123 +0,0 @@ -# Investigation: committed jobs lost across controller failover (bug-795 family) - -> Live investigation 2026-06-20. Status: **root-cause located, fix not yet applied.** This is the long-standing data-loss bug (PREDATES libxce-failover — confirmed by the user; libxce-failover + xce-test-e2e-primitives were *test-infrastructure* improvements, not the durability fix). Owning spec: **ha-commit-durability** (REOPENED for bug-795). Adjacent: ha-convergence-recovery, ha-single-leader-role, ha-peer-connection-reuse. - -## Symptom -Running the **controller failover test** (kill primary / restart nodes while a client submits jobs), committed jobs that were ACKed (`jobsubmit ok`) and visible in `joblist` **disappear** after enough kill/restart cycles. In the captured run, jobs survived terms 2→4 then vanished at the term-4→5 transition; `ctrlstatus` then showed them absent. NOT a `committedLsn` display bug — the data is genuinely gone. - -## DECISIVE EVIDENCE — it is real data loss, not a commit-index issue -Decoded the raw WAL bytes at `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}/data/wal/seg-0000000000000001.wal` (format: `wal_entry_hdr_t` 40B `{term,lsn,timestamp,type,payload_len,crc,prev_crc}` + payload; `src/include/wal_types.h`): - -| Node | WAL entries (physical) | JOB_SUBMITs | State | -|------|------------------------|-------------|-------| -| ctrl-1 | LSN **1–15** (starts NOOP@1) | **12** | full history — but **DOWN** (`connect peer failed -111`) | -| ctrl-2 | LSN **16–18** (starts NOOP@16) | 2 | live, **missing 1–15** | -| ctrl-3 | LSN **13–18** (starts NOOP@13) | 4 | live, **missing 1–12** | - -The committed prefix (LSN 2–12 = the lost jobs) is **physically absent** from both live nodes; it exists only on the down node. `snap/` dirs are **empty** on all nodes (no snapshot). - -## DECISIVE EVIDENCE 2 — catch-up streaming DROPS payloads (bug-847, REPRODUCED 2026-06-20) -A live restart of ctrl-1 (kill ctrl-1 → restart) showed it rejoin and catch up to the cluster length `[1..14]` with **correct LSN/type/CRC chain but EMPTY payloads** — `waldump` reports every JOB_SUBMIT as `PLEN=0 PCRC=00000000 jobid=?`, while ctrl-2/ctrl-3 still hold full 872-byte payloads. So the WAL is **structurally intact but content-empty** — silent corruption that passes every check that doesn't decode the payload (the `PCRC` column is what exposed it). This, not just the missing-prefix gap, is a concrete committed-job-loss mechanism: if the empty-payload node ever leads, the empty entries become authoritative. - -**Root cause (reproduced in a unit test):** the catch-up batch wire path (`xce_wal_batch_encode_raw`/`wal_entry_to_wire` in `src/lib/request.c`) builds a `WalCatchupEntry` struct-list whose per-element `payload :Data` is allocated via `capn_new_list8(cs=root_seg,…)`. When the batch outgrows its first capnp segment, the per-element payload `Data` pointer is **lost across the segment boundary** → on decode `extract_data_bytes(we.payload)` reads `CAPN_NULL` → `raw_payload_len=0` → the receiver writes a payload-less entry. **`test_wal_batch_roundtrip_large_payloads` (14×872B) FAILS "Expected 872 Was 0"; the 2-entry round-trip PASSES** — the threshold is message SIZE (segment overflow), not entry count. **Live replication (`repl_broadcast`) carries payloads fine — only the catch-up/rejoin path corrupts**, which is exactly why loss surfaces only after a node restarts and catches up. Candidate fix: carry all sub-entry payloads as ONE top-level capnp `Data` blob + lengths (single `Data` round-trips correctly), sidestepping per-element cross-segment pointers. RED test is in `src/lib/testsuite/test_wal_wire.c`. - -## ROOT CAUSE (mechanism, code-grounded) -A node that (re)joins with an **empty WAL** is brought up to the cluster's **current committed watermark by WAL streaming `[watermark..committed]` — NOT from LSN 1 — and there is no snapshot to backfill `[1..watermark-1]`.** So it ends up with a permanent prefix gap. When the failover test eventually has the only full-log node down while the others rejoined at high watermarks, the committed jobs are gone for good. - -Why the `seg-...001` file contains LSN 16: a node booting empty defaults `current_lsn=1` (`persist.c:429`); the first write lazily calls `open_new_segment` which names the file by `current_lsn` → `seg-...001` and **`O_TRUNC`s** it (`persist.c:145`); catch-up then writes entries carrying explicit high LSNs (16…) from offset 0. So "writes from the beginning of the file" = **booted empty + caught up from the watermark**, not an in-place overwrite of recovered data. (`replay_wal` at `persist.c:1422` is correct — it opens existing segments **read-only** and advances `current_lsn` past what it recovers.) - -## OPEN FORKS (what to settle next) -1. **Test procedure — clean vs preserve.** `scripts/cleanup.sh` does `rm -rf .../data/*`; `scripts/start.sh` only creates `data/` if missing (preserves). The on-disk files (named `001`, starting at 16/13) match the **cleaned** case. **Confirm with the user whether the failover test cleans data dirs before restart or restarts with data intact.** (Still worth confirming for the exact trigger ordering, but no longer the architectural question — see VERDICT.) - - If preserved + node still returns gapped → **recovery bug** (`replay_wal` failing to restore). - - If cleaned → **catch-up never backfills the prefix + no snapshot** = the architectural gap (CONFIRMED below). -2. **Catch-up start point — ✅ SETTLED (2026-06-20).** Traced. `handle_ctrlhello` (`config_repl.c:1436`) is connection-setup ONLY; catch-up is driven by the heartbeat decision loop. The standby reports its **REAL `applied_lsn`** — `ha_process_standby_state(ha, peer_idx, sr.config_tag, sr.applied_lsn, …)` at `ha.c:3518`, where `sr.applied_lsn` is the standby's own state-report value (an empty standby reports 0). The decision `ha_evaluate_repl_action` (`ha.c:1811`) is CORRECT: `standby_lsn+1 < oldest_wal_lsn → SEND_SNAPSHOT`, `else standby_lsn < committed → STREAM_WAL`. So advertise is NOT the bug — a fresh standby correctly asks for a snapshot when the primary's WAL no longer starts at 1. -3. **Snapshots — ✅ SETTLED: THIS IS THE ROOT CAUSE (2026-06-20).** `write_snapshot` (`persist.c:1051`) writes a **16-byte SENTINEL**: `"XCESNAP\0"` (8) + `checkpoint_lsn` (8) — its own comment says *"Full Cap'n Proto serialization is Task 11."* It carries **ZERO job/queue entities.** `load_snapshot` (`persist.c:1267`) just does `p->checkpoint_lsn = p->current_lsn = stored_lsn` — it **advances the watermark past entries it never received, restoring nothing.** And `persist_checkpoint` → `trim_wal_segments` (`persist.c:1090`) **deletes WAL segments below the checkpoint.** So checkpoint = "trim the committed prefix + write an empty backup": a WAL-eating loop with no recoverable safety net. This is the "entity-bearing snapshots" gap (ha-convergence-recovery backlog) — and it makes any empty-rejoin/prefix-trim below committed **unrecoverable BY DESIGN.** - -## VERDICT (2026-06-20) — architectural root confirmed, code-grounded end-to-end -The loss is **not** an advertise/start-point bug and **not** a recovery bug. It is the **absence of entity-bearing snapshots** combined with **prefix-trimming checkpoints**. Full mechanism: -1. A fresh/empty member (applied_lsn=0), or any member catching up against a primary whose own `oldest_wal_lsn > 1` (its prefix was trimmed by a prior checkpoint). -2. `ha_evaluate_repl_action` → `SEND_SNAPSHOT` (when `1 < oldest_wal_lsn`) or `STREAM_WAL` from the high watermark. -3. SEND_SNAPSHOT path: `persist_checkpoint` writes the **sentinel** (no entities) + **trims the prefix WAL**, then ships the sentinel; receiver `persist_load_snapshot` sets `current_lsn = checkpoint_lsn` with **no entity restore**. -4. STREAM_WAL path: `ha_stream_wal_to_standby` streams `[standby_lsn+1 .. committed]`; `ha_apply_wal_catchup_entry`/`ha_apply_single_entry` **physically persist** those entries (consistent with ctrl-3's on-disk seg holding LSN 13–18) — but the prefix `[1..oldest-1]` is **never streamed and has no snapshot to backfill it.** -5. Net: the member's watermark reaches `committed` while the committed-prefix job entities are gone. **Permanent loss, BY DESIGN.** - -**Fix layer (recommendation — option (b), question-the-architecture):** implement **entity-bearing snapshots** — `write_snapshot` Cap'n-Proto-serializes all live job/queue entities at `checkpoint_lsn`; `load_snapshot` deserializes + `restore_job`/restore_queue them BEFORE advancing `current_lsn`; and `trim_wal_segments` must NOT trim below the LSN covered by a durable entity-bearing snapshot (install-snapshot-before-stream on the receive side). Option (a) "stream `[1..committed]`" only works while some node still retains LSN 1 (false after any trim), so it is a partial mitigation, not the fix. This is the **ha-convergence-recovery** "entity-bearing snapshots" item; route the fix there (or a new `ha-entity-snapshots` spec) under the ha-commit-durability umbrella. - -## SECONDARY FINDINGS -- **Double `WAL_JOB_SUBMIT` per job (bug-845, 2026-06-20, via waldump).** Every committed job appears as TWO consecutive entries with the SAME jobid (ctrl-1: 2,3 / 4,5 / … ; ctrl-3: 14,15 ; ctrl-2: 17,18). **Ruled out array expansion:** `job_submit` assigns array members DISTINCT ids (`base_id+i`, job.c) and the `Job` capnp has a single jobid field, so identical-jobid pairs are NOT two array tasks — a single submit is double-appending. **CONFIRMED a literal byte-for-byte double-append** by the enhanced waldump (payload-CRC + subtime columns, 2026-06-20): each pair is byte-identical — same PCRC (LSN 2,3 both `6d79074e`; 4,5 `c25ac163`; …) and same `subtime` (e.g. 1781959730), so all 12 ctrl-1 submits = 6 jobs × 2. A failover retry would re-stamp `subtime=time(NULL)` and differ → **retry definitively ruled out; it's the SAME encoded `job_t` written twice in one submit flow.** It is replicated *consistently* (F=…555020288 at 14,15 on BOTH ctrl-1 and ctrl-3; G at 17,18 on BOTH ctrl-2/3), so not itself divergence, but it doubles LSN consumption and is a latent cross-node divergence risk if the duplicate count ever varies by primary. Next: trace `handle_jobsubmit`→`persist_job_hook_fun` AFTER_EVENT loop (jlist holding the job twice, or a double hook-fire). -- **Raft cross-node consistency check (user request, 2026-06-20).** Per-LSN comparison of the three dumps: at every OVERLAPPING index the entries AGREE — ctrl-1∩ctrl-3 = {13:NOOP, 14:F, 15:F}; ctrl-2∩ctrl-3 = {16:NOOP, 17:G, 18:G}. So **State Machine Safety (no two members hold different entries at the same index) is NOT violated** in this snapshot; no job sits at different LSNs on different nodes. The broken guarantee is **Leader Completeness / durability**: jobid E=…394588160 was committed+acked at LSN 11–12 (present on ctrl-1) but is ABSENT from the surviving majority (ctrl-2/3) — a committed entry lost (exactly the root-cause mechanism: it was below the survivors' empty-rejoin catch-up start, with no entity snapshot to backfill it). Conversely G=…630517760 is on the survivors but absent on ctrl-1, which was already down before G — losing an entry a crashed minority-node never had is Raft-legal. So the cluster ended with two committed histories sharing only the F segment: ctrl-1's {…,E@11-12,F@14-15} vs survivors' {F@14-15,…,G@17-18}. -- **`term=0` on every WAL entry.** `p->term` (the `hdr.term` source) is **never assigned** the election term anywhere (grep for `p->term =` / `->term =` in persist.c/ha.c returns nothing). So per-entry term is dead. XCE matches logs via the `prev_crc` chain, so possibly benign — but it means **no term-based divergence detection**. Worth a hard look (could let divergent same-LSN entries be treated as matching). -- **Cluster cannot hold a primary** under the test: peer connect `-111` (ECONNREFUSED), RDHUP, rapid term churn 2→5 in ~7 min, elections logged `safe_committed_lsn=X (from 1 granted voters)`. Logged separately in `ha-convergence-recovery/research.md` (2026-06-20 live finding). -- **Crash fixed en-route (unrelated):** `xce_strerror_r` global-buffer-overflow on HA error codes — bug-827, fixed in `src/lib/utils.c` (commit 1aa0761). - -## Is the ha-heartbeat-committed-propagation change implicated? -The loss **predates** it, so it is **not the origin**. But two real concerns it introduces (it writes `last_known_committed_lsn` per-heartbeat, decoupled from WAL possession): -- `ha_enter_real_election` (`ha.c:4137`) truncates WAL above `last_known_committed_lsn` — a heartbeat-fed value now feeds a truncation path (violates the Req 3.1 "advisory-only" boundary I claimed; guarded by `if (committed>0)` so 0 is safe, but still a real boundary leak). -- If the rejoin/catch-up advertised position is derived from `last_known_committed_lsn`, the per-heartbeat freshness would **widen** the prefix gap (fresh node learns committed=13 via heartbeat → advertises 13 → streams from 14). **This is exactly Open Fork #2 — read the CtrlHello advertise path; it also settles aggravation.** -- Candidate-advertise is SAFE: `ha_candidate_advertise_lsn` (`ha.c:2598`) returns bare `persist_durable_floor`, NOT last_known — so a behind node cannot fake election freshness. - -## REPRO / ARTIFACTS (on this host) -- Live deploy with the failed run: `/tmp/xce-ha-deploy/` — controller logs `logs/ctrl-{1,2,3}.log`, WAL `controller/ctrl-N/data/wal/seg-...001.wal`, empty `snap/`. -- Offline WAL decoder (Python, ad-hoc) — walks `wal_entry_hdr_t`, lists `lsn|term|type|payload_len`, flags submits by header timestamp. (To be productized as `waldump`, below.) -- Deployed binary includes controller-status (responderTime) + ha-heartbeat-committed (the `ha.c:4967 "learned committed_lsn=X from heartbeat"` logs are the consume path). - -## NEXT STEPS -1. **Build the WAL/snapshot inspection tooling in `xce_test`** — ✅ `waldump` DONE (2026-06-20). `snapdump`/`walcompare` still TODO. Gap-location is now a one-liner per run. -2. **Read the CtrlHello catch-up advertise/start-point** (`config_repl.c` handle_ctrlhello + the standby's CtrlHelloRequest build) — Open Fork #2. -3. **Confirm whether snapshot creation exists** or is a stub — Open Fork #3 (likely the architectural root: no entity-bearing snapshot ⇒ no prefix backfill). -4. Decide fix layer: (a) full-WAL stream `[1..committed]` when a node has no snapshot, (b) implement entity-bearing snapshots + install-before-stream, or (c) prevent empty-rejoin from advancing committed without holding the entries. Likely a **question-the-architecture** moment (catch-up assumes a snapshot safety net that does not exist). - -## DECISIVE EVIDENCE 3 — bug-851 REPRODUCED & ROOT-CAUSED on an owned cluster (2026-06-21) - -Stood up a fresh local 3-node cluster (`/xce-deploy-ha`, binary built with the 848/849/850 fixes) and drove two scenarios with full log access. Artifacts: `repro-851/{ctrl-1,ctrl-2,ctrl-3}.log` + `waldump-ctrl-{1,2,3}.txt`. - -**Scenario 1 — clean single failover: PASS (848/849/850 validated end-to-end).** -6 jobs → identical `1..13` on all 3 (matching PCRCs). Kill primary ctrl-1 → ctrl-3 promoted: `cluster_init_phase2: persist_recover done — recovered durable_floor=13 … applied_lsn=14 advertised` (bug-849 recovery-on-promotion + bug-848 advertise-real-LSN both fire). 4 jobs on new primary → ctrl-2 converges `14..22`. Restart old primary → recovers `1..13`, catches up `14..22`. **Final: all three WALs byte-identical `1..22`, identical PCRC fingerprint `cc5d361e…`, 0 gaps.** - -**Scenario 2 — chaotic leadership churn (3× kill-the-primary): FAIL — silent committed-prefix divergence.** -After 3 rounds, all three report `committedLsn=54`, all `lsn range 1..54`, 0 gaps, CRC intact — **but the logs DIVERGE at LSN 45–54**: -- ctrl-1: `45=NOOP`, `46..53 = faead246/be0059bf/fe7a3d0b/37d54e4d` → fp `bfeb26ad…`, 25 jobids -- ctrl-2: `45=NOOP`, `46..53 = d32ac747/97c04cbe/d7ba280a/1e155b4c` (DIFFERENT jobs) → fp `216af434…`, 25 jobids -- ctrl-3: `45 = 4c6d24e6` (a JOB, no noop), `45..52 = 4c6d24e6/0887af1f/48fdcbab/8152b8ed` → fp `865304ec…`, 26 jobids - -Three different "committed" tails, each node reporting them committed. This violates the Raft committed-prefix invariant (a committed entry must be byte-identical on every member). - -**ROOT CAUSE (code-grounded, three compounding defects):** -1. **Term not monotonic across the cluster.** Election term = *locally-persisted* term + 1; PreVote explicitly does NOT persist/propagate ("starting PreVote round … no persist"). Under chaotic kills a leader wins term N, appends its own-term noop + jobs, and is killed **before term N is durably propagated to a majority**. A restarted node recovers a *stale* term and elects at the **same** N+1. Logs show **two leaders at term 3** (ctrl-3 @08:40, ctrl-1 @08:47:39) and **two leaders at term 4** (ctrl-2 @08:47:52, ctrl-3 @08:48:04) — each appending its own-term noop at the **same LSN 45**. Persisted terms ended split: ctrl-1=4, ctrl-2=4, ctrl-3=3. -2. **The WAL entry term field is DEAD (always 0).** `wal_entry_hdr_t.term` exists (wal_types.h:30) and the replication envelope carries `term` (typemap.h:381), but the writer does `hdr.term = p->term` (persist.c:588/726/1040) and **`p->term` is NEVER assigned from `ha->term` anywhere in the controller** (zero matches). Every entry, every term, persists `term=0`. waldump confirms term=0 cluster-wide. -3. **No (term,lsn) reconcile → divergent tails never truncated.** Because the WAL has no usable per-entry term, rejoin-reconcile (`ha_convergence.c`) cannot do Raft log-matching: entry 45@ctrl-1 (term 3) and 45@ctrl-2 (term 4) look identical (both term 0, both JOB_SUBMIT/NOOP), so the divergent suffix is taken as already-present and is never overwritten. `ha_compute_safe_committed_lsn` then advances committed (44, 53) "from 1 granted voter" over content that other members never received. - -Net: under leadership churn, acked entries written by a short-lived leader become a permanent divergent tail, and `committed` advances over it → silent loss + cross-node divergence. This is distinct from the catch-up payload-drop (bug-847) and the empty-rejoin gap (bug-828); it is a **Raft-safety gap in term establishment + WAL term recording**. - -**Keystone fix candidate (smallest correct lever):** stamp `hdr.term` from the live raft term (`ha->term`) on the primary append path + carry it through the replication envelope + persist it, THEN make reconcile truncate any local suffix whose (term,lsn) does not match the leader's — standard Raft log-matching. Must be paired with making the election term durably-established-before-acting (no two leaders per term). **HA-core + persistence → full spec workflow required (per CLAUDE.md); not a quick patch.** - -**Secondary (separate) finding — client primary-discovery is positional, not failover-correct.** With `masters = [m0, m1, …]`, the client (`xcecli_ctrlstatus_primary`, client.c:403) only reaches the primary if `m0` IS the primary: a standby answers `primaryName=""` ("unknown" by design, ctrlstatus.c:104) so no redirect hint is captured, and the inner `xcecli_ctrlstatus` SUCCEEDS on the reachable non-primary so transport failover never rotates. Confirmed: `9993`(primary-first)→ok 45ms; `9992,9993`(standby-first)→"no primary"; `9991,9993`(dead-first)→"no primary"; `9993,9991`→ok. Also the dead node returns `ctrlstatus failed: Invalid argument` (bug-846 EINVAL). This is what made earlier remote sessions read as "committed frozen" — the client was pinned to a stale/standby node. Fix options: (a) standby tracks+reports the primary name it heartbeats with (enable redirect), or (b) client rotates to the next master on a reachable-but-non-authoritative reply. - -## TOOLING PLAN (xce_test enhancement — requested by user) -Read-only debug commands in `xce_test`. **NOTE (correction):** `xce_test` does NOT link `persist.c`/`job.c` — it builds from `test.c` only + `libxce`. But `read_Job` is generated into `controller.capnp.c` which IS compiled into `libxce`, so the capnp decode is reachable; `waldump` adds `c_capnproto_dep`+`zlib_dep` to `xce_test_deps` and ships its own zero-alloc jobid decode (reads `struct Job.jobid` after `read_Job`, no `decode_Job`). -- `waldump <data_dir|wal_dir|seg_file>` — ✅ **DONE 2026-06-20.** Per node/segment, lists every entry `lsn | term | type | payload_len | jobid`, flags **LSN gaps** (non-contiguous / missing prefix) + the CRC recovered tip; nonzero exit on gap/CRC-break so it doubles as an E2E assertion. Impl: `src/controller/testsuite/waldump.{c,h}` (module), `cmd_waldump` wired into `top_commands` in `test.c`; unit test `test_waldump.c` (5 cases, hand-built raw segments). **Validated live** against `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}` — reproduced the exact bug table (ctrl-1 full 1–15; ctrl-2 missing prefix 1–15; ctrl-3 missing prefix 1–12). Also surfaced a NEW anomaly: each JOB_SUBMIT appears as a **consecutive duplicate pair** (same jobid at LSN N and N+1) — worth a follow-up. -- `snapdump <snap_dir|snap_file>` — TODO. Decode the `XCESNAP` header (lsn) and report whether it carries entities (it likely does NOT — confirms Fork #3). -- `walcompare <dirA> <dirB>` — TODO. Diff two nodes' logs by `lsn → crc/jobid` to flag divergence/gaps instantly. - -## UPDATE 2026-06-30 — SHARPENED ROOT CAUSE for the chaotic-soak loss: `xce_run_hooks` clobbers the durability `-ETIMEDOUT` - -The 10-round chaotic soak (`/home/claude/tmp/ha-1296`) loss is **NOT** the prefix-gap/entity-snapshot mechanism above. Decisive WAL evidence (waldump): -- All three nodes are **gapless, CRC-intact, full 6504B payloads** (so NOT bug-847 payload-drop, NOT a trimmed prefix — `seg-...001` present on all three, `snap/` empty, no trim fired). -- Lost job `0-0-1782854622-0` exists **only on ctrl-1 @ LSN 30, term 9**; ctrl-2/ctrl-3 hold a **term-10 NOOP @ LSN 30** (the new leader's first entry). Classic divergent uncommitted tail at the term 9→10 boundary. - -ctrl-1's controller log (PID 406309, term-9 leader) shows the durable-majority gate **worked correctly**: -``` -commit: WAIT deadline expired without commit (lsn=29 committed_lsn=0 writable_lsn=28 timeout_ms=500) -commit: WAIT deadline expired without commit (lsn=30 ...) <- the lost job -commit: WAIT deadline expired without commit (lsn=31 ...) -``` -The standby was still catching up (`standby[1] lsn=23 < committed_lsn=29`), so LSN 30 never reached majority → `repl_wait_majority_ack` → `commit_submit` returned **`-ETIMEDOUT`**, and `persist_wal_sync_append` propagated it (persist.c:2550-2551), and `persist_job_hook_fun` returned it (persist.c:2849). - -**The leak is in `xce_run_hooks` (src/lib/hook.c:134-157).** It runs all hooks for the type but `rc` is **overwritten every iteration** and it `return rc` — i.e. it returns ONLY the LAST-registered hook's value (the `break` is `BEFORE_EVENT`-only). Multiple JOB `AFTER_EVENT` hooks are registered (`apiserver_job_hook_fun` apiserver.c:545, `sched_job_hook_fun` sched.c:651, `persist_job_hook_fun` persist.c:615 + re-added on promotion ha.c:3436). When `persist_job_hook_fun` returns `-ETIMEDOUT` but a later-registered hook returns 0, **the `-ETIMEDOUT` is clobbered to 0**. `handle_jobsubmit` (job.c:593-611) then sees `hook_rc == 0`, skips the reject path (job.c:603 — confirmed: NO "rejecting job" line in ctrl-1's log), and replies `jobsubmit ok` (job.c:613-618) for an entry that was never majority-committed. A term-10 leader then legally overwrites LSN 30 → **committed-job-loss** + the harness's `committed_job_loss` count. - -This VIOLATES the CLAUDE.md **Commit Durability** invariant (reply-success ⇒ majority-durable). `handle_jobsubmit`'s gate is correct; the durability signal is destroyed one layer down in the shared hook runner. - -### Fix (surgical, libxce) -`xce_run_hooks` must not let a later hook's success clobber an earlier hook's failure on `AFTER_EVENT`: fold to a **sticky-negative** rc (run every AFTER_EVENT hook, but once any returns < 0 preserve it; return the first/any negative). `BEFORE_EVENT` keeps veto/break semantics. This restores "AFTER_EVENT fails if ANY hook failed," which is exactly what `handle_jobsubmit` already assumes. RED: register two AFTER_EVENT hooks where the first returns `-ETIMEDOUT` and the second returns 0; assert `xce_run_hooks` returns `-ETIMEDOUT`. Blast radius: shared by controller/worker/scheduler — audit any consumer relying on the *last* hook's positive rc (none known; consumers treat non-zero as failure). - -NOTE: the entity-bearing-snapshot / prefix-trim gap documented above is a SEPARATE, still-real defect (it bites a node that rejoins after a checkpoint trims the prefix). It is just not the mechanism this particular soak hit — confirm with a soak that forces a checkpoint+trim. diff --git a/.kiro/specs/ha-commit-durability/brief.md b/.kiro/specs/ha-commit-durability/brief.md deleted file mode 100644 index fcc9b298..00000000 --- a/.kiro/specs/ha-commit-durability/brief.md +++ /dev/null @@ -1,73 +0,0 @@ -# Brief: ha-commit-durability - -## Problem -A 3-node HA controller failover test **permanently lost committed jobs**. The client submitted 10 jobs across several forced primary kills; the 7 earliest (JobIds `0-0-1781708993..1781709035`, WAL LSN ≤20) — which had been replicated and majority-ACKed, and survived the *first* failover — were gone by the end, while only the 3 latest survived. Losing an acknowledged, committed job is a correctness (data-loss) failure, the most severe class for this system. - -## Current State -The HA consistency machinery is implemented (commit engine, WAL replication, convergence/truncate-on-demotion, single-leader own-term gate, PreVote, vote-freshness bug-339). But **"committed" does not imply "durable on a majority"**: - -- **WAL fsync is asynchronous** — a timer thread (`persist.c:173 fsync_worker`) fdatasyncs every `wal_sync_ms`. `persist_append` does **not** fsync before returning; the standby does **not** fsync before sending its `WalAck`. -- **The commit engine advances `committed_lsn` on majority *ACK*** (`commit.c:303 commit_on_ack`, `repl_wait_majority_ack`). An ACK only proves the entry reached the OS page cache — it can be durable on **zero** disks at the moment it is declared committed. -- **Recovery silently truncates to nothing.** When killed nodes restart, `persist_recover` hits a CRC chain break at a zeroed/never-fsync'd region (`ctrl-3.log:1597` `prev_crc=00000000` at offset 40) and stops replay at ~0 entries — but the node still becomes eligible. -- **A hollow node wins and overwrites the good timeline.** With durability lost on the killed nodes and failover quorum collapsed to `required=1`, the freshness gate cannot fence the stale candidate. ctrl-3 became PRIMARY term=4 advertising `committed_lsn=21` over an empty job table (`ctrl-3.log:1612`); the node that *did* hold the data (ctrl-1, applied_lsn=22) converged **down** to it (`ctrl-3.log:1618`). LSN ≤20 discarded forever. - -This violates the Leader-Completeness assumption documented in `src/controller/CLAUDE.md` ("committed entries are on a majority" presumes majority-ACK ⇒ durable-on-a-majority) and the TLA+ mapping `log[i] = WAL on disk`. - -## Desired Outcome -1. An operation reported committed is **durable on a majority of nodes** before it is declared committed — surviving simultaneous crash + restart of any minority. -2. A node that cannot recover its WAL to a valid committed tip is **fenced from leadership** (fail-loud), never silently truncating to zero and then leading; the cluster never converges onto a timeline below its known committed high-water mark. - -## Approach -Two coupled requirements (chosen scope: *Durability + promotion fence*): - -- **R1 Durable commit (fsync-before-ACK):** the standby `fdatasync`s the appended WAL bytes **before** emitting `WalAck`; the primary counts its own vote toward majority only after its local `fdatasync`. `committed_lsn` advances only on a **durable** majority. Preserve throughput via group-commit / batched fdatasync (amortize the sync across pending appends) rather than one sync per entry. Keep `wal_sync_ms` async path only for non-committing/best-effort writes, if any remain. -- **R2 Recovery & promotion fence:** a WAL CRC/replay failure that prevents reconstructing a contiguous committed tip makes the node **ineligible to become primary** until it has caught up from a healthy peer (FATAL or fence, never silent truncate-to-zero-then-lead). A node must not win election or self-promote advertising a `committed_lsn` below the cluster's known committed high-water mark, even under degraded single-voter quorum. - -Honor the user's fail-loud preference: surface the real durability/recovery failure rather than masking it. - -## Scope -- **In**: fsync-before-ACK on the standby apply path and primary self-count; durable-majority gating of `committed_lsn`; group-commit batching to bound the latency cost; fencing an un-recovered/corrupt-WAL node from promotion; preventing a sub-high-water-mark leader from being elected under degraded quorum; unit tests for each; the 3-node live E2E that reproduces this incident (kill-primary-under-load, assert zero committed-job loss). -- **Out**: WAL wire format / ACK message shape (`ha-wal-replication`); per-LSN majority math (`ha-commit-engine` — consumed, the *durability precondition* of its ACK input is what changes); demotion truncate/self-restart and rejoin reconciliation of an *intact* log (`ha-convergence-recovery`); promotion no-op / own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds the durability+high-water *precondition*, not new vote math); entity-bearing snapshots (roadmap); client-side failover (`libxce-failover`). - -## Boundary Candidates -- Durable-commit seam: standby `fdatasync`-before-`WalAck` + primary self-fdatasync before majority self-count (the "ACK ⇒ durable" guarantee). -- Promotion-eligibility seam: recovery-integrity + committed-high-water check gating `raft_cb_become_leader` / PreVote eligibility. - -## Out of Boundary -- Tuning the async `fsync_worker` cadence as a "fix" (it is the defect for committing writes, not a knob to turn). -- Changing how a majority is *counted* (commit-engine) or how votes are *granted* (raft-library) beyond adding the durability/high-water precondition. -- Worker-reported fact resync across restart (`worker-rejoin-resync`, roadmap backstop). - -## Upstream / Downstream -- **Upstream**: `ha-commit-engine` (committed_lsn semantics it now gates on durable ACKs), `ha-wal-replication` (ACK/catch-up machinery this makes durability-correct), `persist.c` fsync infrastructure, `ha-prevote` + vote-freshness (bug-339) eligibility path it strengthens. -- **Downstream**: `ha-convergence-recovery` E2E (its single-outcome guarantee depends on this), `component-failover-adoption` and any real failover deployment. - -## Existing Spec Touchpoints -- **Extends/strengthens**: `ha-wal-replication` (durability precondition on ACK), `ha-commit-engine` (durable-majority), `ha-convergence-recovery` (assumes intact persisted committed state — this spec makes that assumption true). -- **Adjacent (avoid overlapping)**: `ha-single-leader-role`, `raft-library`, `ha-prevote`. - -## Constraints -- C / gnu17, `src/controller/persist.c`, `commit.c`, `repl.c`, `ha.c`, `cluster.c`; TDD mandatory (tests in `testsuite/` before C changes). -- Async-I/O rule: never block the iomgr thread — fdatasync must not stall heartbeat/vote handling; use the dedicated persist/fsync thread + group-commit and signal the commit waiter on completion. -- Must verify against `raft/raft.tla` and the `src/controller/CLAUDE.md` Protocol Verification Checklist (commitIndex monotonic; leader holds all committed entries). -- Live E2E needs `/xce-deploy-ha` (cluster at `/tmp/xce-ha-deploy`); reproduction harness = `xce_test` driving jobsub + primary kills. An automated soak reproduction now exists at `src/tools/test-commit-durability.sh` (rolling SIGKILL+restart under load, asserts every acked job survives). -- Performance: durable-majority adds sync latency to the submit path — group-commit is required, not optional. - -## Reproduction Findings (2026-06-17, from building the E2E) -- **bug-661 is a RACE, not a deterministic failure.** `wal_sync_ms` is hardcoded to `DEFAULT_WAL_SYNC_MS = 10` ms (`config.h:86`) with **no CLI/env/config override**. The committed-but-not-fsync'd window is ~10ms, so a single-node SIGKILL+restart only loses a committed job when the kill lands inside that window AND the truncated node re-leads before catch-up repairs its WAL. The soak reproduced real loss once (4 committed jobs) but passed an 8-round run clean — confirming the rarity. -- **Required test seam (NEW spec item): a configurable fsync interval** (e.g. `--wal-sync-ms`, plumbed into `persist_config_t.wal_sync_ms`). Setting it to several seconds makes every committed entry provably un-fsync'd at kill time, turning the racy soak into a **deterministic** RED. This seam is also the natural way to unit-test fsync-before-ACK. None exists today — add it as part of Requirement 1's testability (cf. the convergence spec's XCE_TESTING restart seam). -- **Acceptance asymmetry**: a soak FAIL proves the bug; a soak PASS does not prove durability (the race may simply not have fired). After the fix, the soak must pass *deterministically regardless of timing* — that, plus the deterministic `--wal-sync-ms` test, is the real acceptance bar. -- **Amplifier confirmed**: loss needs a killed+restarted node to re-lead with a truncated WAL *before* replication catch-up repairs it. The fix must therefore also fence a node that has not caught up to the committed high-water mark from leading (Requirement 2). - -## Design Decision — durability mechanism (2026-06-17, with user) - -**Invariant (user-stated, authoritative):** when the controller replies **ok** to a client, the operation's WAL entry MUST already be persisted (fdatasync'd) on a fault-tolerant majority — self + ⌊N/2⌋ standbys. - -**Exact defect located:** `commit.c:majority_candidate_locked` counts the primary's own contribution as `cap = highest_appended_lsn(p)` — every *appended* (page-cache) entry, not every *fdatasync'd* one. On 3 nodes a commit needs self + 1 standby; the standby ACK is durable (`persist_wal_write_replicated:515` fdatasyncs before replying) but the primary's copy is **not** (it relies on the `fsync_worker` timer). So a "committed" entry is durable on **exactly 1 node** → zero durability fault-tolerance. That is bug-661. - -**Chosen direction (user): synchronous persistence — eliminate `fsync_worker` as the durability mechanism.** -- The submit path **already blocks off the iomgr thread** waiting for majority replication (`job.c:568-569`; `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)`), so adding the primary's own `fdatasync` on that path is liveness-safe — it does not stall heartbeats. -- Minimal correctness change: (1) primary `fdatasync`s its entry before its self-contribution covers that LSN (symmetric with the standby path at `persist.c:515`); (2) `commit.c` self-contribution uses a `durable_lsn` (highest fdatasync'd), not `highest_appended_lsn`; (3) reply stays gated on the commit barrier → reply-ok ⇒ durable-majority. -- **Throughput**: per-entry fdatasync. The dominant amplifier (a job array fanning out to N entries → N fsyncs) is removed by the roadmap item **job-array-single-wal-record** (one WAL record per array → one fsync). Cross-reference that item; this spec assumes it lands or is acceptable interim. -- **Group-commit IS in scope (revised 2026-06-18 at design-review).** Initially deferred, but `/kiro-validate-design` found the highest-volume WAL path is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible, not covered by job-array-single-record — so a pure per-entry synchronous design penalizes the dominant writes. Decision (with user): repurpose `fsync_worker` into a demand-driven group-commit flusher (batch `fdatasync`, advance `durable_lsn`, wake commit-eval) as part of this spec. The roadmap "WAL group-commit batching" item is absorbed here. -- **Open for design:** whether to delete the `fsync_worker` thread entirely or retain a minimal best-effort flush for any non-client-acked / local-only writes (e.g. checkpoint markers already fsync inline; audit `wal_classify` for any write that gates no reply). Confirm no client-acked op bypasses the commit barrier. diff --git a/.kiro/specs/ha-commit-durability/design.md b/.kiro/specs/ha-commit-durability/design.md deleted file mode 100644 index f5e03c21..00000000 --- a/.kiro/specs/ha-commit-durability/design.md +++ /dev/null @@ -1,498 +0,0 @@ -# Design Document — ha-commit-durability - -## Overview - -**Purpose:** Make acknowledgment imply durability in the XCE HA controller. Before this spec, the controller advanced its commit position on a majority of replication **acknowledgments**, but the primary counted its own contribution from a page-cache write (`commit.c:majority_candidate_locked` used `highest_appended_lsn`), while background `fdatasync` ran on a timer (`fsync_worker`). A "committed" operation could therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover. (Post-implementation, `majority_candidate_locked` caps the self-contribution at `persist_durable_lsn(ce->p)` — verified current in `src/controller/commit.c`.) - -**Users:** Cluster operators and every client whose acknowledged job/queue/worker operations must survive controller failover. - -**Impact:** Strengthens the meaning of "committed" from "majority-ACKed (volatile)" to "persisted to stable storage on a fault-tolerant majority." The client reply path is already gated on the commit barrier, so once `committed_lsn` implies a durable majority, the invariant holds with no change to the reply path. Durability becomes **group-commit**: `fsync_worker` is repurposed from a fire-and-forget timer into a demand-driven flusher that, on append, batches `fdatasync` across all pending entries, advances `durable_lsn`, and wakes the commit evaluation. The commit majority counts `durable_lsn` (not appended). This amortizes one `fdatasync` across many entries — important because the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage), which are not client-latency-bound but must be durable before `committed_lsn` passes them. A second concern — a member with a corrupt or truncated log becoming authoritative — is closed by a recovery-integrity verdict and a promotion fence. - -### Goals -- **Durable commit:** when the controller replies *ok*, the operation's log entry is `fdatasync`'d on the acknowledging member plus ⌊N/2⌋ others (Req 1). -- **Recovery integrity & promotion fence:** a member that cannot reconstruct its committed state, or that lags the cluster's committed high-water mark, is barred from leadership until caught up — fail-loud when it cannot recover safely (Req 2). -- **Deterministic verification:** durability timing is controllable in a test mode so the guarantee is provable without racing the clock (Req 3). -- Preserve cluster liveness: durability work never stalls heartbeat/election processing. - -### Non-Goals -- Single-record job-array persistence (`job-array-single-wal-record`, roadmap) — complements this spec by bounding per-array fsync count; independent of the group-commit flusher introduced here. -- Per-entry synchronous fsync as the steady-state mechanism — superseded by group-commit (decision: the high-volume `WAL_MODE_RETAIN` path needs batching; see Components/DurableCommit). -- Replication/ACK wire format (`ha-wal-replication`), per-entry majority arithmetic (`ha-commit-engine`), demotion truncate / rejoin reconciliation of an intact log (`ha-convergence-recovery`), promotion no-op / own-term gate (`ha-single-leader-role`), raw vote-grant arithmetic (`raft-library`). - -## Boundary Commitments - -### This Spec Owns -- The **durability precondition** on the commit majority: a member contributes to the commit majority for an LSN only once that LSN is `fdatasync`'d locally. Introduces and is authoritative for `persist_t.durable_lsn`. -- **Group-commit durability**: `fsync_worker` repurposed from a timer into a demand-driven flusher that batches `fdatasync`, advances `durable_lsn`, and wakes commit evaluation; `persist_wal_async` signals the flusher instead of deferring to a timer. Benefits all members on the high-volume `WAL_MODE_RETAIN` path. -- **Recovery integrity verdict**: `replay_wal`/`persist_recover` distinguish a clean log end from a corruption-truncated end, and re-derive the member's durable floor from the recovered WAL. -- **Promotion eligibility fence**: a recovery-truncated member, or one whose durable position is below the cluster committed high-water mark, is ineligible to become primary until caught up; re-eligible on catch-up. -- **Durable-backed freshness advertisement**: a candidate's advertised log position and a voter's freshness floor derive from durable WAL state, not from volatile in-memory positions that survive a truncating restart. -- **Durability verification seam**: a configurable durability cadence / background-flush control plus an `XCE_TESTING` hook. - -### Out of Boundary -- Counting the majority or advancing `committed_lsn` arithmetic (`ha-commit-engine` — consumed; this spec changes only the *durability precondition* of its inputs). -- The reply ordering / classification sweep (`ha-commit-barrier-rollout` — reused unchanged). -- Demotion truncate + self-restart and rejoin reconciliation of an **intact** log (`ha-convergence-recovery`). This spec fences **promotion**; convergence-recovery handles **demotion** and **rejoin**. -- Vote-grant arithmetic and term handling (`raft-library`); this spec adds only the durability/high-water **precondition** to the existing freshness gate. -- WAL group-commit batching; job-array single-record persistence. - -### Allowed Dependencies -- `ha-commit-engine`: `commit_engine_t`, `commit_submit`, `majority_candidate_locked` (modify self-contribution input only). -- `ha-wal-replication`: standby ACK path (`persist_wal_write_replicated`) and catch-up streaming (consumed; `in_catch_up` reused). -- `ha-commit-barrier-rollout`: `persist_commit_barrier` reply gate (consumed unchanged). -- `persist_t` internals (`current_lsn`, `committed_lsn`, `last_crc`, segment I/O), `ha_state_t` (`applied_lsn`, `last_known_committed_lsn`, `in_catch_up`, `term`). -- Constraint: must not write/persist `committed_lsn` to disk (Raft keeps the commit index volatile); durability is about the **log**, not the commit index. - -### Revalidation Triggers -- Change to the `durable_lsn` semantics or to `majority_candidate_locked`'s self-contribution → re-check `ha-commit-engine` and `ha-convergence-recovery` (which assume durable committed state). -- Change to the promotion-eligibility gate or freshness-floor source → re-check `ha-single-leader-role`, `ha-prevote`, `raft-library`, and the `CLAUDE.md` TLA+ checklist. -- Removal of `fsync_worker` → re-check any caller that assumed deferred fsync (audit `wal_classify`). -- New `wal_sync` override surface → re-check `main.c` flag conventions and deployment scripts. - -## Architecture - -### Existing Architecture Analysis - -- **Commit gate already off the iomgr thread:** the AFTER_EVENT hook fires before the reply and `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)` blocks on a condvar (`job.c:2214`, `do_jobcontrol_op`; AFTER_EVENT hook itself e.g. `job.c:2189`). (Note: citation corrected — file has grown substantially since this spec closed; original cited `job.c:568-569`.) Adding `fdatasync` to this path is liveness-safe. -- **Standby already persists-before-ACK:** `persist_wal_write_replicated` `fdatasync`s before returning (`persist.c:1417`, function defined at `persist.c:1193`). The primary is the asymmetric gap. (Note: citation corrected from the original `persist.c:515`.) -- **Recovery is silent-truncating:** `replay_wal` (`persist.c:4624`) `break`s on CRC chain break / mismatch / short read (now around `persist.c:4704-4784`, several distinct break sites) and returns a verdict; `persist_recover` never sets `committed_lsn` (stays 0). (Note: citation corrected — original cited `persist.c:1021/1050/1042`; `replay_wal` has since moved substantially within the file.) -- **Promotion has no integrity gate:** `raft_cb_become_leader` (`ha.c:1615`) computes `safe_lsn` from votes (`ha.c:1641-1644`) then `persist_truncate_wal_above(safe_lsn)` (`ha.c:1662`) with no check that the local log is intact or ≥ `safe_lsn`. (Note: this describes the pre-spec baseline; citation corrected from the original `ha.c:572/577`. The gate this spec adds is now implemented immediately after, at `ha.c:1699-1822`.) -- **Freshness floor is volatile:** `ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn)` (now `ha.c:5310`, corrected from the original `ha.c:2127`), in-memory atomics not re-derived from durable WAL. -- **Test-seam convention:** `#ifdef XCE_TESTING` hook shims are now spread across dozens of sites in `ha.c` (the file has roughly doubled since this spec closed); `wal_sync_ms` was a hard constant (`config.h:86`, still `DEFAULT_WAL_SYNC_MS = 10`) with no override — this spec added the `--wal-sync-ms` override (confirmed live in `main.c`/`persist.h`). (Note: the original `ha.c:2270-2335` range no longer corresponds to a single representative hook-shim cluster.) - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph RequestPath - Hook[AFTER_EVENT hook off iomgr] - Barrier[persist_commit_barrier] - end - subgraph Persistence - Append[persist_wal_async sync fsync] - DurLsn[durable_lsn] - Recover[persist_recover integrity verdict] - ReplApply[persist_wal_write_replicated standby fsync] - end - subgraph Commit - Majority[majority_candidate_locked durable self] - Committed[committed_lsn] - end - subgraph Election - Fresh[freshness floor from durable WAL] - Promote[raft_cb_become_leader promotion fence] - end - Hook --> Append - Append --> DurLsn - Append --> ReplApply - DurLsn --> Majority - ReplApply --> Majority - Majority --> Committed - Committed --> Barrier - Barrier --> Reply[reply ok to client] - Recover --> Fresh - Recover --> Promote - Fresh --> Promote - DurLsn --> Promote -``` - -**Architecture Integration** -- **Selected pattern:** extend-in-place for the durability precondition (Req 1, 3) plus a small testable fence helper for promotion eligibility (Req 2). No new module — `durable_lsn` is one atomic on `persist_t`. -- **Boundaries:** persistence owns `durable_lsn` and the recovery verdict; the commit engine consumes `durable_lsn`; election consumes the recovery verdict + durable-backed floor. No shared ownership of the commit index. -- **Patterns preserved:** commit-barrier reply gate, `in_catch_up` fence concept, `XCE_TESTING` hooks, the standby persist-before-ACK already in place. -- **Steering compliance:** never block the iomgr thread (fsync lands on the off-iomgr commit path); re-verify against `raft/raft.tla` (commit rule, leader completeness) per `CLAUDE.md`. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), controller daemon | Durable commit, recovery integrity, promotion fence | Extends `persist.c`, `commit.c`, `ha.c`, `cluster.c` | -| Data / Storage | Local WAL segments + `fdatasync` | Synchronous durability; integrity-verified recovery | `posix_fallocate`'d segments; `fdatasync` (data + pre-allocated metadata) | -| Infrastructure / Runtime | `--wal-sync-ms` (new) + `XCE_TESTING` seam | Deterministic durability verification | Follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) precedent | - -## File Structure Plan - -### Modified Files -- `src/controller/persist.c` — `persist_wal_async`: append then **signal the flusher** (no inline per-entry fsync); convert `fsync_worker` from a timer wait into a **demand-driven group-commit flusher** (wake on append-signal, `fdatasync`, advance `durable_lsn` to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in `persist_start`/`persist_stop`/`persist_fini`. `replay_wal`/`persist_recover`: return an integrity verdict; re-derive durable floor from the recovered WAL tip. `persist_wal_sync_append`: **resolved (task 2.4)** — re-routed rather than deleted. It now calls the guarded `sync_append_legacy_commit_store` (`persist.c:5617`), which only advances `committed_lsn` when `written_lsn <= durable_lsn` (regression-pinned by `test_legacy_sync_append_holds_committed_at_durable`). (Note: current state corrected — as of this audit `persist_wal_sync_append` (`persist.c:5639`) DOES have a live production caller, `submit_job_wal_entry_and_commit` (`persist.c:5990-6004`, wired by `raft-apply-decoupling`), contradicting this section's original "dormant, no production caller" framing; the durable-precondition guard was added as designed, so the resolution is correct even though the "dormant" premise no longer holds.) -- `src/include/persist.h` — add `_Atomic uint64_t durable_lsn` and a recovery-verdict type to `persist_t`/the recover signature; keep `fsync_*` fields (now driving the group-commit flusher); add an override field to `persist_config_t` (or its setter). Confirmed current: `durable_lsn` at `persist.h:275`, `RECOVER_CLEAN`/`RECOVER_TRUNCATED` enum, `recover_decision_t`, `recovery_unrecoverable` flag all present as designed. -- `src/controller/commit.c` — `majority_candidate_locked`: use `durable_lsn` (not `highest_appended_lsn`) for the self contribution `cap`. Confirmed current (`commit.c:135`, `persist_durable_lsn(ce->p)`). -- `src/controller/ha.c` — `raft_cb_become_leader`: insert the pre-promotion eligibility gate; `ha_vote_self_freshness`: derive floor from durable WAL state; fence-release on catch-up completion. (Note: the durability-verification `XCE_TESTING` seam actually lives in `src/controller/persist.c` — `persist_test_set_force_fsync_fail` at `persist.c:1547`, `persist_test_durable_lsn_on_disk` at `persist.c:1610` — not in `ha.c` as this section originally stated.) -- `src/controller/cluster.c` — vote / PreVote freshness comparison consume the durable-backed floor; promotion path honors the fence. (Note: citations corrected — real-vote comparison now at `cluster.c:544-557`, PreVote at `cluster.c:643-651`; original cited `cluster.c:380-396`/`467-474`.) -- `src/include/config.h` — keep `DEFAULT_WAL_SYNC_MS`; reconcile the stale `50` in `persistence-and-ha-design.md`. -- `src/controller/main.c` — parse `--wal-sync-ms` (and optionally `--wal-sync-disable`) into `persist_config_t`. -- `src/controller/myapp.c` / `cluster.c` (`persist_config_defaults` callers) — apply the override before `persist_init`. - -### New Files -- `src/controller/testsuite/test_commit_durability.c` — unit/white-box tests for `durable_lsn` advancement, self-counts-durable, recovery verdict, promotion fence (Req 1, 2, 3). -- `src/tools/test-commit-durability.sh` — **exists** (soak E2E); extend to drive the new `--wal-sync-*` seam for a deterministic mode. - -> Each modified function keeps one responsibility: persistence owns durability + recovery verdict; commit owns majority math (consuming `durable_lsn`); election owns eligibility (consuming the verdict + floor). - -## System Flows - -### Durable commit (reply ok ⇒ persisted on a majority) - -```mermaid -sequenceDiagram - participant Client - participant Primary - participant Standby - participant Flusher - Client->>Primary: submit op - Primary->>Primary: persist_wal_async pwrite then signal flusher - Primary->>Standby: replicate entry - Flusher->>Flusher: batched fdatasync then durable_lsn advance then wake commit - Standby->>Standby: persist_wal_write_replicated fdatasync - Standby-->>Primary: WalAck acked_lsn - Primary->>Primary: majority_candidate_locked self equals durable_lsn plus standby acks - Primary->>Primary: committed_lsn advance then commit barrier releases - Primary-->>Client: reply ok -``` - -Key decisions: the reply is released only after `committed_lsn` covers the entry, and `committed_lsn` now requires a **durable** self plus a **durable** standby majority. The `fdatasync` runs on the dedicated flusher thread (the repurposed `fsync_worker`), batched across all pending appends; the AFTER_EVENT caller blocks on the commit barrier (off iomgr) until `durable_lsn` + standby acks cover its entry, so heartbeats are unaffected (Req 1.6) and one `fdatasync` amortizes across the high-volume RETAIN-fact stream. - -### Recovery integrity & promotion fence - -```mermaid -graph TD - Start[startup persist_recover] --> Replay[replay_wal] - Replay --> Verdict{integrity verdict} - Verdict -->|clean eof| Floor[set durable floor from WAL tip] - Verdict -->|corruption truncated| Truncated[mark recovery_truncated] - Floor --> Eligible[eligible to lead] - Truncated --> CanCatchUp{healthy peer reachable} - CanCatchUp -->|yes| Fence[fence from promotion until caught up] - CanCatchUp -->|no| LoudFence[LOUD PERMANENT FENCE: non-promotable, STAYS ALIVE, keeps retrying catch-up — NEVER self-exit] - LoudFence --> Caught - Fence --> Caught{caught up past last committed boundary} - Caught -->|yes| Eligible -``` - -Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state, the controller logs a LOUD operator-visible alert and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up (Req 2.4) — it refuses to serve divergent state by refusing leadership, NOT by terminating. **A controller MUST NEVER self-exit** (`exit`/`abort`) as a designed response to an operational condition (user invariant, 2026-06-18): a self-exited controller needs manual admin recovery and degrades quorum meanwhile, whereas one that stays alive self-heals when a peer returns (Req 2.6). A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5). - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | reply ok ⇒ durable on majority | DurableCommit | `commit barrier`, `durable_lsn` | Durable commit | -| 1.2 | self counts only when durable | DurableCommit | `majority_candidate_locked` | Durable commit | -| 1.3 | standby persists before ACK | DurableCommit (pin) | `persist_wal_write_replicated` | Durable commit | -| 1.4 | no premature ok | DurableCommit | `persist_commit_barrier` | Durable commit | -| 1.5 | survives minority crash+restart | DurableCommit | durable-majority property | Durable commit / soak | -| 1.6 | durability work preserves liveness | DurableCommit | off-iomgr placement | Durable commit | -| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence | -| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence | -| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence | -| 2.4 | loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) | RecoveryIntegrity | loud-fence policy + sticky `recovery_unrecoverable` flag | Recovery fence | -| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both | -| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence | -| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — | -| 3.2 | recoverable after ungraceful kill in verify mode | DurabilityTestSeam | `XCE_TESTING` hook | soak deterministic mode | -| 3.3 | identical external semantics in/out of mode | DurabilityTestSeam | seam affects timing only | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|-------|--------|--------------|--------------------------|-----------| -| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State | -| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; loud permanent non-promotable fence policy (stays alive, NEVER self-exit) | 2.1, 2.4 | replay_wal (P0) | State | -| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State | -| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State | -| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State | - -### Persistence + Commit - -#### DurableCommit - -| Field | Detail | -|-------|--------| -| Intent | Make the primary persist before it contributes to the commit majority | -| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.5 | - -- `persist_wal_async` appends and signals the flusher; it does **not** fsync inline per entry. -- The **group-commit flusher** (repurposed `fsync_worker`) wakes on the append-signal, performs one `fdatasync`, advances `durable_lsn` to the highest synced LSN, and broadcasts the commit-eval condvar — amortizing one sync across all entries pending at wake time (the high-volume RETAIN stream benefits most). -- `majority_candidate_locked` caps the self contribution at `durable_lsn`, not `highest_appended_lsn`. -- Standby persist-before-ACK is preserved and regression-pinned (`persist_wal_write_replicated`, defined at `persist.c:1193`, `fdatasync` call at `persist.c:1417`; note corrected from the original `persist.c:515`); the standby path may later adopt the same flusher (out of scope here). -- Checkpoint markers retain their existing inline `fdatasync` (`persist.c:1761`; note corrected from the original `persist.c:573`, which now falls inside unrelated term-index code added by `ha-term-safety`). -- `durable_lsn` is monotonic, never exceeds the live segment tip; on `persist_truncate_wal_above`, clamp `durable_lsn` to the new tip. - -**Dependencies** -- Inbound: AFTER_EVENT hooks (`job.c`, `queue.c`, `worker.c`, `join.c`), `ha.c` WAL_NOOP — purpose: append path (P0). -- Outbound: `commit_engine_t` — purpose: self contribution input (P0). - -**Contracts**: State [x] - -##### State Management -- **State model:** `_Atomic uint64_t durable_lsn` on `persist_t`; invariant `durable_lsn <= current_lsn - 1`, monotonic non-decreasing except clamp-on-truncate. -- **Persistence & consistency:** `durable_lsn` advances only after a successful `fdatasync` in the flusher; the commit engine reads it for self-coverage so `committed_lsn` ⇒ durable-majority. -- **Concurrency strategy:** the flusher takes `p->lock` to read the appended tip and `fdatasync`, advances `durable_lsn`, releases, then signals commit-eval; `majority_candidate_locked` reads `durable_lsn` via atomic load. Define the append-signal → flush → commit-wake ordering precisely so a WAIT caller cannot miss a wake (lost-wakeup guard). - -**Implementation Notes** -- Integration: the WAIT-mode caller blocks on the commit barrier off the iomgr thread; only the flusher thread blocks on `fdatasync`. Heartbeats unaffected. -- Validation: assert `committed_lsn <= durable_lsn` on the local member after every commit advance (debug build); verify the flusher batches (one fsync advances `durable_lsn` past multiple LSNs under load). -- Risks: **lost-wakeup / batching-window correctness** between `persist_wal_async` signal and the flusher (an entry must never be considered durable before its `fdatasync`); design the signal as level-triggered on the appended-vs-synced gap, not edge-triggered. Bounded flush latency vs throughput is a tunable (`--wal-sync-ms` becomes the max batching delay, not a durability gap). - -### Persistence — RecoveryIntegrity - -| Field | Detail | -|-------|--------| -| Intent | Distinguish clean log end from corruption; re-derive durable floor; loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) | -| Requirements | 2.1, 2.4 | - -**Responsibilities & Constraints** -- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read). -- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN. -- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed. -- Loud-fence policy (NEVER self-exit, user invariant 2026-06-18): if a recovery-truncated / behind member cannot reach a healthy peer to catch up, the controller logs a LOUD operator-visible alert (truncation boundary vs cluster HWM) via `persist_recover_loud_fence_if_unrecoverable`, sets a sticky `recovery_unrecoverable` flag, and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up. It refuses to serve divergent state by refusing leadership — it does NOT `exit`/`abort`. The flag clears (and the fence releases → eligible, Req 2.6) automatically when a peer returns and catch-up reaches a committed-consistent state. - -**Contracts**: State [x] - -##### State Management -- **State model:** verdict enum + `recovery_truncated` flag (on `ha_state_t` or `persist_t`); recovered durable floor = highest intact LSN. -- **Persistence & consistency:** does not persist `committed_lsn`; only re-derives the durable log tip from on-disk segments. -- **Concurrency strategy:** runs single-threaded during `cluster_init_phase2` before the node participates. - -**Implementation Notes** -- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently. -- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability. -- Risks: must not loud-fence on a legitimately empty first-boot WAL (no entries ≠ corruption). - -### Boot Policy Decision Table (task 3.2) - -> **Status:** DECISION recorded (task 3.2); **terminal action corrected by the never-self-exit rework (2026-06-18).** The original table's `FATAL` outcome (a self-`abort`) was REPLACED: the unrecoverable terminal is now a **LOUD PERMANENT FENCE** — non-promotable, **stays alive**, keeps retrying catch-up. This table is the contract that task **3.3** (the loud-fence terminal) and task **4.3** (the promotion fence + release) implement. - -**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. If it cannot reach that state (no healthy peer reachable / cannot reach a committed-consistent state), it enters a **LOUD PERMANENT FENCE**: it logs a loud, operator-visible alert (truncation boundary vs cluster HWM), holds itself **permanently non-promotable**, and **stays alive while continuing to retry catch-up** — it **NEVER self-terminates**. The fence releases (→ ELIGIBLE) automatically once catch-up reaches a committed-consistent state. - -**Invariant (authoritative):** A truncated/behind member always resolves to exactly one of two outcomes: it **catches up → ELIGIBLE**, or it enters a **LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit)**. It is never silently divergent (it refuses to lead), and the process never self-`exit`s/`abort`s as a designed response to this operational condition (user invariant, 2026-06-18) — a self-exited controller needs manual admin recovery and degrades quorum, whereas one that stays alive self-heals when a peer returns (Req 2.6). Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never the unrecoverable terminal on a legitimate sentinel. - -**Inputs (evaluated at boot / promotion-eligibility):** -- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`. -- **durable-vs-HWM** — local durable position relative to the cluster committed high-water mark: `at/above` vs `below`. -- **peer** — a healthy peer reachable for catch-up: `yes` vs `no`. -- **catch-up** — catch-up reaches a committed-consistent state within the bounded window: `succeeds` vs `fails` (n/a when no catch-up is attempted). - -**Outcomes:** -- **ELIGIBLE** — may promote (no fence). -- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6). -- **LOUD PERMANENT FENCE** — log a loud operator-visible alert (truncation/boundary), set the sticky `recovery_unrecoverable` flag, and hold the member permanently non-promotable while it **stays alive** and keeps retrying catch-up — **NEVER self-exit**; releases → ELIGIBLE on catch-up (Req 2.4, 2.6). - -| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req | -|---|---------|----------------|--------------|-----------------|---------|-----------------| -| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) | -| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) | -| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never the unrecoverable terminal when a peer is reachable. (Req 2.2, 2.6) | -| 4 | TRUNCATED | below | no | fails (no committed-consistent state reachable) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Cannot recover locally AND no peer to catch up from ⇒ log loudly + hold non-promotable + alive; releases → ELIGIBLE when a peer returns and it catches up. NEVER `exit`/`abort`. (Req 2.4, 2.6) | -| 4b | CLEAN | below | no | fails (no peer to advance from) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Behind the HWM and no healthy peer to catch up from — same non-terminating loud-fence as row 4; NEVER self-exit. (Req 2.4, 2.2, 2.6) | -| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never the unrecoverable terminal** on a legitimate sentinel. (Req 2.1, 2.2) | -| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) | - -**Notes on the dimensions:** -- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value). -- the catch-up retry is **indefinite** (the member stays alive and keeps trying) and **loud** (a periodic operator-visible re-alert, controlled by `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`); it is never silently divergent because the member refuses leadership the entire time. The previous "bounded window before FATAL" deadline was REMOVED by the never-self-exit rework — there is no deadline-driven termination. -- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer. - -### Election — PromotionFence + FreshnessFloor - -| Field | Detail | -|-------|--------| -| Intent | Bar a truncated/behind member from leadership until caught up; advertise/vote on durable WAL state | -| Requirements | 2.2, 2.3, 2.5, 2.6 | - -**Responsibilities & Constraints** -- Pre-promotion gate in `raft_cb_become_leader`: if recovery-truncated, or local durable position < cluster committed high-water mark, refuse promotion (set/extend `in_catch_up`-style fence) and instead catch up from a healthy primary. -- `ha_vote_self_freshness`: floor derives from the **durable** WAL tip (re-derived on recovery), so a truncated member cannot over-advertise a volatile `last_known_committed_lsn`. Honors `CLAUDE.md`: the post-election safe-commit point still comes from voter replies, never from a stale ctrlhello. -- Fence-release: when catch-up brings the member past its last-known committed boundary, restore eligibility (hook the existing catch-up completion / re-inclusion path). -- Degraded quorum: because a candidate cannot lead beyond its durable tip, a hollow node cannot win even at minimal quorum — preserving Req 1's invariant across leadership changes. - -**Contracts**: State [x] - -**Implementation Notes** -- Integration: reuses `in_catch_up` (`ha.c:917`, `cluster.c:713`; note corrected from the original `ha.c:954`/`cluster.c:529`) and the PreVote/vote freshness comparison (`cluster.c:544-557` real-vote, `cluster.c:643-651` PreVote; note corrected from the original `cluster.c:380-396`/`467-474`); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter." -- Validation: simulate a node restarting truncated; assert it is denied promotion until catch-up, then re-enabled (Req 2.2/2.6). -- Risks: interaction with `ha-single-leader-role` own-term gate and `ha-prevote` — re-verify the combined eligibility predicate against `raft.tla`. - -### Freshness-Floor Formula (task 4.1) - -> **Status:** PINNED (task 4.1, gating). This section is the authoritative contract that tasks **4.2** (implement the floor in `ha.c`/`cluster.c`) and **4.3** (promotion gate) implement against. **Verification method:** this is a **manual conceptual-alignment + Protocol Verification Checklist** argument, NOT a TLC run — there is no TLA+ toolchain in this environment (no `tla2tools.jar`/TLC/java). The argument maps the formula to `raft/raft.tla`'s `mlastLogIndex`/`LastTerm` semantics and to the `src/controller/CLAUDE.md` checklist. Per the spec scope note, `raft.tla` validates the **election/freshness logic** this change touches; it does **not** validate the *durability* of the floor — that is verified by tests 3.4 / 2.x, because `raft.tla` assumes a durable log (see Non-Goals and §5.3 of the broader Raft model). - -#### 1. The pinned formula - -Let, on member `M`: -- `durable_floor(M) = persist_durable_floor(M)` — the durable WAL tip (task 3.4): the highest LSN whose bytes are `fdatasync`'d locally, re-derived on recovery as the **highest intact replayed LSN** (the truncation boundary on `RECOVER_TRUNCATED`, the clean tip on `RECOVER_CLEAN`). This is a durable **last-log index**, not the commit position. -- `learned_committed(M) = last_known_committed_lsn(M)` — the volatile learned cluster commit high-water (a hint, possibly stale across a truncating restart). -- `learned_tag(M) = committed_config_tag(M)`, `local_tag(M) = config_version_tag` — the config epoch (compared first; unchanged by this task). - -**(a) What a candidate ADVERTISES (`mlastLogIndex` analog):** - -``` -cand_lsn = durable_floor(M) // NOT max(applied_lsn, learned_committed) -cand_tag = max(local_tag, learned_tag) // config epoch, unchanged -``` - -The candidate advertises **exactly its durable WAL tip**. It does **not** advertise `applied_lsn` (which can exceed the durable tip — entries applied to volatile state but not yet `fdatasync`'d) and does **not** advertise `learned_committed` (a hint that, after a truncating restart, can exceed the durable tip the node actually holds). This is the core change: a candidate may only claim a log position it can durably back. `cand_tag` is unchanged — config-epoch freshness is orthogonal to log durability. - -**(b) What a voter uses as its freshness FLOOR (the reconciliation):** - -``` -voter_floor_lsn = max( durable_floor(M), learned_committed(M) ) -voter_floor_tag = max( local_tag(M), learned_tag(M) ) // unchanged -``` - -The voter floor is the **max of its own durable tip and its learned cluster-committed high-water**. This is the precise reconciliation of the durable-derived floor against the existing `max(applied_lsn, last_known_committed_lsn)`: -- `applied_lsn` is **replaced** by `durable_floor` (a node must not demand of a candidate a position it itself only holds volatilely; and symmetrically must not under-advertise — see §3/§4). Note `applied_lsn` and `durable_floor` use **different conventions and are not directly comparable as raw counters**: `applied_lsn` is advanced to `last_good_lsn` — the last contiguously-held LSN — via `ha_advance_applied_lsn` (`ha.c:3050`, called at `ha.c:3426-3432`), whereas `durable_floor`/`durable_lsn` is the actual durable LSN *watermark* (the highest LSN whose bytes are `fdatasync`'d). (Note: this audit found the convention itself has since changed — the code comment at `ha.c:3426-3429` cites a later fix, bug-991, that moved `applied_lsn` from `last_good_lsn + 1` [the *next-expected* index this section originally described, cited at the now-stale `ha.c:1234/1253`] to bare `last_good_lsn`, "not `last_good_lsn + 1`," because the `+1` wedged the leader's catch-up streamer. This does not affect the argument here — the two quantities were already established as not directly comparable, and the safety/liveness proof in §3/§4 does not rely on the exact `applied_lsn` convention.) We therefore do **not** rely on a raw `durable_floor ≤ applied_lsn` inequality. The claim that this term never raises the floor beyond what is reachable rests entirely on the `D*` argument in §4: the voter floor's local-state contribution `durable_floor(v)` is bounded above by `D*` (the most-durable reachable candidate's tip) by definition of `D*`, so swapping in `durable_floor` cannot push the floor above some reachable candidate ⇒ cannot newly induce livelock (§4). -- `learned_committed` is **retained** as the safety backstop established by bug-339: a voter must never grant to a candidate below what the cluster has already **committed**, even if this voter's own durable tip is colder. Committed entries are now durable on a majority (Req 1/2.x), so at least a majority of voters carry that committed LSN in their `durable_floor` *and* every voter that learned the commit carries it in `learned_committed` — keeping the floor at the committed high-water on the voter side closes the window where a lagging voter would otherwise grant to a sub-committed candidate. - -**(c) The exact comparison (candidate ≥ voter floor — `logOk` analog):** - -``` -grant_freshness(cand, voter) == - cand_tag > voter_floor_tag - \/ ( cand_tag == voter_floor_tag /\ cand_lsn >= voter_floor_lsn ) -``` - -i.e. deny iff `cand_tag < voter_floor_tag`, or `cand_tag == voter_floor_tag && cand_lsn < voter_floor_lsn`. Term-then-index ordering, matching raft's `LastTerm`-then-`mlastLogIndex` lexicographic order; here the config epoch (`tag`) is the high-order key (raft's `LastTerm`) and the durable LSN is the low-order key (raft's `mlastLogIndex`). Equality at both ⇒ grant (a node at least as fresh, including exactly as fresh, is grantable — preserves the existing `>=` semantics and idempotent re-grant). - -**Candidate and voter use DIFFERENT quantities — this is the load-bearing distinction, and the implementation must keep the two paths separate:** - -> **Citation note (this audit):** the site line numbers below are corrected to their current locations; `ha.c` and `cluster.c` have grown substantially since this spec closed (2026-07-18) from later work (`ha-term-safety`, `raft-apply-decoupling`, `job-array-single-wal-record`, and unlisted in-flight specs). The mechanism itself is unchanged and verified current — a dedicated helper `ha_candidate_advertise_lsn(persist_t *p)` (`ha.c:5347-5349`, added since this spec's original text, which described inline snapshotting instead) now wraps the bare `persist_durable_floor(p)` call for both candidate sites. - -| Role | Site(s) | Quantity it must produce | How | -|------|---------|--------------------------|-----| -| **Candidate** (advertised `cand_lsn`) | `ha.c:7938` (real-vote snapshot), `ha.c:8017` (PreVote snapshot) — corrected from the original `ha.c:3660-3666`/`3735-3741` | **bare `durable_floor(M)`** — NOT folded with `learned_committed` | calls `ha_candidate_advertise_lsn(&c->persist)` (`ha.c:5347-5349`), a thin wrapper over `persist_durable_floor(p)`; do **NOT** route the advertised LSN through `ha_vote_self_freshness` | -| **Voter** (freshness floor `voter_floor_lsn`) | `cluster.c:544-557` (real-vote), `cluster.c:643-651` (PreVote), `ha.c:5436` (`handle_request_vote`, the `XCE_TESTING`-only legacy test gate) — corrected from the original `cluster.c:388-395`/`471-475`/`ha.c:2243` | **`max(durable_floor(M), learned_committed(M))`** — the backstop is retained | keep calling `ha_vote_self_freshness`, changing only its `local_lsn` input from `applied_lsn` to `durable_floor(M)`; the function still folds in `last_known_committed_lsn` | - -**Why the candidate must NOT reuse `ha_vote_self_freshness`.** That function (`ha.c:5310-5324`; note corrected from the original `ha.c:2127-2142`) *unconditionally* computes `*out_lsn = max(local_applied_lsn, last_known_committed_lsn)`. Feeding it `durable_floor` as the local input therefore yields `max(durable_floor, learned_committed)` — **not** the bare `durable_floor` §1a/§3 require. A truncated node with `durable_floor = 4` but a stale volatile `learned_committed = 8` would then advertise `cand_lsn = 8` and **win** — exactly the truncated-candidate-wins hole this task closes (§5a). So 4.2 must give the candidate sites a **candidate-only path** that calls `persist_durable_floor(p)` directly and assigns it to `cand_lsn`, bypassing the `learned_committed` fold. (The candidate's `cand_tag` is still `max(local_tag, learned_tag)`; only the **LSN axis** must bypass the fold.) The two PreVote/real-vote candidate snapshots currently call `ha_vote_self_freshness` for both fields — 4.2 splits the LSN field off onto the bare-`durable_floor` path while leaving the tag field on the existing max. - -**Why safety lives on the advertisement axis and the backstop lives on the voter axis.** The candidate must not **over-advertise** beyond durable state — that is the safety requirement (§3): pinning `cand_lsn = durable_floor` is what denies a truncated candidate. The `learned_committed` backstop belongs on the **voter** axis (§3 dual guard): a voter must refuse any candidate below the cluster-committed boundary it learned, even when the voter's own durable tip momentarily lags. Folding `learned_committed` into the *candidate* advertisement would defeat the very safety property it protects on the *voter* side; folding it into the *voter* floor is exactly the bug-339 backstop. raft.tla confirms the split: `RequestVote` sends `mlastLogIndex |-> Len(log[i])` — the **bare candidate log length**, with no committed term folded into the candidate's advertisement; the committed-high-water guard lives only on the voter's `logOk` floor. This recipe is now internally consistent with §1a (bare candidate `durable_floor`), §3 (quorum-intersection on the advertised `durable_floor`), and §4 (voter floor bounded by `D*`). - -**(d) `safe_committed_lsn` is UNCHANGED and still from voter replies.** This floor governs only **vote granting / advertisement**. The post-election `safe_committed_lsn` is still computed by `ha_compute_safe_committed_lsn` (`ha.c:1144`; note corrected from the original `ha.c:480/562`) from the `peerLastLogLsn` values in the `RequestVoteReply` set, never from this floor and never from a stale ctrlhello (CLAUDE.md "Post-Election Safe Commit Point"). The reply's `peer_last_log_lsn` field (assigned at `cluster.c:596` real-vote, `cluster.c:735` PreVote; note corrected from the original `cluster.c:426`) now carries the durable-backed `voter_floor_lsn`, which is *more* conservative (never above a durable position), so the elected leader's `safe_committed_lsn` is if anything tightened, never loosened — `persist_truncate_wal_above(safe_committed_lsn)` still only removes entries strictly above a majority-backed point. - -#### 2. raft.tla alignment - -`raft.tla` (`HandleRequestVoteRequest`, lines 284-289): - -``` -logOk == \/ m.mlastLogTerm > LastTerm(log[i]) - \/ /\ m.mlastLogTerm = LastTerm(log[i]) - /\ m.mlastLogIndex >= Len(log[i]) -``` - -Mapping: -- `Len(log[i])` (voter's last-log **index**) ↔ `voter_floor_lsn` on the durable axis. Our durable WAL tip `persist_durable_floor` is the faithful analog of `Len(log[i])`: in `raft.tla` the log is **restored intact from stable storage** on restart (the model has no volatile-vs-durable distinction — every entry in `log[i]` is durable by assumption). By making committed entries durable on a majority (2.x) and deriving the advertised/floor index from `persist_durable_floor` (the on-disk tip) rather than a volatile `applied_lsn`, our `cand_lsn`/`voter_floor_lsn` become the genuine "length of the durable log," so the `mlastLogIndex >= Len(log[i])` comparison holds with the same meaning the model assigns it. -- `m.mlastLogIndex` (candidate's advertised index) ↔ `cand_lsn = durable_floor(candidate)` — exactly raft's `mlastLogIndex |-> Len(log[i])` in `RequestVote` (line 196): the candidate advertises *its own log length*, which in our durable-log mapping is its durable tip. The pre-change code advertised `max(applied_lsn, learned_committed)`, which could exceed `Len(durable log)` — an over-advertisement raft.tla cannot express (in the model you cannot advertise an index beyond `Len(log[i])`). The fix restores faithfulness to the model. -- `LastTerm(log[i])` / `m.mlastLogTerm` ↔ the config epoch `tag`. XCE folds per-entry Raft terms into a monotone config-version tag (single-writer log; term changes ride the config epoch); the high-order lexicographic key is preserved. This task does not change tag handling. - -**Caveat (restated):** `raft.tla` does **not** model durability — it assumes `log[i]` survives restart intact, so it cannot itself prove that `persist_durable_floor` *is* durable. That property (the WAL-tip derivation is genuinely on stable storage and survives an ungraceful kill) is the subject of tests 3.4 and 2.x, not this alignment argument. What `raft.tla` *does* validate — and what this section argues we faithfully implement — is the election/freshness **logic**: with our durable tip standing in for `Len(log)`, the `logOk` predicate and the Leader-Completeness proof carry over. - -#### 3. Leader Completeness (safety) — a truncated/behind candidate cannot win - -Claim: under this formula, any candidate that wins an election holds **all committed entries** (raft's Leader Completeness). - -Argument: -1. By Req 1/2.x, an entry committed at LSN `L` is `fdatasync`'d on a **majority** `Q_L` of members; each member in `Q_L` has `durable_floor ≥ L` (durability is a contiguous prefix watermark — CLAUDE.md Commit Ordering — so a durable tip ≥ L implies all entries ≤ L are durable on that member). -2. A candidate `C` needs votes from a **majority** `Q_v`. Any two majorities intersect: `Q_L ∩ Q_v ≠ ∅`, so at least one voter `v` has `durable_floor(v) ≥ L`, hence `voter_floor_lsn(v) ≥ durable_floor(v) ≥ L`. -3. For `v` to grant, the comparison requires `cand_lsn ≥ voter_floor_lsn(v) ≥ L`. Since `cand_lsn = durable_floor(C)` (the candidate cannot advertise above its durable tip), `durable_floor(C) ≥ L` — `C` durably holds entry `L` (again by the prefix-watermark property, all entries ≤ L). -4. Therefore a candidate whose durable tip is **below** the committed high-water `L` (a truncated or behind node) is denied by every voter in the intersection and **cannot assemble a majority**. The pre-change bug was that `C` could advertise `learned_committed` (or volatile `applied_lsn`) above its durable tip and thereby satisfy step 3 *without durably holding `L`* — winning, then truncating committed work away. Pinning `cand_lsn = durable_floor(C)` closes exactly that hole. - -The retained `learned_committed` term in the *voter* floor (§1b) is the dual guard: even a voter whose own durable tip momentarily lags `L` (e.g. just rejoined) still refuses a candidate below the cluster-committed `L` it learned, so the safety margin does not depend on which majority voted. - -#### 4. Liveness (no election livelock) — the most-durable candidate can always win - -Claim: the formula is not "too high" — among any available majority, the member holding the **longest durable log** can always be elected; the floor is always satisfiable by some reachable candidate, so there is no bug-339-style infinite election loop. - -Argument: -1. Let `D* = max{ durable_floor(m) : m in an available majority A }`, achieved by member `C*`. `C*` advertises `cand_lsn = D*`. -2. The committed high-water `L` satisfies `L ≤ D*`: `L` is durable on a majority, that majority intersects `A`, so some member of `A` has `durable_floor ≥ L`, hence `D* ≥ L`. -3. For any voter `v ∈ A`: `voter_floor_lsn(v) = max(durable_floor(v), learned_committed(v))`. We have `durable_floor(v) ≤ D*` by definition of `D*`. And `learned_committed(v) ≤ L ≤ D*` — a member never *learns* a committed LSN above the true committed high-water (`learned_committed` is a lower-or-equal estimate of `L`, advanced only from majority-acked commits / leader heartbeats). Therefore `voter_floor_lsn(v) ≤ D* = cand_lsn(C*)`, so the comparison `cand_lsn ≥ voter_floor_lsn(v)` **holds for every** `v ∈ A`, and `C*` is grantable by the whole majority. -4. Hence the floor is always satisfiable by the most-durable reachable candidate ⇒ an election can always complete ⇒ no livelock. - -Why this avoids the bug-339 pattern specifically: bug-339 was an *infinite* loop because the RDHUP failover path advertised a **stale `committed_lsn`** that no longer matched the floor the (other) voters demanded — the advertiser claimed a position it could not satisfy against the floor, so it kept losing and retrying. Under the pinned formula **no node advertises a position it cannot durably back** (`cand_lsn = durable_floor`), and the voter floor is bounded above by `D*` (step 3, because both `durable_floor(v) ≤ D*` and `learned_committed(v) ≤ L ≤ D*`). The floor is therefore always met by `C*`. The earlier bug-339 fix pinned the voter floor to `max(applied_lsn, last_known_committed_lsn)`; this task swaps the local-state term from the volatile `applied_lsn` to the durable `durable_floor`. Per §1b we do **not** rest on a raw `durable_floor ≤ applied_lsn` comparison (the two use different counter conventions); instead, the no-livelock guarantee rests solely on the `D*` bound of §4 — `durable_floor(v) ≤ D*` for every reachable voter `v` — so the swapped-in term can never push the voter floor above the most-durable reachable candidate `C*`. It therefore cannot reintroduce a too-high (livelock) floor while it closes the too-low (safety) hole on the advertisement side. - -> **The safety/liveness tension is resolved, not papered over.** The "too low ⇒ truncated node wins" failure lives on the **advertisement** axis (what a candidate may claim) and is fixed by `cand_lsn = durable_floor` (§3). The "too high ⇒ livelock" failure lives on the **voter-floor** axis (what a voter demands) and is avoided because the voter floor is bounded above by `D*`, the most-durable reachable candidate's tip (§4). The two fixes act on different sides of the same comparison, so there is no contradiction: §3 raises the *floor for under-durable candidates* without raising it *beyond the best available candidate*. - -#### 5. The two required cases - -**(a) Truncated-candidate DENIED.** Node `T` restarts; its WAL hit a CRC break at LSN 5, so `replay_wal` returns `RECOVER_TRUNCATED` with `durable_floor(T) = 4` (highest intact replayed LSN). Cluster committed high-water `L = 8` (entries 5–8 are durable on the surviving majority `Q`). `T` becomes a candidate and advertises `cand_lsn = durable_floor(T) = 4` (it **cannot** advertise the stale `learned_committed = 8` it may still hold in volatile state — that is exactly the over-advertisement the formula forbids). Any voter `v ∈ Q` has `durable_floor(v) ≥ 8`, so `voter_floor_lsn(v) ≥ 8 > 4 = cand_lsn` ⇒ every such voter **denies** (§1c). `T` cannot reach a majority ⇒ not elected (Leader Completeness preserved, §3). Independently, the promotion fence (task 4.3) bars `T` from `raft_cb_become_leader` because `persist_recovery_truncated(T)` is true and `durable_floor(T) < L` — so even a degenerate quorum cannot promote it. `T` instead catches up from a healthy peer to `durable_floor ≥ 8`, then becomes both grantable and promotable (Req 2.6 / Boot Policy Decision Table row 3). - -**(b) Longest-durable-log STILL ELECTS (no livelock).** Three nodes; the primary committed through **LSN 8**, so `L = 8`. Because committed ⇒ durable on a majority (Req 1/2.x), at least two of the three members hold `durable_floor ≥ 8`; the primary then died. The two survivors are `A` and `B`. - -- *Sub-case L=8 is consistent with both survivors:* the committing majority for `L = 8` was some 2-of-3 set. If the dead primary was **not** in that majority, then both survivors `A` and `B` were in it ⇒ both have `durable_floor ≥ 8`. If the dead primary **was** in it, then exactly one of the two acking members survives; say `A` has `durable_floor = 8` while `B` ended cleanly one entry behind at `durable_floor = 7` (it appended/acked 8 against an earlier *uncommitted* round but its flusher had not advanced past 7 at crash — `B` is `RECOVER_CLEAN`, just behind, not truncated). Either way **some survivor has `durable_floor ≥ 8`**, so `D* ≥ 8`. - -Take the tighter sub-case (`A=8`, `B=7`), where `D* = 8` achieved by `A`. `A` advertises bare `cand_lsn = durable_floor(A) = 8`. Voter `B`: `voter_floor_lsn(B) = max(durable_floor=7, learned_committed(B) ≤ L = 8)`. Here `learned_committed(B) ≤ 8`; if `B` had learned the `L = 8` commit then `voter_floor_lsn(B) = 8 = cand_lsn(A)` ⇒ `B` grants (equality grants, §1c); if `B` only learned up to 7 then `voter_floor_lsn(B) = 7 < 8` ⇒ `B` also grants. **Either way `B` grants `A`**, `A` self-grants ⇒ majority ⇒ **`A` is elected** (no livelock — consistent with §4 step 3, `voter_floor_lsn(v) ≤ D*`). Symmetrically `B` cannot win if `A` participates: `B` advertises bare `cand_lsn = 7`, and `A`'s `voter_floor_lsn = max(8, learned_committed_A ≤ 8) = 8 > 7` ⇒ `A` denies `B`, steering the election to the most-durable node `A` (§4). - -**The dangerous-looking livelock state is UNREACHABLE.** The only way `A` could fail to be grantable by a majority is if some voter `v` had `voter_floor_lsn(v) > cand_lsn(A) = D*` — which by §4 step 3 requires `learned_committed(v) > D*`. But `learned_committed(v) ≤ L` (a node never learns a committed LSN above the true high-water) and `L ≤ D*` (committed ⇒ durable on a majority that intersects the survivors ⇒ some survivor has `durable_floor ≥ L`, so `D* ≥ L`). Hence `learned_committed(v) ≤ L ≤ D*` for every survivor `v`, so no survivor can demand a floor above `D*`, and a two-equal-tip-but-high-`learned_committed` deadlock (both survivors `durable_floor = 5`, both `learned_committed = 8`) **cannot arise when `L = 8` is genuinely committed**: `L = 8` committed forces some survivor to `durable_floor ≥ 8 > 5`, contradicting "both tips = 5". This is precisely the property §4 needs the case to demonstrate. - -The elected `A` then computes `safe_committed_lsn` from the vote replies (durable-backed `peer_last_log_lsn` of `A`=8, `B`=7 ⇒ majority-backed safe point), unchanged by this task. - -#### 6. Protocol Verification Checklist pass (`src/controller/CLAUDE.md`) - -Walking the relevant items under the pinned formula: - -- **Freshness gate — candidate `lastLogTerm >= voter` AND `lastLogIndex >= voter`:** HOLDS. The comparison (§1c) is `cand_tag >= voter_floor_tag` (term axis) AND, at equal tag, `cand_lsn >= voter_floor_lsn` (index axis) — exactly the checklist's two-key gate, now with `lastLogIndex` meaning the **durable** tip on both sides. -- **`safe_committed_lsn` derived from voter replies, not stale ctrlhello:** HOLDS, UNCHANGED. This task does not touch `ha_compute_safe_committed_lsn`; it still reads `votes[].peer_last_log_lsn` (§1d). The floor change makes those replies durable-backed (more conservative), never sourced from a ctrlhello. -- **`commitIndex` / `committed_lsn` monotonic, contiguous prefix:** HOLDS, UNCHANGED. The floor affects election only; commit advancement is untouched here (it is task 2.x). The freshness floor never writes `committed_lsn`. -- **WAL truncation on promotion only removes entries above the safe commit point:** HOLDS. `persist_truncate_wal_above(safe_committed_lsn)` is unchanged; the durable-backed (tightened) `safe_committed_lsn` means truncation is, if anything, more conservative. -- **`durable_lsn` is a prefix watermark (never a per-entry set):** HOLDS and is RELIED UPON — the Leader-Completeness argument (§3) and liveness argument (§4) both use "durable tip ≥ L ⇒ all entries ≤ L durable," which is exactly the prefix-watermark property the checklist asserts. `cand_lsn`/`voter_floor_lsn` are single LSNs (watermarks), not sets. -- **(post `ha-commit-durability`) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority:** the *election-side* analog HOLDS here — a member contributes to a **vote-grant** (and advertises) only at its durably-persisted LSN. The commit-majority arithmetic side is task 2.x; this task makes the freshness/advertisement side durable-backed so the two are consistent. -- **`votedFor` updated atomically before granting; `currentTerm` monotonic:** UNCHANGED by this task (term/vote bookkeeping in `raft_on_vote_request` is untouched); the floor change only gates *whether* `grant` is reached. - -**Concern / caveat (restated):** `raft.tla` validates the freshness/election **logic** above; it does **not** validate that `persist_durable_floor` is genuinely durable (the model assumes a durable log). That durability is verified by tests 3.4 (durable-floor re-derivation on recovery) and 2.x (acknowledged ⇒ durable). No safety/liveness contradiction was found — the formula simultaneously satisfies Leader Completeness (§3) and no-livelock (§4) because the two pressures act on opposite sides of the comparison. - -### Runtime / Test — DurabilityConfig + TestSeam - -| Field | Detail | -|-------|--------| -| Intent | Deterministic, configurable durability timing for verification | -| Requirements | 3.1, 3.2, 3.3 | - -**Responsibilities & Constraints** -- `--wal-sync-ms` plumbed `main.c → persist_config_t` before `persist_init`; default unchanged (`DEFAULT_WAL_SYNC_MS`). Under group-commit this bounds the **maximum batching delay** (a flush fires on the append-signal or when this elapses), never a durability gap — an acked op is always durable regardless of the value. -- The seam's verification role: an `XCE_TESTING` hook (and/or a large `--wal-sync-ms` to widen the batching window) lets a test assert an acknowledged op is recoverable after an immediate ungraceful termination — proving durability comes from the commit gate, not the flush cadence. -- External acknowledgment semantics are identical with or without the mode (timing controllability only). - -**Contracts**: State [x] - -**Implementation Notes** -- Integration: follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`, `main.c:32`) convention; the actual durability test-seam functions live in `src/controller/persist.c` (`persist_test_set_force_fsync_fail` at `persist.c:1547`, `persist_test_durable_lsn_on_disk` at `persist.c:1610`), not at the originally-cited `ha.c:2270-2335`. -- Validation: deterministic E2E mode in `test-commit-durability.sh` drives the seam so the soak becomes a deterministic RED on the unfixed build and GREEN after. -- Risks: ensure the seam is compiled out of production (`#ifdef XCE_TESTING`). - -## Error Handling - -### Error Strategy -- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state. -- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; when catch-up is impossible (no peer) the node enters a LOUD PERMANENT FENCE — non-promotable, **stays alive**, keeps retrying catch-up, NEVER self-exits (Req 2.4, 2.6). -- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark. - -### Monitoring -- Log every `durable_lsn` regression-clamp on truncate, every recovery verdict (with the truncation boundary), every promotion refusal (durable tip vs high-water), and every fence set/release — operator-visible, once per transition (consistent with the quorum-transition logging already in the HA stack). - -## Testing Strategy - -### Unit Tests -- `durable_lsn` advances only after `fdatasync`; `majority_candidate_locked` self-cap equals `durable_lsn` not `highest_appended_lsn` (1.2). -- `replay_wal` returns `RECOVER_TRUNCATED` on an injected CRC break at LSN 1 and `RECOVER_CLEAN` on an intact log (2.1). -- Promotion gate denies a recovery-truncated / behind-high-water member and re-enables it after catch-up (2.2, 2.6). -- `ha_vote_self_freshness` floor reflects the durable WAL tip, not a stale in-memory `last_known_committed_lsn` (2.3). -- Standby `persist_wal_write_replicated` `fdatasync`s before ACK — regression pin (1.3). - -### Integration Tests -- Commit advances `committed_lsn` only when self is durable + a standby ACK is durable; reply released only after (1.1, 1.4). -- Kill the sole durable contributor mid-commit; acknowledged op survives on the majority (1.5). -- Durability work does not delay heartbeat handling beyond tolerance (1.6). - -### E2E / Failover -- `src/tools/test-commit-durability.sh` soak: rolling SIGKILL+restart under load asserts every acknowledged job survives (1.1, 1.5, 2.x). With the `--wal-sync-*` seam, run a **deterministic** variant: widen/disable background timing, kill a durable contributor in-window, assert survival — RED on the current build, GREEN after the fix (3.1, 3.2). - -### Performance -- Measure submit latency delta from synchronous `fdatasync` vs the existing replication round-trip; confirm it is within the round-trip already paid before deciding group-commit is needed (research item). - -## Performance & Scalability -- **Group-commit** amortizes one `fdatasync` across all entries pending at flush time, so the per-entry sync cost falls as load rises — critical for the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage) which is not client-latency-bound. `--wal-sync-ms` bounds the maximum batching delay (a latency/throughput tunable), never a durability gap. The `job-array-single-wal-record` roadmap item independently cuts per-array entry count. Target: zero acknowledged-job loss across failover (correctness) takes precedence over per-submit latency; group-commit keeps the throughput cost sub-linear in write rate. - -## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`) -- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3). -- **Unrecoverable-terminal decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Concrete policy: fence at boot, attempt catch-up from a healthy primary, and on a failure to reach a committed-consistent state (no peer) enter a **LOUD PERMANENT FENCE** — non-promotable, stays alive, keeps retrying catch-up, **NEVER self-exit** (Req 2.2, 2.4, 2.6). **RESOLVED (task 3.2; terminal corrected by the never-self-exit rework 2026-06-18 — the original self-`abort` FATAL was replaced by the loud permanent fence):** see the **Boot Policy Decision Table (task 3.2)** above; tasks 3.3 (loud-fence terminal) and 4.3 (fence + release) implement it. -- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2). -- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never the unrecoverable terminal on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2). -- **Synchronous-append path — RESOLVED (task 2.4), re-verified current:** `persist_wal_sync_append` (`persist.c:5639`) no longer advances the commit position unguarded — it routes through `sync_append_legacy_commit_store` (`persist.c:5617`), which only stores past `committed_lsn` when `written_lsn <= durable_lsn`, closing the ACK-without-durability hole (Req 1.1, 1.2; regression-pinned by `test_legacy_sync_append_holds_committed_at_durable`). (Note: this audit found the path is NOT dormant as originally described here — it now has a live production caller, `submit_job_wal_entry_and_commit` (`persist.c:5990-6004`), introduced by `raft-apply-decoupling`. The guard makes this safe regardless, so no further fix is needed, but the "no production caller today" premise in this section's original text no longer holds.) diff --git a/.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log b/.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log deleted file mode 100644 index 30926a84..00000000 --- a/.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log +++ /dev/null @@ -1,164 +0,0 @@ -[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] My name is ctrl-1 -[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1 -[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1) -[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin -[2026-06-21 08:37:35,671] [1217413] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:37:35,671] [1217413] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:37:35,671] [1217413] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c -[2026-06-21 08:37:35,671] [1217413] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=0) -[2026-06-21 08:37:35,671] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:37:35,674] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started) -[2026-06-21 08:37:40,873] [1217413] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:37:40,874] [1217413] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:37:40,874] [1217413] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=0 current_lsn=1; applied_lsn=0 advertised for catch-up -[2026-06-21 08:37:40,874] [1217413] [ INFO] [raft_cb_become_follower] HA FSM (raft): → CTRL_STANDBY (term=1, first_boot=1) -[2026-06-21 08:37:40,874] [1217413] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:37:50,478] [1217413] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover -[2026-06-21 08:37:50,479] [1217413] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 8ms (priority=0) to avoid split vote (bug-606) -[2026-06-21 08:37:50,679] [1217413] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=2, n_peers=3) — staying CTRL_STANDBY, no persist -[2026-06-21 08:37:50,728] [1217413] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 2) -[2026-06-21 08:37:50,728] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started) -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=0 safe_config_tag=5298030106630493228 (from 1 granted voters) -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=2, first_boot=0) -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992) -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel -[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993) -[2026-06-21 08:37:50,738] [1217413] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0) -[2026-06-21 08:37:50,738] [1217413] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=1, writable_lsn gate armed -[2026-06-21 08:37:53,924] [1217413] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1) -[2026-06-21 08:37:53,925] [1217413] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1) -[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=2 < target=5 → streaming [2+1..5] -[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=3 target_lsn=5 -[2026-06-21 08:39:27,322] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [2+1..5] to standby -[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=3 < target=7 → streaming [3+1..7] -[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=4 target_lsn=7 -[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [3+1..7] to standby -[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=7 < target=11 → streaming [7+1..11] -[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=11 -[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..11] to standby -[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=4 < target=13 → streaming [4+1..13] -[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=5 target_lsn=13 -[2026-06-21 08:39:27,923] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [4+1..13] to standby -[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=8 < target=13 → streaming [8+1..13] -[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13 -[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby -[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=5 < target=13 → streaming [5+1..13] -[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=6 target_lsn=13 -[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [5+1..13] to standby -[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=9 < target=13 → streaming [9+1..13] -[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13 -[2026-06-21 08:39:28,524] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby -[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=6 < target=13 → streaming [6+1..13] -[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=7 target_lsn=13 -[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [6+1..13] to standby -[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=10 < target=13 → streaming [10+1..13] -[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13 -[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby -[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=7 < target=13 → streaming [7+1..13] -[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=13 -[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..13] to standby -[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=11 < target=13 → streaming [11+1..13] -[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13 -[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby -[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=8 < target=13 → streaming [8+1..13] -[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13 -[2026-06-21 08:39:29,526] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby -[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=12 < target=13 → streaming [12+1..13] -[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13 -[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby -[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=9 < target=13 → streaming [9+1..13] -[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13 -[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby -[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=10 < target=13 → streaming [10+1..13] -[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13 -[2026-06-21 08:39:30,327] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby -[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=11 < target=13 → streaming [11+1..13] -[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13 -[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby -[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=12 < target=13 → streaming [12+1..13] -[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13 -[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby -[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] My name is ctrl-1 -[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1 -[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1) -[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin -[2026-06-21 08:43:44,327] [1224672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=2) -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=2, raft_term=2, primary_fd=-1) — skipping own election -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=13 advertised for catch-up -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:43:44,328] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:43:45,250] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:43:45,250] [1224672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6) -[2026-06-21 08:47:39,017] [1224672] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover -[2026-06-21 08:47:39,017] [1224672] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 122ms (priority=0) to avoid split vote (bug-606) -[2026-06-21 08:47:39,217] [1224672] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist -[2026-06-21 08:47:39,262] [1224672] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3) -[2026-06-21 08:47:39,265] [1224672] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started) -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters) -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=0) -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992) -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel -[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993) -[2026-06-21 08:47:39,272] [1224672] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0) -[2026-06-21 08:47:39,272] [1224672] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed -[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=48 → streaming [46+1..48] -[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=48 -[2026-06-21 08:47:41,420] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..48] to standby -[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=48 < target=53 → streaming [48+1..53] -[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=53 -[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..53] to standby -[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] My name is ctrl-1 -[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1 -[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1) -[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin -[2026-06-21 08:47:47,839] [1228504] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=3) -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:47:47,841] [1228504] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up -[2026-06-21 08:47:47,841] [1228504] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:47:47,841] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:47:52,855] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:52,855] [1228504] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:47:52,855] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:47:52,855] [1228504] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8) -[2026-06-21 08:48:04,047] [1228504] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover -[2026-06-21 08:48:04,047] [1228504] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 159ms (priority=0) to avoid split vote (bug-606) -[2026-06-21 08:48:04,248] [1228504] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist -[2026-06-21 08:48:04,295] [1228504] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4) -[2026-06-21 08:48:04,295] [1228504] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started) -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=53 safe_config_tag=5298030106630493228 (from 1 granted voters) -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0) -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992) -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel -[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993) -[2026-06-21 08:48:04,305] [1228504] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0) -[2026-06-21 08:48:04,307] [1228504] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=54, writable_lsn gate armed -[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54) -[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54) diff --git a/.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log b/.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log deleted file mode 100644 index f5bb0bef..00000000 --- a/.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log +++ /dev/null @@ -1,99 +0,0 @@ -[2026-06-21 08:37:36,163] [1217460] [ INFO] [ main] My name is ctrl-2 -[2026-06-21 08:37:36,164] [1217460] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2 -[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2) -[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin -[2026-06-21 08:37:36,173] [1217460] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:37:36,173] [1217460] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-2/conf/current' (rc=-2) — will fetch from primary after election -[2026-06-21 08:37:36,173] [1217460] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=0) -[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election -[2026-06-21 08:37:36,673] [1217460] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded -[2026-06-21 08:37:36,673] [1217460] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery -[2026-06-21 08:37:36,673] [1217460] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:37:46,177] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover -[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8) -[2026-06-21 08:37:50,743] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:37:50,743] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:37:50,743] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:37:50,743] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7) -[2026-06-21 08:40:02,029] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover -[2026-06-21 08:40:02,029] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 585ms (priority=1) to avoid split vote (bug-606) -[2026-06-21 08:40:02,629] [1217460] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=1, n_peers=3) — staying CTRL_STANDBY, no persist -[2026-06-21 08:40:03,270] [1217460] [ INFO] [ha_on_heartbeat_received] HA: primary heartbeat arrived mid-PreVote — cancelling the round and re-attaching as CTRL_STANDBY (leader alive, no term change) (Req 4.2) -[2026-06-21 08:40:03,273] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:40:03,273] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:40:03,273] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:40:03,273] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=9) -[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] My name is ctrl-2 -[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2 -[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2) -[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin -[2026-06-21 08:46:07,149] [1227110] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:46:07,149] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:46:07,149] [1227110] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c -[2026-06-21 08:46:07,149] [1227110] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=3) -[2026-06-21 08:46:07,649] [1227110] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election -[2026-06-21 08:46:07,649] [1227110] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:46:07,649] [1227110] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:46:07,650] [1227110] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=22 current_lsn=23; applied_lsn=22 advertised for catch-up -[2026-06-21 08:46:07,650] [1227110] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:46:07,650] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:46:09,319] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:46:09,319] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6) -[2026-06-21 08:47:38,988] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover -[2026-06-21 08:47:38,988] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 556ms (priority=1) to avoid split vote (bug-606) -[2026-06-21 08:47:39,276] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:39,276] [1227110] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:47:39,277] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:47:39,277] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8) -[2026-06-21 08:47:39,588] [1227110] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606) -[2026-06-21 08:47:51,992] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover -[2026-06-21 08:47:51,992] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 662ms (priority=1) to avoid split vote (bug-606) -[2026-06-21 08:47:52,793] [1227110] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist -[2026-06-21 08:47:52,840] [1227110] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4) -[2026-06-21 08:47:52,843] [1227110] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started) -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters) -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0) -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9991) -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel -[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993) -[2026-06-21 08:47:52,850] [1227110] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0) -[2026-06-21 08:47:52,850] [1227110] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed -[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=48 < target=49 → streaming [48+1..49] -[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=49 -[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..49] to standby -[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=49 < target=53 → streaming [49+1..53] -[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=50 target_lsn=53 -[2026-06-21 08:47:53,596] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [49+1..53] to standby -[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] My name is ctrl-2 -[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2 -[2026-06-21 08:47:59,844] [1228672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2) -[2026-06-21 08:47:59,845] [1228672] [ INFO] [ cluster_init_phase1] HA: restored term=4 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin -[2026-06-21 08:47:59,854] [1228672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:47:59,854] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:59,854] [1228672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c -[2026-06-21 08:47:59,854] [1228672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=4) -[2026-06-21 08:48:00,354] [1228672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-2, voted_term=4, raft_term=4, primary_fd=-1) — skipping own election -[2026-06-21 08:48:00,354] [1228672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:48:00,355] [1228672] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:48:00,357] [1228672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up -[2026-06-21 08:48:00,357] [1228672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:48:00,357] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:48:04,312] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:48:04,312] [1228672] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:48:04,312] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:48:04,312] [1228672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7) diff --git a/.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log b/.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log deleted file mode 100644 index 59dd67ef..00000000 --- a/.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log +++ /dev/null @@ -1,213 +0,0 @@ -[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] My name is ctrl-3 -[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3 -[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3) -[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin -[2026-06-21 08:37:36,672] [1217472] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:37:36,672] [1217472] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-3/conf/current' (rc=-2) — will fetch from primary after election -[2026-06-21 08:37:36,672] [1217472] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=0) -[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election -[2026-06-21 08:37:37,672] [1217472] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded -[2026-06-21 08:37:37,672] [1217472] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery -[2026-06-21 08:37:37,672] [1217472] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:37:47,276] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover -[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8) -[2026-06-21 08:37:50,743] [1217472] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:37:50,743] [1217472] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:37:50,743] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:37:50,743] [1217472] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7) -[2026-06-21 08:40:01,930] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover -[2026-06-21 08:40:01,930] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1150ms (priority=2) to avoid split vote (bug-606) -[2026-06-21 08:40:02,636] [1217472] [ INFO] [ myreq_handler] HA: PreVote DENIED candidate='ctrl-2' term=1 reason=stale_term (my_term=0, floor tag=5298030106630493228 lsn=13) -[2026-06-21 08:40:03,130] [1217472] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist -[2026-06-21 08:40:03,176] [1217472] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3) -[2026-06-21 08:40:03,176] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3) -[2026-06-21 08:40:03,180] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started) -[2026-06-21 08:40:03,223] [1217472] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=13 safe_config_tag=5298030106630493228 (from 1 granted voters) -[2026-06-21 08:40:03,223] [1217472] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:40:03,223] [1217472] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:40:03,224] [1217472] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=14 advertised for catch-up -[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=1) -[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel -[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 0, localhost:9992) -[2026-06-21 08:40:03,229] [1217472] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0) -[2026-06-21 08:40:03,230] [1217472] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=14, writable_lsn gate armed -[2026-06-21 08:40:06,333] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1) -[2026-06-21 08:40:06,333] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:40:06,374] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=15 (committed_tag=5298030106630493228 committed_lsn=14) -[2026-06-21 08:40:33,348] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 10) -[2026-06-21 08:40:33,348] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:41:03,364] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 20) -[2026-06-21 08:41:03,364] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:41:33,382] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 30) -[2026-06-21 08:41:33,382] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:42:03,397] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 40) -[2026-06-21 08:42:03,397] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:42:33,411] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 50) -[2026-06-21 08:42:33,411] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:43:03,425] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 60) -[2026-06-21 08:43:03,425] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:43:33,240] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 70) -[2026-06-21 08:43:33,240] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111 -[2026-06-21 08:43:45,247] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991) -[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent -[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 1, localhost:9991) -[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[0] lsn=13 < committed_lsn=22 → stream WAL batches -[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=14 target_lsn=22 -[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [13+1..22] to standby -[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=14 < target=22 → streaming [14+1..22] -[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=15 target_lsn=22 -[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [14+1..22] to standby -[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=15 < target=22 → streaming [15+1..22] -[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=16 target_lsn=22 -[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [15+1..22] to standby -[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=16 < target=22 → streaming [16+1..22] -[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=17 target_lsn=22 -[2026-06-21 08:43:48,850] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [16+1..22] to standby -[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=17 < target=22 → streaming [17+1..22] -[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=18 target_lsn=22 -[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [17+1..22] to standby -[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=18 < target=22 → streaming [18+1..22] -[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=19 target_lsn=22 -[2026-06-21 08:43:49,251] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [18+1..22] to standby -[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=19 < target=22 → streaming [19+1..22] -[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=20 target_lsn=22 -[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [19+1..22] to standby -[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=20 < target=22 → streaming [20+1..22] -[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=21 target_lsn=22 -[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [20+1..22] to standby -[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=21 < target=22 → streaming [21+1..22] -[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=22 target_lsn=22 -[2026-06-21 08:43:49,852] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby -[2026-06-21 08:43:51,294] [1217472] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=23 (committed_tag=5298030106630493228 committed_lsn=22) -[2026-06-21 08:45:24,291] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1) -[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991) -[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 1) -[2026-06-21 08:45:24,292] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111 -[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent -[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 0, localhost:9991) -[2026-06-21 08:45:51,305] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 10) -[2026-06-21 08:45:51,305] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111 -[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=35 < target=36 → streaming [35+1..36] -[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=36 -[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..36] to standby -[2026-06-21 08:46:09,314] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[1] (localhost:9992) -[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[1] established, ctrlhello sent -[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992) -[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=22 < committed_lsn=44 → stream WAL batches -[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=23 target_lsn=44 -[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [22+1..44] to standby -[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=23 < target=44 → streaming [23+1..44] -[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=24 target_lsn=44 -[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [23+1..44] to standby -[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=24 < target=44 → streaming [24+1..44] -[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=25 target_lsn=44 -[2026-06-21 08:46:12,717] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [24+1..44] to standby -[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=25 < target=44 → streaming [25+1..44] -[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=26 target_lsn=44 -[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [25+1..44] to standby -[2026-06-21 08:46:13,117] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=26 < target=44 → streaming [26+1..44] -[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=27 target_lsn=44 -[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [26+1..44] to standby -[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=27 < target=44 → streaming [27+1..44] -[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=28 target_lsn=44 -[2026-06-21 08:46:13,319] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [27+1..44] to standby -[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=28 < target=44 → streaming [28+1..44] -[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=29 target_lsn=44 -[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [28+1..44] to standby -[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=29 < target=44 → streaming [29+1..44] -[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=30 target_lsn=44 -[2026-06-21 08:46:13,720] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [29+1..44] to standby -[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=30 < target=44 → streaming [30+1..44] -[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=31 target_lsn=44 -[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [30+1..44] to standby -[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=31 < target=44 → streaming [31+1..44] -[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=32 target_lsn=44 -[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [31+1..44] to standby -[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=32 < target=44 → streaming [32+1..44] -[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=33 target_lsn=44 -[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [32+1..44] to standby -[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=33 < target=44 → streaming [33+1..44] -[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=34 target_lsn=44 -[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [33+1..44] to standby -[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=34 < target=44 → streaming [34+1..44] -[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=35 target_lsn=44 -[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [34+1..44] to standby -[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=35 < target=44 → streaming [35+1..44] -[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=44 -[2026-06-21 08:46:14,923] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..44] to standby -[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=36 < target=44 → streaming [36+1..44] -[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=37 target_lsn=44 -[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [36+1..44] to standby -[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=37 < target=44 → streaming [37+1..44] -[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=38 target_lsn=44 -[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [37+1..44] to standby -[2026-06-21 08:46:15,565] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=45 (committed_tag=5298030106630493228 committed_lsn=44) -[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=38 < target=44 → streaming [38+1..44] -[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=39 target_lsn=44 -[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [38+1..44] to standby -[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=39 < target=44 → streaming [39+1..44] -[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=40 target_lsn=44 -[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [39+1..44] to standby -[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=40 < target=44 → streaming [40+1..44] -[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=41 target_lsn=44 -[2026-06-21 08:46:16,125] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [40+1..44] to standby -[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=41 < target=44 → streaming [41+1..44] -[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=42 target_lsn=44 -[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [41+1..44] to standby -[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=42 < target=44 → streaming [42+1..44] -[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=43 target_lsn=44 -[2026-06-21 08:46:16,526] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [42+1..44] to standby -[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=43 < target=44 → streaming [43+1..44] -[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=44 target_lsn=44 -[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [43+1..44] to standby -[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=45 < target=48 → streaming [45+1..48] -[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=46 target_lsn=48 -[2026-06-21 08:47:29,558] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [45+1..48] to standby -[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=51 → streaming [46+1..51] -[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=51 -[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..51] to standby -[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] My name is ctrl-3 -[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3 -[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3) -[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin -[2026-06-21 08:47:36,071] [1228337] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data -[2026-06-21 08:47:36,072] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:36,072] [1228337] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c -[2026-06-21 08:47:36,072] [1228337] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=3) -[2026-06-21 08:47:37,072] [1228337] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election -[2026-06-21 08:47:37,072] [1228337] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4) -[2026-06-21 08:47:37,072] [1228337] [ INFO] [ xce_config_init] xce controller is running -[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue -[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host -[2026-06-21 08:47:37,074] [1228337] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=52 current_lsn=53; applied_lsn=52 advertised for catch-up -[2026-06-21 08:47:37,074] [1228337] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -[2026-06-21 08:47:37,074] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:47:39,277] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:39,277] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:47:39,277] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:47:39,277] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7) -[2026-06-21 08:47:52,078] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover -[2026-06-21 08:47:52,078] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1088ms (priority=2) to avoid split vote (bug-606) -[2026-06-21 08:47:52,855] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:47:52,855] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:47:52,855] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:47:52,855] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7) -[2026-06-21 08:47:53,279] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606) -[2026-06-21 08:48:04,083] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover -[2026-06-21 08:48:04,083] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1) -[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY -[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1092ms (priority=2) to avoid split vote (bug-606) -[2026-06-21 08:48:04,312] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1 -[2026-06-21 08:48:04,312] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c) -[2026-06-21 08:48:04,312] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms) -[2026-06-21 08:48:04,312] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7) -[2026-06-21 08:48:05,284] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606) diff --git a/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-1.txt b/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-1.txt deleted file mode 100644 index 4855e5c9..00000000 --- a/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-1.txt +++ /dev/null @@ -1,57 +0,0 @@ - 1 | 0 | NOOP | 0 | 00000000 | - 2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1 - 3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1 - 4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1 - 5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1 - 6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1 - 7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1 - 8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1 - 9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1 - 10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1 - 11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1 - 12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1 - 13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1 - 14 | 0 | NOOP | 0 | 00000000 | - 15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1 - 16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1 - 17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1 - 18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1 - 19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1 - 20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1 - 21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1 - 22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1 - 23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1 - 24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1 - 25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1 - 26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1 - 27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1 - 28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1 - 29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1 - 30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1 - 31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1 - 32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1 - 33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1 - 34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1 - 35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1 - 36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1 - 37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1 - 38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1 - 39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1 - 40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1 - 41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1 - 42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1 - 43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1 - 44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1 - 45 | 0 | NOOP | 0 | 00000000 | - 46 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1 - 47 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1 - 48 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1 - 49 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1 - 50 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1 - 51 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1 - 52 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1 - 53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1 - 54 | 0 | NOOP | 0 | 00000000 | - segments: 3 entries: 54 submits: 50 - lsn range: 1..54 recovered tip: 54 - gaps: 0 crc chain: intact diff --git a/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-2.txt b/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-2.txt deleted file mode 100644 index facc97d9..00000000 --- a/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-2.txt +++ /dev/null @@ -1,57 +0,0 @@ - 1 | 0 | NOOP | 0 | 00000000 | - 2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1 - 3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1 - 4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1 - 5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1 - 6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1 - 7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1 - 8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1 - 9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1 - 10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1 - 11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1 - 12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1 - 13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1 - 14 | 0 | NOOP | 0 | 00000000 | - 15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1 - 16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1 - 17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1 - 18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1 - 19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1 - 20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1 - 21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1 - 22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1 - 23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1 - 24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1 - 25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1 - 26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1 - 27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1 - 28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1 - 29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1 - 30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1 - 31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1 - 32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1 - 33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1 - 34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1 - 35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1 - 36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1 - 37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1 - 38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1 - 39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1 - 40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1 - 41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1 - 42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1 - 43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1 - 44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1 - 45 | 0 | NOOP | 0 | 00000000 | - 46 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1 - 47 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1 - 48 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1 - 49 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1 - 50 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1 - 51 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1 - 52 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1 - 53 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1 - 54 | 0 | NOOP | 0 | 00000000 | - segments: 3 entries: 54 submits: 50 - lsn range: 1..54 recovered tip: 54 - gaps: 0 crc chain: intact diff --git a/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-3.txt b/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-3.txt deleted file mode 100644 index fad1536b..00000000 --- a/.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-3.txt +++ /dev/null @@ -1,57 +0,0 @@ - 1 | 0 | NOOP | 0 | 00000000 | - 2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1 - 3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1 - 4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1 - 5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1 - 6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1 - 7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1 - 8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1 - 9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1 - 10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1 - 11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1 - 12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1 - 13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1 - 14 | 0 | NOOP | 0 | 00000000 | - 15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1 - 16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1 - 17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1 - 18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1 - 19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1 - 20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1 - 21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1 - 22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1 - 23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1 - 24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1 - 25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1 - 26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1 - 27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1 - 28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1 - 29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1 - 30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1 - 31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1 - 32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1 - 33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1 - 34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1 - 35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1 - 36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1 - 37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1 - 38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1 - 39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1 - 40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1 - 41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1 - 42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1 - 43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1 - 44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1 - 45 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1 - 46 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1 - 47 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1 - 48 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1 - 49 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1 - 50 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1 - 51 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1 - 52 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1 - 53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1 - 54 | 0 | NOOP | 0 | 00000000 | - segments: 2 entries: 54 submits: 51 - lsn range: 1..54 recovered tip: 54 - gaps: 0 crc chain: intact diff --git a/.kiro/specs/ha-commit-durability/requirements.md b/.kiro/specs/ha-commit-durability/requirements.md deleted file mode 100644 index d91d0101..00000000 --- a/.kiro/specs/ha-commit-durability/requirements.md +++ /dev/null @@ -1,61 +0,0 @@ -# Requirements Document - -## Project Description (Input) - -**Who has the problem:** Cluster operators running the XCE HA controller cluster — and every client whose acknowledged job/queue/worker operations must survive controller failover. - -**Current situation:** In a 3-node failover test, committed jobs were permanently lost (`joblist` dropped 10→3; the 7 earliest, already acknowledged, vanished). Root cause (bug-661): **acknowledgment does not imply durability.** The controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution as soon as the entry is written to volatile buffers, not after it is persisted to stable storage — background persistence happens on a timer. A standby already persists before acknowledging, but the primary does not. So a "committed" operation can be durable on as few as one member; a single ill-timed crash + restart, or a restarted member with a truncated log winning leadership, discards it. This violates the documented assumption that a committed operation is held durably on a majority. - -**What should change (chosen scope — durable commit + promotion fence):** -1. **Durable commit:** when the controller acknowledges an operation to a client, that operation's log entry must already be persisted to stable storage on a fault-tolerant majority (the acknowledging member plus ⌊N/2⌋ others). The controller must count its own copy toward the commit majority only once it is durably persisted. -2. **Recovery integrity and leadership fence (fail-loud):** a member that cannot reconstruct its committed state from persisted storage must be barred from leadership until it has caught up, and must never become authoritative while behind the cluster's known committed high-water mark — surfacing the failure loudly rather than silently discarding committed operations. - -The mechanism (synchronous persistence vs. later group-commit batching) is a design decision; group-commit batching is deferred to the roadmap as a contract-preserving throughput optimization. See `.kiro/specs/ha-commit-durability/brief.md` for the full discovery analysis, evidence trail, and design decision. - -## Introduction - -This spec closes the durability gap beneath the HA consistency machinery: it makes acknowledgment imply durability. After this spec, any operation the controller reports as successful is guaranteed to be persisted on a majority of members at the moment of acknowledgment, and therefore survives the loss of any minority of members and any subsequent leadership change. It also fences a member that cannot recover its committed state — or that lags the cluster's committed high-water mark — from becoming the authoritative leader, so a truncated or corrupted log can never overwrite acknowledged operations. The client reply path already waits on the commit barrier; this spec strengthens the meaning of "committed" to "durably persisted on a majority," so the existing reply gate enforces the invariant. A testability provision makes the guarantee deterministically verifiable without depending on race timing. - -## Boundary Context - -- **In scope**: the durable-commit invariant on the primary's commit path (self contributes to the commit majority only when durably persisted); pinning the standby's persist-before-acknowledge behavior; the no-premature-acknowledgment guarantee; survival of acknowledged operations across the crash+restart of any minority; preserving cluster liveness (heartbeat/election) while performing durability work; recovery-integrity and leadership fencing for members that cannot reconstruct committed state or lag the committed high-water mark; degraded-quorum preservation of the invariant; a deterministic durability-verification provision; the soak E2E (`src/tools/test-commit-durability.sh`) plus unit coverage. -- **Out of scope**: the replication/acknowledgment wire format (`ha-wal-replication`); per-entry majority arithmetic and commit-index advancement (`ha-commit-engine` — consumed, this spec changes only the durability *precondition* on its inputs); demotion truncate/self-restart and rejoin reconciliation of an intact log (`ha-convergence-recovery`); promotion no-op and own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds a durability/high-water *precondition*, not new vote logic); WAL group-commit batching (roadmap — a contract-preserving throughput optimization, deferred); single-record job-array persistence (`job-array-single-wal-record`, roadmap); entity-bearing snapshots (roadmap); client-side controller failover (`libxce-failover`). -- **Adjacent expectations**: relies on `ha-commit-engine`'s commit-position semantics (monotonic, per-entry majority) and on `ha-wal-replication`'s acknowledgment and catch-up machinery; the client reply is already gated on the commit barrier (`ha-commit-barrier-rollout`), so strengthening "committed" to "durable-majority" enforces the invariant without changing the reply path. Downstream, `ha-convergence-recovery`'s single-outcome convergence depends on this spec making "persisted committed state" genuinely durable; a returning member that this spec fences from leadership until caught up complements that spec's truncate-and-restart reconciliation. - -## Requirements - -### Requirement 1: Durable Commit — Acknowledgment Implies Majority Persistence - -**Objective:** As a cluster operator, on behalf of every client, I want any operation the controller acknowledges as successful to already be durably persisted on a majority of controllers, so that no acknowledged work is ever lost to a controller failure or failover. - -#### Acceptance Criteria - -1. When the controller acknowledges a state-changing operation to a client as successful, the controller shall have durably persisted that operation's log entry to stable storage on a majority of cluster members — the acknowledging member plus at least ⌊N/2⌋ other members. -2. While evaluating whether an operation is committed, the controller shall count its own log entry toward the commit majority only after that entry is durably persisted to its local stable storage, not when it is merely written to volatile buffers. -3. When a standby acknowledges a replicated log entry, the standby shall durably persist that entry to its local stable storage before sending the acknowledgment. -4. While a state-changing operation's log entry is not yet durably persisted on a majority of members, the controller shall not report that operation as successful to the requester. -5. If any minority of cluster members that contributed to an operation's commit are simultaneously lost and restarted, the acknowledged operation shall remain present and recoverable on the surviving cluster. -6. While performing durability persistence for a commit, the controller shall continue to service heartbeat and election processing without stalling, so that durability work does not induce spurious failover. - -### Requirement 2: Recovery Integrity and Leadership Fence - -**Objective:** As a cluster operator, I want a controller that cannot reconstruct its committed state to be barred from leadership until it has caught up, so that a member with a truncated or corrupted log can never become authoritative and discard acknowledged operations. - -#### Acceptance Criteria - -1. When the controller recovers its persisted log at startup, if the log cannot be reconstructed contiguously up to its last known committed boundary, the controller shall not silently discard committed entries and continue as if its committed state were complete. -2. If a controller's recovered log does not reach its last known committed boundary, the controller shall be ineligible to become primary until it has caught up to the cluster's committed boundary from a healthy member. -3. While a controller's committed position is below the cluster's known committed high-water mark, the controller shall not assume leadership in a way that would discard already-committed operations. -4. If a controller cannot recover to a consistent committed state and cannot catch up from a healthy member, the controller shall surface the condition as a loud, operator-visible alert and hold itself permanently non-promotable while staying alive and continuing to retry catch-up — refusing to serve potentially divergent state by refusing leadership, NOT by terminating the process. A controller MUST NEVER self-terminate (`exit`/`abort`) as a designed response to an operational condition: a self-exited controller requires manual admin recovery and degrades quorum meanwhile, whereas one that stays alive and keeps retrying self-heals automatically when a peer returns (see Requirement 2.6). (A genuine programming-invariant `assert`, or an external admin/supervisor kill, is not such a designed response.) -5. While the cluster operates with reduced redundancy or minimal quorum, the controller shall preserve the durable-commit guarantee of Requirement 1 across leadership changes, so that no acknowledged operation is lost even when quorum is temporarily degraded. -6. When a fenced member finishes catching up to the cluster's committed boundary, the controller shall restore that member's eligibility to participate in leadership. - -### Requirement 3: Deterministic Durability Verification - -**Objective:** As a developer verifying durability, I want the durability behavior to be deterministically controllable in a verification mode, so that the durable-before-acknowledge guarantee can be confirmed without depending on race timing. - -#### Acceptance Criteria - -1. Where a durability-verification mode is enabled, the controller shall make the durability behavior deterministically controllable (for example, by configuring or disabling background persistence) so that an acknowledged operation's durability can be asserted without relying on timing. -2. Where durability-verification mode removes reliance on background persistence, an operation that the controller acknowledged as successful shall be recoverable after an immediate, ungraceful termination of a contributing member. -3. The controller shall preserve the same externally observable acknowledgment semantics whether or not durability-verification mode is enabled, so that the mode changes timing controllability only, not the durability contract. diff --git a/.kiro/specs/ha-commit-durability/research.md b/.kiro/specs/ha-commit-durability/research.md deleted file mode 100644 index 2954ae0c..00000000 --- a/.kiro/specs/ha-commit-durability/research.md +++ /dev/null @@ -1,220 +0,0 @@ -# Gap Analysis — ha-commit-durability - -> Brownfield analysis against the XCE controller (`src/controller/`). Informs `/kiro-spec-design`. -> Line numbers from the working tree at analysis time (2026-06-18); re-confirm before editing. - -## Current-State Summary - -The HA stack is mature and well-factored: per-LSN majority commit engine (`commit.c`), fan-out WAL replication with ACK/catch-up (`repl.c`, `config_repl.c`), a commit barrier that gates client replies (`persist_commit_barrier`), demotion truncate + rejoin reconciliation (`ha_convergence.c`), single-leader own-term gate, PreVote, and a vote-freshness gate. The durability gap is **beneath** all of it: "committed" (majority-ACKed) does not imply "durable on a majority", and recovery/promotion trust in-memory position rather than the durable log. - -The submit reply path is already commit-gated and runs **off the iomgr thread** (`job.c:568-569` — "the AFTER_EVENT hook writes WAL and waits for majority replication"), so durability work can be added to that path without stalling heartbeats. - -## Requirement-to-Asset Map - -### Requirement 1 — Durable Commit (acknowledgment ⇒ majority persistence) - -| Need | Existing asset | Status | -|------|----------------|--------| -| Reply gated on commit | `persist_commit_barrier` → `commit_submit(COMMIT_MODE_WAIT)` (persist.c); callers `job.c:1024`, `queue.c:144`, `worker.c:713`, `join.c:130` | ✅ Exists — reuse; strengthen meaning of "committed" | -| Standby persists before ACK | `persist_wal_write_replicated` `fdatasync` before return (persist.c:515) | ✅ Exists — **pin with regression test** (Req 1.3) | -| Primary self-counts only when durable | `commit.c:majority_candidate_locked` sets `cap = highest_appended_lsn(ce->p)` (appended, not durable) | ❌ **Missing** — core defect (bug-661) | -| Primary persists before self-count | `persist_wal_async` (persist.c:340) appends via `pwrite`, broadcasts, returns **without fsync**; durability deferred to `fsync_worker` timer (persist.c:173) | ❌ **Missing** | -| Track highest durable LSN | none — `persist_t` has `committed_lsn`/`current_lsn` but no `durable_lsn` | ❌ **Missing** (new field) | -| Liveness during durability work | commit-wait path already off iomgr (`job.c:568-569`) | ✅ Constraint satisfied by placement | - -**Minimal change shape:** add `durable_lsn` to `persist_t`, advanced when the entry's bytes are `fdatasync`'d; `persist_wal_async` fsyncs (inline, synchronous design — see brief decision) and sets `durable_lsn` before returning; `majority_candidate_locked` uses `min(durable_lsn, …)` for the self contribution instead of `highest_appended_lsn`. Reply gate then enforces the invariant unchanged. - -### Requirement 2 — Recovery Integrity + Leadership Fence - -| Need | Existing asset | Status | -|------|----------------|--------| -| No silent truncate-to-zero on corrupt recovery | `replay_wal` breaks on CRC chain break / CRC mismatch / short read (persist.c:1021, 1050, 1042) and returns the replayed count as **success**; only `-ENOMEM` propagates (persist.c:1107, 1152) | ❌ **Missing** — silent truncate-accept is the amplifier | -| Durable floor re-derived on restart | `persist_recover` **never sets `committed_lsn`** — it stays 0 after restart; floor is reconstructed only at promotion (persist.c:1114) | ❌ **Missing** — restart advertises in-memory floor unbacked by WAL | -| Pre-promotion integrity / caught-up gate | `raft_cb_become_leader` (ha.c:551) computes `safe_lsn` from votes (ha.c:572) then `persist_truncate_wal_above(safe_lsn)` (ha.c:577) — **no check that local WAL is intact or ≥ safe_lsn** | ❌ **Missing** — hollow node can promote, truncate collapses tip without backfill | -| Freshness floor from durable WAL | `ha_vote_self_freshness` = `max(applied_lsn, last_known_committed_lsn)` — **in-memory atomics**, not re-derived from durable WAL (ha.c:2127; used cluster.c:380-396 vote, cluster.c:467-474 PreVote) | ❌ **Missing** — truncated candidate advertises a floor it can't back | -| Fence "ineligible until caught up" | `in_catch_up` atomic exists but only a **standby** uses it to deny its **own** outgoing vote (ha.c:325, cluster.c:529); no fence for a freshly-recovered/promoting node | ⚠️ Partial — extend concept | -| Re-eligibility on catch-up | standby catch-up apply `ha_apply_wal_catchup_entry` (ha.c:1192); re-inclusion exists | ✅ Exists — hook the fence release here | - -### Requirement 3 — Deterministic Durability Verification - -| Need | Existing asset | Status | -|------|----------------|--------| -| Configurable durability cadence / disable background flush | `wal_sync_ms` = `DEFAULT_WAL_SYNC_MS` (config.h:86, value 10); `persist_config_defaults` (persist.h:30); **no env/CLI/config override** — callers `myapp.c:164`, `cluster.c:1267` pass defaults unmutated | ❌ **Missing** — must add an override | -| Test-mode seam convention | `#ifdef XCE_TESTING` hook shims (ha.c:2270-2335), test-only `handle_request_vote` (ha.c:2161); cluster.c not linked under XCE_TESTING (cluster.c:67) | ✅ Pattern exists — follow it | - -> Doc drift noted: `persistence-and-ha-design.md:4178` says `wal_sync_ms=50`; code is 10 (config.h:86). Reconcile in design. - -## Implementation Approach Options - -**Option A — Extend existing components (recommended).** All change sites are inside existing, well-scoped functions (`commit.c`, `persist.c`, `ha.c`, `cluster.c`). R1 = `durable_lsn` field + 2 edits (`persist_wal_async` fsync, `majority_candidate_locked` self cap). R2 = make `replay_wal` return an integrity verdict + a pre-promotion gate in `raft_cb_become_leader` + floor-from-durable-WAL in `ha_vote_self_freshness`. R3 = config override + an `XCE_TESTING` seam. -- ✅ Leverages the commit-barrier and `in_catch_up` machinery already present; smallest surface; matches TLA+ mapping in `CLAUDE.md`. -- ❌ Touches correctness-critical election/commit code — demands rigorous TDD and TLA+ re-verification. - -**Option B — New durability module.** Extract a `durability`/`durable_lsn` tracker as a new component. -- ✅ Isolated unit testing. ❌ Over-engineered — `durable_lsn` is one atomic on `persist_t`; a new module adds indirection without separation benefit. Not recommended. - -**Option C — Hybrid (recommended for R2 only).** Extend in place for R1/R3; add a small new **recovery-integrity + promotion-eligibility helper** (e.g. `persist_recovered_intact()` / `ha_promotion_durable_ok()`) for R2 so the integrity verdict and the fence predicate are unit-testable and don't bloat `raft_cb_become_leader`. -- ✅ Keeps the election hot path readable; testable fence predicate. Recommended: **A for R1+R3, C for R2.** - -## Effort & Risk - -| Req | Effort | Risk | Justification | -|-----|--------|------|---------------| -| R1 Durable commit | **M** (3–7d) | **Medium-High** | Localized (`durable_lsn` + 2 edits) but correctness-critical; needs TLA+ re-check (commit rule, leader completeness) and durability TDD. Liveness already safe (off-iomgr path). | -| R2 Recovery + fence | **L** (1–2wk) | **High** | Touches recovery, promotion, and the vote-freshness floor; must **compose with `ha-convergence-recovery`** (truncate/rejoin) and `ha-single-leader-role` (own-term gate) without double-handling, and honor `CLAUDE.md`'s "never truncate from stale ctrlhello" rule. Election-safety sensitive. | -| R3 Verification seam | **S** (1–3d) | **Low** | Config override + `XCE_TESTING` hook following an established pattern. | - -## Recommendations for Design Phase - -**Preferred approach:** Option A (extend) for R1 + R3; Option C (extend + small fence helper) for R2. - -**Key design decisions to resolve:** -1. **Durability mechanism** — already decided (brief): synchronous persist-before-self-count, eliminate `fsync_worker` as the durability mechanism; group-commit batching deferred to roadmap. Design must confirm `fsync_worker`'s fate: delete entirely vs retain a best-effort flush for any non-client-acked / local-only write (audit `wal_classify` for writes that gate no reply; checkpoint markers already fsync inline at persist.c:573). -2. **`durable_lsn` semantics** — single atomic on `persist_t`; advanced under `p->lock` after `fdatasync`; consumed by `majority_candidate_locked`. Confirm interaction with `persist_truncate_wal_above` (must not let `durable_lsn` exceed a post-truncation tip). -3. **Recovery integrity verdict** — `replay_wal`/`persist_recover` must distinguish "clean EOF" from "stopped at corruption". Decide: FATAL on corruption (fail-loud, simplest, honors the user's fail-loud preference) vs flag-and-fence (recover what's intact, fence from leadership until caught up). The fail-loud preference (`[[feedback_fail_loud_over_timeout]]`) leans FATAL unless the node can provably catch up. -4. **Freshness floor source** — derive the candidate's advertised floor from the **durable WAL tip**, not in-memory `applied_lsn`/`last_known_committed_lsn` alone, so a truncated node cannot advertise an unbacked floor. Reconcile with `CLAUDE.md`: never truncate using a stale ctrlhello `last_known_committed_lsn`; the safe-commit point still comes from voter replies at promotion. -5. **Compose with convergence-recovery** — R2's pre-promotion fence must not duplicate `ha_convergence.c`'s demotion-truncate/rejoin logic; define the seam (this spec fences *promotion*; convergence handles *demotion* and *rejoin reconciliation* of an intact log). - -**Research items to carry forward:** -- Measure `fdatasync` latency on the target storage vs the existing replication round-trip, to confirm the synchronous-design latency add is acceptable before group-commit is needed. -- Confirm no client-acked operation class bypasses the commit barrier (`wal_classify` audit) — any that do would violate the invariant. -- Decide the `wal_sync_ms` override surface (env var vs CLI flag vs config field) consistent with existing `--sync-timeout-ms` precedent (main.c `OPT_SYNC_TIMEOUT`). -- Re-verify against `raft/raft.tla` and the `CLAUDE.md` Protocol Verification Checklist after the self-count change (commit rule, leader completeness). - ---- - -## Design Synthesis (2026-06-18) - -Decisions resolved into `design.md`: -- **Durability mechanism = synchronous.** Add `persist_t.durable_lsn` (atomic, advanced after `fdatasync`); `persist_wal_async` persists before return; `commit.c:majority_candidate_locked` self-contribution uses `durable_lsn` not `highest_appended_lsn`. **`fsync_worker` removed** as the durability mechanism (checkpoint markers keep their inline fsync). Group-commit batching deferred to roadmap as a strict, contract-preserving optimization. -- **No new module (Option A) for R1/R3; small fence helper (Option C) for R2.** `durable_lsn` is one atomic — a module would be over-abstraction (build-vs-adopt: extend). -- **Recovery: fail-loud + fence.** `replay_wal` returns a verdict (`RECOVER_CLEAN` vs `RECOVER_TRUNCATED`); a truncated member is fenced from promotion and catches up from a healthy peer; FATAL only when it cannot reach a consistent committed state and is required for quorum. Balances `[[feedback_fail_loud_over_timeout]]` with availability. -- **Freshness floor from durable WAL**, not volatile in-memory `last_known_committed_lsn`; safe-commit point at promotion still from voter replies (honors `CLAUDE.md`). -- **Compose with convergence-recovery:** this spec fences *promotion* + verifies recovery integrity; convergence-recovery owns *demotion* truncate and *rejoin* of an intact log. No shared ownership of the commit index (kept volatile per Raft). -- **Liveness:** the added `fdatasync` lands on the already-off-iomgr commit path (`job.c:568-569`), so heartbeats are unaffected — Option A's only real risk is retired. - -Generalizations/simplifications: reuse the existing `in_catch_up` fence concept (extend from streaming-standby to recovery-truncated promoter) rather than inventing a new fence; reuse `persist_commit_barrier` reply gate unchanged; follow `--sync-timeout-ms` and `XCE_TESTING` hook conventions for R3. - ---- - -## Gap Re-Audit (2026-06-18) - -> **Post-implementation gap audit.** All 17 tasks (1.1–5.3) + the never-self-exit -> rework are marked `[x]` (landed 2026-06-18). This pass re-verifies each EARS -> acceptance criterion against the on-disk code (codegraph + grep), tags each -> MET / Missing / Unknown / Constraint, and records the known residual -> test-thoroughness items called out in the Implementation Notes. Line numbers -> from the working tree at audit time; re-confirm before editing. - -### Requirement-to-Asset Map (verified against code) - -#### Requirement 1 — Durable Commit (acknowledgment ⇒ majority persistence) - -| Crit | Asset (file:line) | Tag | -|------|-------------------|-----| -| 1.1 ack ⇒ durable on majority | Reply gated on commit barrier (unchanged) + self durable-capped (`commit.c:93`) + standby persist-before-ACK (`persist.c:761`); end-to-end test `test_durable_majority_commit_end_to_end` | **MET** | -| 1.2 self counts only when durable | `majority_candidate_locked` self-cap `cap = persist_durable_lsn(ce->p)` (`commit.c:93`, was `highest_appended_lsn`); debug invariant `assert(cand <= persist_durable_lsn)` (`commit.c:162`) | **MET** | -| 1.3 standby persists before ACK | `persist_wal_write_replicated` inline `fdatasync` rc-gated, `return -EIO` on failure so the standby does NOT ACK (`persist.c:761-773`); caller `ha_apply_single_entry` ACKs only on rc==0 (no-apply/no-ACK on negative). Pinned by `test_replicated_persist_syncs_before_return` / `test_replicated_sync_failure_no_ack` (counter `test_replicated_fdatasync_count`) | **MET** | -| 1.4 not reported until durable-majority | committed_lsn cannot advance over a non-durable entry (1.2 cap) → existing WAIT-mode reply gate enforces it unchanged | **MET** | -| 1.5 minority loss survives | Property test `test_sole_durable_contributor_loss_survives` (committed ⇒ ≥⌊N/2⌋+1 durable copies, drop one); live failover proof via soak (5.2) | **MET** (live N=2 fault is decision/property-level — see residual G1) | -| 1.6 liveness during durability work | Group-commit flusher (`fsync_worker`) syncs OFF the iomgr/caller thread; `fdatasync` under `p->lock` only (never with `fsync_mutex`); `test_durability_work_off_iomgr_preserves_liveness` asserts flush on a non-caller thread + heartbeat reads <50ms | **MET** | - -Mechanism note: `persist_wal_async` no longer fsyncs inline (`persist.c:643-652` signals the flusher); durability watermark `persist_t.durable_lsn` (`_Atomic uint64_t`) advanced ONLY after a successful `fdatasync` (flusher `persist.c:281-311`, `persist_flush` `persist.c:801-806`). `fsync_worker` repurposed from fixed-interval timer to demand-driven group-commit flusher; checkpoint markers retain their inline `fdatasync` (`persist.c:162`). Legacy `sync_append` ACK-commit store now durable-gated (`sync_append_legacy_commit_store`, store only when `written_lsn <= durable_lsn`). - -#### Requirement 2 — Recovery Integrity + Leadership Fence - -| Crit | Asset (file:line) | Tag | -|------|-------------------|-----| -| 2.1 no silent truncate-accept | `replay_wal(p, from_lsn, recover_verdict_t*)` classifies CRC chain break / CRC mismatch / short header / short payload → `RECOVER_TRUNCATED`; clean EOF / pre-alloc zero tail / empty WAL → `RECOVER_CLEAN` (`persist.c:1411-1605`); verdict stored on `persist_t.recovery_verdict`, accessors `persist_recovery_verdict`/`persist_recovery_truncated` (production, not XCE_TESTING-gated) | **MET** | -| 2.2 ineligible until caught up | Pre-promotion gate in `raft_cb_become_leader` (`ha.c:952-1010`): REFUSE if `persist_recovery_truncated` OR `durable_floor < promotion_safe_lsn` (read AFTER truncate-clamp), set `in_catch_up`, return WITHOUT becoming primary | **MET** | -| 2.3 below-HWM never authoritative | Durable-backed freshness floor: candidate advertises bare `ha_candidate_advertise_lsn = persist_durable_floor` (`ha.c:2587`), voter floor `max(durable_floor, learned)`; promotion gate compares clamped `durable_floor` vs voter-reply `promotion_safe_lsn`; floor re-derived from recovered WAL tip in `persist_recover` (assign, not max) | **MET** | -| 2.4 loud alert, hold non-promotable, STAY ALIVE, never self-exit | `persist_recover_loud_fence_if_unrecoverable` (`persist.c:1787`): loud ERROR with truncation boundary + sticky `_Atomic bool recovery_unrecoverable`, **NO abort/exit/tlog_exit**; `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). grep confirms zero abort/exit on the recovery path; `persist_fatal_hook` removed from production (only a test comment remains) | **MET** | -| 2.5 preserve durable-commit at degraded/minimal quorum | Hollow node refused regardless of quorum size (gate is unconditional on truncated/behind) + 4.2 freshness floor; `test_promotion_gate_hollow_refused_at_minimal_quorum` (decision-level, Req 2.5) | **MET** (decision-level; no dedicated live N=2 fault-injection — residual G1) | -| 2.6 re-eligibility on catch-up | Fence/flag clear automatically when `!truncated && durable_floor >= HWM`: `ha_evaluate_promotion_fence` ELIGIBLE branch clears `in_catch_up` (`ha.c:592-597`); `promotion_fence_timeout_fn` releases + `persist_clear_recovery_unrecoverable` on catch-up (`ha.c:755-763`); gate ELIGIBLE branch clears + cancels timer (`ha.c:986-993`) | **MET** | - -Bounded-window catch-up (task 4.4): `raft_cb_become_leader` NEVER FATALs/aborts synchronously — on FENCE/would-be-unrecoverable it arms a one-shot `promotion_fence_timeout_fn` (`DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000`) that re-reads live state, releases on catch-up, re-emits the loud alert + re-arms if still hollow with no peer — never exits (`ha.c:721-807`). UAF-safe cancel; cancelled in `ha_fini`. - -#### Requirement 3 — Deterministic Durability Verification - -| Crit | Asset (file:line) | Tag | -|------|-------------------|-----| -| 3.1 configurable durability cadence | `--wal-sync-ms` flag (`main.c:67,118`) → `myconf.wal_sync_ms` → `persist_config_set_wal_sync_ms` (`persist.h:130`) applied at BOTH persist_init sites (`myapp.c:169` production, `cluster.c:1284` phase2); 0-sentinel = keep `DEFAULT_WAL_SYNC_MS`. Test seam `persist_test_durable_lsn_on_disk` (read-only on-disk CRC-validated durable LSN) + `persist_test_set_force_fsync_fail` fault injection, all XCE_TESTING-gated (`persist.h:445-506`) | **MET** | -| 3.2 ack ⇒ recoverable after ungraceful kill in verify-mode | Deterministic E2E (`test-commit-durability.sh --deterministic`): with `--wal-sync-ms 30000`, in-window primary `fdatasync >= 1` (=2) precedes ack ("durable-before-ack"), flipped RED→GREEN stable 3/3 / 9-9 probes (5.2) | **MET** | -| 3.3 same observable ack semantics with/without verify-mode | Documented as a property in the test header (`test_commit_durability.c:33-34`): verify-mode "changes nothing observable about the acknowledgment contract" — but there is **NO dedicated A/B assertion** running the same op with mode on vs off and asserting identical observable acknowledgment | **PARTIAL** (residual G2) | - -Constraint compliance: `safe_committed_lsn` arithmetic UNTOUCHED (voter-reply-sourced, honors `CLAUDE.md` "never truncate from stale ctrlhello"). raft.tla election/freshness logic maps faithfully; raft.tla does **NOT** model durability (restores `log` intact on restart, `raft.tla:165-174`) — the commit-durability invariant is verified by tests 2.x/5.1/5.2 + the manual Protocol Verification Checklist, NOT by raft.tla. (Tagged **Constraint** — by-design deferral, see residual G3.) - -### Residual Gaps - -| ID | Gap | Severity | Notes | -|----|-----|----------|-------| -| **G1** | Req 2.5 has no dedicated **minimal-quorum live fault-injection** test (N=2, lose the sole durable contributor across a real failover). Current coverage = decision/property-level (`test_promotion_gate_hollow_refused_at_minimal_quorum`, `test_sole_durable_contributor_loss_survives`) + the rolling-SIGKILL soak. | **LOW** | Invariant is enforced by the unconditional gate + freshness floor and the soak observed zero loss; a targeted N=2 case would harden the assertion against a future regression but is not a correctness gap. | -| **G2** | Req 3.3 has no **verify-mode A/B observable-equivalence** assertion (same op, mode on vs off, identical externally observable acknowledgment). Property is asserted only by header comment + the structural fact that the seam is read-only / production-excluded. | **LOW** | The seam (`persist_test_*`, `--wal-sync-ms`) demonstrably changes only timing, not the reply path (reply gate unchanged). An explicit A/B test would close the letter of 3.3. | -| **G3** | raft.tla does **not** model durable-vs-volatile log; the formal model cannot prove the commit-durability invariant. | **LOW (by design)** | Explicitly deferred (task 5.3 note, requirements out-of-scope). Durability is test-backed. A durable-log TLA+ extension is out of scope unless explicitly opted in. | - -No CRITICAL/HIGH residuals. No criterion is **Missing** or **Unknown**: 16/16 verifiable EARS criteria are MET (1.1–1.6, 2.1–2.6, 3.1–3.2) with 3.3 PARTIAL (property held but not A/B-asserted). The never-self-exit invariant (Req 2.4) is upheld — grep/nm confirm no abort/exit/tlog_exit on the recovery/promotion path. - -### Integration Risks - -- **Group-commit flusher latency on a degrading disk** (Implementation Note 2.1 follow-up): the flusher tight-spins on SUSTAINED `fdatasync` failure (releases `p->lock` each iter so liveness holds, but CPU/log burn). A HANGING (not erroring) `fdatasync` blocks appends since it runs under `p->lock` — pre-existing, not introduced here. Non-blocking; carry to roadmap with group-commit batching. -- **Test-fixture durable priming (bug-704 class)**: any commit-engine / election fixture that primes `current_lsn`/`applied_lsn`/`last_known_committed_lsn` but NOT `durable_lsn` will see floor/advertise=0 and committed-stall, because both the self-cap (2.2) and the freshness floor (4.2) now read durable state. Future tests MUST prime `durable_lsn` to the modeled tip. (4 fixtures already corrected; bug-760 caught one adjacent real-flusher test.) -- **fsync-able filesystem for validation**: `/tmp` overlay in some sandboxes EIOs on `fdatasync`; because committed_lsn is now durable-gated, an EIO correctly HOLDS committed_lsn and surfaces as adjacent-test failures. Final validation MUST run on a real fsync-able fs. Gate on ANSI-stripped `:FAIL:` count (meson `Fail:0` masks Unity sub-group failures). - -### Options for the Open Test-Thoroughness Items (G1, G2) - -**Option A — Close both with targeted unit/integration tests (recommended).** -- G1: add an N=2 minimal-quorum integration test reusing the `test_sole_durable_contributor_loss_survives` two-persist-dir harness, but drive it through a real failover-style commit + drop-the-durable-contributor and assert the acked op is present on the survivor. -- G2: add an A/B test that submits the same op with verify-mode (`--wal-sync-ms` slow / forced) and without, asserting byte-identical externally observable acknowledgment (same rc, same reply gate release semantics). -- **Effort: S (1–3d).** **Risk: Low** — test-only, no production change; reuses existing harnesses/seams. - -**Option B — Cover via the soak/E2E harness only.** -- Extend `test-commit-durability.sh` with an N=2 deterministic round (G1) and a mode-on/mode-off ack-equivalence probe (G2). -- **Effort: S (1–2d).** **Risk: Low-Medium** — E2E is less deterministic than unit; the strace-gap probe is the established deterministic surface, so prefer it over joblist-loss for G2. - -**Option C — Accept as documented residuals (status quo).** -- Record G1/G2 as LOW-severity known limitations; rely on the gate/floor invariants + the zero-loss soak already in place. -- **Effort: none.** **Risk: Low** — acceptable given no CRITICAL/HIGH gap, but leaves the letter of Req 3.3 (A/B) and a dedicated Req 2.5 live case unasserted, so a future regression in those exact paths could go uncaught until soak. - -**Recommendation:** Option A — both items are S-effort, test-only, and turn two LOW residuals into fully-asserted criteria; G3 stays a by-design deferral. - ---- - -## REOPENED 2026-06-19 — cold-restart-all (total-cluster-outage) committed-job loss (bug-795) - -**Trigger:** the `xce-test-e2e-primitives` spec converted `test-commit-durability.sh` to exit-code primitives and ADDED a new `run_cold_restart_recovery` phase (kill ALL 3 controllers simultaneously, then restart ALL from disk, then assert every acknowledged job recovers). Running it live on a 3-node cluster surfaced a NON-DETERMINISTIC committed-job loss that this spec's original validation never exercised. - -**What was observed (3 live soak runs, sandbox 3-node cluster):** -- Rolling **single-node** SIGKILL + restart: ZERO loss every round (consistent with this spec's 5.2 result — single-host loss is non-deterministic and standbys fdatasync-before-WalAck, so a committed job is durable off the primary). -- **Cold-restart-all** (kill ALL 3, restart ALL): a job that returned `jobsubmit ok` AND survived every rolling round VANISHED after the full restart (`expect-jobs expected N, actual N-2`; `expect-job <id> not found`). PASSED in one run, FAILED in another → a **race**. Recovery logs show WAL `truncate` on the rejoining nodes. - -**Why this spec did not catch it:** 5.2's soak was rolling **single-node** kills only; the durability guarantee was validated (and CLAUDE.md flipped to "GUARANTEED") against that scenario. The **total simultaneous outage** path — where ALL members lose their volatile/page-cache tail at once and recovery must reconstruct committed state from on-disk WAL alone — was never tested. When every node dies together, the "durable off the primary" assumption that makes single-host loss non-deterministic no longer applies: only what was `fdatasync`'d on a majority before the ack can survive. - -**Open question — REAL durability gap vs OVERLAYFS artifact (must resolve first):** -- This sandbox's `/tmp` is **overlay** (NOT a real fsync-able fs). This spec's own notes repeatedly flag that overlay `/tmp` can EIO on `fdatasync` and that "final validation must run on a real fsync-able fs." -- If `fdatasync` on overlay is weak (returns rc=0 without true durability) OR if recovery's truncation (`replay_wal` / `persist_truncate_wal_above` / durable-floor re-derivation, tasks 3.1/3.4) is too aggressive after a total outage, a committed entry's tail could be lost or truncated on cold restart. -- The durable-gating (2.2/2.4) should make `committed_lsn` HOLD if `fdatasync` failed — so an un-fdatasync'd job should NOT have returned `jobsubmit ok`. Yet these jobs DID get `jobsubmit ok`. So either `fdatasync` returned success on overlay without real durability, or a non-durable-gated ack path exists under the total-outage timing, or recovery truncates committed entries. - -**Disposition:** the `xce-test-e2e-primitives` soak is the reproduction surface and is working correctly (it detects the loss via exit codes). The fix (if real) belongs HERE. See Phase 6. - -### UPDATE 2026-06-19b — trustworthy soak sharpens the finding (rolling-failover loss + liveness) - -After the `xce-test-e2e-primitives` soak was HARDENED (commit d82cc78: it no longer conflates liveness with durability — durability is asserted ONLY against a confirmed-serving cluster, with a catch-up-lag confirm re-check; a non-serving cluster is INCONCLUSIVE, never "DATA LOSS"), three 6–10-round live runs give a cleaner signal: - -1. **The committed-job loss reproduces in ROLLING single-node failover too — not only cold-restart-all.** One run lost a committed job at round 6 with the cluster CONFIRMED SERVING (`DATA LOSS — persistently absent from a SERVING cluster`); two other runs had zero loss. So the durability race is NON-DETERMINISTIC (~1 in 3 runs here) and broader than the cold-restart-all framing in the original reopen note — it is a general acked-but-not-durable-on-majority race that a kill can expose during normal rolling failover. **This is exactly the bug-661 invariant this spec claimed CLOSED for rolling failover (task 5.2) — it is not fully closed.** (Caveat still applies: sandbox `/tmp` is overlayfs; 6.1 must confirm on a real fsync-able fs.) -2. **Separate LIVENESS concern:** every run shows 2–3 rounds where, after killing the current primary (consistently `ctrl-1`), NO new primary served within 60s (the cluster always eventually recovered). A >60s failover for a single-node kill in a 3-node cluster is poor availability and may share a root cause with the durability race (a behind/just-rejoined node interacting with the promotion fence / freshness floor). Worth a dedicated liveness check in 6.2. - -**Phase 6 scope note:** 6.1's reproduction MUST include BOTH the cold-restart-all step AND the rolling-failover rounds (the hardened soak runs both and reports trustworthy per-round verdicts), and should capture the >60s-election cases for the liveness angle. - -### UPDATE 2026-06-19c — 6.1 ANSWERED: the loss is REAL, not (only) an overlay artifact - -The cold-restart-all / rolling durability loss was initially suspected to be an overlay-`/tmp` `fdatasync`-EIO artifact of the dev sandbox. That is now ruled out as the primary cause: -- **User runs the soak OUTSIDE the container** (real host, real fsync-able fs, NO overlay `fdatasync` EIO) and still observes the DATA LOSS + the liveness failures. So they are genuine, not sandbox `fdatasync` noise. -- In-sandbox confirmation on a **real ext2/3 deploy dir** (`fdatasync` succeeds, ZERO EIO): a 3-round run PASSED clean, but an 8-round run reproduced BOTH a liveness cascade (after killing `ctrl-1` in round 1 the cluster never returned to 3-healthy) AND a cold-restart `expect-jobs` shortfall with a PERSISTENTLY-absent committed job — with **zero `fdatasync` EIO**. (Caveat: the 8-round run's cold-restart followed a degraded run, so the cleanest real-fs evidence is the user's outside-container host.) - -**Verdict for 6.1:** the durability loss and the liveness stall reproduce off overlay → REAL controller findings, not an `fdatasync`-EIO artifact. The overlay EIO remains a SEPARATE, sandbox-only confound (it makes the dev-box `controller_unit_test` flaky — see bug-760 — and inflates loss on overlay deploys), but it is not the root cause of the soak loss. - -**Phase 6 is now at 6.2 (root-cause).** Two real, non-deterministic findings to root-cause: -- **Durability:** an acked (`jobsubmit ok`) job is PERSISTENTLY absent from a SERVING cluster after failover / cold restart — i.e. bug-661 is NOT fully closed (Phase 5 only exercised rolling single-node kills with the racy soak). Buckets: acked-without-durable-majority (commit-gate timing) / recovery truncates committed entries / behind-then-killed node re-leads with a truncated log. -- **Liveness:** after killing the current primary (consistently `ctrl-1` in these runs), the cluster sometimes takes >60s — or fails entirely — to elect a serving primary, and a restarted node sometimes never returns to a healthy 3-node serving set. Possibly coupled to the promotion-fence / freshness-floor interaction with a just-rejoined node. - -The hardened `xce-test-e2e-primitives` soak (trustworthy: liveness vs durability separated, confirm-settle, quiet wait-primary) is the reproduction surface for both. diff --git a/.kiro/specs/ha-commit-durability/spec.json b/.kiro/specs/ha-commit-durability/spec.json deleted file mode 100644 index e54de69c..00000000 --- a/.kiro/specs/ha-commit-durability/spec.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "feature_name": "ha-commit-durability", - "created_at": "2026-06-17T23:40:42Z", - "updated_at": "2026-07-18T00:55:00Z", - "language": "en", - "phase": "implementation-complete", - "reopened_at": "2026-06-19T00:00:00Z", - "reopened_reason": "bug-795: total-cluster-outage (cold-restart-all) committed-job loss surfaced by the xce-test-e2e-primitives soak; Phase 6 added (6.1 reproduce-on-real-fs gates the rest).", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T00:55:00Z", - "note": "Prior state: all 28 tasks (1-6.4) were [x] in tasks.md but phase was stuck at \"tasks-generated\" with no validate_impl record (administrative oversight, same pattern as ha-entity-snapshots). Per-task Implementation Notes in tasks.md already document extensive self-review of each task (including a corrective REWORK note for the never-self-terminate invariant and a cross-task fix, bug-760, for an adjacent test regression) -- this pass focuses on cross-task/feature-level checks.\n\nMECHANICAL: Full ASAN build clean (nix develop + xce-build.sh controller asan), zero compiler errors. Meson test summary Ok:3 Fail:0. Standalone controller_unit_test binary (authoritative per this project's own meson-Fail:0-masks-Unity-subgroups convention): 0 \":FAIL:\" occurrences, 791 individual \": PASS\" test lines, all green (process exit code alone is not gating -- pre-existing unrelated ASAN leak-report noise at exit is a documented quirk, not a functional failure). TBD/TODO grep on the feature's owned files (persist.c, persist.h, commit.c, ha.c, cluster.c, test_commit_durability.c, test-commit-durability.sh) found only pre-existing, unrelated items not introduced by this spec: cluster.c's 3 TODOs are an unrelated config-generation-tool note and a config-group FIXME, and ha.c:1844 is the same pre-existing ha-term-safety-owned stub TODO already flagged as unrelated by the ha-entity-snapshots validate_impl pass. Secrets grep clean.\n\nSTALENESS RE-CHECK (the key judgment call): this spec's own live evidence (deterministic RED->GREEN E2E and rolling-soak zero-loss) was captured 2026-06-18/2026-06-19, and the dedicated cold-restart-all real-fs revalidation (task 6.1-6.4) on 2026-07-02 -- both materially predate today. In the interim, persist.c/ha.c/cluster.c/commit.c (the exact files this spec owns) received substantial further edits: bug-1433/1434 (2026-07-11, added persist_committed_lsn_advance_atomic -- a CAS-safe committed_lsn advance, and removed a force-flush call reachable from a never-blocks vtask class), bug-1417/1419 (vote-index mapping, WAL replay-tally dedup), the job-array-single-wal-record series (new WAL_JOB_ARRAY_SUBMIT record type + apply/replay changes), and bug-1445 (2026-07-17, first-boot promotion-check-vs-own-first-replay race in raft_cb_become_leader -- the SAME function this spec's promotion-eligibility gate lives in). Given this density of churn in exactly the owned files, the old live evidence alone was judged insufficient without a fresh check. Code-level verification: majority_candidate_locked (commit.c:135) still caps the self contribution at persist_durable_lsn (unchanged); persist_committed_lsn_advance_atomic (persist.c:5881) and its two call sites explicitly preserve \"committed_lsn never advances past durable_lsn\" in their own code comments, i.e. the later CAS fix was written to respect this spec's invariant rather than bypass it; bug-1445's new check is an orthogonal job-array live-apply-fence re-check added alongside, not a modification of, this spec's recovery-truncated/durable-floor promotion gate.\n\nFRESH LIVE RE-VALIDATION (this session, genuine ext4 -- confirmed /home/claude, backing device /dev/nvme0n1p2 ext4, NOT the container's overlay /): (a) --deterministic strace mode returned a false RED (3/3 probes \"0 fdatasync in-window\"). Diagnosed, not accepted at face value: this sandbox's date is BusyBox date, which does not support GNU date's %N (nanoseconds) -- confirmed by direct invocation, it emits the literal unexpanded string \"%N\", breaking the script's sub-second window arithmetic. Cross-checked strace's own (unaffected) microsecond timestamps against the controller's committed_lsn advance log lines for all 3 probes: fdatasync consistently PRECEDED the commit-advance/ack by 16-43ms in every probe -- i.e. durable-before-ack genuinely holds on the current build; the script's verdict was a test-harness false positive, not a product regression. Logged as bug-1446 (test-harness only). (b) 4-round rolling SIGKILL+restart soak plus a full cold-restart-all cycle on the same ext4 deploy: 8/8 PASS, 0 job loss -- all 20 committed jobs survived both rolling failover and cold-restart-all. This directly reconfirms Req 1.5 and the Phase-6 total-outage guarantee on today's code, not just June/early-July code.\n\nINTEGRATION: durable_lsn -> majority_candidate_locked self-cap -> committed_lsn -> commit barrier -> client reply chain verified intact end to end (code read + the live soak's observed ordering above). recovery-verdict -> promotion-gate -> freshness-floor chain verified intact (persist_recovery_verdict/persist_durable_floor still consumed by ha.c's promotion gate and cluster.c's vote/PreVote freshness comparisons). No boundary spillover: the only two touches to raft_cb_become_leader since this spec landed (bug-1433/1434's force-flush removal, bug-1445's first-boot fence re-check) are both additive/orthogonal, not modifications of this spec's owned gate logic. src/controller/CLAUDE.md's \"Commit Durability\" guarantee note (Protocol Verification Checklist) is current and accurate, and is explicitly cross-referenced by the newer bug-1434 code comments.\n\nCOVERAGE: all 3 requirements sections (1.1-1.6 Durable Commit, 2.1-2.6 Recovery Integrity and Leadership Fence, 3.1-3.3 Deterministic Durability Verification) map through design.md's Requirements Traceability table to a completed task; no gaps. File Structure Plan matches actual files (persist.c/.h, commit.c, ha.c, cluster.c, config.h, main.c, test_commit_durability.c, test-commit-durability.sh) with no drift. Blocked tasks: none.", - "known_limitations": [ - "bug-1396 (open, low-priority, unconfirmed): an extra/duplicate committed job appeared after one cold-restart-all cycle in the original 2026-07-02 5-run validation -- the opposite symptom of the loss this spec targets (over-counting, not loss), not yet root-caused. Does not block this GO (it is not a durability-invariant violation) but remains open for a future investigation.", - "persist_apply_pump_run_sync (flagged by bug-1433, not introduced by this spec): still reachable from raft_cb_become_leader on a VTASK_LIVENESS (never-blocks) vtask and performs bounded pread() -- lower risk than the fdatasync this spec's own group-commit flusher correctly keeps off that path, but strictly still blocking I/O. Tracked as its own follow-up in bug-1433, adjacent to but outside this spec's boundary.", - "bug-1446 (this session, test-harness only): src/tools/test-commit-durability.sh's --deterministic mode produces a false RED in any environment whose `date` lacks GNU date's %N support (e.g. BusyBox date, as in this sandbox) because its submit-to-ack window arithmetic silently breaks on the unexpanded literal \"%N\". The soak mode used for this session's live re-validation is unaffected and remains the authoritative live check in such environments. A future fix should make the script detect/require GNU date or use an alternative sub-second time source, and fail loud on an incompatible date rather than emitting a false verdict.", - "The group-commit batching mechanism's interaction with the newer job-array-single-wal-record feature (multi-entry array WAL records) was confirmed consistent at the shared committed_lsn/durable_lsn-invariant level (code read) but was not independently re-exercised by a dedicated array-specific durability soak in this pass; job-array-single-wal-record carries its own validate_impl track for that." - ] - } -} diff --git a/.kiro/specs/ha-commit-durability/tasks.md b/.kiro/specs/ha-commit-durability/tasks.md deleted file mode 100644 index 1472c004..00000000 --- a/.kiro/specs/ha-commit-durability/tasks.md +++ /dev/null @@ -1,176 +0,0 @@ -# Implementation Plan - -> TDD is mandatory (project rule): write the failing test in `testsuite/` before the C change (red → green → refactor). -> Re-verify HA changes against `raft/raft.tla` and the `src/controller/CLAUDE.md` Protocol Verification Checklist. -> Most core tasks edit the same correctness-critical files (`persist.c`, `commit.c`, `ha.c`, `cluster.c`) and are therefore sequential by design; only genuinely file-disjoint analysis is marked `(P)`. - -- [x] 1. Foundation: durability verification seam and deterministic reproduction -- [x] 1.1 Add a configurable flush-cadence override - - Parse a `--wal-sync-ms` controller flag (following the existing `--sync-timeout-ms` convention) into the persistence config before persistence starts; default behavior unchanged. - - Observable: starting a controller with `--wal-sync-ms <N>` sets the flush cadence to N; omitting it preserves the current default. - - _Requirements: 3.1_ - - _Boundary: DurabilityConfig_ - -- [x] 1.2 Add the test-only durability-verification hook - - Add an `XCE_TESTING`-gated seam that lets a test assert an acknowledged operation is recoverable after an immediate ungraceful termination; compiled out of production builds. - - Observable: in a test build the hook is callable and the production build links without it. - - _Requirements: 3.2, 3.3_ - - _Boundary: DurabilityTestSeam_ - -- [x] 1.3 Drive a deterministic durability reproduction from the soak harness - - Extend `test-commit-durability.sh` with a deterministic mode that widens the batching window / uses the seam and kills a durable contributor in-window. - - Observable: on the current (unfixed) build the deterministic mode reports DATA LOSS (RED), proving bug-661 without relying on race timing. - - _Requirements: 3.1, 3.2_ - - _Depends: 1.1, 1.2_ - -- [x] 2. Durable commit mechanism (acknowledgment ⇒ majority persistence) -- [x] 2.1 Introduce durable-LSN tracking via a group-commit flusher - - Repurpose the background fsync thread from a timer into a demand-driven flusher: an append signals it; it performs one `fdatasync` over all pending appends, advances a durable-LSN watermark, and wakes the commit evaluation. - - Guard against lost wakeups: the signal is level-triggered on the appended-vs-synced gap so an entry is never considered durable before its `fdatasync` (gating item). - - Observable: a unit test shows one `fdatasync` advances the durable-LSN past several appended LSNs under load, and the durable-LSN never exceeds the synced tip. - - _Requirements: 1.1, 1.6_ - - _Boundary: DurableCommit_ - -- [x] 2.2 Count self toward the commit majority only when durable - - Change the commit-majority evaluation so the local member's self-contribution is capped at the durable-LSN watermark, not the appended tip; clamp the watermark on WAL truncation. - - Observable: a unit test shows `committed_lsn` does not advance over an entry until that entry is durable locally, and the self-cap equals the durable-LSN. - - _Requirements: 1.2, 1.4_ - - _Boundary: DurableCommit_ - -- [x] 2.3 Pin standby persist-before-acknowledge with a regression test - - Add a regression test asserting the standby apply path `fdatasync`s a replicated entry before sending its acknowledgment. - - Observable: the test fails if the standby ACK is emitted before the durable write and passes against the current behavior. - - _Requirements: 1.3_ - - _Boundary: DurableCommit_ - -- [x] 2.4 Reconcile the dormant synchronous-append commit path - - The synchronous-append path advances the commit position directly on majority acknowledgment and is built on the premise that the append already fsync'd inline — a premise group-commit removes. Confirm it has no production caller and delete it, or route it through the same durable-majority evaluation so it can never advance the commit position on a non-durable entry. - - Observable: a code search shows no path advances the commit position on majority ACK without the durable precondition; the dormant path is removed or re-routed, and a test guards against a future caller reintroducing an ACK-without-durability commit. - - _Requirements: 1.1, 1.2_ - - _Boundary: DurableCommit_ - - _Depends: 2.1_ - -- [x] 3. Recovery integrity (no silent truncate; durable floor) -- [x] 3.1 Return a recovery integrity verdict and mark a truncated member non-promotable - - Make WAL replay distinguish a clean log end from a corruption-stopped end (CRC chain break / mismatch / short read) and surface that verdict; a corruption-truncated recovery is never reported as a clean recovery and sets a member-level "recovery-truncated" flag that the promotion gate (4.3) consumes. - - Observable: a unit test injecting a CRC break at the first entry yields a TRUNCATED verdict and sets the non-promotable flag, while an intact log yields CLEAN; an empty first-boot WAL is CLEAN (not corruption). - - _Requirements: 2.1_ - - _Boundary: RecoveryIntegrity_ - -- [x] 3.2 Pin the fail-loud vs fence boot policy as a decision with a verification table - - Pin the concrete policy (gating item, decision only — no behavior change yet): a truncated member fences from leadership at boot, attempts catch-up from a healthy peer, and terminates loudly only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence. Encode the decision as a table of input states → expected outcome. - - Observable: the policy and its decision table are recorded, and skeleton tests assert each row's expected outcome (initially failing), establishing the contract 3.3 and 4.3 implement. - - _Requirements: 2.2, 2.4_ - - _Boundary: RecoveryIntegrity_ - -- [x] 3.3 Implement loud FATAL on bounded-unrecoverable recovery - - Implement the FATAL branch of the 3.2 policy: when a recovery-truncated member cannot reach a committed-consistent state within the bounded catch-up window, terminate loudly with the truncation boundary logged; a reachable healthy peer never triggers FATAL. - - Observable: the 3.2 decision-table rows for the FATAL/no-FATAL cases pass; logs show the truncation boundary on FATAL. - - _Requirements: 2.4_ - - _Depends: 3.1, 3.2_ - - _Boundary: RecoveryIntegrity_ - -- [x] 3.4 Re-derive the durable floor from the recovered WAL tip - - On recovery, publish the member's durable floor as the highest intact replayed LSN so later advertisement is backed by durable state rather than a volatile in-memory position. - - Precision: this floor is the durable **last-log index** (WAL tip), used by the Raft freshness gate — not the commit position; the post-election safe-commit point continues to come from voter replies, not from this floor. - - Observable: a unit test shows that after a truncating restart the published durable floor equals the highest intact LSN (the WAL tip), not a stale higher in-memory value or the commit position. - - _Requirements: 2.3_ - - _Boundary: RecoveryIntegrity_ - -- [x] 4. Promotion fence and durable-backed freshness -- [x] 4.1 (P) Pin the freshness-floor formula and re-verify election safety and liveness - - Define the exact reconciliation of the durable-derived freshness floor against the existing in-memory floor — the durable quantity is the durable **last-log index** (WAL tip), not the commit position — then re-check it against `raft/raft.tla` for Leader Completeness AND liveness (no election livelock) plus the Protocol Verification Checklist (gating item; analysis only, no shared source files). - - Scope of the raft.tla check: it validates the **election/freshness logic** (the `mlastLogIndex`/`LastTerm` semantics this change touches), which raft.tla models directly. It does NOT validate the *durability* of that floor — raft.tla assumes a durable log (see 5.3). The durability of the WAL-tip derivation is verified by tests (3.4), not by raft.tla. - - Observable: the pinned formula is recorded and the model/checklist verification passes for both a truncated-candidate-denied case and a longest-durable-log-still-elects case. - - _Requirements: 2.3_ - - _Boundary: FreshnessFloor_ - -- [x] 4.2 Derive the vote and PreVote freshness floor from durable WAL state - - Implement the pinned formula so a candidate cannot advertise, and a voter cannot grant against, a position beyond durable WAL state; the post-election safe-commit point continues to come from voter replies, never a stale ctrlhello. - - Observable: a node restarted with a truncated log is denied (vote and PreVote), while the node holding the longest durable log still wins — no livelock. - - Note: 4.2 and 4.3 both edit the same election code (`cluster.c` vote/PreVote and `ha.c` promotion); they are sequential (4.2 before 4.3), not parallel. - - _Requirements: 2.3_ - - _Depends: 4.1, 3.4_ - -- [x] 4.3 Add the pre-promotion eligibility gate and fence release - - Before a member becomes primary, refuse promotion if it carries the recovery-truncated flag (from 3.1) or its durable position is below the cluster committed high-water mark; extend the existing catch-up fence to cover the promoting/recovered member and release it when catch-up passes the last committed boundary. Reuse the durable-LSN truncate-clamp from 2.2 so the gate evaluates a clamped durable position at the promotion-path truncate site. - - Observable: a recovery-truncated or behind member is denied promotion until catch-up completes, then regains eligibility; a hollow node cannot win even at minimal quorum. - - _Requirements: 2.2, 2.5, 2.6_ - - _Depends: 4.2, 3.1, 3.2, 2.2_ - -- [x] 4.4 Bounded-window catch-up before FATAL (fence does not FATAL immediately) - - The promotion gate (4.3) currently FATALs a recovery-truncated/behind member immediately at the promotion instant when no granting voter has a live endpoint AT THAT INSTANT — a transient no-peer snapshot during aggressive failover. Per the 3.2 decision table (rows 3/5) a member with a (momentarily) reachable peer must FENCE→catch-up→ELIGIBLE, NEVER immediate-FATAL; FATAL is reached ONLY after a BOUNDED failure to reach a committed-consistent state. Implement the deferred bounded-window catch-up: on FENCE (or would-be-FATAL no-peer), keep the member alive in `in_catch_up`, schedule a one-shot delaymgr timer for `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`, and reach the 3.3 FATAL ONLY from that timer if the member is STILL recovery-truncated AND durable_floor < cluster HWM AND no healthy peer / no catch-up progress when it fires. Re-derive eligibility (release the fence) when catch-up brings durable_floor ≥ HWM. Never FATAL synchronously in `raft_cb_become_leader`; never fence indefinitely. - - Observable: a recovery-truncated node whose peers are transiently unreachable FENCES (stays alive) and is NOT FATAL'd; once a healthy peer is reachable and catch-up passes the committed boundary it regains eligibility; FATAL fires only after the bounded window with no peer/progress (unit-tested via the `persist_fatal_hook` + a short window). The rolling SIGKILL soak (task 5.2) keeps the cluster available (always re-elects) with zero acknowledged-job loss. - - _Requirements: 2.2, 2.4, 2.6_ - - _Depends: 4.3, 3.3_ - - _Boundary: PromotionFence_ - -- [x] 5. Integration and validation -- [x] 5.1 Wire durable-majority commit end to end - - Connect the durable-LSN self-count and the commit barrier so the client reply is released only after the entry is durable on a majority; assert liveness (the flusher does not stall heartbeat/election handling). - - Observable: an integration test confirms reply-ok only after durable-majority, killing the sole durable contributor mid-commit leaves the acknowledged op present on the survivors, and the full controller suite passes under ASAN. - - _Requirements: 1.1, 1.4, 1.5, 1.6, 2.5_ - - _Depends: 2.2, 4.2_ - -- [x] 5.2 Prove zero acknowledged-job loss across failover - - Run the deterministic durability E2E and the rolling SIGKILL+restart soak; both must show every acknowledged job surviving every failover. - - Observable: the deterministic mode that was RED in 1.3 now passes (GREEN), and the soak reports zero DATA LOSS across rounds. - - _Requirements: 1.1, 1.5, 2.2, 2.3, 2.6_ - - _Depends: 5.1, 4.3, 1.3_ - -- [x] 5.3 Re-verify the formal safety checklist and flip the durability gap note - - Re-run the `raft/raft.tla` alignment and the Protocol Verification Checklist (commit rule, leader completeness, `commitIndex` monotonic) against the integrated implementation. - - IMPORTANT — raft.tla does NOT model durability: it restores `log` intact from "stable storage" on restart (`raft.tla:165-174`), i.e. it assumes what bug-661 violates. A passing TLC/alignment run on the unmodified spec therefore does NOT prove the commit-durability invariant. That invariant is verified by the unit/integration/E2E tests (2.x, 5.1, 5.2) plus the manual checklist — not by raft.tla. Do not conclude durability is proven from raft.tla alone. Formal durability proof would require extending raft.tla with a durable-vs-volatile log model (out of scope unless explicitly opted in). - - Update `src/controller/CLAUDE.md`: convert the "Commit Durability — KNOWN GAP (bug-661)" note into the now-true guarantee and tick the post-`ha-commit-durability` checklist item. - - Observable: the checklist passes with the durable self-count and the durable-backed freshness floor in place; the durability invariant is backed by passing tests (not by raft.tla); the CLAUDE.md gap note is removed; any deviation is recorded. - - _Requirements: 1.2, 2.3_ - - _Depends: 5.1_ - -- [x] 6. REOPENED 2026-06-19 — total-cluster-outage (cold-restart-all) committed-job loss (bug-795) <!-- CLOSED 2026-07-02: 6.1 found the loss does NOT reproduce on a real fs — overlayfs fdatasync artifact confirmed. 6.2/6.3 not applicable (conditional on 6.1=REAL, which it wasn't). See Implementation Notes. --> - > Surfaced by the `xce-test-e2e-primitives` soak's new cold-restart-all step (kill ALL 3 + restart ALL). Phase 5 only validated rolling SINGLE-node failover; the total simultaneous outage path was never tested. Sequential by design (each task gates the next). DO NOT start a fix before 6.1 resolves real-vs-artifact. -- [x] 6.1 Reproduce cold-restart-all loss on a REAL fsync-able fs (rule out the overlayfs artifact) - - Run the converted `test-commit-durability.sh` (incl. its `run_cold_restart_recovery` phase) against a live 3-node cluster on a REAL fsync-able filesystem (NOT container overlay `/tmp`) — e.g. a tmpfs-free ext4/xfs deploy dir. Repeat enough rounds to characterize the race (it is non-deterministic: ~1-in-2 in sandbox runs). Capture controller recovery logs (WAL `truncate`, durable-floor, `committed_lsn`) for a losing run. - - Observable: a definitive verdict — EITHER the loss reproduces on a real fs (⇒ genuine total-outage durability gap, proceed to 6.2) OR it does NOT reproduce on a real fs and only occurs on overlay `fdatasync` (⇒ overlayfs artifact; record as an environment limitation, close Phase 6 with a note, and the `xce-test-e2e-primitives` task 7.2 unblocks on a real-fs run). - - _Requirements: 1.1, 1.5_ - - _Boundary: ValidationOnly (no production change) — `src/tools/test-commit-durability.sh` is owned by xce-test-e2e-primitives; consume it as-is._ -- [x] 6.2 Root-cause the total-outage recovery loss (conditional on 6.1 = REAL) <!-- N/A — 6.1 found no loss on real fs --> - - Instrument the recovery path for a simultaneous all-node SIGKILL + restart: determine whether (a) a job returned `jobsubmit ok` without being `fdatasync`'d on a fault-tolerant majority (a Req-1 commit-gate hole specific to the total-outage timing), (b) recovery (`replay_wal` / `persist_truncate_wal_above` / durable-floor re-derivation, tasks 3.1/3.4) truncates committed entries on cold restart, or (c) a just-rejoined-then-immediately-cold-killed BEHIND node re-leads with a truncated log and overwrites the good timeline (a promotion-fence/freshness-floor hole under cold restart, tasks 4.2/4.3). - - Observable: the exact mechanism is identified with a captured trace; classified as commit-gate / recovery-truncate / promotion-fence; mapped to the specific Requirement(s) and code path it violates. - - _Requirements: 1.1, 1.4, 1.5, 2.3, 2.6_ - - _Boundary: RecoveryIntegrity / DurableCommit (controller persist/commit/ha)_ - - _Depends: 6.1_ -- [x] 6.3 Fix the total-outage durability/recovery path + add a deterministic regression (conditional on 6.2) <!-- N/A — 6.2 not applicable --> - - Repair the identified mechanism so an acknowledged job survives a simultaneous all-node kill+restart, honoring the never-self-terminate invariant (loud fence, stay alive, keep retrying — see `src/controller/CLAUDE.md` Process Lifecycle). Add a deterministic cold-restart-all regression (prefer a unit/integration harness over the racy soak; the soak's `run_cold_restart_recovery` step stays as the E2E backstop). - - Observable: the cold-restart-all loss no longer reproduces (real fs); the new regression is RED on the unfixed build and GREEN after; the full controller suite passes under ASAN with `:FAIL:`==0. - - _Requirements: 1.1, 1.4, 1.5, 2.3, 2.6_ - - _Boundary: DurableCommit / RecoveryIntegrity / PromotionFence_ - - _Depends: 6.2_ -- [x] 6.4 Re-validate and reconcile the durability claim - - Re-run the deterministic regression + the cold-restart-all soak on a real fs (GREEN); reconcile `src/controller/CLAUDE.md` "Commit Durability — GUARANTEED" to state the total-outage scenario is now covered (or, if 6.1 found an artifact, qualify the overlay-fs caveat). Unblock `xce-test-e2e-primitives` task 7.2 and re-run its soak to close it. - - Observable: durability guarantee holds for BOTH rolling single-node failover AND total-cluster-outage cold restart, backed by the new regression; CLAUDE.md is accurate; bug-795 closed. - - _Requirements: 1.1, 1.2_ - - _Boundary: DurableCommit + docs_ - - _Depends: 6.3_ - -## Implementation Notes -- 1.1: Config plumbing routes through `myconf_t.wal_sync_ms` + `persist_config_set_wal_sync_ms()` (0-sentinel = keep `DEFAULT_WAL_SYNC_MS`), NOT a new config.h constant (design File Structure Plan keeps `DEFAULT_WAL_SYNC_MS`). The override is applied at BOTH `persist_init` sites — `myapp_run` step-6 (production path) and `cluster_init_phase2` (guarded by `wal_path[0]=='\0'`, skipped in production). They are mutually exclusive; apply any new persist_config override at both sites or it is silently dropped. Pre-existing latent bug (do NOT fix here): `--sync-timeout-ms` is applied only in phase2, so it is ignored on the production startup path — relevant if a later task relies on `sync_timeout_ms`. -- 1.2: Test seam = `persist_test_durable_lsn_on_disk(persist_t*)` (decl `persist.h`, impl `persist.c`, all `#ifdef XCE_TESTING`) — re-opens WAL segment files from disk and CRC-validates the chain (read-only, state-free) to return the highest fully-persisted LSN; new test `testsuite/test_commit_durability.c` (registered in `meson.build` + `controller_test.c`). Production `xce_controller` links without it (proven via `nm`). DRY NOTE: ~115 lines of segment-scan/CRC-chain logic are duplicated from the static `replay_wal`; tasks 3.1 (recovery verdict) and 3.4 (durable floor from WAL tip) touch the SAME scan/CRC logic — consider extracting a shared read-only `wal_scan_durable_lsn`-style helper there and routing this seam through it (avoid a third copy). `XCE_TESTING` is defined only for `controller_unit_test` (`src/controller/meson.build:167`). -- 1.3: Deterministic mode = `bash src/tools/test-commit-durability.sh --deterministic` proves the durability GAP via strace, NOT joblist data-loss. SCOPE CLARIFICATION (review-ratified): single-host SIGKILL committed-job loss is physically NON-deterministic — `persist_wal_async` pwrites inline (page cache survives process death) and standbys fdatasync-before-WalAck (persist.c:515), so a committed job is always durable off the primary; a data-loss run flips GREEN/RED/RED. The deterministic slice: with `--wal-sync-ms 30000` the primary acks "jobsubmit ok" with ZERO primary fdatasync in the submit→ack window (exec-wrap `strace -f -tt -e trace=fdatasync,fsync`; PTRACE_ATTACH `-p` is yama-blocked). RED = in-window primary syncs==0 (15/15 across 5 runs; primary=0 vs standby=2). This is a valid RED→GREEN gate: the task-2.x fsync-before-ack fix makes the primary count ≥1 ⇒ GREEN. Original racy soak retained as the default mode. Boundary: only `src/tools/test-commit-durability.sh` (flag plumbed via deploy-ha.sh start.sh `$*` passthrough — no C edits). For task 5.2's E2E, prefer this strace gap-proof over joblist-loss for a deterministic durability assertion. -- 2.1: `persist_t.durable_lsn` (`_Atomic uint64_t`, accessor `persist_durable_lsn()`) = prefix watermark advanced ONLY after a successful `fdatasync`. `fsync_worker` is now a demand-driven group-commit flusher: `persist_wal_async` pwrites under `p->lock`, releases, then signals `fsync_cond`. Lost-wakeup guard is LEVEL-triggered on `flusher_work_pending_locked` (`durable_lsn < current_lsn-1`), re-checked under `p->lock` before wait and after each flush (reviewer-confirmed sound). Invariants: monotonic, `durable_lsn <= current_lsn-1`, clamped on truncate (`clamp_durable_lsn_locked` at both truncate paths). fail-loud rc-gate: `durable_lsn` does NOT advance on `fdatasync` error (all 4 sites: flusher main/fallback, `persist_stop`, `persist_flush`) — logs ERROR. Test fault-injection seam: `#ifdef XCE_TESTING` toggle `persist_test_set_force_fsync_fail()` (production-excluded, nm-verified). NOT yet consumed by commit (that is 2.2). FOLLOW-UP (non-blocking, out of scope for 2.1): bound the flusher retry on SUSTAINED fdatasync failure (backoff / rate-limited logging) — currently tight-spins (releases `p->lock` each iter, so liveness OK, but CPU/log burn on a degrading disk). NOTE: a HANGING (not erroring) fdatasync blocks appends since it runs under `p->lock` — pre-existing, not 2.1's boundary. -- 2.2: `majority_candidate_locked` (commit.c:93) self-cap = `persist_durable_lsn(ce->p)` (was `highest_appended_lsn`); appended tip kept only for the `assert(cand <= highest_appended_lsn)` never-beyond-log check. So `committed_lsn` cannot advance over an entry until it is durable locally (Req 1.2/1.4). Durable-advance WAKE (the wire 2.1 deferred): persist defines a callback hook `on_durable_advance(void*)`/`durable_cb_ctx` + `persist_set_durable_cb()`; commit registers `commit_on_durable_advance` (ctx=engine) in `commit_engine_init`, unregisters in `commit_engine_fini`. Flusher invokes it ONLY when `durable_lsn` advanced, AFTER releasing `p->lock` → callback takes `ce->lock` + broadcasts `ce->cond`. Respects commit→persist dependency direction (persist never includes commit). Lock order p->lock→ce->lock only (cycle-free). Lost-wakeup-free: broadcast follows the release-store; waiter re-reads durable under ce->lock; timedwait backstop. Shutdown UAF closed: `persist_stop` JOINS the flusher BEFORE `commit_engine_fini` (cluster.c:1675 vs 1695); final stop-flush does NOT fire the callback; fini also unregisters defensively. GOTCHA (bug-704): self-cap broke 4 existing commit-path fixtures (test_repl_majority/coldstart_lsn0/commit_barrier_rollout/demotion_convergence) that primed `current_lsn` but not `durable_lsn` — fix = prime `durable_lsn = appended tip` in any in-memory fixture that sets current_lsn without writing real WAL bytes (the flusher can't advance durable for a fake log). Any future commit-engine test must prime durable_lsn too. -- 2.3: Standby persist-before-ACK pinned + Req-1.3 hole closed. `persist_wal_write_replicated` (persist.c:741) now rc-gates the inline `fdatasync`: on failure (or `flusher_fsync_forced_fail`) it logs ERROR + `return -EIO` (symmetric with the 2.1 flusher gate) — the standby will NOT ACK a non-durable entry. Caller `ha_apply_single_entry` (ha.c:1135-1166) ALREADY routes a negative return to no-apply/no-ACK (`ha_send_wal_ack_reply` reached only on rc==0), so NO ha.c change was needed (boundary stayed persistence-only). Ordering pin = `#ifdef XCE_TESTING atomic_uint test_replicated_fdatasync_count` incremented at the sync site; `test_replicated_persist_syncs_before_return` asserts it increments inside the call (page-cache means a disk re-read can't prove fsync — use the counter). `test_replicated_sync_failure_no_ack` REDs without the gate. Seam production-excluded (nm-verified). HOUSEKEEPING: controller tests `mkdtemp` into `/tmp/xce_*` and don't clean up — thousands accumulate and eventually make `mkdtemp` fail → misleading suite-wide "Expected Non-NULL" cascade. Run `rm -rf /tmp/xce_*` if that appears (pre-existing test pattern, not a regression). -- 2.4: SPEC INACCURACY corrected — `persist_wal_sync_append` is NOT dormant; it is the LIVE job-submit path (`persist_job_hook_fun` calls it at persist.c:1911). Took the "route through durable-majority eval" option (delete infeasible). Production path (ce!=NULL, lsn!=0) was already durable-safe via 2.2's `majority_candidate_locked` self-cap. Closed the hazard in the LEGACY branch: the former unconditional `atomic_store(committed_lsn, written_lsn)` on majority ACK is now `sync_append_legacy_commit_store()` which advances ONLY when `written_lsn <= persist_durable_lsn(p)` (LSN-0 edge: store of 0 valid). Added the design's debug invariant `committed_lsn <= durable_lsn` (`#ifdef XCE_TESTING` assert at commit.c:162 + inside the legacy store) as the future-proof guard. Fixed the stale "after local fsync via persist_wal_async" comment (2.1 deferred the fsync). CODE-SEARCH (authoritative): the ONLY `committed_lsn` stores are — engine advance (commit.c, durable-capped by 2.2), legacy sync_append (now gated), and the two `persist_truncate_wal_above` cutoff stores (NOT ACK-driven; each immediately `clamp_durable_lsn_locked`). ha.c/ha_convergence.c stores are `last_known_committed_lsn` (DIFFERENT field). So no path advances commit on ACK without the durable precondition. Seam `persist_test_sync_append_legacy_commit_store` is production-excluded. -- 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs. -- 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code. -- 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits). -- 3.4: On recovery, `persist_recover` publishes the durable floor = highest intact replayed LSN (recovered WAL tip = `current_lsn-1` after `replay_wal`) into `durable_lsn` via `atomic_store` (ASSIGN, NOT max — so a truncating restart lands at the LOWER intact tip, never over-advertising; this is the crux of Req 2.3). Added `persist_durable_floor()` (NULL-safe thin accessor) as the seam task 4.2 consumes. Floor is the durable LAST-LOG INDEX (WAL tip), NOT the commit position (`committed_lsn`) and NOT a volatile in-memory value. Invariant-safe (recovery/init-time; recovered entries are CRC-validated on-disk = genuinely durable; preserves `durable_lsn <= current_lsn-1`). 4.2 (ha.c/cluster.c) CONSUMES this floor in `ha_vote_self_freshness` — 3.4 only PUBLISHES (persistence-only). Phase 3 (recovery integrity) COMPLETE. -- 4.1: ANALYSIS-ONLY (no .c/.h). Pinned freshness-floor formula in design.md §"Freshness-Floor Formula (task 4.1)" — THE CONTRACT 4.2 implements. CANDIDATE advertises BARE `cand_lsn = persist_durable_floor(p)` (durable WAL tip) — sites ha.c:3660-3666 (real-vote) + ha.c:3735-3741 (PreVote) must call `persist_durable_floor` DIRECTLY, BYPASSING `ha_vote_self_freshness` on the LSN axis (that fn unconditionally folds in `last_known_committed_lsn` → `max(durable_floor, learned)` = the truncated-candidate-wins bug). VOTER floor = `max(durable_floor, last_known_committed_lsn)` — sites cluster.c:388-395/471-475 + ha.c:2243 keep `ha_vote_self_freshness` with `local_lsn=durable_floor` (the learned_committed backstop stays on the VOTER axis only). Grant iff `cand_tag > floor_tag OR (==tag AND cand_lsn >= floor_lsn)` (term-then-index, matches raft.tla `LastTerm`/`mlastLogIndex |-> Len(log)`). `cand_tag`/`floor_tag` = `max(local_tag, learned_tag)`. SAFETY (Leader Completeness): quorum intersection Q_L∩Q_v ⇒ truncated candidate (durable tip < committed L) denied. LIVENESS (no bug-339 livelock): rests on D* (max durable tip in reachable majority); `durable_floor(v) ≤ D*` and `learned_committed(v) ≤ L ≤ D*`, so the most-durable candidate is always grantable. `safe_committed_lsn` STAYS voter-reply-sourced (unchanged). CAVEAT: raft.tla validates election/freshness LOGIC only, NOT durability (no TLC run — no toolchain; conceptual alignment) — durability verified by 2.x/3.4 tests. (1 remediation: candidate-vs-voter contradiction + §5b example fixed.) -- 4.2: Implements the 4.1 formula. New `ha_candidate_advertise_lsn(persist*)` = bare `persist_durable_floor`. CANDIDATE sites advertise bare durable_floor for the LSN (tag still via `ha_vote_self_freshness(...,&tag,NULL)`): ha.c:~3690 (real election), ha.c:~3771 (PreVote), AND myapp.c:~234 (boot election — a THIRD candidate site the brief didn't name; fixed too, else a boot-time truncated candidate over-advertises). VOTER floor: cluster.c:~383 (real) + cluster.c:~471 (PreVote) feed `local_lsn = persist_durable_floor(&c->persist)` into `ha_vote_self_freshness` → floor = `max(durable_floor, learned_committed)`. Comparison unchanged (term-then-index). `ha.c:2243` legacy `handle_request_vote` is XCE_TESTING-only/param-driven — left unchanged. `safe_committed_lsn` UNTOUCHED (voter-reply-sourced). 3 tests in test_ha_vote_gate.c prove: candidate advertises bare durable_floor (not stale learned); voter durable-floor denies truncated; longest-durable grantable / no livelock. bug-339 regression test fixture primed `durable_lsn=10` (model-preserving). INFO: truncated-deny proven at the freshness-comparison-expression level shared by both vote+PreVote paths (no dedicated live PreVote-handler truncation driver — adequate; nice-to-have for 5.x). GOTCHA for future election tests: the freshness floor now reads `persist_durable_floor`, so any vote/election fixture that sets `applied_lsn`/`last_known_committed_lsn` but not durable state will see floor/advertise=0 — prime `durable_lsn` to the modeled tip. -- 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE. -- 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit). -- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die). -- REWORK (2026-06-18, user invariant — a controller MUST NEVER self-terminate): tasks 3.3/4.3/4.4 had implemented Req 2.4 "fail loud" as a SELF-ABORT (`persist_recover_fatal_if_unrecoverable → tlog_exit(); abort()` + `persist_fatal_hook`). That violated the invariant (a self-exited controller needs MANUAL admin recovery + leaves quorum degraded; a live retrying node self-heals when a peer returns) and was the true root cause of the 5.2 soak quorum collapse. CORRECTED: removed the abort/exit terminal entirely. New: `persist_recover_loud_fence_if_unrecoverable(persist_t*)` (NO abort) logs a loud operator-visible ERROR (truncation boundary vs cluster HWM) + sets sticky `_Atomic bool recovery_unrecoverable` (accessors `persist_recovery_unrecoverable`/`persist_clear_recovery_unrecoverable`); the member STAYS ALIVE, permanently non-promotable/fenced (`in_catch_up` + the sticky flag both consulted by the promotion gate), and KEEPS RETRYING catch-up. Decision enum `RECOVER_DECISION_FATAL` → `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). 4.4 timer no longer aborts — it re-emits the loud operator alert + re-arms (operator visibility), never exits. Flag + fence CLEAR automatically on catch-up (not truncated AND durable_floor ≥ HWM) → re-eligible (Req 2.6), fully automatic. `persist_fatal_hook` removed. Verified: grep/nm show NO abort/exit/tlog_exit on the recovery path; ASAN suite :FAIL:==0; reworked tests (row4/4b → loud_fence, unrecoverable-stays-alive-loud, released-on-catchup, cancel-stops-realert) PASS. Spec updated: requirements.md Req 2.4 + design.md (decision table rows 4/4b, RecoveryIntegrity, traceability, FATAL-vs-fence open question) reworded "FATAL/terminate" → "LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — NEVER self-exit)". PROJECT-WIDE GUIDANCE added: `src/controller/CLAUDE.md` §"Process Lifecycle (DESIGN INVARIANT)" + `.kiro/steering/ha-async-io.md` — controllers never self-terminate. (`ha_self_restart` execv-restart in convergence/demotion is a SEPARATE primitive, untouched.) -- 5.2: PASS on the reworked (no-self-exit) build. (a) DETERMINISTIC E2E flipped RED→GREEN, stable 3/3 runs / 9-9 probes — in-window primary fdatasync ≥1 (=2) precedes the ack ("durable-before-ack"), the inverse of 1.3's unfixed RED. (b) ROLLING SIGKILL soak (6 rounds) exit 0 = ZERO acknowledged-job loss, cluster stayed continuously available, 0 self-terminations (the no-self-exit rework removed the abort that collapsed quorum in the first attempt). Validation-only (no production code). NOTE: ran in-sandbox via background+poll on a worktree deploy dir (avoids /tmp overlay EIO); the earlier collapse was the abort terminal, now corrected — a host re-run is still good practice but the in-sandbox soak now passes with the fix. -- 5.3: Formal re-verification (conceptual — no TLA+ toolchain in env; project practice = manual alignment + the CLAUDE.md Protocol Verification Checklist). Checklist walked against the integrated impl, all hold: votedFor/currentTerm (untouched); `commitIndex`/`committed_lsn` monotonic + never exceeds `durable_lsn` (2.2/2.4 debug-assert); leader commits via per-entry majority ack (unchanged); `safe_committed_lsn` from voter replies (untouched, 4.2/4.3); WAL truncation only above the safe point + `durable_lsn` clamped on truncate; freshness gate term-then-index with durable-backed LSN (4.1 pinned vs raft.tla `mlastLogIndex`/`LastTerm`; candidate bare `durable_floor`, voter floor `max(durable_floor, learned)`); contiguous-prefix commit; `durable_lsn` prefix watermark; causal-order append; and the NEW item — a member counts toward the commit majority only at its durable LSN, acknowledged⇒durable-on-majority — now TRUE. raft.tla alignment: the election/freshness logic maps faithfully; raft.tla does NOT model durability (assumes a durable log, restores `log` intact on restart `raft.tla:165-174`) so it CANNOT prove the commit-durability invariant — that invariant is backed by tests 2.x/5.1 + the deterministic RED→GREEN E2E + the zero-loss/available soak (5.2), NOT by raft.tla. `src/controller/CLAUDE.md`: "Commit Durability — KNOWN GAP (bug-661)" flipped to "GUARANTEED / CLOSED (landed 2026-06-18)" with the mechanism list (incl. the no-self-exit loud fence); the post-`ha-commit-durability` checklist item ticked `[x]`. ALL 17 tasks (1.1–5.3) + the no-self-exit rework COMPLETE. -- 6.1-6.4 (2026-07-02): Ran `src/tools/test-commit-durability.sh ~/tmp/xce-durability-deploy 8` (default 8 rolling rounds + `run_cold_restart_recovery`) 5 times against a live 3-node cluster deployed to `~/tmp` — confirmed via `mount`/`df -T` to be genuine ext4 on `/dev/nvme0n1p2`, NOT the container's overlay `/`. **Verdict: the cold-restart-all committed-job LOSS does NOT reproduce on a real fs.** Across 5 full runs (40 rolling-failover rounds + 5 cold-restart-all cycles total): zero instances of a committed job vanishing after a full cold-stop-all + cold-start-all cycle. Confirms bug-795's original sighting was an overlayfs `fdatasync` artifact, not a genuine total-outage durability gap — the branch predicted in task 6.1's own Observable. 6.2/6.3 (conditional on 6.1=REAL) are therefore N/A. One SEPARATE, unrelated anomaly surfaced in run 5/5: `expect-jobs` reported `actual 20` vs `expected 19` after cold-restart-all — an EXTRA committed job, not a missing one (the opposite of bug-795's symptom), root cause not yet investigated (deprioritized — doesn't match the loss scenario this phase was probing for). Logged as **bug-1396** (open, low-priority, unconfirmed whether harness-side ACKED_FILE race or a genuine controller duplication). `src/controller/CLAUDE.md`'s "Commit Durability" caveat updated to reflect the closed investigation (see that file). -- VALIDATION FIX (2026-06-18, `/kiro-validate-impl` finding bug-760): the feature-level validation caught a deterministic 6/6 FAIL of the ADJACENT-spec test `test_fact_hook_append_is_retained_and_nonblocking` (`test_commit_barrier_rollout.c:446`, "Expected 1 Was 0"). ROOT CAUSE = task 2.2's durable-gating: committed_lsn now requires the async group-commit flusher to advance durable_lsn, but this real-persist test checked committed_lsn SYNCHRONOUSLY the instant after the covering ACK (before the flusher fdatasync'd). Task 2.2 had primed durable_lsn in the in-memory `barrier_setup_vcpus` stubs but MISSED this real-flusher test. CORRECTION of the earlier impl-notes (3.1/5.1) claim that this was a pre-existing `/tmp`-overlay fdatasync-EIO artifact: that is FALSE in this env — fdatasync returns rc=0 on /tmp here and the test passed pre-spec; it was a genuine durable-gating regression. FIX: insert `persist_flush(g_fp)` + assert `persist_durable_lsn(g_fp) >= fact_lsn` BEFORE `repl_on_ack` (commit_on_ack runs synchronously inside repl_on_ack, so the now-durable self completes the majority deterministically). Production was never affected (RETAIN facts have no client waiter; WAIT-mode replies block on the commit barrier `commit.c:334`). Verified: full controller suite 0 failures across 4 runs + meson `Ok:1 Fail:0`. GATING NOTE: meson `Fail:0` masks Unity sub-group failures — gate on the ANSI-stripped `:FAIL:` count from the test binary stdout. diff --git a/.kiro/specs/ha-commit-engine/brief.md b/.kiro/specs/ha-commit-engine/brief.md deleted file mode 100644 index bad97d7b..00000000 --- a/.kiro/specs/ha-commit-engine/brief.md +++ /dev/null @@ -1,24 +0,0 @@ -# Brief: ha-commit-engine - -> Sub-spec 1 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> Authoritative requirements/design/research: `.kiro/specs/ha-consistency-invariants/` (requirements.md, design.md §Commit Engine + §Quorum Accounting, research.md). This brief scopes the slice; do not re-derive decisions. - -## Problem -The majority-ACK check is dishonest today: `repl_wait_majority_ack` counts standbys with `acked_lsn > 0` ("ACKed anything ever"), not `acked_lsn >= entry_lsn`; quorum is checked only at the dispatch gate (TOCTOU); catch-up standbys count toward write quorum while excluded from broadcast; zero active standbys short-circuits to success. Even the one existing barrier (job submit) provides no real per-entry guarantee. - -## Desired Outcome -A commit engine (`commit.c`/`include/commit.h`) is the single authority for commit-index advancement: per-LSN majority evaluation (`acked_lsn >= N`, self counted), contiguous + monotonic `committed_lsn`, WAIT/RETAIN completion modes, `n_vcpus`-relative admission cap, membership-change re-evaluation. Wired behind the existing job-submit sync path only (adapter in `repl_wait_majority_ack`). `ha_has_write_quorum` excludes catch-up standbys. - -## Scope -- **In**: `commit.c` + `include/commit.h` per design §Commit Engine service interface; quorum-accounting fix in `ha.c`; `repl_on_ack` → `commit_on_ack` feed; removal of the `n_active == 0 → success` short-circuit for `n_peers > 1`; `n_peers <= 1` self-quorum preserved; `testsuite/test_commit_engine.c`. -- **Out**: classification/rollout to other WAL types (ha-commit-barrier-rollout); own-term gate *consumers* and COMMITTING removal (ha-single-leader-role — but `commit_set_writable_lsn` API ships here, unset = no gate); demotion/egress/catch-up (ha-convergence-recovery). - -## Requirements (umbrella IDs) -2.1 (job-submit path), 2.3, 2.6, 4.1, 4.2, 4.3, 4.4, 6.5, 7.1, 7.4, 7.5. - -## Upstream / Downstream -- **Upstream**: ha-wal-replication (acked_lsn, WalAckReply — complete). -- **Downstream**: ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery (all consume the engine). - -## Constraints -Per umbrella design: `commit_on_ack` callable from iomgr (never blocks); blocking waits on vtask threads only; admission cap expressed relative to `n_vcpus` (never absolute); TDD in testsuite/ first. diff --git a/.kiro/specs/ha-commit-engine/design.md b/.kiro/specs/ha-commit-engine/design.md deleted file mode 100644 index 366a0961..00000000 --- a/.kiro/specs/ha-commit-engine/design.md +++ /dev/null @@ -1,334 +0,0 @@ -# Design: ha-commit-engine - -> Sub-spec 1 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **Commit Engine** and **Quorum Accounting** components (`.kiro/specs/ha-consistency-invariants/design.md`) and reuses the umbrella's `commit.h` service interface verbatim. Discovery is complete via the umbrella's `research.md` (Option C phased hybrid, locked decisions); this spec implements umbrella Migration Strategy phase 1. Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping). - -## Overview - -**Purpose**: This feature gives cluster operators an honest commit barrier for job submission: a new commit engine (`commit.c`) becomes the single authority for `committed_lsn` advancement, evaluating majority per entry (`acked_lsn >= N`, self counted) instead of "ACKed anything ever", and the dispatch-gate quorum count stops counting catch-up standbys. - -**Users**: Cluster operators gain the guarantee that a confirmed job submission is durable on a majority of members; downstream sub-specs (ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) gain the engine interface they build on. - -**Impact**: Replaces the counting logic of `repl_wait_majority_ack` (repl.c:589, current) for WAL ACKs with per-LSN engine evaluation; removes the `n_active == 0 → success` short-circuit for multi-member clusters; moves the `committed_lsn` store out of `persist_wal_sync_append` (now persist.c:5639) into the engine; fixes `ha_has_write_quorum` (now ha.c:4495) to exclude catch-up standbys. Behavior of every other write path is unchanged in this spec. (Note: `persist_wal_sync_append` itself was later changed by ha-term-safety task 3.1 to route the primary's append through the raft library — `ha_primary_log_append` → `raft_log_append` — before this majority-commit barrier runs; see that spec's design.md.) - -### Goals -- Entry N commits only when `1 (self) + count(active standbys with acked_lsn >= N) >= n_peers/2 + 1` -- `committed_lsn` advancement: monotonic, contiguous, never beyond the appended log — owned by the engine alone -- Full `commit.h` service interface shipped (WAIT + RETAIN modes, `commit_set_writable_lsn`, membership re-evaluation) so downstream specs consume without interface churn -- Admission cap on concurrent WAIT blockers, expressed relative to `n_vcpus` (never absolute) -- `n_peers <= 1` self-quorum preserved; `n_peers > 1` zero-peer-ACK never commits - -### Non-Goals -- Wiring any path other than job submit (WAL_JOB_SUBMIT sync append) — barrier-rollout owns the classification sweep -- Setting `writable_lsn` anywhere (own-term gate consumers are ha-single-leader-role; unset = no gate) -- RETAIN-mode production consumers (facts roll out in ha-commit-barrier-rollout; the mode itself ships and is tested here) -- Config-change barrier (`acked_config_lsn` track untouched), demotion/egress/catch-up changes, COMMITTING removal -- Async continuation of commit waits (umbrella-documented future work; blocking condvar + admission cap is the accepted mechanism) (Note: superseded by raft-apply-decoupling, which added a per-LSN continuation registry, `commit_resolve`/async registration, and a dedicated timeout-sweep thread to `commit_engine_t` — see that spec's design.md) - -## Boundary Commitments - -### This Spec Owns -- `src/controller/commit.c` + `src/controller/include/commit.h` — the commit engine: per-LSN ACK evaluation, waiter registry, admission cap, `writable_lsn` storage, `committed_lsn` advancement -- The walAck branch semantics of `repl_wait_majority_ack` (becomes the engine adapter) and the `repl_on_ack → commit_on_ack` feed -- `ha_has_write_quorum` catch-up exclusion and the `repl_active_standby_count` helper -- The `committed_lsn` advancement rule on the job-submit sync path (`persist_wal_sync_append`) -- `testsuite/test_commit_engine.c` and updates to existing tests whose expectations encode the dishonest counting - -### Out of Boundary -- Which WAL types use WAIT vs RETAIN (classification table — ha-commit-barrier-rollout) -- Calling `commit_set_writable_lsn` from promotion, WAL_NOOP, COMMITTING deletion (ha-single-leader-role) -- Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery) -- `acked_config_lsn` / configAck counting in `repl_wait_majority_ack` (config barrier — ha-commit-barrier-rollout); the configAck branch is left byte-for-byte as-is -- WalAckReply wire format, standby apply mechanics, `acked_lsn` update rules (ha-wal-replication — complete upstream) - -### Allowed Dependencies -- `standby_conn_t.acked_lsn` / `in_catch_up` per-standby tracking in `replicator_t` (src/include/repl.h) — read-only consumption; the engine never manages connections -- `persist_t.committed_lsn` atomic (src/include/persist.h:71) — the engine is the only production writer after init/truncation -- iomgr/vtaskmgr threading rules per `ha-async-io.md`: `commit_on_ack` and `commit_on_membership_change` are called on the iomgr thread and must never block on I/O or long waits (bounded mutex hold only); `commit_submit(WAIT)` blocks only on vtask threads -- Dependency direction (umbrella contract): `include/commit.h` depends only on libc/stdatomic types plus forward-declared `persist_t`; `commit.c` is called BY `persist.c` (append/sync path), `repl.c` (ACK + membership events), and `ha.c`/`cluster.c` (init, future promotion); it calls into none of them (Note: superseded — `commit.h` now `#include`s the full `persist.h` and `vtaskmgr.h` rather than only forward-declaring, and `commit.c` now calls INTO `persist.c` (`persist_durable_lsn`, `persist_set_durable_cb`) to implement the durable-majority self-cap; see ha-commit-durability and raft-drive-async design.md) - -### Revalidation Triggers -- Any change to `commit.h` signatures or return-code contract → revalidate ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery -- Any change to `acked_lsn` semantics or WalAckReply payload (ha-wal-replication) → re-verify engine evaluation -- Changing the admission-cap formula or `COMMIT_WAIT_RESERVED` default → re-check operator tuning guidance and barrier-rollout sizing assumptions -- Moving `committed_lsn` ownership (e.g., persisting it) → revalidate ha-convergence-recovery's rejoin reconciliation - -## Architecture - -### Existing Architecture Analysis -- Single-writer model: only the primary appends WAL entries; per-standby `acked_lsn` (Raft matchIndex equivalent) already exists and is updated by `repl_on_ack` (now repl.c:250-285) on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, now ha.c:7249 → ha.c:6630 → repl.c:250) -- ACKs are cumulative: a standby ACKs its highest contiguously applied LSN (ha-wal-replication contract), so per-standby `acked_lsn` is a watermark — engine evaluation needs no per-entry bitmap -- Defects replaced: `repl_wait_majority_ack` counted `acked_lsn > 0` — any ACK ever; `n_active == 0` short-circuited to success; quorum denominator was `n_active` (connected standbys) rather than `n_peers` (configured members); `ha_has_write_quorum` counted `standby_endpoints->count`, which included catch-up standbys. (Note: these are pre-spec line citations that no longer resolve — the dishonest-counting branch was fully deleted, not just bypassed: `repl_wait_majority_ack` now unconditionally delegates the walAck branch to `commit_submit`, per ha-commit-barrier-rollout task 2.2. Current `ha_has_write_quorum` is at ha.c:4495-4523.) -- `persist_wal_sync_append` (now persist.c:5639) is the only sync (blocking) write path — job submit via `persist_job_hook_fun` (now persist.c:6509); it blocks a vtask pool thread (default 8, `--vcpus` override, sized in `cluster.c` ~1743-1745; `commit_engine_init` is wired at cluster.c:1820, not cluster.c:1082 as originally cited); no liveness risk since wakeups come from iomgr (plus, since ha-commit-durability, from the group-commit flusher's durable-advance callback — see below) -- `replicator_t` is embedded in `persist_t` (`p->repl`), and `ha_state_t`/`cluster_t` reach it via `c->persist.repl` — the engine can reach the standby registry through its `persist_t*` - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph vtask_thread - JS[job submit hook<br/>persist_wal_sync_append] - end - subgraph iomgr_thread - PCM[peer_ctrl_msg_handler<br/>repl_ack_decode_and_dispatch] - ROA[repl_on_ack] - end - subgraph commit_engine[commit engine - commit.c] - SUB[commit_submit WAIT/RETAIN] - OA[commit_on_ack] - MC[commit_on_membership_change] - CL[(committed_lsn<br/>in persist_t)] - end - GATE[ha_dispatch_gate<br/>ha_has_write_quorum] - REG[(replicator_t standbys<br/>acked_lsn, in_catch_up)] - - GATE -->|fast reject 6.1| JS - JS -->|entry lsn N| SUB - PCM --> ROA - ROA -->|standby_idx, acked_lsn| OA - ROA -->|updates| REG - OA -->|reads| REG - OA -->|advance + broadcast| CL - SUB -->|wait predicate on| CL - MC -->|wake + re-evaluate| SUB - GATE -->|repl_active_standby_count| REG -``` - -**Architecture Integration** (sliced from umbrella): -- Selected pattern: single commit engine as the authority for `committed_lsn`; the dispatch gate stays a fast-rejection filter (6.1), the engine is the authoritative ACK-time verifier (1.2, 4.1) — closing the TOCTOU by construction: gate-time connectivity never substitutes for acknowledgment (1.3) -- Existing patterns preserved: hook-based persistence dispatch, full fan-out broadcast, `errnox` early-return style, `ack_cond` signaling (kept for the untouched configAck track) -- New component rationale (umbrella): requirements families 2.x/4.x are one capability; placing them in `repl.c` would bloat connection management with consensus logic -- Steering compliance: iomgr callbacks never block (bounded mutex only); blocking waits on vtask threads with `n_vcpus`-relative admission cap; TDD with testsuite/ tests first - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), pthreads, stdatomic | commit engine | no new dependencies, no .capnp changes | -| Data / Storage | existing WAL (persist.c) | `committed_lsn` atomic in `persist_t` | no schema change; WAL_NOOP is NOT in this slice | -| Build / Test | meson, testsuite/ pattern | `test_commit_engine.c` registered in `controller_unit_test_src` | XCE_TESTING seams | - -## File Structure Plan - -### Directory Structure -``` -src/controller/ -├── commit.c # NEW — commit engine: per-LSN evaluation, waiter registry, -│ # admission cap, writable_lsn, committed_lsn advancement -├── include/commit.h # NEW — engine public API (controller-internal; umbrella-verbatim) -└── testsuite/ - └── test_commit_engine.c # NEW — unit + integration tests for the engine and wiring -``` - -### Modified Files -- `src/include/repl.h` — `replicator_t` gains `struct commit_engine *ce` (forward-declared, NULL until wired); declares `repl_set_commit_engine()`, `repl_active_standby_count()`; `repl_wait_majority_ack()` gains an `entry_lsn` parameter -- `src/controller/repl.c` — `repl_on_ack` forwards `(standby_idx, acked_lsn)` to `commit_on_ack` after updating `acked_lsn`; `repl_add_standby` / `repl_remove_standby` / `repl_start_catchup` / `repl_complete_catchup` call `commit_on_membership_change`; walAck branch of `repl_wait_majority_ack` delegates to `commit_submit(WAIT)` when the engine is wired (legacy counting survives only for engine-less unit-test contexts and is retired by ha-commit-barrier-rollout); `repl_active_standby_count` helper added -- `src/controller/persist.c` — `persist_wal_sync_append` passes `written_lsn` to the adapter; deletes its own `committed_lsn` store (persist.c:1035) and the `has_standbys → -ENODEV` special case on the engine-wired path (the engine decides: self-quorum vs timeout) -- `src/controller/ha.c` — `ha_has_write_quorum` counts via `repl_active_standby_count` (connected AND not `in_catch_up`); `ha_state_t` gains a `replicator_t *repl` back-pointer set at cluster init -- `src/controller/ha.h` — `ha_state_t` field addition (`replicator_t *repl`) -- `src/controller/include/cluster.h` / `cluster.c` — `cluster_t` gains `commit_engine_t commit`; `commit_engine_init(&c->commit, &c->persist, ha->n_peers, c->vtaskmgr.n_vcpus)` + `repl_set_commit_engine` called at init phase after vtask pool sizing (cluster.c:1082) and peer count are known; `commit_engine_fini` on teardown -- `src/controller/meson.build` — add `commit.c` to controller sources and `testsuite/test_commit_engine.c` to `controller_unit_test_src` -- `src/controller/testsuite/test_repl_majority.c` — walAck cases updated to engine-wired per-LSN semantics (configAck cases untouched) - -## System Flows - -### Job submit through the engine (umbrella "request-shaped write", sliced) - -```mermaid -sequenceDiagram - participant C as client - participant V as vtask thread - participant CE as commit engine - participant R as repl broadcast - participant S as standby - participant IO as iomgr thread - C->>V: job submit - V->>V: ha_dispatch_gate (incl. ha_has_write_quorum, catch-up excluded) - V->>V: insert job (provisional, unchanged order) - V->>R: persist_wal_async: append lsn=N + broadcast - V->>CE: commit_submit(N, WAIT, sync_timeout_ms) - CE->>CE: admission cap check (n_vcpus - reserved) - S->>IO: WalAck acked_lsn >= N - IO->>CE: repl_on_ack → commit_on_ack - CE->>CE: majority for N? advance committed_lsn, broadcast cond - CE-->>V: 0 (committed) | -ETIMEDOUT | -XCE_ERR_NOT_READY - alt committed - V->>C: success - else timeout / not ready - V->>C: error (operation failed) - end -``` - -Flow decisions: the admission cap rejects before blocking when `inflight_waits >= max(1, n_vcpus - COMMIT_WAIT_RESERVED)` (7.1, 7.2); majority is evaluated against the entry's LSN at ACK time — gate-time connectivity never substitutes (1.2, 1.3, 4.1); a timeout leaves `committed_lsn` unmoved past the entry (2.1) and the entry in the log (outcome converges per umbrella 2.5, owned downstream). - -### Commit evaluation (inside `commit_on_ack`) - -Under the engine mutex: collect `acked_lsn` of all standbys with `ep != NULL && !in_catch_up` into a scratch array, sort descending. With self counted, entry N has majority when at least `n_peers/2` active standbys have `acked_lsn >= N`; therefore the new commit candidate is `arr[n_peers/2 - 1]` (0-indexed; 0 candidates → no advance) clamped to the highest appended LSN. Store `committed_lsn = max(committed_lsn, candidate)` (2.2); cumulative ACK watermarks make the result contiguous by construction (2.3). Broadcast the condvar; each waiter re-checks its own predicate. Catch-up standbys are excluded from the count (umbrella-locked, conservative — their ACKs count after `repl_complete_catchup` triggers `commit_on_membership_change` re-evaluation). - -(Note: superseded by ha-commit-durability, Req 1.2 (bug-661 fix) — the candidate is no longer clamped only to the highest *appended* LSN; self's contribution is additionally capped at `persist_durable_lsn(ce->p)`, the highest LSN fdatasync'd locally, so an entry that is only page-cache-resident cannot count self toward its own majority. See that spec's design.md for the durable-majority barrier.) - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1 | submit confirmed only after per-entry majority | Commit Engine, Job-Submit Adapter | job submit | -| 1.2 | ACK counts toward N only if acked_lsn >= N | Commit Engine | commit evaluation | -| 1.3 | connectivity never substitutes for ACKs | Commit Engine (ACK-time eval) | job submit | -| 2.1 | no commit-index advance without majority | Commit Engine | commit evaluation | -| 2.2 | monotonic committed_lsn | Commit Engine (max() store) | commit evaluation | -| 2.3 | contiguous, bounded by appended log | Commit Engine (watermark median, clamp) | commit evaluation | -| 3.1, 3.2, 3.3 | admission quorum: only members able to receive | Quorum Accounting (`ha_has_write_quorum`, `repl_active_standby_count`) | job submit (gate) | -| 4.1 | post-acceptance shrink → fail, not confirm | Commit Engine (WAIT timeout, no false success) | job submit | -| 4.2 | membership change re-evaluates pending waits | Commit Engine (`commit_on_membership_change`), ACK Feed | — | -| 4.3, 4.4 | zero peer ACKs never majority (n_peers>1) | Commit Engine (short-circuit removal) | commit evaluation | -| 5.1 | lag never blocks majority confirmation | Commit Engine (majority threshold only) | commit evaluation | -| 6.1 | reject without quorum | Quorum Accounting (gate fast-reject) + Commit Engine | job submit | -| 6.2 | auto-resume on quorum restore | Quorum Accounting (stateless re-evaluation per request) | — | -| 6.3 | standalone self-quorum | Commit Engine (`n_peers <= 1` immediate commit) | — | -| 7.1, 7.2 | n_vcpus-relative admission cap, NOT_READY beyond | Commit Engine (admission counter) | job submit | -| 7.3 | ACK processing unaffected at cap | ACK Feed (iomgr, non-blocking) | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| Commit Engine | controller/commit.c | authority for commit-index advancement and waiter completion | 1.1–1.3, 2.1–2.3, 4.1–4.4, 5.1, 6.3, 7.1–7.2 | repl ACK events (P0), persist_t committed_lsn (P0) | Service, State | -| ACK Feed | controller/repl.c | forward ACK + membership events to the engine from iomgr | 4.2, 7.3 | commit engine (P0), iomgr handler chain (P0) | Event | -| Quorum Accounting | controller/ha.c, repl.c helper | honest gate-time quorum, catch-up exclusion | 3.1–3.3, 6.1, 6.2 | repl registry (P0) | Service | -| Job-Submit Adapter | controller/persist.c, repl.c | route the sync append wait through the engine | 1.1, 6.3 | commit engine (P0) | Service | - -### Commit Engine (`commit.c` / `include/commit.h`) - -| Field | Detail | -|-------|--------| -| Intent | Single authority for per-entry commit evaluation, `committed_lsn` advancement, and waiter completion | -| Requirements | 1.1–1.3, 2.1–2.3, 4.1–4.4, 5.1, 6.3, 7.1–7.2 | - -**Responsibilities & Constraints** -- Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`, now declared persist.h:274; only the engine stores to it in production after init/truncation paths — the store formerly in `persist_wal_sync_append`'s engine-wired branch is deleted; truncation stores remain in `persist_truncate_wal_above`, now persist.c:6764 — note the field comment there records several more legitimate `committed_lsn` writers added since, e.g. `persist_committed_lsn_advance_atomic`, all monotonic-max) -- Commit rule (Raft §5.3, umbrella-verbatim): entry N is committed when `count(self + active standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.1–2.3) (Note: superseded in part by ha-commit-durability — self's contribution to the count is additionally capped at the local durable watermark, not just the appended tip; see `majority_candidate_locked` in commit.c) -- Own-term gate storage: `writable_lsn` defaults to 0 = no gate (this spec never sets it); when set (by ha-single-leader-role / ha-term-safety's `ha_arm_own_term_gate`), no waiter completes successfully until `committed_lsn >= writable_lsn` -- Admission cap: at most `max(1, n_vcpus - COMMIT_WAIT_RESERVED)` (reserved default = 2) concurrent WAIT blockers; excess fails fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`) (Note: superseded by raft-drive-async task 1.6 — this local cap is now only the fallback used when no `vtaskmgr_t` pool is wired (`commit_set_pool`); the production path folds admission onto the shared vtaskmgr pool's single LivenessAdmission reservation instead. See that spec's design.md.) -- `n_peers <= 1`: self-ACK constitutes majority; WAIT returns 0 immediately and advances `committed_lsn` to the entry (6.3 — intentional change: standalone nodes now advance `committed_lsn` on sync appends, Raft-correct). For `n_peers > 1`, zero peer ACKs never commit (4.3, 4.4) - -**Dependencies** -- Inbound: `repl_on_ack` (iomgr) — ACK events (P0); `persist_wal_sync_append` — WAIT registration (P0); `cluster.c` init — lifecycle (P0); membership events from repl registry mutations (P0) -- Outbound: none (reads `p->repl` registry and stores `p->committed_lsn`); signals waiters via condvar broadcast with per-LSN predicates - -**Contracts**: Service [x] / State [x] / Event [x] - -##### Service Interface (umbrella-verbatim, `src/controller/include/commit.h`) -```c -/* include/commit.h — all functions thread-safe; _on_ack callable from iomgr (non-blocking) */ -typedef enum { COMMIT_MODE_WAIT, COMMIT_MODE_RETAIN } commit_mode_t; - -int commit_engine_init(commit_engine_t *ce, persist_t *p, int n_peers, int n_vcpus); -void commit_engine_fini(commit_engine_t *ce); - -/* Register entry at append time. RETAIN returns 0 immediately. - * WAIT blocks up to timeout_ms: 0 = committed; -ETIMEDOUT = not committed - * (entry remains in log; outcome resolves per umbrella 2.5); -XCE_ERR_NOT_READY = - * admission cap exhausted (only cause). An unmet own-term gate does NOT fail - * fast: the gate is part of the completion predicate, so WAIT blocks within - * timeout_ms until the gate opens, surfacing -ETIMEDOUT if it never does. */ -int commit_submit(commit_engine_t *ce, uint64_t lsn, commit_mode_t mode, - uint32_t timeout_ms); - -/* iomgr path: record standby ACK, evaluate majority, advance committed_lsn - * contiguously, wake satisfied waiters. Never blocks. */ -void commit_on_ack(commit_engine_t *ce, int standby_idx, uint64_t acked_lsn); - -/* Promotion: set own-term gate to the no-op entry's LSN. - * (API ships here; no caller in this spec — unset = no gate.) */ -void commit_set_writable_lsn(commit_engine_t *ce, uint64_t noop_lsn); - -/* Quorum-membership change (standby add/remove/catch-up transition): - * re-evaluate pending entries; standbys in catch-up are excluded. */ -void commit_on_membership_change(commit_engine_t *ce); -``` -- Preconditions: entries submitted in LSN order (single-writer guarantees this); `commit_engine_init` called after `n_peers` and `n_vcpus` are final -- Postconditions: `committed_lsn` never decreases (2.2); never exceeds the highest majority-acked contiguous LSN nor the highest appended LSN (2.1, 2.3) -- Invariants: no waiter completes successfully while `committed_lsn < its lsn` or (gate set) `committed_lsn < writable_lsn`; `inflight_waits <= max(1, n_vcpus - COMMIT_WAIT_RESERVED)` - -##### State Management -- State: per-standby view delegated to `standby_conn_t.acked_lsn` / `in_catch_up` (no duplication); internal: sorted matchIndex scratch array (`MAX_STANDBYS`), waiter accounting (per-LSN predicate over shared condvar), `writable_lsn`, `inflight_waits` counter, `n_peers`, `n_vcpus`, `persist_t *p` back-reference -- Concurrency: one mutex + one condvar (broadcast on `committed_lsn` advance and on membership change; waiters re-check per-LSN predicate — umbrella-accepted broadcast cost); `committed_lsn` read lock-free via the existing atomic; mutex hold in `commit_on_ack` is bounded (scan + sort of ≤ MAX_STANDBYS entries, no I/O) — safe on iomgr -- Re-evaluation semantics (4.2): membership change broadcasts the condvar so pending waiters re-check against updated membership; waiters whose entries gained majority complete with 0, others keep waiting until their deadline and fail with `-ETIMEDOUT` (4.1 — no confirmation ever happens without actual majority ACK; early-fail on shrink is permitted but not required by the umbrella, and is deliberately not implemented to allow reconnection within the timeout) - -**Implementation Notes** -- Integration: instance embedded in `cluster_t`; wired to repl via `repl_set_commit_engine(&c->persist.repl, &c->commit)`; XCE_TESTING tests construct the engine standalone against a stub `persist_t` + `replicator_t` -- Validation: assert monotonicity and the inflight invariant under `XCE_TESTING`; log a counter of committed-vs-appended lag (retained-uncommitted depth) for observability -- Risks: condvar broadcast storms under high write concurrency — accepted at current scale (umbrella); revisit with async continuation (out of scope) - -### ACK Feed (`repl.c`) - -| Field | Detail | -|-------|--------| -| Intent | Deliver ACK and membership events from the iomgr thread to the engine without blocking | -| Requirements | 4.2, 7.3 | - -Summary: `repl_on_ack` (repl.c:176) — after `atomic_store(&s->acked_lsn, acked_lsn)` and the existing `ack_cond` signal (kept for configAck) — calls `commit_on_ack(r->ce, slot_idx, acked_lsn)` when `r->ce != NULL`. `repl_add_standby`, `repl_remove_standby`, `repl_start_catchup`, `repl_complete_catchup` call `commit_on_membership_change(r->ce)` after mutating the registry. All call sites are iomgr-driven; `commit_on_*` never block (7.3). `replicator_t` gains the forward-declared `struct commit_engine *ce` pointer (NULL = not wired) so `src/include/repl.h` keeps zero controller-header dependencies. - -### Quorum Accounting (`ha.c` + `repl.c` helper) - -| Field | Detail | -|-------|--------| -| Intent | Honest gate-time write-admission quorum: count only members able to receive new entries | -| Requirements | 3.1–3.3, 6.1, 6.2 | - -Summary (umbrella §Quorum Accounting, sliced): new `int repl_active_standby_count(replicator_t *r)` counts slots with `ep != NULL && !in_catch_up` under `r->lock`. `ha_has_write_quorum` (ha.c:1507) replaces `standby_endpoints->count` with this helper via a `replicator_t *repl` back-pointer on `ha_state_t` (set during cluster init; when NULL — legacy test contexts — fall back to the current `standby_endpoints` count). Quorum condition unchanged: `n_peers <= 1 → true`; else `active >= n_peers / 2` (3.1–3.3). The gate stays a fast-rejection filter (6.1, `-XCE_ERR_NOT_READY` per existing `ha_dispatch_gate`); it is stateless per request, so quorum restoration auto-resumes acceptance with no operator action (6.2); the engine re-verifies authoritatively at ACK time (1.3, 4.1). Quorum-loss transition logging stays out (ha-convergence-recovery). - -### Job-Submit Adapter (`persist.c` + `repl.c`) - -| Field | Detail | -|-------|--------| -| Intent | Route the existing job-submit sync wait through the engine; preserve external return-code contract | -| Requirements | 1.1, 4.3, 4.4, 6.3 | - -Summary (umbrella migration note: "`repl_wait_majority_ack` becomes a thin adapter"): `repl_wait_majority_ack(r, timeout_ms, ack_op)` gains an `entry_lsn` parameter. walAck branch: when `r->ce` is wired, `return commit_submit(r->ce, entry_lsn, COMMIT_MODE_WAIT, timeout_ms)`; legacy counting survives only for engine-less unit-test contexts (deprecated comment; retired by ha-commit-barrier-rollout when call sites move to the engine directly). configAck branch: unchanged, `entry_lsn` ignored. `persist_wal_sync_append` (persist.c:1003): passes `written_lsn`; on the engine path, deletes the `has_standbys → -ENODEV` special case and its own `committed_lsn` store — the engine returns 0 (and has advanced `committed_lsn`) for `n_peers <= 1` (6.3), and `-ETIMEDOUT` for `n_peers > 1` with no covering ACKs (4.3, 4.4 — the `n_active == 0 → success` short-circuit is thereby removed from production). Return contract to `persist_job_hook_fun` callers is unchanged (0 / `-ETIMEDOUT` / now also `-XCE_ERR_NOT_READY`), and the existing error propagation to the job-submit client (job.c reference path) carries it (1.1). - -## Data Models - -### Domain Model (changes only) -- **commit_engine_t** (in-memory only; commit state volatile, Raft-conformant — umbrella-locked): `{ pthread_mutex_t lock; pthread_cond_t cond; persist_t *p; int n_peers; int n_vcpus; int inflight_waits; uint64_t writable_lsn; uint64_t scratch[MAX_STANDBYS]; }` (Note: superseded — the struct has since grown a `vtaskmgr_t *pool` field (raft-drive-async), a `uint64_t raft_commit_index` advisory field (ha-term-safety), and `commit_continuation_t *continuations`/`n_continuations`/`sweep_thread`/`sweep_stop` for async continuation resolution (raft-apply-decoupling). See `src/controller/include/commit.h` and those specs' design.md.) -- **replicator_t**: + `struct commit_engine *ce` (NULL until wired) -- **ha_state_t**: + `replicator_t *repl` (NULL in legacy test contexts) -- **cluster_t**: + `commit_engine_t commit` -- Invariants: `committed_lsn` monotonic, contiguous, ≤ highest appended LSN; `inflight_waits` within cap; engine never duplicates per-standby state - -## Error Handling - -### Error Strategy -- `commit_submit(WAIT)` returns `0` (committed), `-ETIMEDOUT` (not committed within timeout — entry remains in log; cluster-wide outcome convergence is umbrella 2.5, owned by ha-convergence-recovery), `-XCE_ERR_NOT_READY` (admission-cap exhaustion only — an unmet own-term gate is not a fail-fast cause: it is part of the WAIT completion predicate, so the waiter blocks within its timeout until the gate opens and surfaces `-ETIMEDOUT` otherwise) — distinct log messages per cause, both retriable -- Errors propagate via the existing `errnox` early-return pattern: `persist_wal_sync_append` → `persist_job_hook_fun` → job-submit reply; no in-memory rollback (provisional-state model, umbrella-locked) -- Gate rejection (no write quorum) keeps the existing `-XCE_ERR_NOT_READY` dispatch-gate behavior (6.1) -- `commit_engine_init` validates `n_peers >= 0`, `n_vcpus >= 1`, non-NULL `persist_t`; returns `-EINVAL` otherwise - -### Monitoring -- Log counters: inflight WAIT count vs cap; committed-vs-appended lag; per-cause rejection WARNs (rate-limited to transitions where applicable). Metrics export out of scope. - -## Testing Strategy - -### Unit Tests (`testsuite/test_commit_engine.c`, XCE_TESTING, stub persist/replicator) -- Per-LSN majority: with n_peers=3, ACK for LSN M does not commit entry N>M (`acked_lsn > 0` regression dies) (1.2); ACK `>= N` from one standby + self commits N (1.1) -- Contiguity + monotonicity: out-of-order/cumulative ACK watermarks advance `committed_lsn` to the median watermark, never past appended log, never backwards (2.1–2.3) -- Zero-peer rule: n_peers=3, no standby ACKs → WAIT returns `-ETIMEDOUT`, `committed_lsn` unmoved (4.3, 4.4); n_peers=1 → WAIT returns 0 immediately, `committed_lsn` advanced (6.3) -- Admission cap: with n_vcpus=8 reserved=2, the 7th concurrent WAIT gets `-XCE_ERR_NOT_READY`; cap floor `max(1, ...)` at tiny n_vcpus; ACKs still advance commits for admitted waiters while cap is saturated (7.1–7.3) -- Own-term gate storage: `writable_lsn` unset → no gating; set above entry → waiter held until `committed_lsn >= writable_lsn` (interface contract for downstream; no production caller here) -- Membership re-evaluation: catch-up standby's ACK excluded; after `commit_on_membership_change` (catch-up complete) its watermark counts and the pending waiter completes (4.2); waiter never falsely completes on shrink (4.1) -- Lag tolerance: n_peers=3, one standby far behind — majority ACK still commits (5.1) - -### Integration Tests (testsuite/) -- Quorum accounting: `repl_active_standby_count` excludes `in_catch_up`; `ha_has_write_quorum` false when the only connected standby is in catch-up (n_peers=3), true again after `repl_complete_catchup`; gate re-accepts after reconnection without any reset call (3.1–3.3, 6.1, 6.2) -- ACK feed: `repl_on_ack` with wired engine updates `acked_lsn` AND advances `committed_lsn` when majority reached (4.2 wiring) -- Sync-append path: `persist_wal_sync_append` with wired engine — success only on per-entry majority; `-ETIMEDOUT` with stale `acked_lsn > 0` from an earlier entry (the dishonest counting regression test); `committed_lsn` unmoved on timeout (1.1, 1.2, 2.1) -- `test_repl_majority.c` updated: walAck cases wire an engine and assert per-LSN semantics; configAck cases byte-identical - -### Regression -- Full controller testsuite + ASAN (`/xce-build controller`) green; no production walAck path reaches the legacy `acked_lsn > 0` counting - -## Performance & Scalability -- Commit latency unchanged from the ha-wal-replication baseline (majority RTT + standby fsync); `commit_on_ack` adds an O(MAX_STANDBYS log MAX_STANDBYS) scan per ACK under a short mutex — negligible -- Throughput under degradation bounded by the admission cap — deliberate CP rejection, `n_vcpus`-relative; sizing guidance is operator-doc material (umbrella research 3(e)), not spec constants - -## Migration Strategy -This spec is phase 1 of the umbrella's hard-ordered landing ("barrier before state removal"): commit engine + per-LSN evaluation behind the existing job-submit path — a behavioral fix only, no new write paths. The adapter shell (`repl_wait_majority_ack`) and the engine-less legacy fallback are explicitly transitional; ha-commit-barrier-rollout moves call sites onto the engine directly and retires the fallback. Rollback trigger: controller testsuite regression or job-submit E2E failure — reverting this spec restores the prior (dishonest but functional) barrier, leaving the system no less consistent than before. diff --git a/.kiro/specs/ha-commit-engine/requirements.md b/.kiro/specs/ha-commit-engine/requirements.md deleted file mode 100644 index 8e016f2d..00000000 --- a/.kiro/specs/ha-commit-engine/requirements.md +++ /dev/null @@ -1,107 +0,0 @@ -# Requirements Document - -> Sub-spec 1 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.1 (job-submit path), 2.3, 2.6, 4.1–4.4, 6.5, 7.1, 7.4, 7.5 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. - -## Project Description (Input) - -The majority-ACK check in the XCE controller is dishonest today: `repl_wait_majority_ack` counts standbys with `acked_lsn > 0` ("ACKed anything ever"), not `acked_lsn >= entry_lsn`; quorum is checked only at the dispatch gate (TOCTOU); catch-up standbys count toward write quorum while excluded from broadcast; and zero active standbys short-circuits to success. Even the one existing commit barrier (job submit) provides no real per-entry guarantee. Cluster operators need a commit engine (`commit.c` / `include/commit.h`) that is the single authority for commit-index advancement: per-LSN majority evaluation (`acked_lsn >= N`, self counted), contiguous and monotonic `committed_lsn`, WAIT/RETAIN completion modes, an `n_vcpus`-relative admission cap, and membership-change re-evaluation — wired behind the existing job-submit sync path only, with the quorum-accounting holes in `ha.c` fixed. - -## Introduction - -This spec delivers the commit engine and quorum-accounting integrity slice of the umbrella contract. After this spec, a job submission is confirmed to its requester only when its specific log entry has been acknowledged by a majority of cluster members (counting the primary itself); the commit index advances monotonically and contiguously, never on stale or substitute evidence; write admission counts only members actually able to receive entries; and a multi-member cluster never treats zero peer acknowledgments as a majority. Standalone (single-member) self-quorum behavior is preserved. The engine ships the full service interface (WAIT and RETAIN completion modes, own-term gate setter) so downstream sub-specs can consume it, but only the job-submit path is wired in this spec. - -## Boundary Context - -- **In scope**: The commit engine (per-LSN majority evaluation, commit-index advancement, WAIT/RETAIN completion, admission cap, membership-change re-evaluation); the quorum-accounting fix in the dispatch gate (catch-up exclusion); feeding standby acknowledgments into the engine; removal of the zero-active-standbys success short-circuit for multi-member clusters; preservation of single-member self-quorum; unit/integration tests for all of the above. -- **Out of scope**: Classification and rollout of the barrier to other WAL types (ha-commit-barrier-rollout); own-term gate *consumers* and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role — the `commit_set_writable_lsn` API ships here, but unset means no gate); demotion/egress-filter/catch-up changes (ha-convergence-recovery); config-change barrier error propagation (ha-commit-barrier-rollout); election and vote correctness (raft-library); replication wire format (ha-wal-replication). -- **Adjacent expectations**: Upstream, this spec relies on the ha-wal-replication machinery (per-standby `acked_lsn` tracking and WalAckReply processing — complete). Downstream, ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery consume the engine's service interface; changes to that interface require their revalidation. - -## Requirements - -### Requirement 1: Per-Entry Majority Commit for Job Submission - -**Objective:** As a cluster operator, I want a job submission confirmed only after its specific log entry is durable on a majority of cluster members, so that no acknowledged submission can be lost by a single node failure or failover. - -#### Acceptance Criteria - -1. When the primary receives a job submission, the controller shall confirm the submission to its requester only after the submission's log entry has been acknowledged by a majority of cluster members, counting itself. *(umbrella 2.1, job-submit path)* -2. When the controller evaluates whether the entry with LSN N is committed, the controller shall count a standby's acknowledgment toward N only if that standby's acknowledged LSN is greater than or equal to N. *(umbrella 4.2)* -3. The controller shall not substitute connectivity observed at operation-acceptance time for actual per-entry acknowledgments when determining that an entry is committed. *(umbrella 4.2)* - -### Requirement 2: Commit Index Integrity - -**Objective:** As a cluster operator, I want the commit index to advance only on real majority evidence, monotonically and without gaps, so that the committed prefix of the log is always a safe recovery point. - -#### Acceptance Criteria - -1. If an entry has not achieved majority acknowledgment, the controller shall not advance its commit index past that entry. *(umbrella 2.3)* -2. The controller shall advance its commit index monotonically; the commit index shall never decrease. *(umbrella 2.6)* -3. When acknowledgments arrive in any order, the controller shall advance the commit index contiguously — never beyond the lowest entry that lacks majority acknowledgment, and never beyond the highest locally appended entry. *(umbrella 2.3, 2.6)* - -### Requirement 3: Admission Quorum Accounting - -**Objective:** As a cluster operator, I want write admission to count only cluster members that can actually receive new log entries, so that a write is never admitted against a quorum that does not exist. - -#### Acceptance Criteria - -1. When the primary evaluates whether to accept a state-changing operation, the controller shall count toward quorum only cluster members currently able to receive new log entries. *(umbrella 4.1)* -2. The controller shall exclude standbys undergoing catch-up from the write-admission quorum count. *(umbrella 4.1)* -3. The controller shall exclude disconnected members from the write-admission quorum count. *(umbrella 4.1)* - -### Requirement 4: Acknowledgment-Time Quorum Integrity - -**Objective:** As a cluster operator, I want commit decisions re-verified at acknowledgment time against current membership, so that an operation accepted under a quorum that subsequently vanished is reported as failed rather than confirmed. - -#### Acceptance Criteria - -1. If the set of members able to acknowledge shrinks below a majority after an operation is accepted but before majority acknowledgment is obtained, the primary shall report the operation as failed rather than confirm it. *(umbrella 4.3)* -2. When cluster membership changes (standby connect, disconnect, catch-up entry, or catch-up completion), the controller shall re-evaluate pending commit waits against the updated membership. *(umbrella 4.3)* -3. While the cluster is configured with more than one member, the primary shall never treat zero peer acknowledgments as constituting a majority. *(umbrella 4.4)* -4. While the cluster is configured with more than one member and no standby acknowledgment covering the entry arrives within the commit timeout, the controller shall complete the commit wait with a failure result, not success. *(umbrella 4.4 — removes the zero-active-standbys success short-circuit)* - -### Requirement 5: Standby Lag Never Blocks Confirmation - -**Objective:** As a cluster operator, I want majority to be the only commit threshold, so that one slow or lagging standby cannot stall confirmed writes. - -#### Acceptance Criteria - -1. While one or more standbys lag behind the replication stream, the primary shall continue to confirm operations that achieve majority acknowledgment; full replication shall not be a confirmation precondition. *(umbrella 6.5)* - -### Requirement 6: Quorum Loss, Restoration, and Standalone Behavior - -**Objective:** As a cluster operator, I want a primary without majority to refuse state changes with a clear error and to resume automatically when majority returns, while a single-member cluster keeps confirming locally. - -#### Acceptance Criteria - -1. While the primary cannot reach a majority of cluster members, the primary shall reject cluster-state-changing operations with an error indicating that the cluster cannot currently commit. *(umbrella 7.1)* -2. When majority connectivity is restored, the primary shall resume accepting state-changing operations without operator intervention. *(umbrella 7.4)* -3. While the cluster is configured with a single member, the controller shall treat its own acknowledgment as majority and confirm operations locally, preserving existing standalone behavior. *(umbrella 7.5)* - -### Requirement 7: Commit-Wait Admission Control - -**Objective:** As a cluster operator, I want the number of concurrently blocked commit waits bounded relative to my configured worker-thread budget, so that slow commits degrade into fast, retriable rejections instead of exhausting the request-processing capacity I tuned with `--vcpus`. - -#### Acceptance Criteria - -1. The controller shall bound the number of concurrent blocking commit waits relative to the configured worker-thread count (`--vcpus`), never as an absolute constant. *(umbrella design §Commit Engine admission cap; locked decision, umbrella research item 3(d–e))* -2. If admitting a new blocking commit wait would exceed the admission bound, the controller shall reject the operation with a retriable not-ready error instead of blocking. *(umbrella design §Commit Engine admission cap)* -3. While the admission bound is reached, the controller shall continue to process incoming acknowledgments and advance the commit index for already-admitted waits. *(umbrella design §Commit Engine; ACK processing on the I/O thread is never blocked)* - -## Umbrella Traceability - -| This spec | Umbrella ID | Notes | -|-----------|-------------|-------| -| 1.1 | 2.1 | Job-submit path only; other operation types roll out in ha-commit-barrier-rollout | -| 1.2, 1.3 | 4.2 | Per-entry ACK verification (`acked_lsn >= N`) | -| 2.1 | 2.3 | No commit-index advance without majority | -| 2.2 | 2.6 | Monotonic commit index | -| 2.3 | 2.3, 2.6 | Contiguous advancement, bounded by appended log | -| 3.1, 3.2, 3.3 | 4.1 | Admission quorum counts only members able to receive; catch-up + disconnected excluded | -| 4.1, 4.2 | 4.3 | Post-acceptance quorum shrink → fail; membership-change re-evaluation | -| 4.3, 4.4 | 4.4 | Zero peer ACKs never a majority for n_peers > 1 | -| 5.1 | 6.5 | Majority is the only commit threshold | -| 6.1 | 7.1 | Reject state changes without quorum | -| 6.2 | 7.4 | Auto-resume on quorum restoration | -| 6.3 | 7.5 | Standalone self-quorum preserved | -| 7.1, 7.2, 7.3 | umbrella design §Commit Engine (admission cap) | Locked design decision from umbrella research item 3(d–e); `n_vcpus`-relative, never absolute | diff --git a/.kiro/specs/ha-commit-engine/research.md b/.kiro/specs/ha-commit-engine/research.md deleted file mode 100644 index 2eacdbde..00000000 --- a/.kiro/specs/ha-commit-engine/research.md +++ /dev/null @@ -1,98 +0,0 @@ -# Research Log: ha-commit-engine - -Date: 2026-06-11. Discovery status: **complete via umbrella** — this sub-spec inherits the ha-consistency-invariants discovery (`.kiro/specs/ha-consistency-invariants/research.md`: gap analysis, Option C selection, deep-dives §A–§C, locked decisions). No decisions are re-derived here. This log records only the slice-scoped code verification performed for this spec's design. - -## Inherited Locked Decisions (umbrella — do not re-litigate) -- Commit engine as single authority; WAIT/RETAIN completion modes (research synthesis) -- Admission cap relative to `n_vcpus` (never absolute); `COMMIT_WAIT_RESERVED` default 2 (research item 3(d–e)) -- Blocking condvar on vtask threads accepted; async continuation deferred; no liveness risk (ACKs on iomgr) (research item 3(b)) -- Catch-up standbys excluded from quorum accounting and commit counting until `repl_complete_catchup` (umbrella §Quorum Accounting + commit_on_membership_change contract) -- Commit state volatile (no persisted commitIndex); convergence/rejoin owned by ha-convergence-recovery (research §C) -- Own-term gate via WAL_NOOP at promotion — API (`commit_set_writable_lsn`) ships in this spec, consumers in ha-single-leader-role (research item 4) - -## Slice Verification (2026-06-11, this spec) - -| Fact | Evidence | Implication | -|------|----------|-------------| -| `standby_conn_t.acked_lsn` atomic exists, updated by `repl_on_ack` | src/include/repl.h:24; repl.c:176, store at repl.c:187 | matchIndex bookkeeping ready; engine consumes read-only | -| Dishonest counting: `acked_lsn > 0` | repl.c:419 | replaced by per-LSN engine evaluation | -| `n_active == 0 → success` short-circuit | repl.c:432-435 | removed for n_peers>1 via engine (engine knows n_peers; repl does not) | -| Quorum denominator is `n_active`, not `n_peers` | repl.c:437 | engine uses configured `n_peers` per umbrella commit rule | -| `ha_has_write_quorum` counts `standby_endpoints->count` (includes catch-up) | ha.c:1507-1526; gate call at ha.c:1537 | fix via new `repl_active_standby_count` (ep && !in_catch_up); `in_catch_up` at repl.h:26 | -| Sync barrier: `persist_wal_sync_append` | persist.c:1003-1049; committed_lsn store at 1035; `has_standbys → -ENODEV` degraded-success at ~1027-1031; truncation store at persist.c:1438 | adapter passes `written_lsn`; 1035 store moves into engine; -ENODEV case replaced by engine decision; 1438 store stays | -| ACKs processed on iomgr | `peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` (ha.c:2632) → `repl_on_ack` call at ha.c:2748 | `commit_on_ack` feed point; must never block | -| Membership mutation points | repl.h:50-51 (`repl_add_standby`/`repl_remove_standby`), repl.h:76/79 (`repl_start_catchup`/`repl_complete_catchup`); removal call sites ha.c:3176, 3866 | `commit_on_membership_change` call sites | -| vtask pool default 8, `--vcpus` override | cluster.c:1076-1083 | `n_vcpus` input to `commit_engine_init`; init point after vtaskmgr sizing | -| `replicator_t` embedded in `persist_t` (`p->repl`); cluster reaches via `c->persist.repl` | persist.c:1032; ha.c:538, 2748 | engine reaches standby registry via its `persist_t*`; `commit_engine_init(ce, p, ...)` signature suffices | -| `ha_state_t` has `n_peers` and `raft_c`, but no repl pointer | ha.h:180-185 | add explicit `replicator_t *repl` back-pointer (raft_c only set by election path — not reliable for the gate) | -| repl.h/persist.h live in src/include (lib-visible); controller-local headers in src/controller/include | find results; cluster.h, worker.h | `commit.h` goes to src/controller/include; repl.h uses forward-declared `struct commit_engine *` to avoid dependency inversion | -| `repl_wait_majority_ack` callers: persist.c:1032 (walAck), config_repl.c:916 (configAck) | grep | signature gains `entry_lsn`; configAck branch untouched (out of boundary) | -| Existing tests encode old semantics | testsuite/test_repl_majority.c (walAck cases at :85-:177); meson controller_unit_test_src list at meson.build:74-110 | walAck cases must be updated; new test file registered in meson | -| `XCE_ERR_NOT_READY` defined | ha.c:35 (`= E_XCE_NOT_READY`) | reuse for admission-cap rejection | - -## Design Synthesis Outcomes (this slice) -- **Adapter placement** (brief-mandated): `repl_wait_majority_ack` stays as the shell; gains `entry_lsn` param because the replicator has no LSN knowledge; engine-less fallback retained ONLY for unit-test contexts and explicitly retired by ha-commit-barrier-rollout. Production walAck always goes through the engine. -- **Engine reachability**: forward-declared `struct commit_engine *ce` on `replicator_t` (keeps src/include/repl.h free of controller headers); engine instance embedded in `cluster_t`; init after vtask sizing + peer count known. -- **Standalone behavior change (intentional)**: with the engine wired, `n_peers <= 1` sync appends now advance `committed_lsn` (previously the -ENODEV path left it unmoved). Raft-correct (self-quorum), satisfies spec req 6.3; flagged in design. -- **Membership-shrink semantics**: re-evaluation = wake + re-check; no forced early-fail (umbrella 4.3 requires fail-not-confirm, satisfied by timeout; reconnection within the deadline may still legitimately commit). -- Review gate: design verified against the umbrella contract — interface verbatim, no contradiction with locked decisions, all 20 slice requirement IDs traced. - ---- - -## Gap Re-Audit (2026-06-18) - -**Type:** Post-implementation gap audit (not a pre-impl gap analysis). Roadmap declares `ha-commit-engine` IMPLEMENTED 12/12 (2026-06-12, validate-impl GO, 467 tests green). `tasks.md` shows 6 `[ ]` checkboxes; all 6 are **section/group headers** ("1.", "2.", "3.", "4.", "5.", "6."), not leaf tasks — every leaf sub-task (1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1) is `[x]`. The "12/12" count is the leaf count. The unchecked group headers are an artifact of the checklist format, documented as such in §Implementation Notes — they are **not** deferred work and **not** core gaps. - -### Verdict - -**STATUS: CLOSED.** All 7 requirements (20 EARS acceptance criteria) are implemented and wired in production. Verified against current on-disk source (codegraph verbatim reads + grep, 2026-06-18). The engine has since been extended by `ha-commit-durability` (durable self-cap) — that extension strengthens, and does not regress, the ha-commit-engine contract. - -### Requirement-to-Asset Map (verified, file:line) - -| Req | Criterion | Asset (verified) | Status | -|-----|-----------|------------------|--------| -| 1.1 | confirm only after per-entry majority | `commit_submit(WAIT)` predicate `wait_done_locked` commit.c:184-189; sync path persist.c:1897-1900 → `repl_wait_majority_ack` walAck delegation repl.c:447 → `commit_submit` | MET | -| 1.2 | count ACK toward N only if acked_lsn >= N | `majority_candidate_locked` reads `s->acked_lsn`, sorts desc, candidate = `scratch[n_peers/2-1]` commit.c:107-134 | MET | -| 1.3 | connectivity never substitutes for ACKs | ACK-time eval only; watermarks read from registry at `commit_on_ack` commit.c:366-381; gate (ha_has_write_quorum) is fast-reject only, engine re-verifies | MET | -| 2.1 | no commit advance without majority | `commit_advance_locked` stores only `majority_candidate_locked` result commit.c:145-168 | MET | -| 2.2 | monotonic committed_lsn | `if (cand <= cur) return;` + `atomic_store(max)` commit.c:150-165; XCE_TESTING monotonic assert | MET | -| 2.3 | contiguous, bounded by appended log | candidate clamped to `cap` (durable, ≤ `highest_appended_lsn`) commit.c:136-137, 52-55; cumulative watermarks → contiguous | MET (strengthened: cap is now durable_lsn per ha-commit-durability) | -| 3.1/3.2/3.3 | admission quorum counts only members able to receive; catch-up + disconnected excluded | `repl_active_standby_count` (ep != NULL && !in_catch_up) repl.c:668-686; consumed by `ha_has_write_quorum` ha.c:2049-2058 | MET | -| 4.1 | post-acceptance shrink → fail not confirm | WAIT times out (`-ETIMEDOUT`) when majority never reached; no false-success path; `commit_on_membership_change` re-evals monotonically (shrink never regresses) commit.c:397-409 | MET | -| 4.2 | membership change re-evaluates pending waits | `commit_on_membership_change` wired at 4 mutators: repl.c:130 (add), :162 (remove), :598 (start_catchup), :621 (complete_catchup) | MET | -| 4.3/4.4 | zero peer ACKs never majority (n_peers>1); no zero-active short-circuit | engine derives from `n_peers` not n_active; `k = n_peers/2-1`, absent peers zero-filled commit.c:117-134; legacy `n_active==0→success` short-circuit removed (sync path persist.c:1897 engine branch; configAck has explicit "no n_active shortcut" repl.c:506-510) | MET | -| 5.1 | lag never blocks majority | majority is the only threshold; far-lagging standby sorts low, does not affect `scratch[n_peers/2-1]` commit.c:129-134 | MET | -| 6.1 | reject without quorum | `ha_has_write_quorum` false → `ha_dispatch_gate` -XCE_ERR_NOT_READY ha.c:2031-2059 | MET | -| 6.2 | auto-resume on quorum restore | gate is stateless per request (recomputes `repl_active_standby_count` each call); no reset needed | MET | -| 6.3 | standalone self-quorum | `n_peers <= 1` → `commit_advance_locked` immediate, returns cap commit.c:100-102, 330-332; sync path -ENODEV special case removed on engine branch | MET | -| 7.1/7.2 | n_vcpus-relative admission cap, NOT_READY beyond | `wait_cap = max(1, n_vcpus - COMMIT_WAIT_RESERVED)` commit.c:175-180; `inflight_waits >= cap → -XCE_ERR_NOT_READY` commit.c:313-320 | MET | -| 7.3 | ACK processing unaffected at cap | `commit_on_ack` takes only ce->lock, never the inflight gate; advances + broadcasts regardless of cap commit.c:366-381 | MET | - -### The 6 Open `[ ]` Tasks — Classified - -All 6 are group-header checkboxes whose every child is `[x]`. None is deferred work; none is a core gap. They are a **checklist-rendering artifact** (parent headers left unchecked while leaves complete). - -| Header | Children | Classification | Effort/Risk | -|--------|----------|----------------|-------------| -| 1. Commit engine core | 1.1 [x], 1.2 [x] | DONE (artifact) | none | -| 2. WAIT/self-quorum/zero-peer/cap/gate | 2.1 [x], 2.2 [x] | DONE (artifact) | none | -| 3. Membership re-evaluation | 3.1 [x], 3.2 [x] | DONE (artifact; jointly completed — RED unachievable, §Notes) | none | -| 4. Quorum-accounting fix | 4.1 [x], 4.2 [x] | DONE (artifact) | none | -| 5. Integration (wire engine behind job submit) | 5.1 [x], 5.2 [x], 5.3 [x] | DONE (artifact) | none | -| 6. Validation / dishonest-path audit | 6.1 [x] | DONE (artifact; 6.1 audit record in §Notes) | none | - -**Bottom line: 0 deferred follow-ups remain as core gaps. The "deferred follow-ups in §Implementation Notes" referenced by the roadmap are advisory/out-of-scope items, not open tasks** — see the genuine residuals below; all are explicitly out-of-spec-boundary and owned elsewhere. - -### Genuine Residuals (advisory, all out-of-boundary or already resolved downstream) - -These are the §Implementation Notes "deferred follow-ups". None blocks CLOSED status; each is owned by a later spec or is cosmetic. - -1. **LSN-0 first-entry legacy edge** (Notes 5.1, 6.1 audit). The first-ever WAL entry (LSN 0) routes to the legacy branch (persist.c:1903-1929) because `commit_submit` rejects `lsn==0`. — **RESOLVED downstream:** ha-single-leader-role's promotion no-op entry makes LSN 0 never a client write in practice; repl.c:438-440 now confirms LSN 0 immediately. Once-per-cluster-lifetime, no consistency loss. *Effort: 0 (closed). Risk: none.* -2. **`ce->n_peers` not re-derived on dynamic membership change** (Notes 6.1(b)). If `ha->n_peers` is reassigned post-init (e.g. ha.c RDHUP re-election), the engine's `n_peers` is stale. — **Out of boundary** (dynamic membership = future work); current cluster topology is fixed at init. *Option A: assert n_peers stability at re-election callback; Option B: re-derive `ce->n_peers` in a `commit_set_n_peers` setter called from the election path; Option C: leave as documented latent gap. Effort: A ~0.5d, B ~1d (+revalidate barrier-rollout/durability). Risk: LOW today (no dynamic membership in production); MEDIUM if dynamic reconfiguration is ever added without addressing it.* -3. **Pre-existing ASAN leaks in 6 test groups** (Notes 6.1 LEAK FOLLOW-UP: worker_join 33KB, rdhup_integration, failover, start_election_async, liveness, connection_reuse). — **Out of scope** (pre-existing, production-side allocations; suite runs with `detect_leaks=0`). *Option: dedicated leak-cleanup pass. Effort: ~1-2d. Risk: LOW (test-harness hygiene, not a correctness gap).* -4. **Stale comment persist.c:1151-1154 / pre-engine contract text** (Notes 6.1). One-line doc follow-up. *Effort: trivial. Risk: none.* -5. **Untracked `src/lib/testsuite/test_wal_wire.c`** referenced by committed meson.build (Notes 5.1). Repo hygiene. *Effort: trivial (git add). Risk: none.* - -### Drift Note (informational) - -The engine source has been extended by **ha-commit-durability** (implemented 2026-06-18) AFTER ha-commit-engine: `majority_candidate_locked` now caps the self-contribution at `persist_durable_lsn` (commit.c:93) rather than the appended tip, and a `commit_on_durable_advance` flusher wake was added (commit.c:218). This is additive — it tightens Req 2.3 (committed never beyond durable, not merely never beyond appended) and closes bug-661. The ha-commit-engine contract is unchanged and remains fully satisfied. Original research-table anchors (e.g. repl.c:432, persist.c:1035, ha.c:1507) have drifted; verified-current anchors are in the Requirement-to-Asset Map above. diff --git a/.kiro/specs/ha-commit-engine/spec.json b/.kiro/specs/ha-commit-engine/spec.json deleted file mode 100644 index 307b8f9a..00000000 --- a/.kiro/specs/ha-commit-engine/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "ha-commit-engine", - "created_at": "2026-06-11T09:07:03Z", - "updated_at": "2026-06-12T01:10:00-07:00", - "language": "en", - "phase": "implemented", - "umbrella": "ha-consistency-invariants", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "All 12 tasks implemented 2026-06-11/12 via /kiro-impl autonomous mode (each task independently reviewed). kiro-validate-impl: GO. Suite: 46 groups / 467 tests / 0 failures. Deferred follow-ups recorded in tasks.md Implementation Notes (6.1): LSN-0 interim, 6 pre-existing leak groups, persist.c:1151 stale comment, n_peers==0 legacy-path advisory. Next sub-spec: ha-commit-barrier-rollout." -} \ No newline at end of file diff --git a/.kiro/specs/ha-commit-engine/tasks.md b/.kiro/specs/ha-commit-engine/tasks.md deleted file mode 100644 index f5ec28de..00000000 --- a/.kiro/specs/ha-commit-engine/tasks.md +++ /dev/null @@ -1,101 +0,0 @@ -# Implementation Plan: ha-commit-engine - -> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. All new tests live in `testsuite/test_commit_engine.c` (brief-scoped single test file) plus targeted updates to `test_repl_majority.c` — this shared-file reality is why no `(P)` markers are applied: the test file and `repl.c` are touched by multiple groups, so the plan is sequential. -> Verify exact line numbers with grep before editing (research.md table has the verified anchors; files drift). - -- [x] 1. Commit engine core: per-LSN evaluation and commit-index integrity -- [x] 1.1 Write RED tests for per-LSN majority evaluation and commit-index integrity - - Create `testsuite/test_commit_engine.c` against a stub `persist_t` + `replicator_t` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (meson.build); create `src/controller/include/commit.h` with the umbrella-verbatim interface (types + prototypes only) so tests compile - - Tests: with n_peers=3, a standby ACK for LSN M does not commit entry N>M (kills `acked_lsn > 0` counting); ACK watermark >= N from one standby plus self commits N; cumulative/out-of-order watermarks advance `committed_lsn` to the majority median, never past the highest appended LSN, never backwards; one far-lagging standby does not block majority commit; RETAIN mode returns 0 immediately without blocking - - Observable completion: build succeeds with the new header and test file; all new test cases FAIL (red) against a stub `commit.c` or unlinked implementation - - _Requirements: 1.2, 2.1, 2.2, 2.3, 5.1_ - -- [x] 1.2 Implement the commit engine core to green - - Create `src/controller/commit.c` (add to meson controller sources): `commit_engine_init`/`commit_engine_fini` (validate args, one mutex + one condvar), `commit_on_ack` (bounded scan of active standbys' `acked_lsn` into the scratch array, sort descending, candidate = `arr[n_peers/2 - 1]` clamped to appended log, monotonic `max()` store into `persist->committed_lsn`, condvar broadcast — never blocks), `commit_submit(RETAIN)` immediate return - - Catch-up standbys (`in_catch_up`) excluded from the evaluation count (umbrella-locked conservative rule) - - XCE_TESTING asserts for monotonicity - - Observable completion: all task 1.1 tests PASS; engine is the only `committed_lsn` writer in `commit.c` paths - - _Requirements: 1.2, 2.1, 2.2, 2.3, 5.1_ - -- [x] 2. WAIT mode, self-quorum, zero-peer rule, admission cap, own-term gate storage -- [x] 2.1 Write RED tests for blocking-wait semantics and admission control - - Tests in `test_commit_engine.c`: WAIT blocks until majority ACK then returns 0; WAIT returns `-ETIMEDOUT` at deadline with `committed_lsn` unmoved past the entry; n_peers=1 → WAIT returns 0 immediately and `committed_lsn` advances to the entry; n_peers=3 with zero standby ACKs → `-ETIMEDOUT`, never success (short-circuit regression); with n_vcpus=8 and reserved=2, a 7th concurrent WAIT gets `-XCE_ERR_NOT_READY` while the 6 admitted waits proceed; cap floor `max(1, n_vcpus - reserved)` holds for tiny n_vcpus; while the cap is saturated, `commit_on_ack` still advances `committed_lsn` and completes admitted waiters; `writable_lsn` unset (0) → no gating; `commit_set_writable_lsn(G)` with G > entry → waiter completes only once `committed_lsn >= G` - - Observable completion: new cases FAIL (red) against the task-1 engine - - _Requirements: 1.1, 4.3, 4.4, 6.3, 7.1, 7.2, 7.3_ - -- [x] 2.2 Implement WAIT mode, admission counter, and writable_lsn gate to green - - `commit_submit(WAIT)`: admission check (`inflight_waits >= max(1, n_vcpus - COMMIT_WAIT_RESERVED)` → `-XCE_ERR_NOT_READY`, distinct log message), `pthread_cond_timedwait` loop with per-LSN predicate (`committed_lsn >= lsn && (writable_lsn == 0 || committed_lsn >= writable_lsn)`), `-ETIMEDOUT` on deadline; `n_peers <= 1` immediate self-quorum success advancing `committed_lsn`; `commit_set_writable_lsn` stores the gate (no caller in this spec) - - XCE_TESTING assert: inflight counter within cap and never negative - - Observable completion: all task 2.1 tests PASS; cap is expressed only relative to the `n_vcpus` init argument (no absolute constant other than the reserved headroom) - - _Requirements: 1.1, 4.3, 4.4, 6.3, 7.1, 7.2, 7.3_ - -- [x] 3. Membership-change re-evaluation -- [x] 3.1 Write RED tests for membership re-evaluation (completed jointly with 3.2 — RED unachievable, see Implementation Notes) - - Tests in `test_commit_engine.c`: a catch-up standby's ACK watermark does not count toward a pending entry; after clearing `in_catch_up` and calling `commit_on_membership_change`, the pending waiter completes if the standby's watermark covers it; on standby removal that makes majority unreachable, a pending WAIT never returns success and fails with `-ETIMEDOUT` at its deadline; `commit_on_membership_change` returns without blocking - - Observable completion: new cases FAIL (red) - - _Requirements: 4.1, 4.2_ - -- [x] 3.2 Implement commit_on_membership_change to green (verified no-op — task 1.2 already implemented the full contract; tests 17-20 are the green evidence) - - Re-run the commit evaluation under the engine mutex against current registry state and broadcast the condvar so all pending waiters re-check their predicates; no forced early-fail (reconnection within the deadline may still commit — design decision) - - Observable completion: all task 3.1 tests PASS - - _Requirements: 4.1, 4.2_ - -- [x] 4. Quorum-accounting fix: catch-up exclusion at the dispatch gate -- [x] 4.1 Write RED tests for write-admission quorum counting - - Tests: `repl_active_standby_count` counts only slots with `ep != NULL && !in_catch_up`; `ha_has_write_quorum` with n_peers=3 returns false when the only connected standby is in catch-up, true with one active standby, false with zero connected; after simulated reconnect (registry restored) the gate accepts again with no reset call (stateless re-evaluation); n_peers=1 always true - - Declare `repl_active_standby_count` in src/include/repl.h so tests compile - - Observable completion: new cases FAIL (red) — current `ha_has_write_quorum` counts `standby_endpoints->count` including catch-up (grep ha.c for the current anchor near ha.c:1507) - - _Requirements: 3.1, 3.2, 3.3, 6.1, 6.2_ - -- [x] 4.2 Implement the quorum-accounting fix to green (incl. reviewer-mandated wired gate test 29; reviewed in main context — reviewer subagent unavailable) - - Add `repl_active_standby_count(replicator_t *r)` to repl.c (under `r->lock`); add `replicator_t *repl` back-pointer to `ha_state_t` (ha.h); `ha_has_write_quorum` uses the helper when the back-pointer is wired, falls back to the existing `standby_endpoints` count when NULL (legacy test contexts); quorum condition itself unchanged (`active >= n_peers / 2`, `n_peers <= 1 → true`) - - Observable completion: all task 4.1 tests PASS; existing dispatch-gate tests still green - - _Requirements: 3.1, 3.2, 3.3, 6.1, 6.2_ - -- [x] 5. Integration: wire the engine behind the job-submit sync path -- [x] 5.1 Write RED integration tests for the ACK feed and the sync-append path - - Tests in `test_commit_engine.c`: `repl_on_ack` with a wired engine updates `acked_lsn` AND triggers commit evaluation (majority → `committed_lsn` advances, waiter wakes); `repl_start_catchup`/`repl_complete_catchup`/`repl_remove_standby`/`repl_add_standby` trigger re-evaluation; `persist_wal_sync_append` with wired engine returns 0 only on per-entry majority, returns `-ETIMEDOUT` when standbys' `acked_lsn` covers only an EARLIER entry (stale-ACK regression — connectivity/old ACKs never substitute), leaves `committed_lsn` unmoved on timeout, and for n_peers=1 returns 0 with `committed_lsn` advanced (replaces the old `-ENODEV` degraded path) - - Update `testsuite/test_repl_majority.c` walAck cases to wire an engine and assert per-LSN semantics; configAck cases byte-identical - - Observable completion: new/updated cases FAIL (red) against unwired production code - - _Requirements: 1.1, 1.2, 1.3, 2.1, 4.2, 4.4, 6.3_ - -- [x] 5.2 Implement the repl-side wiring: ACK feed, membership hooks, adapter - - repl.h: forward-declare `struct commit_engine`, add `ce` pointer to `replicator_t`, declare `repl_set_commit_engine()`; add `uint64_t entry_lsn` parameter to `repl_wait_majority_ack` (update the config_repl.c:916 caller to pass 0 — configAck branch behavior untouched) - - repl.c: `repl_on_ack` calls `commit_on_ack(r->ce, slot_idx, acked_lsn)` after the `acked_lsn` store when wired; `repl_add_standby`/`repl_remove_standby`/`repl_start_catchup`/`repl_complete_catchup` call `commit_on_membership_change` when wired; walAck branch of `repl_wait_majority_ack` delegates to `commit_submit(ce, entry_lsn, COMMIT_MODE_WAIT, timeout_ms)` when wired, with the legacy counting kept only for `ce == NULL` test contexts and marked deprecated (retired by ha-commit-barrier-rollout) - - Observable completion: ACK-feed and membership-hook tests from 5.1 PASS; configAck tests unaffected - - _Depends: 2.2, 3.2_ - - _Requirements: 1.1, 4.2_ - -- [x] 5.3 Implement engine instantiation and the sync-append path switch - - cluster.h/cluster.c: embed `commit_engine_t commit` in `cluster_t`; call `commit_engine_init(&c->commit, &c->persist, n_peers, n_vcpus)` + `repl_set_commit_engine` at the init point where both the vtask pool size (cluster.c:1082 anchor) and `ha->n_peers` are final; set the `ha_state_t.repl` back-pointer here; `commit_engine_fini` on teardown - - persist.c `persist_wal_sync_append`: pass `written_lsn` to the adapter; on the engine-wired path remove the `has_standbys → -ENODEV` special case and the `committed_lsn` store (persist.c:1035 anchor — the engine now owns advancement; truncation store at persist.c:1438 stays); return contract to callers: 0 / `-ETIMEDOUT` / `-XCE_ERR_NOT_READY` propagated through the existing job-submit error path - - Observable completion: all task 5.1 tests PASS, including stale-ACK regression and single-node `committed_lsn` advancement; job submit on a wired cluster confirms only on per-entry majority - - _Requirements: 1.1, 1.2, 1.3, 2.1, 4.4, 6.3_ - -- [x] 6. Validation: full regression and dishonest-path audit -- [x] 6.1 Run the full controller regression and audit the legacy paths - - Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix any regression introduced by the short-circuit removal or the `committed_lsn` ownership move (existing persist/repl/ha suites) - - Audit: grep confirms no production (engine-wired) walAck path reaches the `acked_lsn > 0` counting or the `n_active == 0 → success` branch; both survive only under the deprecated `ce == NULL` test fallback - - Observable completion: full testsuite green under ASAN; audit results recorded in the task completion notes - - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 5.1, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3_ - -## Implementation Notes - -- (1.1) THREE PRE-EXISTING suite failures, masked until now (bug-411): test_committed_lsn_does_not_advance_on_timeout (-110), test_repl_ack_after_promote_gate_allows_jobsub (-10013), test_rdhup_primary_dispatch_gate_opens (-10013). Masking cause: each test group runs its own UNITY_BEGIN/END so the process exit code reflects only the LAST group. They live in repl/persist sync-append and quorum-gate paths — tasks 1.2/2.x/5.x touch those areas and 6.1 requires full green, so they must be fixed (or proven fixed by the engine) along the way. -- (1.1) Full-suite output under default ASAN dies at exit via LeakSanitizer (pre-existing leaks in worker_join/rdhup_integration/start_election_async/liveness groups) without flushing stdout — use ASAN_OPTIONS=detect_leaks=0 for full-run logs or run groups standalone with -t. -- (1.1) repl_add_standby returns the SLOT INDEX, not 0 (bug-410 note for fixture writers). -- (1.1) commit.c is registered ONLY in controller_unit_test_src for now; production target linkage lands with task 5.3. -- (2.1) WAIT/cap/gate tests (8-16) read `g_ce.inflight_waits` under `g_ce.lock` — task 2.2 must keep that field accurate while waiters block. Timing floors have ~20ms slack vs sleeps (reviewer advisory: watch for flake on loaded boxes). Unity assertions are main-thread-only after joins — keep that discipline in future threaded tests. -- (3.1/3.2) Joint completion: task 1.2's commit_on_membership_change (commit.c:334-346) already implemented the full 3.2 contract, making 3.1's RED unachievable without dishonest tests. Tests 17-20 landed green as regression coverage; reviewer confirmed mutation-killing power (each test fails against the relevant mutant) and zero production changes. Lesson for later RED tasks: when a prior task's "minimal" allowance fully implements a later contract, complete the pair jointly under review rather than weakening tests. -- (4.1) REVIEWER ADVISORY for 4.2: the gate-side fix (ha_state_t.repl back-pointer + catch-up exclusion through ha_has_write_quorum) has NO RED test — 4.1's tests would all pass even if 4.2 only implemented the helper. 4.2 MUST add a wired gate-level case (n_peers=3, catch-up-only standby → NOT_READY via ha_dispatch_gate) as its own green evidence. Tests 26-28 pin the NULL-back-pointer legacy fallback and n_peers<=1 contracts 4.2 must preserve. Stub repl_active_standby_count returns -1 (repl.c:557) — replace, don't wrap. -- (5.1) **LSN-0 edge for 5.2/5.3 (reviewer-verified)**: commit_submit rejects lsn==0 with -EINVAL (commit.c:229), but the FIRST-EVER WAL entry is LSN 0 (current_lsn starts at 0; written_lsn = current_lsn - 1 at persist.c:1020). When 5.3 passes written_lsn, the first sync append on a fresh cluster would get -EINVAL. 5.3 must handle written_lsn==0 (options: treat entry 0 as valid in the engine, or shift LSN numbering, or special-case — decide per design). Tests 32-34 sidestep via a priming async append; production must not. -- (5.1) persist.c currently passes entry_lsn=0 to the adapter — if 5.2 lands walAck delegation before 5.3 passes written_lsn, the wired sync path returns -EINVAL. Land 5.2's delegation gated on ce!=NULL AND entry_lsn!=0 falling back to legacy, OR land 5.2+5.3 wiring in lockstep. -- (5.1) Baseline failure root causes identified: test_committed_lsn_does_not_advance_on_timeout (test_repl_unit.c:294) expects the OLD contract (rc==0 + sync_rc out-param) but persist_wal_sync_append now rejects-on-timeout — STALE TEST EXPECTATION, fix the test at 6.1, not the code. test_p2_integration.c:test_single_node_committed_lsn_unchanged pins the -ENODEV degraded path that 5.3 replaces — update it at 5.3/6.1. -- (5.1) Repo hygiene (pre-existing, out of scope): untracked src/lib/testsuite/test_wal_wire.c is referenced by committed src/lib/meson.build:176 — should be added to git eventually. -- (5.2) commit.c linked into production controller_src ONE TASK EARLY (meson.build:3) — necessary (repl.c references engine symbols) and reviewer-verified inert (zero production callers of commit_engine_init/repl_set_commit_engine until 5.3; repl_init memsets ce=NULL → all new paths dormant). -- (5.2) repl_broadcast's over-limit eviction (in_catch_up=1, repl.c:268-277) deliberately does NOT feed commit_on_membership_change — reviewer-proven no-op for waiter correctness (shrink can never raise the majority candidate; monotonic store). Design lists exactly four mutators; eviction recovery fires the hook at repl_complete_catchup. -- (5.2) r->ce is read unlocked under a set-once-before-traffic contract (repl.h:47-52). 5.3 must wire it at cluster init BEFORE replication traffic; never NULL it concurrent with traffic (fini ordering is 5.3's responsibility). -- (5.2) Remaining RED going into 5.3: test 33 (Expected -110 Was 0 — persist passes entry_lsn=0 → legacy fallback) and test 34 (Expected 0 Was -19 — -ENODEV special case). These plus test 32's honest-delegation green are 5.3's completion evidence. [RESOLVED at 5.3: both green] -- (5.3) For 6.1 audit: (a) baseline failure #1 (test_committed_lsn_does_not_advance_on_timeout) PHYSICALLY lives in test_p2_integration.c:277 — Unity misattributes it across groups (bug-411 artifact); its root cause is a stale expectation (old rc==0+sync_rc contract). (b) Legacy cluster_init() wires the engine with n_peers==0 (self-quorum) — confirmed test-only (zero production callers; production = myapp_run two-phase), but ce->n_peers is never re-derived if ha->n_peers is reassigned post-init (ha.c:3484 RDHUP re-election) — latent gap for future dynamic membership; consider an assert or re-derivation at the first-boot election callback. (c) Pre-existing teardown race (not 5.3's): persist_fini runs before vtaskmgr_fini in cluster_fini. -- (5.3) cluster_t lives in src/include/xce/cluster_def.h (not cluster.h); engine embedded there. Init point: cluster_init_phase2 after vtaskmgr guard, idempotence-guarded by c->commit.p==NULL (phase2 can run twice). n_vcpus = c->vtaskmgr.n_vcpus (resolved). -- (6.1) AUDIT RECORD (reviewer-confirmed): delegation guard repl.c:435-437 (is_wal_ack && ce && entry_lsn!=0 → commit_submit); legacy acked_lsn>0 counting (repl.c:469) and n_active==0→success (repl.c:482-484) strictly below it, DEPRECATED-marked, walAck-reachable only when ce==NULL || entry_lsn==0; persist-side mirror guard persist.c:1037 with DEPRECATED legacy branch (1044-1064); production always wires ce at cluster_init_phase2 (cluster.c:1134-1148) — sole production legacy exposure = documented LSN-0 first entry (once per cluster lifetime; eliminated by ha-single-leader-role no-op); configAck never delegates (is_wal_ack false; barrier-rollout hardens that track). Census: 46 groups / 467 tests / 0 failures (standalone default-ASAN per group; meson harness uses pre-existing detect_leaks=0 from commit 8468a83). Baseline fixes: stale timeout expectation → reject-on-timeout contract (engine-wired fixture); 2 gate fixtures honestly populated with standby_endpoints. LEAK FOLLOW-UP (out of spec scope): 6 groups leak production-side allocations under default ASAN — worker_join 33,648B/357; rdhup_integration 1,776B/24 (ha.c:3457 elect_ctx deliberate, ha.c:2293/2299 delaymgr objects); failover 232B/2; start_election_async 272B/1; liveness 216B/5; connection_reuse 48B/2. Stale comment persist.c:1151-1154 (pre-engine contract text) — one-line follow-up. diff --git a/.kiro/specs/ha-config-bootstrap/brief.md b/.kiro/specs/ha-config-bootstrap/brief.md deleted file mode 100644 index fb5c37c0..00000000 --- a/.kiro/specs/ha-config-bootstrap/brief.md +++ /dev/null @@ -1,1208 +0,0 @@ -# Brief: ha-election-raft-config - -## Problem - -The current LSF-style priority-based election has three critical safety gaps: - -1. **Config freshness not guaranteed**: A node with missing or stale config can be elected primary and serve incorrect cluster state to workers/schedulers. - -2. **False self-elections during concurrent startup**: When pc13v1 starts while pc13v (the priority-0 primary) is still initializing, pc13v's peer listener port is not open yet. pc13v1's election probe gets ECONNREFUSED, falsely concludes the primary is absent, and self-elects. Both nodes declare themselves primary. - -3. **No bootstrap path for genuine failover**: When a standby is promoted during real failover (original primary down), it has no mechanism to pull config from peer standbys if its local `conf/current` is missing. - -Reproduction from logs: -- pc13v starts at 12:30:03, pc13v1 starts at 12:30:22 (19s later) -- pc13v1 probes pc13v:9990 during election, times out within 3046ms -- pc13v is healthy but still in `load_primary_config()` or early phase2 — listener not open -- pc13v1 wins: "no higher-priority candidate responded" -- pc13v1 calls `load_primary_config()` → "Primary has no config at '/tmp/xce/pc13v1/conf/current' (rc=-2)" → fatal exit - -## Current State - -- **Election** (ha.c:243–306): LSF-style priority-based. Priority-0 always wins unconditionally; priority-N probes [0..N-1] and self-elects on timeout. No quorum. No config-version check. Logically correct for simple cases but unsafe for config freshness. -- **Listener timing** (myapp.c:21–96): `bind()+listen()` happens inside `cluster_init_phase2()` AFTER election and config load. Creates a 2–3 second window where the primary is unreachable to probes. -- **Config versioning** (config_repl.c): SHA-256 tagged, atomic writes, retention policy, staleness detection (`conf_stale` flag). Complete and correct. -- **Config fetch** (config_repl.c:891–965): standby→primary pull framework exists; `do_one_config_fetch()` returns `-ENOSYS` (stub, Task 11 on branch). -- **Bootstrap after failover**: not implemented. Design §14.4 deferred quorum consensus. -- **Simultaneous primaries**: `ha_check_priority_stepdown()` (ha.c:322) has a term=0 exact check that fails to resolve races when both nodes increment term before heartbeat exchange. - -## Availability Trade-off (CP chosen) - -This design chooses **Consistency over Availability** (CP in CAP terms): - -- A cluster of N nodes requires a **majority (≥(N+1)/2)** for both **election** and **write commits**. -- **Election**: A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored. -- **Writes (WAL replication)**: Primary waits for majority standby ACKs before confirming a write (job submit, queue change, state update). In-flight writes that cannot reach majority are rejected. This adds **~1 LAN RTT per write** (~0.1–5ms depending on network). -- **Zero data loss on failover**: A promoted standby is guaranteed to have all committed WAL entries because majority replication ensures durability before write ACK. -- **Operational implication**: Plan for N≥3 nodes. A 2-node cluster has no meaningful majority; it is equivalent to single-node with a cold standby. For HPC, zero data loss (no lost job submissions, no stale queue state) outweighs the write-latency cost. - -## Desired Outcome - -1. **Raft-style quorum election (CP)**: A candidate must obtain ACKs from a strict majority of peers (≥(N+1)/2) before becoming primary. A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored. - -2. **Config freshness encoded in votes**: Each vote reply includes the peer's config tag and WAL LSN. The candidate with the freshest config (highest tag, then highest LSN) among quorum members wins tiebreaks. - -3. **No-config nodes bootstrap after election (deferred fetch)**: A node with `config_tag=0` - never initiates candidacy. It probes peers to determine whether config exists anywhere, - but **defers the actual fetch until the primary is known** — the election proves who - has the freshest state, so bootstrap fetches from the proven winner rather than - speculatively guessing. Three sub-cases: - - `tag=0`, peers have config → set `defer_config_fetch`, participate as voter only, fetch from winner after election - - `tag=0`, all peers also `tag=0` → first-boot cluster, proceed to election as-is - - `tag=0`, no peer reachable → retry: probe loop with exponential backoff (500ms→10s); - log WARN every attempt; block indefinitely until a peer responds or admin kills - -4. **Listener before election**: Peer listener socket opens before election so concurrent-startup probes reach an open port. False self-elections during overlapping startup are eliminated. - -5. **Config freshness preserved through all paths**: A node always has a valid config - before entering phase 2 — either from local disk (normal case) or fetched from the - proven election winner after the election resolves (no-config case, deferred fetch in - Change C). A quorum winner therefore always has `config_tag > 0` when peers do, and - serves correct cluster state immediately after promotion with no additional bootstrap step. - -6. **Most up-to-date primary**: Among all reachable nodes, only the one with the - highest config_tag (then highest LSN) can become primary. A candidate that wins - the vote count but sees a fresher responding peer yields back to standby — - the fresher peer runs its own election and wins. Non-responding (partitioned or - crashed) peers never block election. - -7. **No simultaneous primaries**: Only one node can obtain a quorum at any given term. Term-based fencing prevents split-brain on partition heal. `voted_for` tracking (persisted to `ha-term.bin`) enforces one vote per term per node across restarts. - -8. **Majority-commit WAL replication (CP writes)**: Primary waits for majority (≥(N+1)/2) standby ACKs before marking a WAL entry committed and confirming the write to the caller. In-flight writes that cannot reach majority within a timeout are rejected. Zero data loss on failover: promoted standby has all committed entries. - -9. **Majority-commit config push (CP config)**: Config upgrades (`OP_upgradeConfig`) use the - same `repl_broadcast()` + `repl_wait_majority_ack()` infrastructure as WAL replication — - the primary waits for ≥(N+1)/2 standby ACKs before confirming a config change to the - operator. Operator changes cannot be silently lost on failover. - -10. **Hard SSL/cert gate**: Controller refuses to boot if TLS cert/key files are missing or unreadable. Operator-friendly error message with file path. - -11. **Observable via logs**: Election results, vote tallies, quorum counts, config tag comparisons, bootstrap operations, WAL commit quorum status, config push quorum status, and cert validation logged at INFO or WARN severity. - -## Approach - -**Replace LSF priority election with Raft-style quorum voting AND add majority-commit WAL replication for full CP durability. Config upgrades use the same majority-commit infrastructure for uniform CP semantics.** - -Eight focused changes: - -### Change A — Move listener, iomgr, and vtaskmgr before election - -Extract `bind()+listen()`, `iomgr_init()`, and `vtaskmgr_init()` from -`cluster_init_phase2()` so the full connection framework is running before election. -`vtaskmgr_init()` uses the new `--vcpus N` CLI parameter (Change D item 1) instead of -`cluster_config->vcpus`, removing the last config dependency on this startup block. - -New startup order in `myapp.c`: -``` -validate_peer_count() ← Change D: odd-N check before any init -validate_certs() ← Change D: cert pre-flight -cluster_init_phase1() ← ha_init() reads ha-term.bin: restores term + voted_for - ← conf_read_tag() reads config_version_tag from disk (0 if absent) - ← Node now has durable identity before network is up -create_ssl_context() -iomgr_init() + iomgr_set_ssl_ctx() -cluster_open_listener(port) ← bind + listen; register fd with iomgr -vtaskmgr_init(vcpus) -iomgr_start() -vtaskmgr_start() -atomic_store(ha.role, CTRL_INIT) ← node now answers OP_requestVote + OP_probeConfigTag -bootstrap_config_if_needed() ← Change C: if config_tag==0, probe peers; defers fetch until primary known -ha_wait_for_heartbeat_or_timeout() ← Raft: sem_timedwait(election_timeout + rand_jitter) - ← heartbeat received → CTRL_STANDBY, skip election - ← timeout fires → CTRL_ELECTING → am_i_primary_quorum() -[primary or standby path] -load_primary_config() / ha_connect_to_primary() -cluster_init_phase2_rest() ← kvmgr, xstring, jobmgr, delaymgr, apiserver, etc. -cluster_run() -``` - -With iomgr and vtaskmgr running before election, incoming `OP_requestVote` messages -are handled by the normal dispatch switch — no special pre-framework path needed. -Outgoing vote requests use the existing `connect_ssl_raw()` + `xce_encode()/xce_decode()` -pattern (same as config fetch), keeping per-peer threads lightweight. - -**Full 4-state controller FSM (`ha_role_t`)**: - -The controller lifecycle is a single FSM stored in `c->ha.role` (atomic): -``` -CTRL_INIT ──── timeout + jitter ──→ CTRL_ELECTING ──── quorum won ──→ CTRL_PRIMARY - └─── heartbeat from │ │ - existing primary ─→ └─── quorum lost ─→ CTRL_STANDBY ────┘ - │ - EPOLLRDHUP / heartbeat timeout - │ - CTRL_ELECTING -``` - -**Dispatch gate** (top of vtaskmgr switch, before any `case`): -```c -switch (atomic_load(&c->ha.role)) { -case CTRL_INIT: -case CTRL_ELECTING: - // Only vote + config-probe pass; all others get NOT_READY and retry (Change G) - if (msg->op != OP_requestVote && msg->op != OP_probeConfigTag) { - xce_send_error(task, XCE_ERR_NOT_READY, "controller not ready"); - return; - } - break; -case CTRL_STANDBY: - // Primary-only ops rejected with XCE_ERR_STANDBY + redirect_host/port (Change G) - break; -case CTRL_PRIMARY: - break; // full dispatch -} -``` - -**State transition triggers**: -- `CTRL_INIT → CTRL_ELECTING`: `ha_wait_for_heartbeat_or_timeout()` times out — no - heartbeat received within `election_timeout + rand_jitter`. Different jitter per node - (xorshift64 from `jitter_seed`) means nodes fire at staggered times, so the first - candidate typically wins unopposed. This is Raft's primary split-vote prevention. -- `CTRL_INIT → CTRL_STANDBY`: heartbeat or `OP_ctrlHello` received from an existing - primary during the wait window — node skips election entirely. -- `CTRL_ELECTING → CTRL_PRIMARY`: `am_i_primary_quorum()` returns 1. -- `CTRL_ELECTING → CTRL_STANDBY`: `am_i_primary_quorum()` returns 0. -- `CTRL_STANDBY → CTRL_ELECTING` (fast): `EPOLLRDHUP` on `primary_ctrl_fd`. -- `CTRL_STANDBY → CTRL_ELECTING` (fallback): heartbeat timeout vtask fires — primary - is alive on TCP but not sending heartbeats. Vtask is reset on each heartbeat received; - fires after `election_timeout + rand_jitter` of silence. - -**`ha_wait_for_heartbeat_or_timeout()`** in `myapp.c`: -```c -struct timespec dl = now_plus_ms(c->ha.config.election_timeout_ms + - ha_rand_jitter(&c->ha)); -int hb = (sem_timedwait(&c->ha.heartbeat_seen, &dl) == 0); -if (hb) { - atomic_store(&c->ha.role, CTRL_STANDBY); -} else { - atomic_store(&c->ha.role, CTRL_ELECTING); - rc = am_i_primary_quorum(c); -} -``` -`ha.heartbeat_seen` is posted by the heartbeat dispatch handler when any heartbeat -or `OP_ctrlHello` is received while `ha.role == CTRL_INIT`. - -### Change B — Raft-style quorum voting in `am_i_primary()` - -Replace the current timeout-based probe with a full request-vote protocol: - -**New wire message: OP_requestVote** (Cap'n Proto) -```c -struct RequestVoteRequest { - term @0 :UInt64; // candidate's term - candidate_id @1 :Text; // candidate's name (from --peers) - config_tag @2 :UInt64; // candidate's config SHA-256 tag (0 if none) - last_log_lsn @3 :UInt64; // candidate's WAL applied_lsn - // NOTE: priority removed — Raft config_tag+LSN freshness fully determines winner; - // jitter breaks split votes. Priority plays no role in quorum election. -} - -struct RequestVoteReply { - term @0 :UInt64; // peer's current term - vote_granted @1 :Bool; // true = ACK - peer_config_tag @2 :UInt64; // peer's config tag (for config comparison) - peer_last_log_lsn @3 :UInt64; // peer's WAL LSN (for log comparison) -} -``` - -**Election uses iomgr + vtaskmgr (FSM model)**: - -Since iomgr and vtaskmgr are running before election (Change A), vote requests are -sent via iomgr's async connect path and replies arrive through the normal vtaskmgr -dispatch switch. No election-specific threads are created. - -**New `ha_vote_entry_t`** struct (ha.h) — per-peer election slice; only meaningful as -`ctx->votes[i]` inside `ha_elect_ctx_t`: -```c -typedef struct { - xce_candidate_t *peer; - uint64_t my_term; // term sent in RequestVote - uint64_t my_tag; // config_tag sent in RequestVote - uint64_t my_lsn; // WAL LSN sent in RequestVote - int vote_granted; // 1 = peer granted vote - int replied; // 1 = peer responded (any reply, including denial) - uint64_t peer_tag; // peer's config_version_tag from reply - uint64_t peer_lsn; // peer's applied_lsn from reply - // NOTE: no pthread_t — FSM uses iomgr async sends + vtaskmgr dispatch; - // ha_elect_ctx_t.done semaphore blocks am_i_primary_quorum() -} ha_vote_entry_t; -``` - -**New `ha_elect_ctx_t`** struct (ha.h) — holds all election state; single allocation via -C99 flexible array member (one `calloc`, one `free`): -```c -typedef struct { - int n_peers; - int votes_granted; // includes self-vote (starts at 1) - int n_replied; // peers that responded (any reply) - uint64_t max_resp_tag; // freshest config_tag seen across all replies - uint64_t max_resp_lsn; // freshest lsn seen across all replies - int result; // set before sem_post: 1=primary, 0=standby, <0=error - sem_t done; // startup thread blocks here; dispatch posts when done - ha_vote_entry_t votes[]; // C99 flexible array — n_peers-1 entries -} ha_elect_ctx_t; -// Allocate: calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0])) -// Free: free(ctx) — single allocation, no separate votes free needed -``` - -**`am_i_primary_quorum()`** (ha.c) — initiates election, blocks until done: -```c -int am_i_primary_quorum(cluster_t *c) { - int n_peers = c->pcfg.n_candidates; - if (n_peers == 1) return 1; - - uint64_t my_term = atomic_fetch_add(&c->ha.term, 1) + 1; - ha_persist_term(c, my_term, c->myconf->name); - - ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0])); - ctx->n_peers = n_peers; - ctx->votes_granted = 1; // self-vote - ctx->max_resp_tag = c->myconf->config_version_tag; - ctx->max_resp_lsn = atomic_load(&c->persist.applied_lsn); - sem_init(&ctx->done, 0, 0); - atomic_store(&c->ha.elect_ctx, ctx); // dispatch reads this - - // Send RequestVote to all peers via iomgr (async non-blocking connects) - for (int i = 0, j = 0; i < n_peers; i++) { - if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue; - ctx->votes[j].peer = &c->pcfg.candidates[i]; - ctx->votes[j].my_term = my_term; - ctx->votes[j].my_tag = c->myconf->config_version_tag; - ctx->votes[j].my_lsn = atomic_load(&c->persist.applied_lsn); - iomgr_async_vote_request(c, &ctx->votes[j]); // non-blocking - j++; - } - - // Arm election timeout vtask — fires ha_elect_timeout() if peers don't reply - vtaskmgr_schedule(c, c->ha.config.election_timeout_ms, ha_elect_timeout, ctx); - - sem_wait(&ctx->done); // startup blocks here; dispatch or timeout signals it - - int result = ctx->result; - ha_elect_ctx_free(ctx); - atomic_store(&c->ha.elect_ctx, NULL); - return result; -} -``` - -**`handle_request_vote_reply()`** — called from vtaskmgr dispatch switch on -`OP_requestVoteReply`; accumulates votes and signals completion: -```c -void handle_request_vote_reply(cluster_t *c, RequestVoteReply *reply, int peer_idx) { - ha_elect_ctx_t *ctx = atomic_load(&c->ha.elect_ctx); - if (!ctx) return; // no election in progress - - ha_vote_entry_t *v = &ctx->votes[peer_idx]; - v->replied = 1; - v->peer_tag = reply->peer_config_tag; - v->peer_lsn = reply->peer_last_log_lsn; - if (reply->vote_granted) { v->vote_granted = 1; ctx->votes_granted++; } - if (v->peer_tag > ctx->max_resp_tag || - (v->peer_tag == ctx->max_resp_tag && v->peer_lsn > ctx->max_resp_lsn)) { - ctx->max_resp_tag = v->peer_tag; ctx->max_resp_lsn = v->peer_lsn; - } - ctx->n_replied++; - - int quorum = (ctx->n_peers + 1) / 2; - // Signal when quorum achieved or all peers replied (or timed out) - if (ctx->votes_granted >= quorum || ctx->n_replied == ctx->n_peers - 1) - ha_elect_evaluate_and_signal(c, ctx); -} -``` - -**`ha_elect_timeout()`** vtask — fires when `election_timeout_ms` expires: -counts unreplied peers as non-respondents, calls `ha_elect_evaluate_and_signal()`. - -**`ha_elect_evaluate_and_signal()`** — applies gates, sets `ctx->result`, signals: -- Defense-in-depth assertion (config_tag=0 BUG check) -- Hard gate B (yield if responding peer has fresher data → `result=0`) -- Quorum met → `result=1` -- Quorum lost → schedule jitter vtask, retry once with incremented term; - if second round also fails → `result=0`; finally `sem_post(&ctx->done)` - -**Vote handler on peers** (new function in ha.c): -```c -void handle_request_vote(cluster_t *c, RequestVoteRequest *req, RequestVoteReply *reply) { - pthread_mutex_lock(&c->ha.lock); - uint64_t my_term = atomic_load(&c->ha.term); - - if (req->term < my_term) { // stale candidate — reject - reply->term = my_term; - reply->vote_granted = 0; - goto done; - } - if (req->term > my_term) { // update term from candidate - atomic_store(&c->ha.term, req->term); - my_term = req->term; - // Clear voted_for on new term transition - c->ha.voted_for_term = 0; - c->ha.voted_for_name[0] = '\0'; - } - - // Catch-up gate (Clarification 6): behind nodes must not influence elections - if (c->ha.in_catch_up) { - reply->vote_granted = 0; - goto fill_reply; // still send peer_config_tag/lsn so candidate has full picture - } - - uint64_t my_tag = c->myconf->config_version_tag; - uint64_t my_lsn = atomic_load(&c->persist.applied_lsn); - int log_ok = (req->config_tag > my_tag) || - (req->config_tag == my_tag && req->last_log_lsn >= my_lsn); - - // One vote per term: grant only if not yet voted, or voted for same candidate - int already_voted = (c->ha.voted_for_term == my_term && - strcmp(c->ha.voted_for_name, req->candidate_id) != 0); - reply->vote_granted = (log_ok && !already_voted) ? 1 : 0; - - if (reply->vote_granted) { - c->ha.voted_for_term = my_term; - strncpy(c->ha.voted_for_name, req->candidate_id, XCE_MAXNAME_LEN - 1); - ha_persist_term(c, my_term, req->candidate_id); // durable before reply - } -done: - reply->term = my_term; - reply->peer_config_tag = my_tag; - reply->peer_last_log_lsn = my_lsn; - pthread_mutex_unlock(&c->ha.lock); -} -``` - -**Election safety extensions**: - -1. **Term + `voted_for` persistence** (`ha-term.bin` in `{workdir}/`): - New `ha_persist_term(cluster_t *c, uint64_t term, const char *voted_for)` writes - a 4-field struct atomically (write + rename) before any vote is granted or term - incremented. On startup, `ha_init()` reads `ha-term.bin` and restores `ha.term`, - `ha.voted_for_term`, `ha.voted_for_name`. Without this, a restarting node resets - to term=0 and can double-vote in a term it already participated in. - -2. **New fields in `ha_state_t`** (`ha.h`): - ```c - uint64_t voted_for_term; // term of last granted vote - char voted_for_name[XCE_MAXNAME_LEN]; // candidate name of last granted vote - int primary_ctrl_fd; // persistent control channel fd (Change F; -1 = none) - bool defer_config_fetch; // Change C: set when tag=0 and peers have config; - // cleared after post-election fetch completes - ``` - -3. **Retire `ha_check_priority_stepdown()`** (ha.c:322–368): - This function's guard `if (my_term > 0 || sender_term > 0) return` makes it - permanently unreachable after the first quorum election (term is always ≥ 1 - thereafter). Change B explicitly removes the call site from the heartbeat handler - and deletes the function body. Term-based leadership is fully handled by the - `reply.term > my_term` check in `handle_request_vote()` and the existing - `ha_do_demote()` path. - -**Key properties**: -- **Most up-to-date wins**: A candidate that won quorum still yields if any *responding* - peer has higher config_tag or (same tag, higher LSN). Non-responding peers never - block election — only reachable peers with fresher state can cause a yield. -- **Config freshness hard gate**: Yielding due to fresher peer returns 0 (standby, not - fatal). Only `config_tag == 0` with peers having `config_tag > 0` is fatal (-ENODATA), - because that means the node has never been configured at all. -- **Split vote prevention (primary)**: Each node waits `election_timeout + rand_jitter` - before candidacy (Change A `ha_wait_for_heartbeat_or_timeout()`). Different jitter - values stagger candidacy attempts so the first node typically wins unopposed — the - same xorshift64 `jitter_seed` used elsewhere. This is Raft's fundamental split-vote - prevention, not just recovery. -- **Split vote recovery (safety net)**: If a split vote still occurs inside - `ha_elect_evaluate_and_signal()`, a second attempt is scheduled with `rand_jitter` - delay and incremented term. If that also fails, node returns 0 (standby). This - safety net handles the rare case where multiple nodes have identical jitter values. -- **No priority tiebreak**: Priority is not used in Raft quorum election. When - config_tag and LSN are identical across all peers, the jitter-based pre-election - timeout is the tiebreaker. The `ha.my_priority` field is preserved in `ha_state_t` - for backward compatibility but plays no role in election outcome. - -### Change C — Pre-election config bootstrap (deferred fetch) - -A node with `config_tag=0` is a config-seeker, not a candidate. Before calling -`am_i_primary_quorum()`, `bootstrap_config_if_needed()` determines whether config -exists among peers, but **defers the actual fetch until the primary is known** — -avoiding a speculative download from a peer that may not win the election. - -**Rationale**: Fetching from the highest-tag *reachable* peer before election can -waste I/O: a fresher peer may start late, win the election, and push its config -anyway. Since the election already proves who has the freshest state, bootstrap -should defer to that proof rather than guessing. - -New function `bootstrap_config_if_needed()` in config_repl.c (called from myapp.c): -```c -void bootstrap_config_if_needed(cluster_t *c) { - if (c->myconf->config_version_tag != 0) return; // already have config - - int retry_ms = 500; - for (;;) { - uint64_t best_tag = 0; - bool any_reachable = false; - - for (int i = 0; i < c->pcfg.n_candidates; i++) { - if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue; - uint64_t peer_tag = 0; - int rc = xce_probe_config_tag(c->pcfg.candidates[i].host, - c->pcfg.candidates[i].port, &peer_tag); - if (rc != 0) continue; // peer unreachable — skip - any_reachable = true; - if (peer_tag > best_tag) best_tag = peer_tag; - } - - if (!any_reachable) { - WARN("No config and no peer reachable — retrying in %dms (kill to abort)", - retry_ms); - usleep((useconds_t)retry_ms * 1000); - retry_ms = MIN(retry_ms * 2, 10000); - continue; - } - - if (best_tag == 0) { - INFO("All peers also have no config — first-boot cluster, proceeding"); - return; // all-zero cluster: first boot, proceed to election as-is - } - - // Config exists among peers — defer fetch until primary is known. - // The election will prove who has the freshest state. - INFO("Config exists among peers (best tag=%016" PRIx64 ") — " - "deferring fetch until primary is known", best_tag); - c->ha.defer_config_fetch = true; - return; - } -} -``` - -The function never returns an error. If no peer is reachable, it retries with -exponential backoff. If all peers are `tag=0`, it returns for first-boot election. -If config exists among peers, it sets `defer_config_fetch` and returns — the node -will fetch from the proven winner after the election resolves. - -`xce_probe_config_tag()` is a lightweight new RPC (single round-trip) that returns -the peer's current `config_version_tag` without transferring the full config payload. -Reuses the existing SSL connection setup pattern (`connect_ssl_raw()`). - -**Cap'n Proto schema for `OP_probeConfigTag`**: -```c -struct ProbeConfigTagRequest { - name @0 :Text; // requester's node name (for logging on peer side) -} -struct ProbeConfigTagReply { - config_tag @0 :UInt64; // peer's current config_version_tag (0 if no config) -} -``` -Answered by all nodes in any FSM state (including `CTRL_INIT`) — the dispatch gate -explicitly allows `OP_probeConfigTag` through before the controller is ready for -normal requests (Change A). Note: the actual config transfer (`OP_replConfig`) does -NOT need to be role-agnostic — with deferred fetch, `do_one_config_fetch()` is called -post-election when the serving peer is already in `CTRL_STANDBY` or `CTRL_PRIMARY` -and full dispatch is active. Only the probe op needs the `CTRL_INIT` exemption. - -Integrated into `myapp.c` (before election): -```c -bootstrap_config_if_needed(&c); // returns immediately: tag>0, first-boot, or defer - -// The tag=0 node participates in the election as a voter but never wins -// against nodes with tag>0 (log_ok gate ensures this). It votes, discovers -// the winner via voted_for, and fetches from the proven primary. -rc = am_i_primary_quorum(&c); -if (rc < 0) return rc; - -if (c.ha.defer_config_fetch && rc != 1) { - // Fetch from the proven winner (known via voted_for or heartbeat source). - // Guard rc != 1: a winning node has tag>0 so defer_config_fetch is never - // set for it, but the explicit guard prevents double load_primary_config() - // if logic shifts during implementation. - do_one_config_fetch(c->ha.primary_host, c->ha.primary_port, 0); - load_primary_config(&c); - c->ha.defer_config_fetch = false; -} - -if (rc == 1) { - rc = load_primary_config(&c); // load into memory (already on disk) - if (rc != 0) return rc; - atomic_store(&c->ha.role, CTRL_PRIMARY); -} -``` - -### Change D — Startup pre-flight checks + ASAN leak fix - -`main.c`, `src/controller/cluster.c`, `lib/netio.c`, `ha.c` - -1. **`--vcpus N` CLI parameter** (`main.c`, `src/controller/cluster.c`): - Add a `--vcpus N` flag parsed in `main.c` alongside the existing `--peers` and `--name` - flags. Default: `8` when not specified. Values `< 1` are rejected at startup with - `"ERROR: --vcpus must be ≥ 1"` and exit `-EINVAL`. Remove the - `vcpus` field from `cluster_config_t` (currently read from the config file). Pass the - parsed value directly to `vtaskmgr_init()` — this removes `vtaskmgr_init()`'s - dependency on `cluster_config`, enabling it to run before election (Change A). - - **Removed from cluster config schema**: the `vcpus:` key in the config file is no - longer read. Existing configs that include it receive a logged deprecation warning - (`"WARN: 'vcpus' in config file is deprecated — use --vcpus N instead"`) but the - controller does not fail. - - **Rationale**: CPU count is a per-host capacity decision, not a cluster-wide policy. - Different controller nodes in the same cluster may have different hardware. Keeping - vcpus in the shared config forces all nodes to use the same value and couples an - infrastructure tunable to the cluster state config file. - -2. **Peer count validation in `main.c`** (before `cluster_init_phase1()`): - The `--peers` list must contain exactly 1 node (single-node) or an **odd number ≥ 3** - (HA mode). Even numbers cannot form a strict majority and are rejected at startup: - ```c - int n = pcfg.n_candidates; - if (n != 1 && n % 2 == 0) { - fprintf(stderr, - "ERROR: --peers must list an odd number of controllers (1, 3, 5, ...).\n" - " %d controllers cannot form a majority. Use 1 for single-node\n" - " or add/remove one controller to reach an odd count.\n", n); - return -EINVAL; - } - ``` - This removes the need for the `n_peers == 2` special case inside - `am_i_primary_quorum()` — the quorum formula `(N+1)/2` is correct for all - validated (odd) values of N. - -3. **Cert pre-flight in `main.c`** (before `myapp_run()`): explicit `access()` check on - `{workdir}/certs/xce.key` and `{workdir}/certs/xce.crt`. On failure: print - operator-friendly `"ERROR: TLS cert not found: <path>"` and exit `-ENOENT`. - Controller never starts without valid certs — same hard-prerequisite model as config. - -4. **Fix ASAN leak in `ha_fini()` (ha.c:451)**: add `SSL_CTX_free(ha->ssl_ctx)` call. - This closes the ~5320-byte direct+indirect leak shown in the pc13v1 ASAN output. - -### Change E — Majority-commit WAL replication - -`src/controller/wal.c` (or `persist.c`), `src/controller/ha.c`, `src/controller/cluster.c` - -**Principle**: Primary does not ACK a write to its caller until ≥(N+1)/2 standbys have -confirmed they wrote the WAL entry to disk. This guarantees any promoted standby has -all committed entries — zero data loss on failover. - -**New wire message: OP_walAck** (Cap'n Proto): -```c -struct WalAckRequest { - term @0 :UInt64; // primary's current term (for stale-detection) - lsn @1 :UInt64; // WAL entry LSN being ACKed -} -struct WalAckReply { - term @0 :UInt64; // standby's term (primary steps down if reply.term > mine) - lsn @1 :UInt64; // highest LSN standby has durably written - success @2 :Bool; -} -``` - -**Primary-side commit protocol** (`persist.c` / `wal.c`): -1. Write WAL entry to local disk (existing) -2. Fan-out `OP_walAck` to all connected standbys in parallel (async send) -3. Wait for ≥(N+1)/2 − 1 standby ACKs (own write counts as 1) within - `commit_timeout_ms` (configurable, default 5000ms) -4. On quorum ACK: mark entry committed, return success to caller -5. On timeout without quorum: return `-ETIMEDOUT` to caller; entry is - written locally but not committed — next primary will re-evaluate on - WAL replay - -**Standby-side WAL write** (existing replication path extended): -- On receiving replicated WAL entry: write to disk, fsync, send `OP_walAck` reply -- No change to standby application logic - -**Backpressure**: Primary maintains a per-standby `pending_ack_count`. If a standby -falls more than `max_pending_acks` (default 64) entries behind, it is removed from the -active replication set and reconnects via existing catch-up path. - -**Single-node mode**: quorum = 1, commit is immediate (own write = majority). No change. - -**Clarification 1 — Durability boundary (which entry types block on majority-commit)**: - -Not all WAL entry types have equal durability requirements. The rule: - -| Entry type | Commit model | Rationale | -|---|---|---| -| `WAL_JOB_SUBMIT` | Majority-commit (blocking) | Job loss on failover is visible to user | -| `WAL_JOB_START` / `WAL_JOB_FINISH` | Majority-commit (blocking) | State transitions must survive failover | -| `WAL_QUEUE_CHANGE` | Majority-commit (blocking) | Queue config changes must be durable | -| `WAL_HOST_STATE` | Best-effort async | Reconstructed from worker reconnect on failover | -| `WAL_CHECKPOINT_REF` | Best-effort async | Checkpoint is self-contained; ref is advisory | - -The existing `persist_wal_sync_append()` path (used for `WAL_JOB_SUBMIT` today with -ANY-standby wait) is extended to use majority quorum. The existing `persist_wal_async()` -path remains fire-and-forget for best-effort types. No new caller API — entry type -determines which path is taken inside `persist.c`. - -**Clarification 2 — `committed_lsn` data model**: - -Two new fields added to `persist_t` (persist.h): -```c -_Atomic uint64_t committed_lsn; // highest LSN majority-ACKed; safe to serve/trim -uint64_t applied_lsn; // existing: highest LSN written locally (may exceed committed) -``` - -`committed_lsn` is advanced by `repl_wait_majority_ack()` after quorum is reached. -On promotion, the new primary replays WAL from its checkpoint up to `committed_lsn` -only — entries above `committed_lsn` are discarded (never committed to majority, so -no client received success for them). WAL trim guard updated: trim threshold = `committed_lsn`, -not `applied_lsn`. - -Catch-up standbys (in_catch_up == 1) are excluded from quorum counting — they are not -yet in the active replication set. This means a cluster with 2-of-3 nodes catching up -cannot make progress until at least one catch-up completes. This is the correct CP -behavior: prefer unavailability over serving uncommitted state. - -**Clarification 3 — `repl_broadcast()` prerequisite**: - -`repl_broadcast()` (repl.c:183–196) is currently a stub. Majority-commit requires a -real broadcast implementation. This is co-developed with Change E — they are the same -work unit. Change E scope explicitly includes: -- Implement `repl_broadcast()` with per-standby TCP send via existing iomgr pipeline -- Implement `repl_wait_majority_ack()` replacing the existing `repl_wait_ack()` (any→majority) -- Add per-standby send error callback: failed send → decrement active standby count, - trigger catch-up reconnect (existing path) - -If `repl_broadcast()` stub lands separately before Change E, Change E still works — -it replaces the stub. No ordering dependency beyond Task 11 (config fetch). - -**New function `persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)`** (`persist.c`): - -Removes all WAL entries with LSN > `cutoff_lsn` from local storage. Called in two -contexts (Clarifications 4 and 5). Implementation: scan WAL segment files from newest -to oldest; truncate the file containing `cutoff_lsn` at the byte offset immediately -after that entry; delete any segment files that start above `cutoff_lsn`. Updates -`p->applied_lsn = cutoff_lsn` atomically after truncation. Must be idempotent (calling -twice with same cutoff is safe). New function — does not currently exist in `persist.c`. - -**Clarification 4 — `committed_lsn` must be piggybacked on heartbeats**: - -A promoted standby needs to know where to truncate its WAL. Without this it only has -`applied_lsn`, which may include uncommitted orphan entries written by the old primary -but never majority-ACKed. - -- Extend `CtrlHeartbeat` Cap'n Proto struct with `committed_lsn @N :UInt64` -- Standbys store it in `ha.last_known_committed_lsn` (`_Atomic uint64_t`, init 0) on - each heartbeat receipt -- On promotion, immediately after `am_i_primary_quorum()` returns 1: - ```c - persist_truncate_wal_above(&c->persist, c->ha.last_known_committed_lsn); - c->persist.committed_lsn = c->ha.last_known_committed_lsn; - ``` -- First-boot failover (no heartbeat ever received): `last_known_committed_lsn = 0`, - so WAL replay starts from checkpoint only — correct and safe - -**Clarification 5 — Log divergence on standby re-join**: - -The existing catch-up path assumes the standby is simply *behind* the primary (a clean -subset). After a primary crash, a standby may have orphaned WAL entries above the new -primary's `committed_lsn` — entries the old primary replicated but never majority-ACKed. -Replaying those entries would corrupt the standby's state. - -Catch-up handshake extension (Change E scope): -1. Primary sends its `committed_lsn` in the catch-up initiation reply -2. Standby calls `persist_truncate_wal_above(&c->persist, primary_committed_lsn)` before - requesting WAL replay — discards any orphaned entries above that boundary -3. Standby sets `applied_lsn = primary_committed_lsn` as the new replay base -4. Normal WAL catch-up (existing mechanism) proceeds from there - -**Clarification 6 — Catch-up standbys excluded from voting (not just quorum count)**: - -Clarification 2 excludes `in_catch_up==1` standbys from the majority quorum count. -They must also be prevented from *granting* votes — a behind-node's stale LSN could -tip an election to a less-fresh candidate. - -In `handle_request_vote()`, before the `log_ok` freshness check: -```c -if (c->ha.in_catch_up) { - reply->vote_granted = 0; - // still populate peer_config_tag/peer_last_log_lsn — candidate needs the full picture - goto fill_reply; -} -``` - -**Clarification 7 — `applied_lsn` in vote request must be post-fsync**: - -`RequestVoteRequest.last_log_lsn` is sourced from `atomic_load(&c->persist.applied_lsn)`. -This is correct only if `applied_lsn` is advanced *after* `fsync()` completes in -`persist.c`. A node that claims an LSN it has not durably written could win an election -and then fail to replay entries it advertised. Verify in `persist.c` before implementing -Change B — if `applied_lsn` is advanced pre-fsync anywhere, fix that first. - -### Change F — Persistent standby↔primary control channel - -**Files**: `src/controller/ha.c`, `src/controller/myapp.c`, `src/controller/cluster.c`, -`src/controller/repl.c` - -**Background**: Standbys currently detect primary failure through `ha_check_heartbeat_nocluster()` -(ha.c:380–405), which polls a `last_heartbeat` timestamp with a 10s floor. Failover detection -latency is up to 10 seconds. Additionally, each config fetch and WAL replication message opens a -new TCP+SSL connection (`connect_ssl_raw()`, config_repl.c:685–775), adding per-message handshake -latency for WAL streaming (Change E). - -**Changes**: - -1. **Standby discovers and connects to primary** — new `ha_find_and_connect_to_primary(cluster_t *c)` - (called in `myapp.c` standby path after election). Opens a TCP+SSL connection to the primary - peer and registers the fd with epoll `EPOLLIN | EPOLLRDHUP`. The connection is stored in - `ha_state_t` as `ha->primary_ctrl_fd`. On connect, the standby sends an extended `OP_ctrlHello` - (item 5) carrying identity and state. The primary registers the standby, initiates catch-up, - and begins streaming any missing WAL before the standby enters the active replication set. - - **Primary discovery algorithm** (how the standby finds the winner's address): - - Two startup paths lead here with different amounts of prior knowledge: - - - **Heartbeat path** (`CTRL_INIT → CTRL_STANDBY`): `ha_wait_for_heartbeat_or_timeout()` received - a heartbeat/`OP_ctrlHello` from an existing primary. The sender's host/port is in the packet. - Store immediately as `ha.primary_host/port`. No further discovery needed — connect directly. - - - **Election path** (`CTRL_ELECTING → CTRL_STANDBY`): lost `am_i_primary_quorum()`. Discovery - uses a three-tier strategy exploiting Raft's structural property that the majority voted for - the winner: - - ``` - Tier 1 (fast path — O(1) for ≥(N+1)/2 nodes): - If ha.voted_for_name is set: - try ha_connect_to_primary(voted_for peer) - on success: store ha.primary_host/port, done - on XCE_ERR_NOT_READY: peer still electing — backoff, retry tier 1 - - Tier 2 (redirect — O(1) once any majority member connected): - Probe remaining peers (random order) with OP_ctrlHello - A CTRL_STANDBY peer that already knows the primary responds with - XCE_ERR_STANDBY + redirect_host/port (same field as Change G) - Follow redirect immediately — connect to that address - - Tier 3 (blind scan fallback — for split-vote minorities): - If redirect not received: try remaining peers sequentially - A CTRL_PRIMARY peer accepts OP_ctrlHello normally → connected - A CTRL_INIT/CTRL_ELECTING peer returns XCE_ERR_NOT_READY → skip+retry - Repeat with 100ms → 5s exponential backoff until primary found - ``` - - **Why this converges quickly**: In a 3-node cluster, at least 2 nodes voted for the winner. - Those 2 connect via Tier 1 immediately. The 3rd probes either of them (Tier 2), gets a - redirect, and connects. Total: 2 direct + 1 redirect = all standbys connected with at most - one extra probe per minority node. In a 5-node cluster, at least 3 connect via Tier 1; the - remaining 2 get redirected. - - **`ha.primary_host/port` update**: set on successful connect; cleared to empty on - `EPOLLRDHUP` (primary disconnect). During the election window it is empty; standbys receiving - `OP_ctrlHello` while `ha.primary_host` is empty reply with `XCE_ERR_STANDBY` and no redirect - (graceful — probing node falls through to Tier 3). - - **No new message types**: the redirect uses the existing `XCE_ERR_STANDBY` error reply fields - already extended in Change G (`redirect_host`, `redirect_port`). The standby dispatch gate - (already returning `XCE_ERR_STANDBY` for primary-only ops) populates these fields from - `ha.primary_host/port` when they are known. - -2. **Standby join synchronization — full state sync on connect** (cluster.c, ha.c, repl.c): - - A standby with stale WAL cannot safely promote — its in-memory state (jobs, queues, - resources) would be out of date, and promoted writes would replay entries clients already - saw as committed. Every standby must be fully current before it enters the quorum. - - **Extended `OP_ctrlHello` Cap'n Proto schema**: - ```c - struct CtrlHelloRequest { - name @0 :Text; // peer name (from --peers) - key @1 :Data; // cluster auth key: SHA-256(cluster_name + shared_secret) - // shared_secret read from myconf->cluster_key (new config field) - // primary rejects connections where HMAC does not match its own - applied_lsn @2 :UInt64; // standby's current applied_lsn (0 if fresh node) - config_tag @3 :UInt64; // standby's current config_version_tag (0 if none) - } - struct CtrlHelloReply { - committed_lsn @0 :UInt64; // primary's current committed_lsn - needs_snapshot @1 :Bool; // true if standby's WAL predates oldest primary segment - config_tag @2 :UInt64; // primary's current config_version_tag - } - ``` - - **New wire messages for sync**: - ```c - struct SnapshotTransfer { - snapshot_lsn @0 :UInt64; // LSN of this snapshot (= checkpoint_lsn after fresh checkpoint) - payload @1 :Data; // serialized snapshot blob (from persist_checkpoint()) - } - struct WalCatchupEntry { - lsn @0 :UInt64; // WAL entry LSN - type @1 :UInt8; // WAL entry type (WAL_JOB_SUBMIT etc.) - payload @2 :Data; // WAL entry payload - } - ``` - `OP_snapshotTransfer` and `OP_walCatchupEntry` flow on the persistent control channel fd — - no new connections. - - **Primary-side `handle_ctrlhello()` sequence** (new handler in cluster.c): - ``` - 1. Extract name, applied_lsn, config_tag from CtrlHelloRequest - 2. repl_add_standby(&c->repl, ep, host, port) - 3. needs_snap = repl_needs_snapshot(&c->repl, applied_lsn, oldest_wal_lsn()) - 4. Reply: CtrlHelloReply{c->persist.committed_lsn, needs_snap, myconf->config_version_tag} - 5. repl_start_catchup(&c->repl, ep, applied_lsn) // sets standby's in_catch_up=1 - 6. If needs_snapshot: - persist_checkpoint(&c->persist) // flush in-memory state to disk - send OP_snapshotTransfer(snapshot_lsn, blob) - applied_lsn = snapshot_lsn // advance standby's effective base - 7. Streaming loop — runs until standby is fully current: - target_lsn = atomic_load(&c->persist.committed_lsn) // read current live value - stream WAL entries [applied_lsn+1 → target_lsn] via OP_walCatchupEntry - wait for OP_walAck with acked_lsn >= target_lsn - if atomic_load(&c->persist.committed_lsn) > target_lsn: - applied_lsn = target_lsn; continue loop // new entries committed, catch up again - // standby is now at current committed_lsn — exit loop - 8. repl_complete_catchup(&c->repl, ep) // in_catch_up=0; standby enters live replication - 9. If standby config_tag != primary config_tag: - push_config_to_standbys(c, blob, blen, tag) - ``` - - **`repl_broadcast()` skips `in_catch_up==1` standbys**: while the streaming loop (step 7) - is in progress, `repl_broadcast()` does not send to this standby. The loop itself handles - delivery of all committed entries — including entries committed after the handshake — by - re-reading `committed_lsn` on each iteration. The loop exits only when the standby's - `acked_lsn` equals the *current* `committed_lsn`, at which point `repl_complete_catchup()` - fires and live broadcast takes over. There is no window where entries can be missed. - - **Standby-side sequence** (after receiving CtrlHelloReply in ha.c / myapp.c): - ``` - 1. Store primary_committed_lsn = reply.committed_lsn - 2. persist_truncate_wal_above(&c->persist, primary_committed_lsn) // Clarification 5 - 3. ha.last_known_committed_lsn = primary_committed_lsn // Clarification 4 - 4. ha.in_catch_up = 1 // deny votes during sync - 5. If reply.needs_snapshot: - wait for OP_snapshotTransfer - write received blob as local snapshot file - load_snapshot(&c->persist, path, snapshot_lsn) - applied_lsn = snapshot.snapshot_lsn - 6. Receive OP_walCatchupEntry stream; call apply_wal_entry() for each - 7. Send OP_walAck after each entry (or configurable batch): - OP_walAck{acked_lsn = last applied LSN} - 8. When primary sends no more OP_walCatchupEntry (catch-up loop ended on primary side): - ha.in_catch_up = 0 // standby now in live replication; votes granted normally - ``` - - `apply_wal_entry()` is the same WAL replay function used during startup recovery — it - applies a single WAL entry to the in-memory state (jobs, queues, resources). Must exist - or be extracted from the startup replay path before this change lands. - - **`repl_needs_snapshot()` already exists** in repl.c — compares `standby_applied_lsn` - against `oldest_wal_lsn` (the trim floor). If standby is behind the trim floor, - `persist_checkpoint()` is called on the primary to produce a fresh snapshot, which is - then streamed in full. This is the only path where `persist_checkpoint()` is called - on-demand rather than on schedule. - - **`push_config_to_standbys()` is declared** in config_repl.h; config sync at join - handles the case where a standby was offline during a config upgrade. - -2. **Heartbeat runs on control channel** — primary sends periodic heartbeats on the controller↔controller - channel (in addition to existing worker heartbeats). Standby updates `last_heartbeat` on receipt, - but the epoll path is the fast lane. - -3. **Replace timestamp polling with dual-trigger failover detection** — - `ha_check_heartbeat_nocluster()` (ha.c:380–405) is retired. Two complementary paths: - - **Fast path (`EPOLLRDHUP`)**: TCP disconnect detected immediately; `primary_ctrl_fd` - closed and nulled before `am_i_primary_quorum()` is called. Sub-second latency. - - **Fallback path (heartbeat timeout vtask)**: armed after each heartbeat is received - (`vtaskmgr_schedule(election_timeout + rand_jitter, ha_heartbeat_timeout_cb)`); reset - on every subsequent heartbeat. Fires when primary is silent but TCP is alive — covers - network partitions that keep TCP open, primary overload, or missed heartbeat bursts. - On fire: close+null `primary_ctrl_fd`, set `CTRL_ELECTING`, call `am_i_primary_quorum()`. - -4. **WAL streaming over persistent channel (Change E complement)** — `repl_broadcast()` - (repl.c:183–196) sends WAL entries on the persistent control channels established by each - standby rather than opening per-entry connections. No new connection setup per WAL write. - -**Role flip on promotion**: When a standby wins election and becomes primary, it must -close and discard its `primary_ctrl_fd` (the old connection to the now-dead primary) -*before* calling `am_i_primary_quorum()` triggers the role transition. After promotion, -the new primary accepts `OP_ctrlHello` connections from the remaining standbys — the -role of `primary_ctrl_fd` inverts from "my connection out" to "their connections in", -tracked in a per-standby fd table rather than a single field. - -**Reconnect behavior**: On primary_ctrl_fd disconnect, standby closes and nulls -`primary_ctrl_fd` first, then calls `am_i_primary_quorum()`. -If it becomes primary, the old primary_ctrl_fd is discarded. If it remains standby, it reconnects -with exponential backoff (100ms, 200ms, 400ms, max 5s) until the new primary is reachable. - -**Single-node mode**: `ha_connect_to_primary()` is a no-op when `n_peers == 1`. - -**Key properties**: -- Sub-second failover detection (`EPOLLRDHUP` vs 10s timeout) -- No per-entry TCP+SSL handshake overhead for WAL replication -- Single liveness signal: connection alive = peer alive -- Backward-compatible: standby reconnect on disconnect triggers election (same semantics as today, - faster latency) - -### Change G — libxce client-side retry for transient controller errors - -**Files**: `src/lib/` (libxce — client request/connection layer) - -**Background**: With `CTRL_ELECTING` gating requests during the election window -(Change A), and standbys already returning `XCE_ERR_STANDBY` to misdirected -requests, every daemon (worker, scheduler) needs to handle these transient errors -gracefully. Without built-in retry in libxce, each daemon would need its own -ad-hoc loop — fragile and inconsistent. - -**Changes**: - -1. **Retry policy in libxce's client request path** (the function wrapping - `xce_encode()` + send + recv + `xce_decode()`): - - On `XCE_ERR_NOT_READY`: back off and retry against same endpoint - - On `XCE_ERR_STANDBY`: **redirect** — the standby includes the current primary's - `host:port` in the error payload when known (`ha.primary_host/port`). libxce - reconnects directly to the redirected endpoint instead of blind backoff. - When primary is unknown (e.g. during startup), falls back to normal retry. - - On TCP-level `ECONNREFUSED` or `ETIMEDOUT`: reconnect and retry - (handles the window before `cluster_open_listener()` runs on a starting peer) - - Non-retriable errors (`XCE_ERR_AUTH`, `XCE_ERR_INVALID`, `XCE_ERR_NOENT`, - `XCE_ERR_PERM`, etc.): surface immediately to caller - - **`XCE_ERR_STANDBY` wire extension**: add optional `redirect_host` (Text) and - `redirect_port` (UInt16) fields to the error reply Cap'n Proto struct. Standbys - populate these when `ha.primary_host` is non-empty; clients that don't understand - the new fields simply retry with backoff (backward-compatible). - -2. **Retry configuration** (sensible defaults, overridable per client): - ```c - typedef struct { - int max_retries; // default 20 - int initial_retry_ms; // default 50 - int max_retry_ms; // default 5000 - double backoff_multiplier; // default 2.0 (exponential) - } xce_retry_config_t; - ``` - Defaults cover a full election round (up to 20 × 50ms–5000ms) without - requiring callers to configure anything. Daemons that need tighter or looser - bounds pass a custom `xce_retry_config_t`. - -3. **Transparent to callers**: existing daemon code makes one `xce_request()` call; - libxce handles retry internally. No changes needed in worker, scheduler, or - apiserver code. - -**Key properties**: -- Decouples daemon startup order — a worker that starts before the primary is ready - simply waits for the retry window to expire rather than crashing or looping manually -- `XCE_ERR_STANDBY` retry enables transparent primary failover: after failover, the - next request to the old primary (now standby) gets `XCE_ERR_STANDBY`, libxce retries - against the same host/port and the new primary responds -- Backoff prevents thundering-herd when 50 workers restart simultaneously after a failover - -### Change H — Majority-commit config push - -`src/controller/config_repl.c`, `src/controller/repl.c` - -**Principle**: Config upgrades (`OP_upgradeConfig`) use the same majority-commit -infrastructure as WAL replication (Change E). The primary does not ACK a config -change to the operator until ≥(N+1)/2 standbys have confirmed durable write. - -**Why this matters**: Without majority-commit, a primary could accept a config change, -ACK to the operator, then crash. The new primary serves the old config, and the -operator's change is silently lost. For strict CP, config writes must have the same -durability guarantee as WAL writes — even though they are orders of magnitude rarer. - -**Changes**: - -1. **Reuse Change E infrastructure** — `repl_broadcast()` (sends to all connected - standbys on persistent channels) and `repl_wait_majority_ack()` (blocks until - ≥(N+1)/2 ACKs or `commit_timeout_ms`) already built for WAL. Both functions - accept a message-type parameter so they can serve WAL (`OP_walAck`) and config - (`OP_upgradeConfigAck`) ACKs without duplication. Zero new infrastructure. - -2. **`push_config_to_standbys()` extended** (config_repl.c): - ```c - int push_config_to_standbys(cluster_t *c, const char *blob, size_t len, uint64_t tag) { - // 1. Write locally (existing atomic write+rename — unchanged) - int rc = config_repl_write_local(c, blob, len, tag); - if (rc != 0) return rc; - - // 2. Fan-out OP_upgradeConfig via repl_broadcast (same channel as WAL) - rc = repl_broadcast(c, OP_upgradeConfig, blob, len); - if (rc != 0) return rc; - - // 3. Wait for ≥(N+1)/2 standby ACKs (own write = 1 vote) - rc = repl_wait_majority_ack(c, c->ha.config.commit_timeout_ms); - - // 4. On quorum: advance committed_config_tag and return success - if (rc == 0) { - atomic_store(&c->myconf->committed_config_tag, tag); - } - return rc; // -ETIMEDOUT on failure — operator retries - } - ``` - -3. **Standby-side config ACK** — on receiving `OP_upgradeConfig`, standby: - - Writes config blob to disk (existing atomic write path) - - Loads config into memory - - Sends `OP_upgradeConfigAck` with `acked_tag = written_tag` - - ACK semantics identical to `OP_walAck` — durable write complete - -4. **`OP_upgradeConfigAck` Cap'n Proto**: - ```c - struct UpgradeConfigAck { - term @0 :UInt64; // standby's term (primary steps down if ack.term > mine) - acked_tag @1 :UInt64; // config tag standby has durably written - success @2 :Bool; - } - ``` - -5. **`committed_config_tag` field** added to cluster config state (analogous to - `committed_lsn` for WAL). Advanced only on majority ACK. Stored alongside - `config_version_tag` in durable state. On promotion, a new primary serves only - configs up to `committed_config_tag`. **Piggybacked on heartbeats** (same as - `committed_lsn` — Clarification 4): standbys store the latest value in - `ha.last_known_committed_config_tag` so a promoted standby knows the safe - boundary without an extra round-trip. First-boot promotion (no heartbeat - received): value is 0, standby serves whatever config it loaded from disk — - correct for first-boot where no upgrade has been committed. - -**Key properties**: -- Same `commit_timeout_ms` as WAL (configurable, default 5000ms) -- Single-node mode: quorum=1, commit is immediate (own write = majority) -- Config upgrades are operator-initiated and rare — the blocking ACK wait adds - negligible overhead to cluster operations -- Catch-up standbys (`in_catch_up==1`) are excluded from quorum count, - consistent with WAL majority rules -- If commit times out, operator gets `-ETIMEDOUT` and can retry — libxce retry - (Change G) handles this transparently for the API server - -## Scope - -- **In**: - - Change A: Extract `cluster_open_listener()`, `iomgr_init()`, and `vtaskmgr_init()` - from phase2; call all three before election in myapp.c so the full connection - framework is running during election (incoming `OP_requestVote` handled by normal - dispatch switch); add `CTRL_ELECTING` role + pre-dispatch gate returning - `XCE_ERR_NOT_READY` for non-vote requests during election window - - Change B: Full Raft-style quorum voting in `am_i_primary_quorum()` - - New `OP_requestVote` wire protocol (Cap'n Proto request + reply) - - `handle_request_vote()` handler on all peers with `voted_for` enforcement - - Config tag + LSN freshness comparison in vote grant and candidate yield gate - - Hard gate A (fatal): refuse primary if config_tag=0 and any peer has tag>0 - - Hard gate B (yield): winner yields if any *responding* peer has fresher data - - Split vote retry with jitter (one retry after timeout) - - Term + `voted_for` persistence to `ha-term.bin` (survives restart) - - New `ha_state_t` fields: `voted_for_term`, `voted_for_name`, `primary_ctrl_fd` - - Retire `ha_check_priority_stepdown()` (permanently unreachable after first election) - - Change C: `bootstrap_config_if_needed()` pre-election bootstrap — probes peers - for config tags, defers fetch until primary is known (avoids speculative download - from a peer that may not win the election); tag=0 node participates as voter only; - new `xce_probe_config_tag()` lightweight RPC; role-agnostic - `handle_config_request()` so peers can serve config to a bootstrapping node - - Change D: `--vcpus N` CLI parameter + remove `vcpus` from cluster config (deprecation - warning for existing configs); odd-N peer-count validator at startup (`-EINVAL` on - even count); cert pre-flight check in `main.c`; `SSL_CTX_free` ASAN fix in `ha.c` - - Change E: Majority-commit WAL replication — `OP_walAck` protocol, primary - commit gating, standby ACK path, backpressure, `commit_timeout_ms` config - - Change F: Persistent standby↔primary control channel — `ha_connect_to_primary()`, - `EPOLLRDHUP`-based failover detection, heartbeat on control channel, WAL streaming - over persistent connection - - Change G: libxce client-side retry — exponential backoff on `XCE_ERR_NOT_READY`, - `XCE_ERR_STANDBY`, `ECONNREFUSED`, `ETIMEDOUT`; transparent to daemon callers; - configurable via `xce_retry_config_t` with safe defaults - - Change H: Majority-commit config push — `OP_upgradeConfig` uses the same - `repl_broadcast()` + `repl_wait_majority_ack()` infrastructure as WAL replication - (Change E). Primary waits for ≥(N+1)/2 standby ACKs before confirming a config - upgrade to the operator. Reuses `commit_timeout_ms`. Zero new infrastructure cost. - - Unit and integration tests for all eight changes -- **Out**: - - Raft snapshot protocol — existing dual-zarray COW is orthogonal and unchanged - - Joint consensus for cluster membership changes — candidate list is static - - do_one_config_fetch() stub wiring (Task 11) — prerequisite; must land first or co-developed - - mTLS between controllers — trusted-LAN model with server-side TLS is sufficient - -## Boundary Candidates - -1. **Election protocol** — full replacement of `am_i_primary()` with quorum voting -2. **Config wire protocol** — new OP_requestVote, OP_probeConfigTag, OP_upgradeConfigAck; existing OP_replConfig used for post-election deferred fetch -3. **Primary startup path** — bootstrap integration in myapp.c -4. **Listener lifecycle** — extracted from phase2 to phase1 - -## Out of Boundary - -- Standby fetch path (`fetch_config_from_primary()`) — untouched -- Config format, versioning, or storage layout — untouched -- Raft snapshot protocol — existing dual-zarray COW is orthogonal -- Joint consensus for cluster membership changes — candidate list is static -- mTLS — trusted-LAN model with server-side TLS is sufficient - -## Upstream / Downstream - -- **Upstream**: Task 11 (`do_one_config_fetch()` stub) on persistent_ha branch — with deferred fetch (Change C), the probe-only path in `bootstrap_config_if_needed()` does not call `do_one_config_fetch()`. Task 11 is hit only at the post-election deferred fetch step in `myapp.c`. If Task 11 is not yet landed, the deferred fetch returns `-ENOSYS` and the bootstrapping standby fails to load config — a fatal but clearly logged error. Both can be co-developed; Task 11 must land before or with Change C for a functional cluster. -- **Downstream**: Phase 3 hardening (membership changes, leader transfer), ops tooling for election/commit observability, monitoring metrics for WAL replication lag. - -## Existing Spec Touchpoints - -- **Adjacent**: none of the active specs (resval-id-decoupling, scheduler-controller-msg-gap, worker-join-refactor) touch config replication, HA election, or cluster init. - -## Constraints - -- C (gnu17), no C++, no new external dependencies. -- ASAN build must pass (`nix build .#xcecore_debug`). -- **`--peers` must list 1 or an odd number ≥ 3 controllers.** Even counts are rejected - at startup with `-EINVAL`. This simplifies quorum math: `(N+1)/2` is exact for all - valid N; no special-case branches needed inside `am_i_primary_quorum()`. -- **Static membership**: the candidate list is fixed at startup from `--peers` and never - changes while the cluster is running. No dynamic join/leave of controller peers. - Worker and scheduler nodes may still join dynamically — only controller peer membership - is static. This eliminates the need for joint consensus (Raft §6) and locks the quorum - denominator for the lifetime of the cluster. -- Single-node mode unaffected: quorum=1 everywhere (election, WAL commit = own write). -- TDD required: failing test first, then implementation. -- All new C test cases go in `src/controller/testsuite/` before implementation. -- `make test` must pass without a live server. -- `commit_timeout_ms` must be configurable (default 5000ms) to allow tuning per deployment latency. -- **WAL write stall during standby catch-up (operational constraint)**: In a 3-node cluster, - quorum for WAL commit is 2 (primary + 1 standby). Between primary winning election and the - first standby completing `repl_complete_catchup()`, the active replication set has 0 standbys. - Every `persist_wal_sync_append()` call blocks for `commit_timeout_ms` and returns `-ETIMEDOUT`. - This is CP-correct behavior — the cluster is unavailable until quorum is formed. Operators - should expect a stall of up to `commit_timeout_ms` per write during the initial catch-up - window on first boot or after a restart. Jobs submitted during this window receive a retriable - error; libxce retry (Change G) handles this transparently for workers and schedulers. - -## Deployment Reference - -### Controller node topology - -| Cluster size | Failure tolerance | WAL ACKs needed | Recommendation | -|---|---|---|---| -| 1 node | 0 | self only | dev/test only | -| 3 nodes | 1 failure | 1 standby ACK | minimum for production | -| 5 nodes | 2 failures | 2 standby ACKs | recommended for critical HPC | -| 7 nodes | 3 failures | 3 standby ACKs | rarely justified; adds latency | - -### Job submission throughput (Raft overhead only) - -Basis: LAN environment, 5 controller nodes, quorum = 3 (primary + 2 standby ACKs). - -**Critical path per `WAL_JOB_SUBMIT`:** - -``` -client → primary: RTT/2 + primary WAL fsync -primary → 2 standbys: RTT/2 + standby WAL fsync (parallel, overlaps with above) -2nd ACK → primary: RTT/2 -primary confirms: RTT/2 -───────────────────────────────────────────────── -Total: 1 LAN RTT + max(fsync_primary, fsync_standby) -``` - -| Parameter | Optimistic | Conservative | -|---|---|---| -| LAN RTT (controller↔controller) | 0.1 ms | 0.5 ms | -| NVMe fsync latency | 0.05 ms | 0.2 ms | -| Effective commit latency | ~0.2 ms | ~0.7 ms | - -**Throughput estimates:** - -| Submit model | Commit latency | Jobs/sec | -|---|---|---| -| Serial single stream | 0.3–1 ms | 1,000–3,300 | -| 10 concurrent clients | 0.5 ms | ~20,000 | -| 100 concurrent clients | 0.5 ms | ~200,000 (non-Raft bottlenecks dominate before this) | - -**Practical HPC ceiling**: 5,000–20,000 jobs/sec before jobmgr queue locking and -in-memory state mutations become the constraint. - -**5-node vs 3-node write latency**: negligible. Majority-commit waits for the -**2nd-fastest** ACK out of 4 standbys (5-node) vs the **1st** out of 2 (3-node). -Under uniform LAN latency the difference is sub-millisecond. The extra two nodes -cost nothing in write throughput but double the failure tolerance. - -**When Raft becomes a bottleneck**: automated pipelines submitting >10,000 short -jobs/sec. Mitigation: batch multiple job submits into a single WAL entry at the -API layer before this design makes that necessary. diff --git a/.kiro/specs/ha-config-bootstrap/design.md b/.kiro/specs/ha-config-bootstrap/design.md deleted file mode 100644 index e908a4a2..00000000 --- a/.kiro/specs/ha-config-bootstrap/design.md +++ /dev/null @@ -1,844 +0,0 @@ -# Design: HA Config Bootstrap & Raft-Style Quorum Election - -## Overview - -This feature eliminates three critical safety gaps in the XCE controller's LSF-style priority-based election: unconfigured nodes can win elections and serve stale cluster state; concurrent-startup nodes falsely self-elect because the peer listener is not yet open; and a promoted standby with missing local config has no bootstrap path. The fix replaces priority election with a Raft-style quorum protocol that encodes config freshness in votes, moves the listener before election, and adds majority-commit durability for both WAL writes and config upgrades. - -**Purpose**: Delivers full CP high-availability to cluster operators — zero data loss on failover, no split-brain on concurrent startup, and no silent config loss on primary crash. The design chooses Consistency over Availability (CAP-CP): a minority partition cannot elect a primary; the cluster waits until majority is restored. - -**Impact**: Replaces `am_i_primary()` and `ha_check_heartbeat_nocluster()` with Raft quorum election and `EPOLLRDHUP`-based failover; extends the controller lifecycle from a 2-state (`CTRL_PRIMARY`/`CTRL_STANDBY`) to a 4-state FSM adding `CTRL_INIT` and `CTRL_ELECTING`; adds majority-commit gating to `persist_wal_sync_append()` and `push_config_to_standbys()`; establishes a persistent TLS control channel between each standby and the primary; adds transparent retry to libxce. - -### Goals - -- Raft-style quorum election with config-freshness tiebreaking (CP semantics, no minority-partition primaries) -- Zero data loss on failover via majority-commit WAL and config replication -- Sub-second failover detection via `EPOLLRDHUP` + heartbeat timeout vtask -- Transparent recovery for unconfigured nodes joining an existing cluster (deferred config fetch) -- Transparent client retry in libxce — worker/scheduler/apiserver code unchanged - -### Non-Goals - -- Raft snapshot protocol (existing dual-zarray COW is orthogonal and unchanged) -- Joint consensus for dynamic controller membership changes (`--peers` list is static at startup) -- mTLS between controllers (server-side TLS sufficient for trusted-LAN model) -- Config file format or storage layout changes -- Worker/scheduler daemon protocol changes - ---- - -## Boundary Commitments - -### This Spec Owns - -- Controller election protocol: full replacement of `am_i_primary()` with `am_i_primary_quorum()` (Note: `am_i_primary_quorum()`'s blocking pthread-per-peer implementation was itself later removed — production election now runs through async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; superseded by raft-library and election-iomgr-refactor, see those specs' design.md) -- Controller FSM: 4-state lifecycle (`CTRL_INIT`, `CTRL_ELECTING`, `CTRL_PRIMARY`, `CTRL_STANDBY`) and all state transitions -- Pre-election startup sequence: listener binding, iomgr/vtaskmgr init before election -- Config bootstrap: `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, deferred fetch after election -- Startup validation: `--vcpus N` CLI, even-N peer count rejection, TLS cert pre-flight, ASAN leak fix -- Majority-commit WAL: `repl_broadcast()`, `repl_wait_majority_ack()`, `committed_lsn`, `persist_truncate_wal_above()` -- Majority-commit config push: `OP_upgradeConfigAck`, `committed_config_tag`, heartbeat piggybacking -- Persistent control channel: `ha_find_and_connect_to_primary()`, `EPOLLRDHUP` failover, WAL streaming over channel, three-tier primary discovery -- Election state durability: `ha-term.bin`, `ha_persist_term()`, `voted_for_term/name` across restarts -- libxce client retry: `xce_retry_config_t`, retry wrapper, `XCE_ERR_STANDBY` redirect fields in wire protocol -- Wire protocol extensions: `OP_requestVote`, `OP_probeConfigTag`, `OP_walAck`, `OP_upgradeConfigAck`, `OP_snapshotTransfer`, `OP_walCatchupEntry` -- Structured log observability for all new code paths (election, bootstrap, WAL commit, config push, FSM transitions) - -### Out of Boundary - -- `apply_wal_entry()` — must be extracted from the startup WAL replay path as a prerequisite to Change F; not authored in this spec -- `persist_checkpoint()` trigger policy — existing on-demand call in `repl_needs_snapshot()` path; unchanged -- Worker/scheduler daemon code — libxce retry is transparent; no caller code changes -- apiserver Go tier — no changes -- `scheduler-controller-msg-gap` spec and `worker-join-refactor` spec — orthogonal; no shared files -- Dynamic controller membership changes (joint consensus) -- mTLS between controllers - -### Allowed Dependencies - -- `iomgr` (epoll/`EPOLLIN`/`EPOLLRDHUP`): P0 — control channel and dispatch loop -- `vtaskmgr`: P0 — `vtaskmgr_init(int nvcpus)` confirmed to take vcpus directly, no cluster_config dependency -- `endpoint_connect()` / `connect_ssl_raw()`: P0 — SSL connection setup; TCP connect phase is blocking; vote request connections require per-peer pthreads (Note: superseded by election-iomgr-refactor — vote requests now use async iomgr endpoints, not per-peer pthreads; see that spec's design.md) -- `repl_add/remove/start_catchup/complete_catchup/needs_snapshot/on_ack/trim_lsn`: P0 — existing repl slot management reused unchanged -- `do_one_config_fetch()`: P1 — wired on branch (`2aae45f`); deferred fetch depends on it -- `xce_encode()` / `xce_decode()` / `xce_request()`: P0 — wire protocol encoding -- `ha_do_promote()` / `ha_do_demote()`: P0 — role transitions, reused -- `src/schema/controller.capnp`: P0 — new opcodes appended after existing @31 -- `xcelog.h` INFO/WARN/ERROR macros: P2 — observability - -### Revalidation Triggers - -- Changes to `ha_state_t` field layout require re-checking all ha.c callers -- Changes to `RequestReply` or `CtrlHeartbeat` capnp structs require checking all decode sites -- Moving `cluster_open_listener()` before election changes startup order; any code assuming phase2 runs before network is up must be re-verified -- `committed_lsn` cross-cuts `persist_t`, `repl.c`, and `ha_state_t`; changes to any require re-checking the others -- `ha-term.bin` format change requires migration or version field - ---- - -## Architecture - -### Existing Architecture Analysis - -The current controller has a 2-state election: `am_i_primary()` spawns per-peer pthreads that probe higher-priority peers and self-elect on timeout. `cluster_init_phase2()` contains `bind()+listen()`, `iomgr_init()`, and `vtaskmgr_init()` — all run after election, creating a 2–3 second window where the port is unreachable to election probes. - -Confirmed gaps from codebase inspection (see `research.md` §1 for full inventory): -- `ctrl_role_t` has only `CTRL_PRIMARY=0`, `CTRL_STANDBY=1` -- `ha_state_t` lacks: `voted_for_term`, `voted_for_name`, `primary_ctrl_fd`, `defer_config_fetch`, `elect_ctx`, `last_known_committed_lsn`, `heartbeat_seen`, `last_known_committed_config_tag` -- `repl_broadcast()` is a stub returning 0 without sending; `repl_wait_majority_ack()` does not exist -- `persist_t` lacks `committed_lsn`; `persist_truncate_wal_above()` does not exist -- `XCE_ERR_STANDBY` reply has no `redirect_host/port` fields — must be added to capnp schema -- `ha-term.bin` does not exist; `ha_init()` always starts from term=0 -- `ha_check_priority_stepdown()` is permanently unreachable after first quorum election (term always ≥ 1) -- `ha_check_heartbeat_nocluster()` uses 10s timestamp polling — replaced by dual-trigger failover - -### Architecture Pattern & FSM - -**Selected pattern**: 4-state FSM with Raft-style majority-quorum consensus (CP). Each startup step gates the next. - -```mermaid -stateDiagram-v2 - [*] --> CTRL_INIT : startup complete listener open - CTRL_INIT --> CTRL_STANDBY : heartbeat received from existing primary - CTRL_INIT --> CTRL_ELECTING : election timeout fires - CTRL_ELECTING --> CTRL_PRIMARY : quorum won and no fresher peer - CTRL_ELECTING --> CTRL_STANDBY : quorum lost or fresher peer yields - CTRL_STANDBY --> CTRL_ELECTING : EPOLLRDHUP or heartbeat timeout vtask - CTRL_PRIMARY --> CTRL_STANDBY : higher-term reply received -``` - -```mermaid -graph TB - subgraph Startup - V[validate_peer_count + validate_certs] --> P1[cluster_init_phase1] - P1 --> Net[iomgr_init + cluster_open_listener + vtaskmgr_init] - Net --> Boot[bootstrap_config_if_needed] - Boot --> Wait[ha_wait_for_heartbeat_or_timeout] - end - subgraph Election - Wait --> Quorum[am_i_primary_quorum] - Quorum --> Vote[OP_requestVote per peer thread] - Vote --> Eval[ha_elect_evaluate_and_signal] - end - subgraph Replication - WAL[persist_wal_sync_append] --> Bcast[repl_broadcast] - Bcast --> MAck[repl_wait_majority_ack] - MAck --> Commit[advance committed_lsn] - end - subgraph ControlChannel - SB[CTRL_STANDBY] --> Conn[ha_find_and_connect_to_primary] - Conn --> Epoll[EPOLLIN + EPOLLRDHUP] - Epoll --> Sync[state sync WAL streaming] - end -``` - -**Note (election mechanism superseded)**: The diagrams above show this spec's original blocking pthread-per-peer `am_i_primary_quorum()` election path. That implementation was removed by election-iomgr-refactor (task 5.3) and replaced by async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; `handle_request_vote()` is now test-only, production vote dispatch is `cluster.c` → `raft_on_vote_request()`. The 4-state FSM, majority-commit, and control-channel shapes shown remain accurate. See election-iomgr-refactor and raft-library design.md for the current mechanism. - -### Technology Stack - -| Layer | Choice | Role | Notes | -|-------|--------|------|-------| -| Runtime | C gnu17 | All new code | No C++ | -| Wire format | Cap'n Proto | 6 new opcodes in `controller.capnp` | Appended after existing @31 | -| SSL | OpenSSL (existing) | Vote request and probe connections | Reuses `connect_ssl_raw()` | -| Consensus | Raft subset (custom) | Leader election + majority-commit | No external library; bounded scope justified by static membership and no snapshot protocol need (Note: a dedicated `src/lib/raft/` in-repo library was later extracted from this inline logic; superseded by raft-library, see that spec's design.md) | -| IPC | iomgr epoll | `EPOLLIN` + `EPOLLRDHUP` on control channel | `EPOLLRDHUP` for sub-second failover | -| Persistence | Flat binary (`ha-term.bin`) | Term + voted_for across restarts | Atomic write+rename (existing pattern) | -| Build | Meson + Nix | capnp schema regeneration | `src/schema/controller.capnp` | - ---- - -## File Structure Plan - -### Directory Structure - -``` -src/ -├── schema/ -│ └── controller.capnp # Modified: 6 new opcodes + message structs -├── controller/ -│ ├── ha.h # Modified: CTRL_INIT/CTRL_ELECTING enum values; -│ │ # ha_vote_entry_t, ha_elect_ctx_t types; -│ │ # new ha_state_t fields; ha_persist_term() decl -│ ├── ha.c # Modified: replace am_i_primary() -> am_i_primary_quorum(); -│ │ # add handle_request_vote(), handle_request_vote_reply(), -│ │ # ha_wait_for_heartbeat_or_timeout(), ha_elect_evaluate_and_signal(), -│ │ # ha_find_and_connect_to_primary(), ha_persist_term(); -│ │ # retire ha_check_priority_stepdown(), ha_check_heartbeat_nocluster(); -│ │ # SSL_CTX_free ASAN fix in ha_fini() -│ ├── persist.h # Modified: add committed_lsn to persist_t -│ ├── persist.c # Modified: persist_truncate_wal_above() (new); -│ │ # upgrade persist_wal_sync_append() to repl_wait_majority_ack() -│ ├── repl.h # Modified: repl_wait_majority_ack() decl; -│ │ # pending_ack_count in standby_conn_t -│ ├── repl.c # Modified: implement repl_broadcast() (was stub); -│ │ # repl_wait_majority_ack() (new); backpressure; catch-up exclusion -│ ├── config_repl.h # Modified: bootstrap_config_if_needed() and xce_probe_config_tag() decls -│ ├── config_repl.c # Modified: bootstrap_config_if_needed() (new); -│ │ # xce_probe_config_tag() (new); -│ │ # upgrade push_config_to_standbys() to majority-commit -│ ├── cluster.h # Modified: remove vcpus from cluster_config_t -│ ├── cluster.c # Modified: extract cluster_open_listener() from phase2; -│ │ # extended handle_ctrlhello(); OP_snapshotTransfer and -│ │ # OP_walCatchupEntry dispatch handlers -│ ├── myapp.c # Modified: new startup order (listener before election); -│ │ # bootstrap_config_if_needed() call; deferred fetch integration -│ ├── main.c # Modified: --vcpus N CLI arg; even-N peer count validation; -│ │ # cert pre-flight access() check -│ └── testsuite/ -│ ├── test_ha_election.c # New: quorum election, vote protocol, FSM transitions -│ ├── test_ha_bootstrap.c # New: bootstrap_config_if_needed, probe RPC, deferred fetch -│ ├── test_ha_durability.c # New: ha-term.bin persist/restore, CRC corruption -│ ├── test_persist_truncate.c # New: persist_truncate_wal_above, boundary segment, idempotency -│ ├── test_repl_majority.c # New: repl_broadcast, repl_wait_majority_ack, backpressure -│ └── test_ha_startup.c # New: startup validation (vcpus, peer count, cert pre-flight) -└── lib/ - ├── request.h # Modified: xce_retry_config_t; xce_request_with_retry() decl - ├── request.c # Modified: retry wrapper; redirect handling; backoff logic - └── testsuite/ - └── test_libxce_retry.c # New: retry logic, backoff, redirect, non-retriable errors -``` - -### Modified Files Summary - -| File | Change Type | Requirements | -|------|-------------|--------------| -| `src/schema/controller.capnp` | Extend: 6 opcodes + structs | 2, 3, 5, 6, 7, 8 | -| `src/controller/ha.h` | Extend: enum + types + fields | 1, 2, 3, 5, 6, 9 | -| `src/controller/ha.c` | Major: replace election core; add control channel | 1, 2, 3, 6, 9 | -| `src/controller/persist.h` | Extend: committed_lsn | 5 | -| `src/controller/persist.c` | New function + upgrade sync path | 5 | -| `src/controller/repl.h` | Extend: standby_conn_t field + decl | 5, 8 | -| `src/controller/repl.c` | Implement stubs + add majority logic | 5, 6, 8 | -| `src/controller/config_repl.h` | Extend: two new function decls | 3, 8 | -| `src/controller/config_repl.c` | New functions + upgrade push | 3, 8 | -| `src/controller/cluster.h` | Remove vcpus field | 4 | -| `src/controller/cluster.c` | Extract listener; extend ctrlhello handler | 1, 6 | -| `src/controller/myapp.c` | New startup sequence | 1, 3 | -| `src/controller/main.c` | CLI validation + pre-flight | 4 | -| `src/lib/request.h` | New type + decl | 7 | -| `src/lib/request.c` | Retry wrapper | 7 | - ---- - -## System Flows - -### Flow 1: Controller Startup & Election - -```mermaid -sequenceDiagram - participant Main as main.c - participant MyApp as myapp.c - participant HA as ha.c - participant Cluster as cluster.c - participant Config as config_repl.c - Main->>Main: validate_peer_count odd-N check - Main->>Main: validate_certs access() - Main->>MyApp: myapp_run() - MyApp->>Cluster: cluster_init_phase1 reads ha-term.bin via ha_init - MyApp->>Cluster: iomgr_init + cluster_open_listener + vtaskmgr_init - Note over Cluster: listener open BEFORE election - MyApp->>Config: bootstrap_config_if_needed - Note over Config: tag=0 and peers have config? set defer_config_fetch and return - MyApp->>HA: ha_wait_for_heartbeat_or_timeout - alt heartbeat received in window - HA->>MyApp: set CTRL_STANDBY skip election - else election timeout fires - HA->>HA: am_i_primary_quorum send OP_requestVote per peer - HA->>MyApp: CTRL_PRIMARY or CTRL_STANDBY - end - alt defer_config_fetch and not primary - MyApp->>Config: do_one_config_fetch from proven winner - end - MyApp->>Cluster: cluster_init_phase2_rest kvmgr jobmgr apiserver etc. -``` - -### Flow 2: Quorum Election Vote Protocol - -```mermaid -sequenceDiagram - participant Cand as Candidate ha.c - participant PA as Peer A ha.c - participant PB as Peer B ha.c - Cand->>Cand: increment term persist ha-term.bin - Cand->>PA: OP_requestVote term config_tag last_lsn via per-peer thread - Cand->>PB: OP_requestVote term config_tag last_lsn via per-peer thread - PA->>PA: handle_request_vote check voted_for and log_ok - PA->>Cand: OP_requestVoteReply granted=true peer_tag peer_lsn - PB->>PB: handle_request_vote check voted_for and log_ok - PB->>Cand: OP_requestVoteReply granted=true peer_tag peer_lsn - Cand->>Cand: ha_elect_evaluate_and_signal - Note over Cand: votes_granted ge quorum AND no fresher peer? CTRL_PRIMARY -``` - -Key decisions: fresher-peer yield gate fires when any *responding* peer has higher `config_tag` or equal tag with higher LSN; non-responding peers never block election. Split-vote triggers one retry with incremented term and randomized jitter delay. - -### Flow 3: WAL Majority-Commit - -```mermaid -sequenceDiagram - participant Caller as Job submit caller - participant Persist as persist.c - participant Repl as repl.c - participant SA as Standby A - participant SB as Standby B - Caller->>Persist: persist_wal_sync_append WAL_JOB_SUBMIT - Persist->>Persist: write + fsync local advance applied_lsn - Persist->>Repl: repl_broadcast OP_walAck payload over persistent channel - Repl->>SA: WAL entry on control channel fd - Repl->>SB: WAL entry on control channel fd - SA->>Repl: OP_walAck lsn=N - Repl->>Persist: repl_wait_majority_ack quorum reached - Persist->>Persist: advance committed_lsn to N - Persist->>Caller: return 0 success - SB->>Repl: OP_walAck lsn=N arrives later counted but not blocking -``` - -Entry types `WAL_HOST_STATE` and `WAL_CHECKPOINT_REF` bypass this flow via `persist_wal_async()` (best-effort, unchanged). - -### Flow 4: Standby Failover Detection and Re-election - -```mermaid -sequenceDiagram - participant IOM as iomgr epoll - participant HA as ha.c standby - participant Peers as Peer controllers - IOM->>HA: EPOLLRDHUP on primary_ctrl_fd - HA->>HA: close primary_ctrl_fd set CTRL_ELECTING - HA->>HA: persist_truncate_wal_above last_known_committed_lsn - HA->>Peers: am_i_primary_quorum new term OP_requestVote - alt quorum won - HA->>HA: CTRL_PRIMARY accept inbound control channels - else quorum lost - HA->>Peers: ha_find_and_connect_to_primary three-tier discovery - HA->>HA: CTRL_STANDBY - end -``` - -Fallback trigger: heartbeat timeout vtask fires after `election_timeout_ms + rand_jitter` of silence on an open TCP channel (covers network partitions that keep TCP alive but suppress heartbeats). - ---- - -## Requirements Traceability - -| Requirement | Summary | Components | Key Interface | Flow | -|-------------|---------|------------|---------------|------| -| 1.1–1.4 | Listener + iomgr/vtaskmgr before election; dispatch gate for init/electing states | `cluster.c`, `ha.c`, `myapp.c` | `cluster_open_listener()`, vtaskmgr dispatch gate | Flow 1 | -| 2.1–2.12 | Quorum election: voted_for enforcement, freshness gate, split-vote retry, single-node shortcircuit | `ha.c` | `am_i_primary_quorum()`, `handle_request_vote()` (Note: both superseded by `ha_start_election_async()`/raft library, `handle_request_vote()` now test-only — see election-iomgr-refactor), `OP_requestVote` | Flow 2 | -| 3.1–3.5 | Pre-election config probe, deferred fetch, retry backoff | `config_repl.c`, `myapp.c` | `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, `OP_probeConfigTag` | Flow 1 | -| 4.1–4.4 | --vcpus CLI, even-N rejection, cert pre-flight, ASAN fix | `main.c`, `ha.c` | CLI argp, `access()`, `SSL_CTX_free` | Flow 1 | -| 5.1–5.8 | Majority-commit WAL: committed_lsn, broadcast, ack-wait, backpressure, truncation | `persist.c`, `repl.c`, `ha.c` | `repl_wait_majority_ack()`, `persist_truncate_wal_above()`, `OP_walAck` | Flow 3 | -| 6.1–6.8 | Persistent control channel: EPOLLRDHUP failover, state sync, catch-up, three-tier discovery | `ha.c`, `cluster.c`, `repl.c` | `ha_find_and_connect_to_primary()`, extended `OP_ctrlHello`, `OP_snapshotTransfer`, `OP_walCatchupEntry` | Flow 4 | -| 7.1–7.6 | Client retry: backoff, redirect, non-retriable passthrough | `request.c` (libxce) | `xce_request_with_retry()` (Note: removed, superseded by `client_failover.c` StatusClassifier/EndpointRotator — see libxce-failover), `xce_retry_config_t`, redirect fields in `OP_requestVoteReply` | — | -| 8.1–8.5 | Majority-commit config push, committed_config_tag, heartbeat piggybacking | `config_repl.c`, `repl.c` | `push_config_to_standbys()`, `OP_upgradeConfigAck` | Flow 3 analog | -| 9.1–9.3 | Term + voted_for durability across restarts via ha-term.bin | `ha.c` | `ha_persist_term()`, `ha_init()` read path | — | -| 10.1–10.5 | Structured logs for election outcome, bootstrap, WAL commit, config push, FSM transitions | All new code paths | `xcelog.h` INFO/WARN macros | All flows | - ---- - -## Components and Interfaces - -### Summary Table - -| Component | File(s) | Intent | Req Coverage | Key Dependencies | -|-----------|---------|--------|-------------|-----------------| -| Controller FSM | `ha.h`, `ha.c` | 4-state lifecycle + election logic | 1, 2, 9 | iomgr P0, vtaskmgr P0 | -| Config Bootstrap | `config_repl.c` | Pre-election probe + deferred fetch | 3 | `do_one_config_fetch` P1 | -| Startup Validation | `main.c`, `ha.c` | CLI, peer count, cert pre-flight, ASAN | 4 | argp, `access()` | -| WAL Majority-Commit | `persist.c`, `repl.c` | committed_lsn, broadcast, ack-wait | 5 | iomgr P0 | -| Persistent Control Channel | `ha.c`, `cluster.c` | TCP+SSL channel, EPOLLRDHUP, state sync | 6 | `repl.c` P0, iomgr P0 | -| Majority-Commit Config | `config_repl.c` | Config push with majority ACK | 8 | `repl_broadcast` P0 | -| Term Durability | `ha.c` | ha-term.bin persist/restore | 9 | atomic write+rename | -| Client Retry | `request.c` | Transparent retry + redirect | 7 | `XCE_ERR_*` codes P0 | -| Wire Protocol | `controller.capnp` | New opcodes and message types | 2, 3, 5, 6, 7, 8 | capnp toolchain P0 | - ---- - -### Controller Core (ha.h / ha.c) - -#### Controller FSM - -| Field | Detail | -|-------|--------| -| Intent | 4-state FSM governing election, standby, and primary lifecycle; owns all election logic and term durability | -| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1–2.12, 9.1–9.3 | - -**Responsibilities & Constraints** -- Owns `ctrl_role_t` enum — add `CTRL_INIT=2`, `CTRL_ELECTING=3` (values above existing 0/1) -- Single allocation for `ha_elect_ctx_t` via C99 flexible array — one `calloc`, one `free` -- All role transitions via `atomic_store(&c->ha.role, ...)` — no lock held across transition -- `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()` are deleted in this spec -- (Note: `am_i_primary_quorum()`'s production implementation and `handle_request_vote()`/`handle_request_vote_reply()` were themselves later removed/demoted to test-only by election-iomgr-refactor — production election is now `ha_start_election_async()` → raft library; see that spec's design.md) - -**New `ha_state_t` fields**: -```c -uint64_t voted_for_term; // Req 9: term of last granted vote -char voted_for_name[XCE_MAXNAME_LEN]; // Req 9: candidate of last granted vote -int primary_ctrl_fd; // Req 6: persistent channel fd (-1 = none) -bool defer_config_fetch; // Req 3: tag=0 node defers fetch to post-election -_Atomic void *elect_ctx; // Req 2: ha_elect_ctx_t* during election; NULL otherwise -_Atomic uint64_t last_known_committed_lsn; // Req 5: from primary heartbeats -sem_t heartbeat_seen; // Req 1: posted by heartbeat handler during CTRL_INIT -uint64_t last_known_committed_config_tag; // Req 8: from primary heartbeats -``` - -**New types** (ha.h): -```c -typedef struct { - xce_candidate_t *peer; - uint64_t my_term; - uint64_t my_tag; - uint64_t my_lsn; - int vote_granted; - int replied; - uint64_t peer_tag; - uint64_t peer_lsn; -} ha_vote_entry_t; - -typedef struct { - int n_peers; - int votes_granted; // starts at 1 (self-vote) - int n_replied; - uint64_t max_resp_tag; - uint64_t max_resp_lsn; - int result; // 1=primary, 0=standby, <0=error - sem_t done; // startup thread blocks here; dispatch posts when done - ha_vote_entry_t votes[]; // C99 flexible array: n_peers-1 entries - // Allocate: calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0])) -} ha_elect_ctx_t; -``` - -**`ha_persist_term()` contract**: -- Precondition: called before any vote is granted or term incremented -- Action: writes `ha_term_file_t` atomically to `{workdir}/ha-term.bin` (write + rename) -- Postcondition: file reflects current state; durable before any network reply - -**Dispatch gate** (vtaskmgr switch, req 1.3): -```c -// CTRL_INIT or CTRL_ELECTING: only OP_requestVote and OP_probeConfigTag pass through -switch (atomic_load(&c->ha.role)) { -case CTRL_INIT: -case CTRL_ELECTING: - if (msg->op != OP_requestVote && msg->op != OP_probeConfigTag) { - xce_send_error(task, XCE_ERR_NOT_READY, "controller not ready"); - return; - } - break; -case CTRL_STANDBY: - // Primary-only ops return XCE_ERR_STANDBY + redirect_host/port when known - break; -case CTRL_PRIMARY: - break; -} -``` - -**Contracts**: Service [x] / State [x] - -**Implementation Notes** -- Risk: `endpoint_connect()` has a blocking TCP connect phase. Vote request sends require per-peer pthreads (not pure iomgr async). Election context `ha_elect_ctx_t` accumulates replies; threads post to `ctx->done` semaphore when quorum is reached or all peers replied. (Note: this risk was resolved by election-iomgr-refactor — vote sends are now pure async iomgr endpoints via `ha_start_election_async()`, not per-peer pthreads; `ha_elect_ctx_t` gained a refcount and several fields since, see ha-elect-ctx-lifecycle design.md) -- Integration: `ha_wait_for_heartbeat_or_timeout()` uses `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + rand_jitter`. Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`. -- `voted_for` enforcement: per-term, one vote per node. If `req->term > my_term`: update term, clear `voted_for_term`/`voted_for_name`. If `req->term < my_term`: deny. If already voted for a different candidate in same term: deny. - ---- - -### Persistence Layer (persist.h / persist.c) - -#### WAL Majority-Commit - -| Field | Detail | -|-------|--------| -| Intent | Gate job/queue WAL writes on majority-standby durability; new committed_lsn field and truncation function | -| Requirements | 5.1–5.8 | - -**New `persist_t` field**: -```c -_Atomic uint64_t committed_lsn; // highest majority-ACKed LSN; invariant: committed_lsn <= applied_lsn -``` - -**`persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)` contract**: -- Precondition: `cutoff_lsn <= p->applied_lsn`; may be called after promotion or standby re-join -- Action: find WAL segment containing `cutoff_lsn` by filename (filename encodes start LSN as hex); scan entries in that segment sequentially until LSN > cutoff_lsn; truncate at that byte offset; delete all segments starting above cutoff_lsn; atomically update `p->applied_lsn = cutoff_lsn` -- Postcondition: no WAL entry with LSN > cutoff_lsn exists on disk; idempotent (calling twice is safe) -- Note: WAL segments are 64 MB named `seg-{LSN_hex}.wal`; no built-in LSN-to-offset index; sequential scan required - -**Entry type dispatch** (5.1, 5.2): -- `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE` → `persist_wal_sync_append()` → majority-commit path (blocking) -- `WAL_HOST_STATE`, `WAL_CHECKPOINT_REF` → `persist_wal_async()` → best-effort (unchanged) - -**WAL trim guard**: trim threshold changes from `applied_lsn` to `committed_lsn` to prevent trimming uncommitted entries. - -**Contracts**: Service [x] / State [x] - ---- - -### Replication Layer (repl.h / repl.c) - -#### Majority-Commit Broadcast - -| Field | Detail | -|-------|--------| -| Intent | Implement repl_broadcast (was stub); add repl_wait_majority_ack; manage per-standby backpressure | -| Requirements | 5.1–5.8, 8.1–8.5 | - -**New `standby_conn_t` field**: `int pending_ack_count` — backpressure counter; default max 64 - -**`repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len)` contract**: -- Sends payload to all connected standbys where `!in_catch_up && pending_ack_count <= max_pending_acks` -- Sends over persistent control channel fd (no new connection per entry) -- On send failure: remove standby from active set, trigger catch-up reconnect, log WARN -- Returns 0 when fanout initiated (not when ACKed) -- (Note: signature and mechanism superseded — current `repl_broadcast(replicator_t *r, const wal_entry_hdr_t *hdr, const void *payload, uint32_t payload_len, uint64_t prev_log_term)` sends a per-entry AppendEntries carrying the leader's per-entry term for the raft library's log-replication reconcile, with NACK-driven retry; see raft-log-replication, ha-term-safety, raft-adapter-conformance design.md) - -**`repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)` contract**: -- Blocks (cond_timedwait) until `acked_count + 1 >= (n_active + 1) / 2` - - `+1` counts own write; `n_active` excludes `in_catch_up==1` standbys (5.4, Clarification 6) -- `ack_op` parameter selects `OP_walAck` vs `OP_upgradeConfigAck` to disambiguate concurrent commits -- Returns 0 on quorum, `-ETIMEDOUT` on timeout -- Single-node shortcircuit: `n_active == 0` → quorum immediately from own write (5.8, 8.5) -- (Note: signature and mechanism superseded — current `repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn, uint32_t timeout_ms, uint8_t ack_op)` delegates the walAck track to a commit engine (`commit_submit`) rather than counting acks directly, and gates commit on `durable_lsn` (fdatasync-confirmed), not raw ACK count; see ha-commit-engine and ha-commit-durability design.md) - -**Contracts**: Service [x] / State [x] - ---- - -### Config Replication (config_repl.h / config_repl.c) - -#### Config Bootstrap - -| Field | Detail | -|-------|--------| -| Intent | Pre-election config tag discovery and deferred post-election fetch for unconfigured nodes | -| Requirements | 3.1–3.5 | - -**`bootstrap_config_if_needed(cluster_t *c)` contract**: -- Returns immediately if `c->myconf->config_version_tag != 0` -- Probes all peers via `xce_probe_config_tag()`; three outcomes: - 1. All peers `tag=0` → first-boot cluster, proceed to election (3.3) - 2. Any peer has `tag > 0` → set `c->ha.defer_config_fetch = true`, return (3.2) - 3. No peer reachable → retry with exponential backoff 500ms→10ms cap 10s; log WARN each attempt (3.4) -- Never returns an error; blocks until resolution or process killed -- (Note: this full contract was later simplified/removed — the production `bootstrap_config_if_needed()` was removed in election-iomgr-refactor task 5.3; only a stub remains for linkage, mirroring the simplified two-branch logic now inlined ahead of `ha_start_election_async()` in myapp.c. See election-iomgr-refactor design.md) - -**`xce_probe_config_tag(const char *host, uint16_t port, uint64_t *tag_out)` contract**: -- Single-RTT SSL request via `connect_ssl_raw()` + xce encode/decode -- Request: `OP_probeConfigTag` with `ProbeConfigTagRequest{name}` (requester name for peer logging) -- Reply: `ProbeConfigTagReply{config_tag}` (0 if peer has no config) -- Peer answers in any FSM state — dispatch gate exempts `OP_probeConfigTag` alongside `OP_requestVote` -- Returns 0 on success, `-ECONNREFUSED` / `-ETIMEDOUT` on unreachable peer - -#### Majority-Commit Config Push - -| Field | Detail | -|-------|--------| -| Intent | Upgrade push_config_to_standbys() from fire-and-forget to majority-commit using shared repl infrastructure | -| Requirements | 8.1–8.5 | - -**`push_config_to_standbys()` extended contract**: -1. Write config blob locally (existing atomic write+rename — unchanged) -2. `repl_broadcast(c, OP_upgradeConfig, blob, len)` -3. `repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck)` -4. On quorum: `atomic_store(&c->myconf->committed_config_tag, tag)` (8.1) -5. On timeout: return `-ETIMEDOUT`; operator retries (8.3) - -**`committed_config_tag`**: new field in cluster config state; piggybacked on `CtrlHeartbeat` (`committed_config_tag @N :UInt64`). Standbys store in `ha.last_known_committed_config_tag`. On promotion: serve only configs up to `last_known_committed_config_tag` (8.4). - -**Contracts**: Service [x] / State [x] - ---- - -### Persistent Control Channel (ha.c / cluster.c) - -#### Primary Discovery and Control Channel - -| Field | Detail | -|-------|--------| -| Intent | Persistent TLS channel per standby; EPOLLRDHUP failover; full state sync on join | -| Requirements | 6.1–6.8 | - -**Three-tier primary discovery** (post-election, standby path): -1. **Tier 1** (O(1) for majority): if `ha.voted_for_name` set, try direct connect to voted-for peer; on `XCE_ERR_NOT_READY` peer still electing — backoff, retry Tier 1 -2. **Tier 2** (redirect): probe remaining peers; a `CTRL_STANDBY` peer that knows the primary replies with `XCE_ERR_STANDBY` + `redirect_host/port`; follow redirect immediately -3. **Tier 3** (blind scan): try remaining peers sequentially with 100ms→5s exponential backoff until primary found - -**Extended `OP_ctrlHello` schema** (controller.capnp): -``` -struct CtrlHelloRequest { - name @0 :Text; - key @1 :Data; # SHA-256(cluster_name + shared_secret); auth gate - applied_lsn @2 :UInt64; - config_tag @3 :UInt64; -} -struct CtrlHelloReply { - committed_lsn @0 :UInt64; - needs_snapshot @1 :Bool; - config_tag @2 :UInt64; -} -``` - -**Failover detection** — dual-trigger replacing timestamp polling (6.2, 6.3, 6.4): -- **Fast**: `EPOLLRDHUP` on `primary_ctrl_fd` → immediately close, null fd, set `CTRL_ELECTING` -- **Fallback**: heartbeat timeout vtask armed on each heartbeat receipt; fires after `election_timeout_ms + rand_jitter` of silence; same action as EPOLLRDHUP path - -**`XCE_ERR_STANDBY` redirect extension** (req 7.2, Tier 2): -Add to `RequestReply` (controller.capnp): -``` -redirect_host @N :Text; # empty string when primary unknown -redirect_port @M :UInt16; # 0 when primary unknown -``` -Standbys populate from `ha.primary_host/port` when known; clients without redirect fields fall back to normal retry. - -**State sync sequence on standby join** (6.5, 6.6, 6.7): - -Primary side (`handle_ctrlhello()`): -1. `repl_add_standby()` → `repl_start_catchup()` sets `in_catch_up=1` -2. Reply `CtrlHelloReply{committed_lsn, needs_snapshot, config_tag}` -3. If `needs_snapshot`: `persist_checkpoint()` → send `OP_snapshotTransfer` -4. Stream `OP_walCatchupEntry` [applied_lsn+1 → current committed_lsn] -5. On `OP_walAck` reaching committed_lsn → `repl_complete_catchup()` (`in_catch_up=0`) -6. If standby config_tag ≠ primary config_tag → `push_config_to_standbys()` - -Standby side: -1. Store `primary_committed_lsn = reply.committed_lsn`; update `ha.last_known_committed_lsn` -2. `persist_truncate_wal_above(primary_committed_lsn)` — discard orphaned uncommitted entries -3. `ha.in_catch_up = 1` — deny votes during sync (Clarification 6) -4. If `reply.needs_snapshot`: receive `OP_snapshotTransfer`, write, call `load_snapshot()` -5. Apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each -6. On catch-up loop end from primary: `ha.in_catch_up = 0` - -**Role flip on promotion** (6.8): standby closes and nulls `primary_ctrl_fd` before returning from `am_i_primary_quorum()` (Note: this call site moved to the raft library's `on_become_leader` callback path, `raft_cb_become_leader`, since `am_i_primary_quorum()` was removed — see election-iomgr-refactor); new primary accepts inbound `OP_ctrlHello` connections in a per-standby fd table. - -**Contracts**: Service [x] / State [x] - -**Implementation Notes** -- Risk: `apply_wal_entry()` must exist before Change F lands; extract from startup WAL replay path as a prerequisite task -- Risk: `persist_truncate_wal_above()` requires sequential scan of partial WAL segment; validate with unit tests before integrating into standby join path -- Integration: `repl_broadcast()` skips `in_catch_up==1` standbys during the streaming loop; `repl_complete_catchup()` fires only when standby's `acked_lsn` equals current `committed_lsn` - ---- - -### Client Retry (src/lib/request.c) - -**(Note: superseded by libxce-failover)** — `xce_request_with_retry()` described below was later removed as an "orphaned no-op" (per `src/lib/request.c` comment at the definition of `XCE_RETRY_DEFAULTS`); it was superseded by a `StatusClassifier`/`EndpointRotator` failover loop in `src/lib/client_failover.c`, driven from the xcecli layer. `xce_retry_config_t` and `XCE_RETRY_DEFAULTS` (the tunable struct/defaults — max_retries=20, initial_retry_ms=50, max_retry_ms=5000, backoff_multiplier=2.0) still exist verbatim in `src/include/request.h` and are consumed by the new failover loop. See libxce-failover design.md for the current mechanism. - -#### libxce Retry Wrapper - -| Field | Detail | -|-------|--------| -| Intent | Transparent retry for transient controller errors; existing daemon code unchanged | -| Requirements | 7.1–7.6 | - -**`xce_retry_config_t`**: -```c -typedef struct { - int max_retries; // default 20 - int initial_retry_ms; // default 50 - int max_retry_ms; // default 5000 - double backoff_multiplier; // default 2.0 (exponential) -} xce_retry_config_t; -``` - -**`xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg)` contract**: -- NULL `cfg` → uses defaults -- `XCE_ERR_NOT_READY`: backoff, retry same endpoint (7.1) -- `XCE_ERR_STANDBY` with `redirect_host/port`: reconnect to redirect, retry (7.2) -- `XCE_ERR_STANDBY` without redirect: backoff, retry same endpoint (7.2 fallback) -- `ECONNREFUSED` / `ETIMEDOUT`: reconnect, retry (7.3) -- `XCE_ERR_AUTH`, `XCE_ERR_PERM`, `XCE_ERR_INVALID`, `XCE_ERR_NOENT`: surface immediately (7.4) -- Exhausted `max_retries`: surface last error to caller - -**Contracts**: Service [x] - -**Implementation Notes** -- Current `xce_request()` is async send-only; the retry wrapper handles the synchronous request-reply cycle around it -- Backoff jitter uses simple `rand_r()` to avoid thundering-herd (7.6) - ---- - -### Wire Protocol (src/schema/controller.capnp) - -#### New Opcodes and Message Types - -Append after existing `upgradeConfig @31`: - -| Opcode | @N | Purpose | Req | -|--------|----|---------|-----| -| `requestVote` | @32 | Raft vote request/reply | 2 | -| `probeConfigTag` | @33 | Lightweight config tag probe | 3 | -| `walAck` | @34 | WAL majority-commit ACK | 5 | -| `upgradeConfigAck` | @35 | Config push majority-commit ACK | 8 | -| `snapshotTransfer` | @36 | Snapshot blob from primary to standby | 6 | -| `walCatchupEntry` | @37 | WAL entry during catch-up streaming | 6 | - -**Key struct definitions** (to add to controller.capnp): -``` -struct RequestVoteRequest { - term @0 :UInt64; - candidateId @1 :Text; - configTag @2 :UInt64; - lastLogLsn @3 :UInt64; -} -struct RequestVoteReply { - term @0 :UInt64; - voteGranted @1 :Bool; - peerConfigTag @2 :UInt64; - peerLastLogLsn @3 :UInt64; -} -struct ProbeConfigTagRequest { name @0 :Text; } -struct ProbeConfigTagReply { configTag @0 :UInt64; } -struct WalAckRequest { term @0 :UInt64; lsn @1 :UInt64; } -struct WalAckReply { term @0 :UInt64; lsn @1 :UInt64; success @2 :Bool; } -struct UpgradeConfigAck { term @0 :UInt64; ackedTag @1 :UInt64; success @2 :Bool; } -struct SnapshotTransfer { snapshotLsn @0 :UInt64; payload @1 :Data; } -struct WalCatchupEntry { lsn @0 :UInt64; entryType @1 :UInt8; payload @2 :Data; } -``` -(Note: `WalCatchupEntry.payload` is now `List(UInt8)`, not `Data` — bug-1505 capnpc-c codec fix, unrelated to any listed spec) - -Also extend: -- `CtrlHeartbeat`: add `committedLsn @N :UInt64` and `committedConfigTag @M :UInt64` for piggybacking (req 5.6, 8.4) -- `RequestReply`: add `redirectHost @N :Text` and `redirectPort @M :UInt16` for standby redirect (req 7.2) - ---- - -## Data Models - -### New Persistent State - -**`ha-term.bin`** — fixed-size binary, atomic write+rename: -```c -typedef struct { - uint64_t magic; // 0xDEADBEEFCAFEBABE - uint64_t term; - uint64_t voted_for_term; - char voted_for_name[XCE_MAXNAME_LEN]; - uint32_t crc32; // CRC32 of all preceding fields -} ha_term_file_t; -``` -- Read in `ha_init()` at startup; missing file → term=0, no prior vote (first boot) -- CRC mismatch → log WARN, initialize to term=0 (conservative: safe to vote again) - -### New In-Memory State - -| Struct | Field | Type | Invariant | -|--------|-------|------|-----------| -| `persist_t` | `committed_lsn` | `_Atomic uint64_t` | `committed_lsn <= applied_lsn` always | -| `ha_state_t` | 8 new fields | various | See Components section | -| `standby_conn_t` | `pending_ack_count` | `int` | `<= max_pending_acks` while in active set | - -### Consistency Guarantees - -- WAL trim guard: threshold = `committed_lsn` not `applied_lsn`; prevents trimming entries a promoted standby may need for divergence resolution -- `committed_lsn <= applied_lsn` asserted at promotion time; violated invariant is a programming error -- `ha-term.bin` is written atomically before any vote reply is sent; a crash between write and reply means the term is persisted but the vote may not have been counted — conservative and safe -- (Note: majority-commit via `committed_lsn` alone is superseded — ha-commit-durability added a `durable_lsn` field: a member counts toward commit majority only once its LSN is `fdatasync`-confirmed, not merely ACKed/appended; `committed_lsn` still exists but the true commit/promotion-safety floor is `persist_durable_floor()`. See ha-commit-durability design.md) - ---- - -## Error Handling - -### Election Errors - -| Error | Condition | Response | -|-------|-----------|----------| -| `config_tag=0` and any peer has `tag > 0` | Hard gate A | Fatal exit `-ENODATA`; log ERROR with peer's tag value | -| Quorum not reached after 2 rounds | Normal election result | Enter `CTRL_STANDBY`; log WARN with vote tally | -| Fresher responding peer yield gate B | Winner yields to peer with higher tag or LSN | Enter `CTRL_STANDBY`; log INFO with peer tag/lsn seen | -| `ha-term.bin` CRC mismatch at startup | File corruption or incomplete write | Log WARN; initialize term=0; safe to proceed | -| Split vote after 2nd round | Split-vote safety net | Enter `CTRL_STANDBY`; log WARN | - -### WAL and Config Commit Errors - -| Error | Condition | Response | -|-------|-----------|----------| -| `-ETIMEDOUT` from `repl_wait_majority_ack` | No quorum within `commit_timeout_ms` | Return to caller; entry persisted locally but not committed; log WARN | -| Standby backpressure exceeded | `pending_ack_count > max_pending_acks` | Remove from active set; trigger catch-up reconnect; log WARN | -| Standby send failure in `repl_broadcast` | TCP error on control channel fd | Same as backpressure — remove and trigger reconnect | - -### Startup Validation Errors - -| Error | Condition | Response | -|-------|-----------|----------| -| Even peer count | `n % 2 == 0 && n > 1` | Fatal `-EINVAL`; operator-friendly stderr with correct format | -| Cert missing or unreadable | `access(path, R_OK) != 0` | Fatal `-ENOENT`; print full path | -| `--vcpus < 1` | CLI argument | Fatal `-EINVAL`; print message | -| `vcpus` key in config file | Deprecation | Log WARN; do not fail startup | - -### Operational Constraint — WAL Stall Window - -Between primary winning election and first standby completing `repl_complete_catchup()`, the active replication set has 0 standbys. Every `persist_wal_sync_append()` call blocks for `commit_timeout_ms` and returns `-ETIMEDOUT`. This is correct CP behavior. Libxce retry (req 7) handles this transparently for workers and schedulers. Operators must be warned in deployment documentation. - -### Monitoring (Req 10) - -All new paths emit structured `xcelog.h` log entries: -```c -// Election completion (10.1) -INFO("election complete: role=%s term=%lu votes=%d/%d quorum=%d", - role_str, term, granted, n_peers, quorum); -// Bootstrap probe result (10.2) -INFO("config_bootstrap: outcome=defer best_peer_tag=%016llx", best_tag); -// WAL commit quorum (10.3) -INFO("wal_commit: lsn=%lu acked=%d/%d latency_ms=%d", lsn, acked, n_active, ms); -// Config push quorum (10.4) -WARN("config_push_timeout: tag=%016llx timeout_ms=%d", tag, commit_timeout_ms); -// FSM transition (10.5) -INFO("ha_role: %s -> %s term=%lu", from_str, to_str, term); -``` - ---- - -## Testing Strategy - -### Unit Tests (`src/controller/testsuite/`) - -| File | Verifies | Req | -|------|---------|-----| -| `test_ha_election.c` | Quorum met on 2/3 votes; voted_for enforcement across terms; freshness gate yields to fresher peer; split-vote triggers retry; single-node self-elects immediately | 2.1–2.12 | -| `test_ha_bootstrap.c` | All-zero first-boot proceeds; peer-has-config sets defer_config_fetch; no-peer-reachable retries with backoff; retry succeeds on 3rd attempt | 3.1–3.5 | -| `test_ha_durability.c` | ha_persist_term writes then ha_init restores term and voted_for; CRC corruption zeroes state safely; missing file zeroes state | 9.1–9.3 | -| `test_persist_truncate.c` | Truncation at segment boundary deletes whole segment; truncation mid-segment scans to correct offset; calling twice with same cutoff is idempotent | 5.6, 5.7 | -| `test_repl_majority.c` | Majority reached on 2/3 ACKs; timeout returns -ETIMEDOUT; single-node returns 0 immediately; in_catch_up standby excluded from quorum count | 5.1–5.8, 8.1–8.5 | -| `test_ha_startup.c` | Even peer count rejected; cert missing returns -ENOENT with path; --vcpus 0 rejected; vcpus in config logs deprecation warning but does not fail | 4.1–4.4 | - -### Unit Tests (`src/lib/testsuite/`) - -| File | Verifies | Req | -|------|---------|-----| -| `test_libxce_retry.c` | NOT_READY triggers backoff and retry same host; STANDBY with redirect connects to redirected host; STANDBY without redirect falls back to backoff; AUTH error surfaces immediately; max_retries exhausted surfaces last error; 50 concurrent clients with backoff do not starve | 7.1–7.6 | - -### Integration Tests (multi-node, `make test-integration`) - -| Scenario | Verifies | Req | -|----------|---------|-----| -| Concurrent-startup: both nodes start within 3s | Only one node wins election; no false self-election | 1.1, 2.1 | -| Config-fresh node vs stale node election | Stale node yields; fresh node wins | 2.7 | -| Unconfigured node joins live cluster | defer_config_fetch set; deferred fetch succeeds; node enters standby with correct config | 3.2, 3.5 | -| Primary crash → standby promotion | Sub-second EPOLLRDHUP detection; promoted standby has all committed WAL entries | 5.6, 6.2 | -| WAL commit with all standbys down | Returns -ETIMEDOUT per entry; single-standby quorum resumes on reconnect | 5.5 | -| Config upgrade under network partition | Operator receives -ETIMEDOUT; retry after heal succeeds | 8.3 | -| Restart after participating in term N | Restored voted_for prevents double-vote | 9.1, 9.2 | - -### Performance Tests - -| Test | Target | Req | -|------|--------|-----| -| Election round-trip latency (3-node LAN) | < 100ms p99 | 2.1 | -| WAL majority-commit overhead (3-node LAN) | < 5ms p99 per WAL_JOB_SUBMIT | 5.1 | -| Failover detection latency | < 1s from TCP disconnect to election start | 6.2 | -| 50 concurrent libxce clients post-failover | All succeed within 30s; no starvation | 7.6 | - ---- - -## Security Considerations - -- **Cert pre-flight** (4.2): Controller refuses to start with missing or unreadable TLS cert/key. Prevents silent unencrypted operation. -- **Cluster auth key in `OP_ctrlHello`** (6.1): `key = SHA-256(cluster_name + shared_secret)` gates standby join. `cluster_key` is a new field read from the existing config file (not CLI, to avoid shell history exposure). Primary rejects `OP_ctrlHello` where HMAC does not match. -- **`voted_for` persistence** (9.1): Prevents double-vote within a term across restarts. A node that crashes after voting and restarts must not vote for a second candidate in the same term. -- **Term-based fencing**: A primary that receives any reply with `term > my_term` immediately steps down (`ha_do_demote()`). Prevents a reconnected stale primary from continuing after partition heal. -- **No priority tiebreak**: Raft freshness (config_tag + LSN) fully determines winner. The `ha.my_priority` field is preserved in `ha_state_t` for backward compatibility but plays no role in election outcome. - -## Performance & Scalability - -- **Write latency**: 1 LAN RTT + max(fsync_primary, fsync_standby) per majority-commit entry. At 0.1–0.5ms LAN RTT and 0.05–0.2ms NVMe: ~0.2–0.7ms commit latency. Applies only to `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE`. -- **Throughput ceiling**: 1,000–20,000 job submits/sec before jobmgr locking dominates. Raft overhead is not the bottleneck at HPC-typical rates. -- **5-node vs 3-node write latency**: negligible. 5-node waits for 2nd-fastest of 4 standbys vs 1st of 2; sub-millisecond difference on uniform LAN. -- **Single-node shortcircuit**: all majority-commit paths check `n_peers == 1` first; self-only write immediately satisfies quorum. -- **Backpressure**: `max_pending_acks = 64` (configurable) prevents any single slow standby from stalling primary fanout. -- **`commit_timeout_ms`**: configurable, default 5000ms; tune per deployment network latency. diff --git a/.kiro/specs/ha-config-bootstrap/election-trace.md b/.kiro/specs/ha-config-bootstrap/election-trace.md deleted file mode 100644 index 8c744f4c..00000000 --- a/.kiro/specs/ha-config-bootstrap/election-trace.md +++ /dev/null @@ -1,358 +0,0 @@ -# Election Trace: 3-Node Cold Start (2026-06-04 00:15:58) - -## Parameters - -| Parameter | Value | -|-----------|-------| -| election_timeout_ms | 5000 | -| election_priority_step_ms | 500 | -| heartbeat_interval_ms | 3000 | -| committing_timeout_ms | 10000 | -| heartbeat_timeout_ms | 10000 | -| cluster size | 3 (ctrl-1 idx=0, ctrl-2 idx=1, ctrl-3 idx=2) | -| initial_state | ha-term.bin wiped, config_tag=4986624fd309142c on all 3 | - -## Startup Sequence (myapp_run) - -For each node, the startup order is: -1. cluster_init_phase1 (raft_init with n_peers=1 placeholder, term=0) -2. SSL context -3. **iomgr_init + iomgr_start** ← iomgr thread RUNNING -4. cluster_register_listener ← listener in epoll -5. delaymgr_init + start -6. persist_init + start -7. load_primary_config -8. **stagger nap** + ha_start_election_async -9. cluster_run - -**Key**: iomgr is running (step 3) BEFORE the stagger nap (step 8). -This means incoming vote requests are processed during the nap. - -## Detailed Trace - -### T+0.000s — ctrl-1 starts (pid 532871) - -``` -restored term=0 from ha-term.bin -ha->term=0, voted_for_name="" -raft STATE (from ha_init): n_peers=1, self_idx=0, term=0, role=FOLLOWER, voted_for=-1 - -myapp_run step 7: Pre-election config loaded: tag=4986624fd309142c - -ha_start_election_async: - ha->raft_c = c ← SET to non-NULL - rcfg.initial_term = atomic_load(&ha->term) = 0 - rcfg.initial_voted_for = -1 (voted_for_name="") - raft_init: n_peers=3, self_idx=0, s->term=0, s->role=FOLLOWER - raft_start_election: - s->term: 0 → 1 - s->voted_for = 0 (self) - s->voted_for_term = 1 - s->votes_granted = 1 - s->role = CANDIDATE - on_persist(term=1, voted_for=0): - ha->term = 1 - ha->voted_for_name = "ctrl-1" (c is non-NULL) - on_send_vote_request to peer 1 (ctrl-2), term=1 - on_send_vote_request to peer 2 (ctrl-3), term=1 - on_schedule_timeout(5000ms) - - CMD: INFO "→ CTRL_ELECTING (async, n_peers=3)" - CMD: INFO "returning (n_peers=3, raft election started)" - -→ cluster_run() starts iomgr event loop -→ vote connections to ctrl-2 and ctrl-3 attempted -→ ctrl-2 and ctrl-3 NOT STARTED YET → connection refused → peer counted as DENY -→ ctrl-1 waits for election timeout (5000ms) - votes_granted=1 (self only), quorum=2, not reached -``` - -### T+0.301s — ctrl-2 starts (pid 532883) - -``` -restored term=0 -ha->term=0, voted_for_name="" -raft STATE (from ha_init): n_peers=1, self_idx=0, term=0, role=FOLLOWER, voted_for=-1 - -myapp_run steps 1-7 complete. iomgr running. listener in epoll. - -→ enters 500ms stagger nap (my_priority=1 × 500ms) - -DURING NAP: iomgr is running, listener accepting connections. - But ctrl-1's vote connections already failed (ctrl-2 wasn't up yet). - No new vote requests arrive during this nap window. - ha->term stays at 0. - voted_for_name stays at "". -``` - -### T+0.603s — ctrl-3 starts (pid 532895) - -``` -restored term=0 -ha->term=0, voted_for_name="" -raft STATE (from ha_init): n_peers=1, self_idx=0, term=0, role=FOLLOWER, voted_for=-1 - -→ enters 1000ms stagger nap (my_priority=2 × 500ms) -``` - -### T+0.801s — ctrl-2 nap ends, starts election - -``` -already_voted check: voted_for_name="" (EMPTY!) → NO → starts election - -ha_start_election_async: - ha->raft_c = c ← SET to non-NULL - rcfg.initial_term = ha->term = 0 - raft_init: n_peers=3, self_idx=1, s->term=0 - raft_start_election: - s->term: 0 → 1 - s->voted_for = 1 (self) - s->votes_granted = 1 - on_persist(term=1, voted_for=1): - ha->term = 1 - ha->voted_for_name = "ctrl-2" (c non-NULL) - on_send_vote_request to peer 0 (ctrl-1), term=1 - on_send_vote_request to peer 2 (ctrl-3), term=1 - - CMD: INFO "→ CTRL_ELECTING" - CMD: INFO "returning" -→ cluster_run -``` - -### T+0.810s — ctrl-2's votes arrive - -**To ctrl-1 (index 0):** -``` -ctrl-1 raft state: s->term=1, s->role=CANDIDATE, s->voted_for=0(self) -raft_on_vote_request(term=1, from_idx=1): - Step 1: term(1) < s->term(1)? NO (equal) - Step 2: term(1) > s->term(1)? NO (equal) - Step 3: role is CANDIDATE → passes (not LEADER) - Step 4: s->voted_for(0) != from_idx(1) && s->voted_for_term(1) == term(1) - → "already voted for different candidate" → DENY -Result: DENY (already voted for self at term=1) -``` - -**To ctrl-3 (index 2)** — ctrl-3 is in 1000ms nap, iomgr running: -``` -ctrl-3 raft state: n_peers=1, self_idx=0, s->term=0, s->role=FOLLOWER, s->voted_for=-1 -(placeholder from ha_init — n_peers/selx are WRONG for this context) - -raft_on_vote_request(term=1, from_idx=1): - Step 1: term(1) < s->term(0)? NO - Step 2: term(1) > s->term(0)? YES - s->term = 1 - s->voted_for = -1 - s->voted_for_term = 0 - s->role = FOLLOWER - was_leader = false - Step 3: role is FOLLOWER → passes - Step 4: voted_for(-1) != from_idx(1)? NO (voted_for is -1 → doesn't match "!= -1" check) - Step 5: on_vote_eligible → always returns 1 - Step 6: on_persist(term=1, from_idx=1): - ha->term = 1 ← SET! - c = ha->raft_c = NULL! ← NOT YET SET (ha_start_election_async not called) - voted_for(1) != my_priority(2) → memset(voted_for_name, 0, ...) ← CLEARED! -Result: GRANT ← VOTE GRANTED - -SIDE EFFECT: ha->term=1, BUT voted_for_name is EMPTY (BUG) -``` - -### T+0.865s (00:15:59.374) — ctrl-2 wins term=1 - -``` -ctrl-2 receives ctrl-3's vote reply: vote_granted=1 -votes_granted: self(1) + ctrl-3(1) = 2 = quorum (ceil(3/2)=2) - -raft_on_vote_reply → quorum reached → on_become_leader(term=1): - raft_cb_become_leader(term=1): - is_first_boot = !phase2_done = true - cluster_init_phase2(c) ← first boot init - phase2_done = 1 - ha->role = CTRL_PRIMARY_COMMITTING - CMD: INFO "→ CTRL_PRIMARY_COMMITTING (term=1, first_boot=1)" - - Promote vote eps → standby channels - CMD: INFO "promoted vote ep[0] → standby channel" - CMD: INFO "promoted vote ep[1] → standby channel" - - push_config_to_standbys: config blob sent to ctrl-1 and ctrl-3 - ha_send_ctrlhello_to_all_standbys: ctrlhello sent - - ha_start_primary_heartbeat: armed every 3000ms - - Schedule committing_timeout(10000ms): - deadline: T+10.865s (00:16:09.374) - - CMD: INFO "Config committed on majority (tag=4986624fd309142c, pushed=2/2)" - CMD: INFO "primary heartbeat armed (interval=3000ms)" -``` - -### T+0.915s (00:15:59.424) — ctrl-3 receives ctrlhello (still in nap) - -``` -During ctrl-3's nap (iomgr running): - ctrl-2 sends ctrlhello on promoted vote ep → arrives at ctrl-3 listener - - handle_ctrlhello: - primary_ctrl_fd = fd ← SET - CMD: INFO "primary control channel established (fd=6)" - CMD: INFO "standby ctrl-2 catch-up complete (applied_lsn=0 → committed_lsn=0)" - - handle_repl_config_push: - config saved and loaded (tag=4986624fd309142c) - - ha_arm_heartbeat_timeout_vtask: - liveness poll armed (timeout=10000ms, poll=1500ms) - - NOTE: ha->role stays at CTRL_STANDBY (set by ha_init). - voted_for_name stays at "" (cleared by BUG). - ha->term stays at 1 (set by raft_cb_persist). -``` - -### T+1.603s (00:16:00.119) — ctrl-3 nap ends - -``` -already_voted check: voted_for_name="" (EMPTY!) → NO → starts election - NOTE: primary_ctrl_fd != -1, raft.term = 1, but these aren't checked! - -ha_start_election_async: - ha->raft_c = c ← NOW non-NULL - rcfg.initial_term = ha->term = 1 ← SET TO 1! - rcfg.initial_voted_for = -1 (voted_for_name is empty) - raft_init: n_peers=3, self_idx=2, s->term=1 - raft_start_election: - s->term: 1 → 2 ← TERM=2! - s->voted_for = 2 (self) - s->votes_granted = 1 - on_persist(term=2, voted_for=2): - ha->term = 2 - ha->voted_for_name = "ctrl-3" - - CMD: INFO "→ CTRL_ELECTING" - CMD: INFO "returning" -→ cluster_run -→ sends vote requests at TERM=2 to ctrl-1 and ctrl-2 -``` - -### T+1.610s — ctrl-3's votes at term=2 arrive - -**To ctrl-1:** -``` -ctrl-1: s->term=1, role=CANDIDATE, voted_for=0(self) -raft_on_vote_request(term=2, from_idx=2): - term(2) > s->term(1) → term upgrade: - s->term = 2, voted_for = -1, role = FOLLOWER - was_leader = false (was CANDIDATE) - role=FOLLOWER → passes - voted_for=-1 → passes - on_persist(term=2, from_idx=2): - ha->term = 2 - ha->voted_for_name = "ctrl-3" (c is non-NULL for ctrl-1) -Result: GRANT -``` - -**To ctrl-2:** -``` -ctrl-2: s->term=1, role=LEADER (in COMMITTING), voted_for=1(self) -raft_on_vote_request(term=2, from_idx=2): - term(2) > s->term(1) → term upgrade: - s->term = 2, voted_for = -1, role = FOLLOWER - was_leader = true → on_become_follower! - raft_cb_become_follower: - cancel election timeout (none active) - ha->role = CTRL_STANDBY ← STEPS DOWN! - CMD: INFO "→ CTRL_STANDBY" (NOT IN LOG!) - ha_arm_heartbeat_timeout_vtask - role=FOLLOWER → passes - on_persist(term=2, from_idx=2): - result: ha->term=2, voted_for_name="ctrl-3" -Result: GRANT - -NOTE: committing timeout from term=1 is NOT cancelled! - It will fire at T+10.865s (00:16:09.374). -``` - -### T+1.615s (00:16:00.170) — ctrl-3 wins term=2 - -``` -ctrl-3: self(1) + ctrl-1(1) = 2 = quorum -raft_cb_become_leader(term=2): - ha->role = CTRL_PRIMARY_COMMITTING - CMD: INFO "→ CTRL_PRIMARY_COMMITTING (term=2, first_boot=1)" - - push_config_to_standbys: - repl_wait_majority_ack TIMEOUT (500ms) - CMD: WARN "timeout waiting for majority quorum" - CMD: WARN "Config push: timeout waiting for majority ACK" - - ha_start_primary_heartbeat(3000ms) - - Schedule committing_timeout(10000ms): - deadline: T+11.615s (00:16:10.725) -``` - -### T+5.200s (00:16:03.722) — ctrl-1 election timeout - -``` -ctrl-1's initial election timeout fires (5000ms from T+0.000s) -But ctrl-1 already stepped down to FOLLOWER at T+1.610s (saw term=2). -raft_cb_become_follower: - cluster_init_phase2 (first_boot) - ha->role = CTRL_STANDBY - CMD: INFO "→ CTRL_STANDBY (term=2, first_boot=1)" - ha_arm_heartbeat_timeout_vtask (10000ms) -``` - -### T+10.865s (00:16:09.425) — ctrl-2 committing timeout FIRES - -``` -committing_timeout_fn checks t->ha->role: - If raft.c fix worked: ha->role = CTRL_STANDBY → NO-OP (correct!) - If raft.c fix didn't work: ha->role = CTRL_PRIMARY_COMMITTING → promotes - -Actual log: ctrl-2 → CTRL_PRIMARY (term=2) - CMD: WARN "committing timeout — majority ack not received..." - CMD: INFO "→ CTRL_PRIMARY (term=2)" - -CONCLUSION: raft.c fix did NOT work in this test run, - or ctrl-2 never received ctrl-3's vote request at term=2. -``` - -### T+11.615s (00:16:10.725) — ctrl-3 committing timeout fires - -``` -committing_timeout_fn: ha->role = CTRL_PRIMARY_COMMITTING → promotes -ha_do_promote: term 2→3 -CMD: INFO "→ CTRL_PRIMARY (term=3)" -``` - -## Root Causes Identified - -### Bug 1: `ha->raft_c` is NULL during vote processing in nap - -During the stagger nap, other nodes' vote requests arrive and are processed. -`raft_cb_persist` uses `ha->raft_c` to look up candidate names, but this is -NULL until `ha_start_election_async` runs (after the nap). - -**Effect**: `ha->term` is correctly set, but `ha->voted_for_name` is CLEARED. -The `already_voted` check in myapp_run sees an empty `voted_for_name` and -starts an unnecessary election. - -**Fix**: Set `ha->raft_c = c` earlier, at startup, before iomgr starts. - -### Bug 2: Committing timeout not cancelled on step-down - -When `raft_on_vote_request` (with my fix) calls `on_become_follower` for a -node that was LEADER/COMMITTING, the committing timeout is not cancelled. -The delaymgr task still fires later. - -**Fix**: In `raft_cb_become_follower`, cancel any pending committing timeout. - -### Bug 3: `already_voted` check incomplete - -Only checks `ha->voted_for_name` — misses the case where: -- `ha->raft.term > 0` (saw a higher term from another candidate) -- `ha->primary_ctrl_fd >= 0` (ctrlhello already received from winner) - -**Fix**: Also check `raft.term > 0` and `primary_ctrl_fd >= 0`. diff --git a/.kiro/specs/ha-config-bootstrap/manual-tests.md b/.kiro/specs/ha-config-bootstrap/manual-tests.md deleted file mode 100644 index 3b24bef6..00000000 --- a/.kiro/specs/ha-config-bootstrap/manual-tests.md +++ /dev/null @@ -1,388 +0,0 @@ -# Manual Integration Tests: ha-config-bootstrap - -This document describes how to run the integration tests for tasks 11.1–11.5. -Run each test, record the output, and paste the filled **Results Template** at -the bottom back to the developer. - -## Environment - -``` -Worktree: /home/rshen/projects/xce/worktrees/persistent_ha -Build dir: /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build -Controller: src/xce.build/controller/xce_controller -CLI: src/xce.build/frontend/xcecli/xce (or /tmp/xce_static after make build-cli) -Unit tests: src/xce.build/controller/controller_unit_test - src/xce.build/lib/xcelib_test -``` - -## Build First - -```bash -# From the worktree -/xce-build # full dev build (preferred: uses nix develop + meson) - -# Or manually from the src directory: -nix develop /home/rshen/projects/xce/xce/src --command bash -c \ - "cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja" -``` - -## Unit Tests (Sanity Check Before Integration) - -```bash -cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build - -./controller/controller_unit_test 2>/dev/null | tail -5 -./lib/xcelib_test 2>/dev/null | tail -3 -``` - -Expected: all tests PASS, no FAIL lines, exit 0. - ---- - -## Test Infrastructure - -> **Design constraint:** The controller rejects an even number of peer nodes -> (`--peers` list), because Raft quorum requires an odd cluster size (1, 3, 5…) -> to guarantee a safe majority. **All tests use a 3-node cluster.** - -### Deploy via `/xce-deploy-ha` skill - -All tests use the `/xce-deploy-ha` skill which automates deployment: - -```bash -# 1. Build (if not already built) -/xce-build controller - -# 2. Deploy 3-node cluster -/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993 -``` - -This deploys to **`/tmp/xce-deploy/`** with: -- `certs/` — shared TLS cert and key (copied from `ha/data/`) -- `controller/ctrl-1/conf/` — initial config (primary only; standbys receive via push) -- `controller/ctrl-{1,2,3}/data/` — WAL, snapshots, `ha-term.bin` -- `logs/ctrl-{1,2,3}.log` — per-node log files -- `scripts/start.sh` and `scripts/cleanup.sh` — lifecycle helpers - -Nodes are named `ctrl-1`, `ctrl-2`, `ctrl-3` (index 0 = highest priority = ctrl-1). - -### Start and verify - -```bash -# Start all 3 (staggered by 0.5s) -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 & -sleep 0.5 -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 & -sleep 0.5 -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 & - -# Wait 15s for election + config replication, then verify -grep -i "CTRL_PRIMARY\|raft.*leader\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3 -find /tmp/xce-deploy/controller/ctrl-2/conf /tmp/xce-deploy/controller/ctrl-3/conf -type f | sort -``` - -### Clean restart (preserves config, wipes data) - -```bash -kill -9 $(pgrep xce_controller) 2>/dev/null || true -bash /tmp/xce-deploy/scripts/cleanup.sh -# Then restart nodes as above -``` - -**What cleanup.sh preserves:** `controller/ctrl-1/conf/*`, `$TOP/certs/` -**What cleanup.sh removes:** all `controller/ctrl-{1..N}/data/` dirs, `controller/ctrl-{2..N}/conf/*`, all `logs/*.log` - -### Manual restart with config preserved (for failover testing) - -To restart without wiping standby configs (so standbys retain `config_tag > 0`): - -```bash -kill -9 $(pgrep xce_controller) 2>/dev/null || true -rm -rf /tmp/xce-deploy/controller/ctrl-{1,2,3}/data /tmp/xce-deploy/logs/*.log -mkdir -p /tmp/xce-deploy/controller/ctrl-{1,2,3}/data -# Then restart nodes as above -``` - ---- - -## Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication - -**Requirements:** 1.1, 2.1, 2.7, 3.2, 3.5 - -**Scenario:** ctrl-1 has the initial config; ctrl-2 and ctrl-3 start with -`config_tag=0` (no config). All three start concurrently — ctrl-1 wins election -(freshest config), then pushes the config to ctrl-2 and ctrl-3 via replication. - -**Steps:** Deploy and start as described in Test Infrastructure above. - -**Expected log patterns — Election (current implementation):** -- ctrl-1: `raft_cb_become_leader` → `CTRL_PRIMARY_COMMITTING` → `CTRL_PRIMARY` -- ctrl-2/ctrl-3: `myapp_run: CTRL_STANDBY — waiting for primary to connect and push config` - -**Expected log patterns — Config replication (primary push model):** -- ctrl-1: `Config committed on majority (tag=<hex>, pushed=2/2 standbys)` -- ctrl-2/ctrl-3: `handle_repl_config_push: config saved and loaded (tag=<hex>)` -- Config files appear in both standby workdirs: - ```bash - ls /tmp/xce-deploy/controller/ctrl-2/conf/ - ls /tmp/xce-deploy/controller/ctrl-3/conf/ - # Expected: cluster.bin.<hex_tag> and current - ``` - -**Checklist:** -- [ ] Exactly ctrl-1 becomes `CTRL_PRIMARY` -- [ ] ctrl-2 and ctrl-3 become `CTRL_STANDBY` -- [ ] No crash or ASAN error -- [ ] Config replicated to ctrl-2: `cluster.bin.*` exists in `/tmp/xce-deploy/controller/ctrl-2/conf/` -- [ ] Config replicated to ctrl-3: `cluster.bin.*` exists in `/tmp/xce-deploy/controller/ctrl-3/conf/` -- [ ] Config tag on standbys matches primary's tag - ---- - -## Test 11.2 — Failover and WAL Durability - -**Requirements:** 5.6, 6.2, 5.5 - -**Prerequisites:** 3-node cluster running from Test 11.1. - -### Scenario 1: Primary crash → liveness timeout → re-election - -1. Identify which node is primary: `grep CTRL_PRIMARY /tmp/xce-deploy/logs/ctrl-*.log` -2. Kill the primary process: `kill -9 <pid>`. -3. Watch the two surviving standbys: `tail -f /tmp/xce-deploy/logs/ctrl-{2,3}.log` - -**Expected on ctrl-2 and ctrl-3 (both standbys, within ~10s of primary kill):** -``` -HA: liveness poll: no primary activity for 10s → failover -HA: primary_ctrl_fd RDHUP → CTRL_ELECTING -``` -**Expected on the winner (standby with config_tag > 0 campaigns and wins):** -``` -raft_cb_become_leader -CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY -``` - -> **Note (2026-06-03):** Failover election requires standbys to have a -> non-zero `config_tag` loaded at startup. After the initial deployment, -> standbys receive config via `handle_repl_config_push`. On subsequent -> restarts, if `cleanup.sh` wipes `ctrl-{2,3}/conf/`, the standbys restart -> with `config_tag=0` and participate as voters only (they will not campaign). -> Preserve `ctrl-{2,3}/conf/` across restarts for failover to complete. -> See test variant below in results. - -### Scenario 2: WAL commit with both standbys down → ETIMEDOUT - -1. Kill both standbys (leaving only the primary). -2. Submit a job: -```bash -/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \ - --server 127.0.0.1:9901 jobsubmit -c "hostname" -``` - -**Expected in primary log:** `repl_wait_majority_ack: timeout` — NOT a crash. - -### Scenario 3: Reconnected standby catches up - -1. Restart one of the killed standbys. -2. Watch logs on the restarted node. - -**Expected:** -``` -CtrlHelloReply from primary: committed_lsn=... -CTRL_STANDBY -``` - -**Checklist:** -- [ ] RDHUP failover detected within 1s of primary kill -- [ ] One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other) -- [ ] WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode) -- [ ] Restarted standby reconnects and enters CTRL_STANDBY - ---- - -## Test 11.3 — Unconfigured Node Bootstrap - -**Requirements:** 3.2, 3.5 - -**Scenario:** ctrl-3 starts with no local cluster config (config_tag=0); ctrl-1 and ctrl-2 already hold a valid config from Test 11.1. - -1. Start ctrl-1 and ctrl-2 first (let one become primary). -2. Ensure ctrl-3 has no conf: `rm -rf /tmp/xce-deploy/controller/ctrl-3/conf/* 2>/dev/null` (or rely on fresh deploy where only ctrl-1 has conf). -3. Start ctrl-3. - -**Expected in ctrl-3 log:** -``` -ha_start_election_async: no local config (tag=0, n_peers=3) -myapp_run: CTRL_STANDBY — waiting for primary to connect and push config -handle_repl_config_push: config saved and loaded (tag=...) -``` - -**Expected in `/tmp/xce-deploy/controller/ctrl-3/conf/`:** a `cluster.bin.<tag>` and `current` pointer file appear after primary push. - -**Checklist:** -- [ ] ctrl-3 detects config_tag=0, participates as voter only -- [ ] ctrl-3 yields election to the existing primary (fresher config) -- [ ] ctrl-3 receives config via `handle_repl_config_push` from primary -- [ ] ctrl-3 enters CTRL_STANDBY with correct config -- [ ] Config files appear in `/tmp/xce-deploy/controller/ctrl-3/conf/` after push - ---- - -## Test 11.4 — Term Durability Across Restart - -**Requirements:** 9.1, 9.2 - -**Scenario:** After election, `ha-term.bin` is written; on restart, term is restored. - -1. Run 3-node cluster from Test 11.1, wait for election to complete. -2. Check that `ha-term.bin` was written on the primary: -```bash -xxd /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin | head -4 -# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0 -``` -3. Kill ctrl-1. -4. Restart ctrl-1: `VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &` -5. Check startup log for term restore. - -**Expected in startup log:** -``` -HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin -``` -(See `cluster_init_phase1` log line) - -**Checklist:** -- [ ] `ha-term.bin` file exists in `/tmp/xce-deploy/controller/ctrl-1/data/` after election -- [ ] Magic bytes are `0xDEADBEEFCAFEBABE` (appears as `beba feca efbe adde` in `xxd` little-endian) -- [ ] Term field (bytes 8–15) is > 0 after a vote was granted -- [ ] On restart: log shows `HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin` -- [ ] Node re-joins correctly without double-voting - ---- - -## Test 11.5 — Full ASAN Build Verification - -**Requirements:** 4 - -**Steps:** - -```bash -# Run unit tests (already built with ASAN via /xce-build) -cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build - -./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)' -./lib/xcelib_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)' -``` - -**Expected output:** No lines (zero ASAN errors introduced by this feature). - -Pre-existing leaks from `test_worker_join` in controller_unit_test are acceptable — -they existed before this branch and are not new. - -```bash -# Count pre-existing vs new leaks: -./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l -``` - -**Checklist:** -- [ ] `controller_unit_test` exits 0, all tests PASS -- [ ] `xcelib_test` exits 0, all tests PASS -- [ ] Zero `AddressSanitizer` errors (use-after-free, buffer overflow) -- [ ] No new `LeakSanitizer` leaks from ha-config-bootstrap code paths - ---- - -## Results Template - -Copy, fill in, and paste back: - -``` -## Test Run Results - -Date: 2026-06-03 -Commit: 374f3bf chore(ha-config-bootstrap): gap analysis audit + fix parent task headers -Build: /xce-build controller completed successfully? YES - -### Unit Tests -- controller_unit_test: PASS (1/1, 157s) -- xcelib_test: PASS - -### 11.1 Concurrent Startup + Config Replication (3 nodes) -- Election result: PASS - - Primary elected: ctrl-1 - - Time to election: ~16s (first boot: term=1 standby → phase2 → term=2 leader → term=3 PRIMARY) - - Two standbys confirmed: YES -- Config replication result: PASS - - cluster.bin exists in /tmp/xce-deploy/controller/ctrl-2/conf/: YES (cluster.bin.4986624fd309) - - cluster.bin exists in /tmp/xce-deploy/controller/ctrl-3/conf/: YES (cluster.bin.4986624fd309) - - Config tag on standbys matches primary: YES (4986624fd309142c) -- Log excerpt: - ctrl-1: `raft_cb_become_leader → CTRL_PRIMARY_COMMITTING (term=2)` - ctrl-1: `Config committed on majority (tag=4986624fd309142c, pushed=2/2 standbys)` - ctrl-1: `ha_do_promote: → CTRL_PRIMARY (term=3)` - ctrl-2: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)` - ctrl-3: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)` - -### 11.2 Failover and WAL Durability (3 nodes) -- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS (with note) - - Time to detect liveness loss: ~11s (liveness poll timeout 10s) - - Both standbys correctly detected: `liveness poll: no primary activity → failover` - - Both standbys transitioned: `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING` - - Note: With clean-restart (configs preserved), failover election succeeds. - With cleanup.sh (wipes standby confs), standbys restart with tag=0 and - cannot campaign (voter-only mode). See additional notes. - - Log excerpt (detection): - ctrl-2: `liveness_poll_fn: no primary activity for 11s → failover` - ctrl-2: `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING` - ctrl-3: same pattern -- Scenario 2 (degraded WAL timeout, both standbys down): NOT TESTED (requires xcecli) -- Scenario 3 (reconnect one standby): NOT TESTED (covered by Test 11.1 re-run) - -### 11.3 Unconfigured Node Bootstrap (ctrl-2/ctrl-3 join ctrl-1+cluster) -- Result: PASS -- cluster.bin created in /tmp/xce-deploy/controller/ctrl-2/conf/? YES -- cluster.bin created in /tmp/xce-deploy/controller/ctrl-3/conf/? YES -- Log excerpt: - ctrl-2: `ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only` - ctrl-2: `myapp_run: CTRL_STANDBY — waiting for primary to connect and push config` - ctrl-2: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)` - ctrl-3: same pattern - -### 11.4 Term Durability -- ha-term.bin exists: YES (all 3 nodes) -- xxd output (ctrl-1, first 2 lines): - 00000000: beba feca efbe adde 0200 0000 0000 0000 ................ - 00000010: 0200 0000 0000 0000 6374 726c 2d31 0000 ........ctrl-1.. - Magic: 0xDEADBEEFCAFEBABE (little-endian) ✓ - Term: 2 ✓ - voted_for: "ctrl-1" ✓ -- Restart log (term restore line): - `cluster_init_phase1: HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin` -- Result: PASS - -### 11.5 ASAN Build -- controller_unit_test ASAN errors: 0 (only pre-existing LeakSanitizer from test_worker_join/makeInstance) -- xcelib_test ASAN errors: 0 -- Result: PASS - -### Additional Notes / Unexpected Behaviour -1. **Architecture change from original design**: The election path now uses - `ha_start_election_async` + raft library (election-iomgr-refactor), not - `am_i_primary_quorum`. Config flows via primary push (`handle_repl_config_push`) - not standby fetch (`do_one_config_fetch`). - -2. **Config tag persistence on restarts**: After initial deployment, standbys - receive config via push. On clean restart (data wipe + preserve conf), - standbys load `config_tag > 0` from disk and can campaign in failover - elections. On full cleanup (`cleanup.sh` wipes conf/), standbys restart - with `config_tag=0` and participate as voters only. - -3. **Liveness detection**: Works correctly — both standbys detect primary loss - within ~10s (liveness_poll_fn timeout). The `ha_arm_heartbeat_timeout_vtask` - delaymgr-based mechanism replaces the old watchdog pthread. - -4. **Dead code removed during this session**: `am_i_primary()`, `probe_thread()`, - `wait_any_probe()`, `ha_watchdog_thread()`, `ha_start_watchdog()`, - `ha_stop_watchdog()`, `ha_install_sigusr1()`, `ha_check_heartbeat_nocluster()`, - `start_heartbeat_listener()`, `stop_heartbeat_listener()`, and associated - `test_am_i_primary` suite. All ~630 lines removed. -``` diff --git a/.kiro/specs/ha-config-bootstrap/requirements.md b/.kiro/specs/ha-config-bootstrap/requirements.md deleted file mode 100644 index 0fb6f23e..00000000 --- a/.kiro/specs/ha-config-bootstrap/requirements.md +++ /dev/null @@ -1,151 +0,0 @@ -# Requirements: HA Config Bootstrap & Raft-Style Quorum Election - -## Introduction - -The current LSF-style priority-based controller election has three critical safety gaps: (1) no guarantee of config freshness on the elected primary, (2) false self-elections during concurrent startup when the peer listener port is not yet open, and (3) no bootstrap path for a standby promoted during genuine failover whose local config is missing. This feature replaces priority-based election with Raft-style quorum voting, adds majority-commit WAL and config replication for full CP durability, establishes persistent standby-to-primary control channels, and hardens the controller startup sequence. The system shall maintain CP semantics: consistency over availability, with zero data loss on failover. - -## Boundary Context - -- **In scope**: quorum-based controller election, pre-election config bootstrap for unconfigured nodes, listener binding before election, startup pre-flight validation (peer count, certs, vcpus), majority-commit WAL replication with entry-type differentiated durability, majority-commit config push, persistent standby-to-primary control channels with catch-up synchronization, client-side transparent retry for transient controller errors, election term and voted_for durability across restarts, observable election and commit events via structured logs -- **Out of scope**: Raft snapshot protocol (existing COW is orthogonal), joint consensus for cluster membership changes (candidate list is static at startup from `--peers`), mTLS between controllers (trusted-LAN model with server-side TLS is sufficient), config file format or storage layout changes -- **Adjacent expectations**: the existing `do_one_config_fetch()` stub must be completed for config bootstrap to function in the deferred-fetch path; static controller peer membership (no dynamic join/leave) is a hard constraint; single-node mode (quorum=1) must remain fully operational without behavioral changes - -## Requirements - -### Requirement 1: Listener and Connection Framework Before Election - -**Objective:** As a cluster operator, I want peer listener sockets to be open before election begins, so that concurrent-startup nodes can reach each other's vote handlers rather than falsely concluding peers are absent. - -#### Acceptance Criteria -1. When a controller starts, the system shall bind and listen on its peer port before initiating any election probe. -2. When a controller starts, the system shall initialize the I/O manager and virtual task manager before initiating any election probe, so that incoming vote requests are handled by the normal dispatch loop. -3. When the controller is in the startup-initializing or electing state, the system shall reject non-vote, non-config-probe requests with a not-ready error response. -4. When the controller is in standby state, the system shall reject primary-only operations with a standby error response that includes a redirect to the known primary host and port. - ---- - -### Requirement 2: Quorum-Based Controller Election - -**Objective:** As a cluster operator, I want the controller election to require a strict majority of peer votes, so that a minority partition cannot elect a primary and split-brain is prevented. - -#### Acceptance Criteria -1. When a controller enters the electing state, the system shall send vote requests to all peer controllers and require acknowledgments from a strict majority (greater than or equal to half the cluster size rounded up) before becoming primary. -2. When a controller receives a vote request from a candidate with a lower term, the system shall deny the vote. -3. When a controller receives a vote request from a candidate with a higher term, the system shall update its local term and clear any prior vote record for the previous term. -4. When a controller receives a vote request from a candidate whose config tag or WAL log sequence number is less fresh than its own, the system shall deny the vote. -5. When a peer controller is in catch-up synchronization and has not yet caught up to the primary's committed state, the system shall deny its vote in any election. -6. When a controller has already granted a vote to one candidate in a given term, the system shall deny any subsequent vote request from a different candidate in the same term. -7. When a candidate wins the vote count but any responding peer has a higher config tag or equal config tag with a higher WAL log sequence number, the system shall yield to that fresher peer and enter standby state. -8. When a candidate with no local config (config tag zero) determines that any peer has a config with a non-zero tag, the system shall refuse to become primary and exit with a fatal error. -9. When the cluster is a single node, the system shall self-elect as primary immediately with no quorum requirement. -10. When a split vote occurs where no candidate achieves a quorum, the system shall retry the election once with a randomized delay and incremented term. -11. If the second election attempt also fails to achieve a quorum, the system shall enter standby state. -12. When every responding peer has an identical config tag and WAL log sequence number as the candidate, the system shall use randomized pre-election jitter as the tiebreaker without considering priority values. - ---- - -### Requirement 3: Config Bootstrap for Unconfigured Nodes - -**Objective:** As a cluster operator, I want a controller node that starts with no local configuration to automatically discover and fetch configuration from the proven election winner after the election resolves, so that fresh nodes can join an existing cluster without manual intervention. - -#### Acceptance Criteria -1. When a controller starts with no local config (config tag zero), the system shall probe all peer controllers for their config tag before declaring candidacy. -2. When a controller with no local config detects that at least one peer has a non-zero config tag, the system shall participate in the election as a voter only and defer the config fetch until the election winner is known. -3. When a controller with no local config and all reachable peers also have no config, the system shall proceed to election as a normal candidate with config tag zero. -4. When a controller with no local config cannot reach any peer, the system shall retry with exponential backoff (starting at 500 milliseconds, doubling up to 10 seconds) until a peer responds or the process is killed by an administrator. -5. When the election resolves and a deferred config fetch is pending, the system shall fetch the full configuration from the proven election winner. - ---- - -### Requirement 4: Startup Validation and Safety Checks - -**Objective:** As a cluster operator, I want the controller to validate its startup prerequisites before initialization, so that misconfigured deployments are detected immediately with clear, actionable error messages. - -#### Acceptance Criteria -1. When the `--peers` list contains an even number greater than 2, the system shall reject the configuration at startup with an error message explaining that an odd number of controllers is required for majority formation. -2. When the TLS certificate or key file is missing or unreadable at the expected path, the system shall reject startup with an error message that includes the full file path. -3. When the `--vcpus` value is less than 1, the system shall reject startup with an error message. -4. When the cluster config file contains a `vcpus` key, the system shall log a deprecation warning directing operators to use the `--vcpus` CLI parameter instead. - ---- - -### Requirement 5: Majority-Commit WAL Replication - -**Objective:** As a cluster operator, I want WAL entries for critical state changes to be committed only after a majority of controllers have durably written them, so that zero data loss is guaranteed on failover. - -#### Acceptance Criteria -1. When the primary writes a job submission, job state transition, or queue change WAL entry, the system shall not acknowledge the write to the caller until a majority of controllers (including the primary itself) have confirmed durable write of that entry. -2. When the primary writes a host state or checkpoint reference WAL entry, the system shall acknowledge the write without waiting for standby confirmation. -3. When a standby controller receives a WAL entry from the primary, the system shall write it to durable storage and reply with an acknowledgment carrying the written log sequence number. -4. When a standby controller falls more than the configurable maximum pending acknowledgments behind the primary, the system shall remove that standby from the active replication set and trigger catch-up reconnection. -5. If the primary cannot obtain majority acknowledgment within the configured commit timeout, the system shall return a timeout error to the caller. -6. When a standby controller is promoted to primary, the system shall truncate any WAL entries above the primary's last known committed log sequence number before replaying the WAL. -7. When a standby controller reconnects to a primary via catch-up, the system shall truncate any uncommitted WAL entries above the primary's committed log sequence number before requesting WAL replay. -8. When the cluster is a single node, the system shall consider the local write alone as satisfying the majority requirement. - ---- - -### Requirement 6: Persistent Standby-to-Primary Control Channel - -**Objective:** As a cluster operator, I want standby controllers to maintain a persistent connection to the primary, so that failover detection is sub-second and WAL replication avoids per-entry connection overhead. - -#### Acceptance Criteria -1. When a standby controller enters standby state after election, the system shall establish and maintain a persistent TLS connection to the elected primary. -2. When the persistent control channel disconnects, the system shall detect the failure immediately and trigger a new election. -3. When heartbeat messages are received on the persistent control channel, the system shall reset the heartbeat timeout and defer election. -4. When no heartbeat is received within the election timeout plus randomized jitter on the persistent control channel, the system shall trigger a new election. -5. When a standby with an outdated config tag or applied log sequence number connects to the primary, the system shall synchronize all missing state (config snapshot and WAL entries) before admitting the standby to the active replication set. -6. When a standby's applied log sequence number is older than the primary's oldest retained WAL segment, the system shall transfer a full state snapshot before streaming WAL entries. -7. When a standby connects with a config tag that differs from the primary's, the system shall push the primary's current configuration after catch-up synchronization completes. -8. When a standby wins an election and becomes primary, the system shall close its outbound connection to the old primary and begin accepting inbound control channel connections from other standbys. - ---- - -### Requirement 7: Client-Side Transparent Retry for Transient Controller Errors - -**Objective:** As a daemon developer, I want libxce to transparently retry requests when the controller returns transient errors, so that worker, scheduler, and API server code does not need ad-hoc retry loops. - -#### Acceptance Criteria -1. When libxce receives a not-ready error from a controller, the system shall retry the request with exponential backoff against the same endpoint. -2. When libxce receives a standby error that includes a redirect to the current primary, the system shall reconnect to the redirected host and port and retry the request. -3. When libxce receives a TCP-level connection refused or timeout error, the system shall reconnect and retry the request. -4. When libxce receives a non-retriable error (authentication failure, invalid request, permission denied, not found), the system shall surface the error to the caller immediately without retry. -5. The system shall provide configurable retry parameters (maximum retries, initial delay, maximum delay, backoff multiplier) with safe production defaults. -6. When multiple clients retry simultaneously after a failover, the system shall stagger their retry attempts through exponential backoff to prevent thundering herd. - ---- - -### Requirement 8: Majority-Commit Config Replication - -**Objective:** As a cluster operator, I want configuration changes pushed to the cluster to be committed only after a majority of controllers have durably written them, so that operator changes are not silently lost on primary failover. - -#### Acceptance Criteria -1. When the primary pushes a configuration upgrade, the system shall not acknowledge the change as committed until a majority of controllers (including the primary itself) have confirmed durable write. -2. When a standby controller receives a configuration upgrade from the primary, the system shall write the config blob to durable storage and reply with an acknowledgment. -3. If the primary cannot obtain majority acknowledgment for a configuration upgrade within the configured commit timeout, the system shall return a timeout error to the caller. -4. When a standby is promoted to primary, the system shall serve only configurations up to the last known committed config tag received via heartbeat from the previous primary. -5. When the cluster is a single node, the system shall consider the local write alone as satisfying the majority requirement for config upgrades. - ---- - -### Requirement 9: Election State Durability Across Restarts - -**Objective:** As a cluster operator, I want the controller's election term and vote record to survive process restarts, so that a restarting node cannot double-vote in a term it already participated in. - -#### Acceptance Criteria -1. When a controller's term is incremented or its vote is granted to a candidate, the system shall durably persist the current term and voted-for candidate name before responding to the vote request. -2. When a controller starts up, the system shall restore the most recently persisted term and voted-for record from durable storage before participating in any election. -3. When a persisted term file does not exist at startup (first boot), the system shall initialize the term to zero with no prior vote recorded. - ---- - -### Requirement 10: Election and Replication Observability - -**Objective:** As a cluster operator, I want election results, vote tallies, config bootstrap operations, WAL commit quorum status, and config push quorum status to appear in structured logs at INFO or WARN severity, so that cluster health and HA behavior can be monitored without external tooling. - -#### Acceptance Criteria -1. When an election completes, the system shall log at INFO level the outcome (primary won or standby), the term, the number of votes granted, and the quorum size. -2. When a config bootstrap probe determines the config situation (all peers tag zero, config exists on peers, or no peer reachable), the system shall log the finding at INFO or WARN level. -3. When a WAL commit achieves quorum, the system shall log at INFO level; when a WAL commit times out, the system shall log at WARN level. -4. When a configuration upgrade achieves quorum, the system shall log at INFO level; when a configuration upgrade times out, the system shall log at WARN level. -5. When a controller transitions between the init, electing, primary, and standby states, the system shall log the transition at INFO level. diff --git a/.kiro/specs/ha-config-bootstrap/research.md b/.kiro/specs/ha-config-bootstrap/research.md deleted file mode 100644 index bdb0e541..00000000 --- a/.kiro/specs/ha-config-bootstrap/research.md +++ /dev/null @@ -1,794 +0,0 @@ ---- - -# Cross-Spec Gap Audit (2026-06-03) - -> Triggered by: `/kiro-validate-gap ha-config-bootstrap election-iomgr-refactor, check am_i_primary() still be used?` -> Scope: confirm cross-spec coverage status; answer `am_i_primary()` question definitively. - -## Analysis Summary - -- **ha-config-bootstrap** remains at ~95% (all unit tasks done; tasks 11.1–11.5 are manual integration tests). -- **election-iomgr-refactor** is 100% complete and already marked `implemented`. -- The two specs are consistent and non-overlapping in implementation: election-iomgr-refactor superseded the blocking-pthread election path defined in ha-config-bootstrap, keeping all requirements satisfied through a different (better) implementation path. -- `am_i_primary()` question: **answered — it is dead code, safe to delete, but not harmful to leave.** - -## `am_i_primary()` Status - -### What it is - -`am_i_primary()` (ha.c:869) is the original **LSF priority-based election probe** — it spawns one pthread per peer and self-promotes if no higher-priority peer responds before a deadline. It was the pre-Raft election mechanism. - -### Is it still called? - -**No.** Confirmed by grep: - -- **`myapp.c`**: comment at line 48 explicitly says `am_i_primary_quorum` (the intermediate Raft replacement) was **removed** in Req 9.3. `am_i_primary()` is never referenced in myapp.c. -- **`cluster.c` line 1015**: The line `/* Determine initial role based on am_i_primary() result set by myapp_run */` is a **stale comment** — the actual code underneath calls `ha_is_primary(&c->ha)` (the atomic role check), not `am_i_primary()`. The comment is a lie left over from the old startup sequence. -- **`ha.c`**: The function is defined but has no callers in production code. The only cross-file reference to `am_i_primary` (not `am_i_primary_quorum`) in non-test code is the cluster.c comment above. - -### What the tests use - -- `test_ha_role.c` (lines 241–396): three test cases test `am_i_primary()` directly — these are legacy P1-6 tests covering the old priority-based election. They still pass because the function body is intact. -- `test_ha_bootstrap_support.c` and `test_ha_election.c`: both define stubs for `am_i_primary_quorum` (the intermediate function, not `am_i_primary`), used to isolate the quorum election logic under test. - -### Decision: can `am_i_primary()` be deleted? - -**Yes — safe to delete in a cleanup pass.** However: -- The three test cases in `test_ha_role.c` would also need to be deleted. -- The stale comment in cluster.c line 1015 should be updated to say "role set by ha_elect_evaluate_and_signal via vtask". -- `start_heartbeat_listener()` / `stop_heartbeat_listener()` stubs (ha.c:951-960) referenced in the same comment block can also be cleaned up. - -**Recommendation**: Leave `am_i_primary()` as-is during active HA development. Mark as cleanup tech debt. Its presence does not affect correctness, ASAN, or test pass rate. - -## Requirement Coverage Cross-Check - -The election-iomgr-refactor implemented `ha_start_election_async` which replaced both `am_i_primary()` and `am_i_primary_quorum()`. The ha-config-bootstrap tasks 4.3 and 9.4 originally called `am_i_primary_quorum()` from myapp.c — these are now superseded by `ha_start_election_async`. Both specs' requirements are fully met by the current codebase. - -No new code needed for either spec. - ---- - -# Gap Analysis — Completion Audit (2026-06-02) - -> Analysis date: 2026-06-02. Phase: `tasks-generated`. All leaf implementation tasks `[x]`. -> Purpose: verify actual codebase coverage of all 10 requirements; identify -> what remains before the spec can be marked `implemented`. - -## Analysis Summary - -- **Scope**: 10 requirements across 8 implementation changes (A–H), covering - Raft-style quorum election, majority-commit WAL + config replication, persistent - control channels, config bootstrap, term durability, and client-side retry. -- **Implementation status**: **~95% complete**. All 22 leaf subtasks (1.1–10.2) - are implemented and verified. All 9 requirements have implementation assets in - the live codebase (verified via grep audit). The parent group headers in - tasks.md show `[ ]` only because they are section labels, not actionable tasks. -- **Remaining work**: Integration tests 11.1–11.5 (manual cluster validation). -- **No design changes needed**: The implementation already follows the approved - design; `xce_request_with_retry`, `repl_wait_majority_ack`, `ha_find_and_connect_to_primary`, - `ha_persist_term`, `bootstrap_config_if_needed`, and all 9 capnp extensions are - in the live codebase. -- **Recommendation**: Run the 5 integration tests from `manual-tests.md` - against a 3-node cluster, then mark the spec `implemented`. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **ha-config-bootstrap** is **~95% complete**. All 10 requirements have verified implementation assets. All 22 leaf subtasks (1.1–10.2) are done. -- **Remaining work**: Tasks 11.1–11.5 (manual integration tests documented in `manual-tests.md`). -- **Cross-spec verification**: `election-iomgr-refactor` superseded the blocking-pthread election path while preserving all ha-config-bootstrap requirements. The two specs are consistent. `ha-repl-hardening` extends Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence — no conflicts. -- **Cleanup debt**: `am_i_primary()` (ha.c:869), `ha_wait_for_heartbeat_or_timeout`, and `start_heartbeat_listener()`/`stop_heartbeat_listener()` stubs are dead code — safe to delete in a dedicated cleanup commit (tracked in election-iomgr-refactor/research.md). - -## Requirement-to-Asset Map — Final State - -| Requirement | Verified Key Assets | Status | -|-------------|-------------------|--------| -| Req 1: FSM init + dispatch gate | `CTRL_INIT`, `CTRL_ELECTING` in ha.h:59-60; `ha_dispatch_gate` in ha.c; `ha_wait_for_heartbeat_or_timeout` | ✅ Implemented | -| Req 2: Quorum election | `ha_elect_evaluate_and_signal` (ha.c:1604); `am_i_primary_quorum` replaced by `raft_start_election`; `handle_request_vote` term/freshness gate | ✅ Implemented | -| Req 3: Config bootstrap | `bootstrap_config_if_needed`, `xce_probe_config_tag`, `defer_config_fetch` — all in ha.c/config_repl.c | ✅ Implemented | -| Req 4: Contiguous WAL + ASAN | WAL schema capnp, `persist_*` callbacks, `SSL_CTX_free` fix | ✅ Implemented | -| Req 5: Majority-commit WAL | `repl_broadcast`, `repl_wait_majority_ack`, `committed_lsn` in persist_t, `persist_truncate_wal_above` | ✅ Implemented | -| Req 6: Persistent control channel | `ha_find_and_connect_to_primary`, extended ctrlhello handler, `EPOLLRDHUP` failover, `ha_on_primary_ctrl_fd_rdhup` | ✅ Implemented | -| Req 7: State sync | `apply_wal_entry`, `repl_start_catchup/complete_catchup`, ctrlhello + WAL catch-up flow | ✅ Implemented | -| Req 8: Majority-commit config | `OP_upgradeConfigAck` handler, `committed_config_tag`, `ha_repl_ack_received` | ✅ Implemented | -| Req 9: Voted-for persistence | `ha_persist_term` (ha.c:1323); `ha_init` restore path for `ha-term.bin` | ✅ Implemented | -| Req 10: Client-side retry | `xce_request_with_retry` (request.c:1664); `xce_retry_config_t` | ✅ Implemented | - -## Remaining Gaps - -### Gap 1: Tasks 11.1–11.5 — Integration validation (IP = In Progress via manual-tests.md) - -The `manual-tests.md` file contains step-by-step procedures for all 5 -integration tests. They require a running 3-node cluster: - -| Test | What it validates | Status | -|------|-------------------|--------| -| 11.1 | Concurrent startup + config-fresh election + config replication | Manual test doc written; not yet run | -| 11.2 | Failover (RDHUP → promote → WAL survived); WAL commit under partition | Manual test doc written; not yet run | -| 11.3 | Unconfigured node bootstrap (`config_tag=0`) | Manual test doc written; not yet run | -| 11.4 | Config upgrade majority-commit; term durability across restart | Manual test doc written; not yet run | -| 11.5 | Full ASAN build verification | Manual test doc written; not yet run | - -**Note**: These are manual cluster tests by design. The automated hook-based -unit tests in `testsuite/` (tests in `test_ha_bootstrap.c`, `test_ha_durability.c`, -`test_p1_integration.c`, `test_p2_integration.c`, `test_p3_hardening.c`, -`test_p3_manual_failover.c`) cover the in-process logic. Tasks 11.1–11.5 -validate end-to-end process behavior that the hook seams cannot reach. - -### Gap 2: Parent group headers in tasks.md show `[ ]` (Cosmetic) - -Tasks 1., 2., 3., 4., 5., 6., 7., 8., 9., 10. are section grouping -labels — all their child subtasks are `[x]`. The parent headers can be marked -`[x]` for clarity. - -### Constraint: `xce_request_with_retry` not in lib header (known limitation) - -The function lives in `src/lib/request.c:1664` but is not declared in -`src/lib/include/xce/request.h`. Callers must add `extern int xce_request_with_retry(...)` -locally. Intentional — the libxce `request.h` only exposes the public API; -internal retry logic is a controller-side concern. - -## Effort and Risk Assessment - -| Item | Effort | Risk | Justification | -|------|--------|------|---------------| -| Tasks 11.1–11.5 (integration) | M (1–2 days) | Low | Procedures documented; run cluster + verify logs | -| Fix parent checkboxes | S (1 commit) | None | Cosmetic | - -## Conclusion - -The `ha-config-bootstrap` spec is **functionally complete** — all 10 -requirements are implemented, tested, and verified at the unit level. -The 5 remaining integration tests (11.1–11.5) are manual cluster validations -that exercise the full system behavior. The `manual-tests.md` file contains -all the procedures. Once these are run, the spec can be marked `implemented` -and closed. - -*Generated: 2026-05-27. Branch: persistent_ha.* - ---- - -## Analysis Summary - -- **Scope**: 10 requirements across 8 distinct changes (A–H). The codebase has strong foundational infrastructure (iomgr, repl slot management, WAL, SSL) but the HA election core, quorum commit, and control channel are mostly absent or stubbed. -- **Largest gaps**: Raft-style quorum election (Req 2) and persistent control channel with full state sync (Req 6) are full rewrites/additions with no current equivalent logic; majority-commit WAL (Req 5) requires implementing the two stub functions at the heart of replication. -- **Strong reusable assets**: `repl_add_standby/remove/start_catchup/complete_catchup/needs_snapshot` are real implementations; the SSL connect pattern (`connect_ssl_raw`), xorshift64 PRNG, `ha_do_promote/demote`, and iomgr epoll machinery are all directly reusable. -- **Risk concentration**: Req 2 and Req 6 carry High risk; Req 5 carries High risk due to `committed_lsn` cross-cutting `persist_t`, `repl.c`, and `ha_state_t`. The remaining requirements are Medium or Low. -- **Recommended approach**: Hybrid (Option A) — implement changes A→D→B→C→E→F→G→H in dependency order, writing C tests first per TDD, one atomic commit per change. - ---- - -## 1. Current State Investigation - -### 1.1 HA Core (ha.h / ha.c) - -**`ctrl_role_t` enum** — only two values: `CTRL_PRIMARY = 0`, `CTRL_STANDBY = 1`. -Missing: `CTRL_INIT`, `CTRL_ELECTING` (required by Req 1). - -**`ha_state_t`** — current fields of interest: - -| Field | Status | Notes | -|---|---|---| -| `term` (_Atomic uint64_t) | Exists | Used for fencing | -| `role` (_Atomic int32_t) | Exists | Cast to ctrl_role_t | -| `my_priority` | Exists | LSF priority index — kept for compat but unused in Raft | -| `applied_lsn` (_Atomic uint64_t) | Exists | Reusable for vote freshness check | -| `primary_host[XCE_MAXNAME_LEN]` | Exists | Reusable | -| `primary_port` | Exists | Reusable | -| `last_heartbeat` (struct timespec) | Exists | Replaces with `EPOLLRDHUP` path in Req 6 | -| `in_catch_up` (_Atomic int) | Exists | Reusable for Clarification 6 vote exclusion | -| `jitter_seed` (uint64_t) | Exists | xorshift64, directly reusable | -| `ssl_ctx` (void*) | Exists | Reusable | -| `standby_endpoints` (zarray_t*) | Exists | Superseded by per-standby fd table in Req 6 | -| `voted_for_term` (uint64_t) | **MISSING** | Req 9 | -| `voted_for_name` (char[]) | **MISSING** | Req 9 | -| `primary_ctrl_fd` (int) | **MISSING** | Req 6 | -| `defer_config_fetch` (bool) | **MISSING** | Req 3 | -| `elect_ctx` (_Atomic ptr) | **MISSING** | Req 2 | -| `last_known_committed_lsn` (_Atomic uint64_t) | **MISSING** | Req 5 | -| `heartbeat_seen` (sem_t) | **MISSING** | Req 1 | - -**`ha_config_t`** — missing: `commit_timeout_ms`. All election fields present (election_timeout_ms, jitter_max_ms, etc.) but values tuned for LSF model. - -**`am_i_primary()`** — LSF priority probe via per-peer pthreads. No quorum, no config_tag/LSN in vote, no term/voted_for persistence. Must be replaced entirely with `am_i_primary_quorum()`. - -**`ha_check_priority_stepdown()`** — permanently unreachable once term > 0; exists only for the term=0 simultaneous-boot race. Must be retired (Req 2). - -**`ha_check_heartbeat_nocluster()`** — timestamp polling with configurable timeout. Replaced by `EPOLLRDHUP` fast path + heartbeat timeout vtask (Req 6). Function itself can be deleted. - -**`ha_watchdog_thread()`** — sets `failover_requested` but cannot call `ha_promote_to_primary()` (no `cluster_t`). Partially functional stub. Replaced by vtask-based heartbeat monitor (Req 6). - -**`start_heartbeat_listener()` / `stop_heartbeat_listener()`** — no-op stubs with comment "full implementation in cluster.c Task 7". **Missing** (Req 1). - -**`ha_fini()`** — missing `SSL_CTX_free(ha->ssl_ctx)`. ASAN-visible leak (~5 KB). - -### 1.2 Startup Sequence (myapp.c / main.c) - -**myapp.c current order**: -``` -cluster_init_phase1() -initialize_openssl() + create_ssl_context() -am_i_primary() ← election here -load_primary_config() / fetch_config_from_primary() -cluster_init_phase2() ← iomgr, vtaskmgr, bind+listen inside -cluster_start() ← more network init -cluster_run() -``` - -**Bind+listen** happens inside `cluster_init_phase2()` or `cluster_start()` — **after** election. Gap: concurrent-startup nodes probe an unbound port (Req 1). - -**No `bootstrap_config_if_needed()`** — standby path goes straight to `fetch_config_from_primary()` post-election without checking whether local config_tag is 0 or whether peers have config. Gap (Req 3). - -**main.c**: -- `--vcpus N` arg: **MISSING**. vcpus comes from cluster config read inside phase2. This creates a dependency that prevents vtaskmgr from running before election (Req 1, 4). -- Peer count validation: **MISSING**. Even-N peer lists accepted silently (Req 4). -- Cert pre-flight `access()` check: **MISSING**. Cert paths computed but SSL context created later in myapp.c, not with early-exit error in main.c (Req 4). - -### 1.3 Replication (repl.c / repl.h) - -| Function | Status | Notes | -|---|---|---| -| `repl_init / fini` | Real | Lifecycle ok | -| `repl_add_standby` | Real | Slot management implemented | -| `repl_remove_standby` | Real | Implemented | -| `repl_start_catchup` | Real | Sets in_catch_up=1 | -| `repl_complete_catchup` | Real | Clears in_catch_up, broadcasts cond | -| `repl_needs_snapshot` | Real | Compares standby_lsn vs oldest_wal_lsn | -| `repl_on_ack` | Real | Updates acked_lsn, signals cond | -| `repl_trim_lsn` | Real | WAL trim guard implemented | -| `repl_broadcast` | **STUB** | Returns 0 without sending anything (Req 5, 8) | -| `repl_wait_ack` | Partial | Waits for ANY non-catch-up standby, not majority (Req 5) | -| `repl_wait_majority_ack` | **MISSING** | Req 5, 8 | - -**`standby_conn_t`** — missing `pending_ack_count` for backpressure (Req 5). - -### 1.4 Persistence (persist.h / persist.c) - -- `current_lsn` and `checkpoint_lsn` exist. **`committed_lsn`** — **MISSING** (Req 5). -- `persist_wal_sync_append()` uses `repl_wait_ack()` (any-standby) — must upgrade to `repl_wait_majority_ack()`. -- `persist_truncate_wal_above()` — **MISSING** (Req 5, 6). -- `persist_config_t.sync_timeout_ms` exists — analogous to `commit_timeout_ms` needed for majority-commit. - -### 1.5 Config Replication (config_repl.c) - -- `do_one_config_fetch()` — **wired** (recent commit `2aae45f` on branch: "wire do_one_config_fetch() — full SSL roundtrip"). Not a stub. -- `bootstrap_config_if_needed()` — **MISSING** (Req 3). -- `xce_probe_config_tag()` — **MISSING**. New lightweight RPC needed (Req 3). -- `push_config_to_standbys()` — declared in config_repl.h; likely fire-and-forget (no majority wait). **Gap**: must upgrade to majority-commit (Req 8). -- `OP_probeConfigTag` Cap'n Proto schema — **MISSING** (Req 3). -- `OP_upgradeConfigAck` Cap'n Proto schema — **MISSING** (Req 8). -- `committed_config_tag` — **MISSING** (Req 8). - -### 1.6 Control Channel (ha.c / cluster.c) - -- No persistent standby-to-primary TCP connection. Each config fetch and WAL entry opens a new SSL connection via `connect_ssl_raw()`. -- `ha_find_and_connect_to_primary()` — **MISSING** (Req 6). -- Extended `OP_ctrlHello` with applied_lsn, config_tag — **MISSING** (Req 6). -- `OP_snapshotTransfer`, `OP_walCatchupEntry` — **MISSING** (Req 6). -- `handle_ctrlhello()` extended handler — **MISSING** (Req 6). -- Three-tier primary discovery — **MISSING** (Req 6). -- EPOLLRDHUP failover detection — **MISSING** (Req 6). -- Heartbeat timeout vtask replacing watchdog thread — **MISSING** (Req 6). - -### 1.7 Election State Durability - -- `ha-term.bin` file — **does not exist**. -- `ha_persist_term()` — **MISSING** (Req 9). -- `ha_init()` zeros term and has no `ha-term.bin` read path. - -### 1.8 libxce Client Retry (src/lib/) - -- `ha_should_redirect()` / `ha_no_primary_known()` exist in ha.c and are used by join.c. -- No transparent retry loop for `XCE_ERR_NOT_READY`, `XCE_ERR_STANDBY`, `ECONNREFUSED`. -- `xce_retry_config_t` — **MISSING** (Req 7). -- `redirect_host` / `redirect_port` fields in `XCE_ERR_STANDBY` error reply — **MISSING** (Req 7). - ---- - -## 2. Requirement-to-Asset Map - -| Req | Key Gap | Key Reusable Asset | Gap Tag | -|---|---|---|---| -| 1 — Listener before election | `CTRL_INIT`/`CTRL_ELECTING` enum values; pre-dispatch gate in vtaskmgr; extract bind+listen from phase2; `ha_wait_for_heartbeat_or_timeout()`; `heartbeat_seen` semaphore | iomgr epoll, vtaskmgr dispatch switch, xorshift64 jitter | **Missing** | -| 2 — Quorum election | Entire `am_i_primary_quorum()` and vote protocol; `ha_vote_entry_t`, `ha_elect_ctx_t`; `handle_request_vote()` / `_reply()`; `ha_elect_evaluate_and_signal()`; OP_requestVote capnp schema | `ha_do_promote/demote`, jitter PRNG, `applied_lsn`, iomgr async connect | **Missing** | -| 3 — Config bootstrap | `bootstrap_config_if_needed()`; `xce_probe_config_tag()`; OP_probeConfigTag capnp; `defer_config_fetch` flag in ha_state_t | `do_one_config_fetch()` (wired), `connect_ssl_raw()` | **Missing** | -| 4 — Startup validation | `--vcpus N` CLI arg; even-N peer count check; cert pre-flight `access()`; `SSL_CTX_free` ASAN fix | argp option parsing pattern in main.c | **Missing** | -| 5 — Majority-commit WAL | `repl_broadcast()` implementation; `repl_wait_majority_ack()`; `committed_lsn` in persist_t; `persist_truncate_wal_above()`; `committed_lsn` in heartbeat; `last_known_committed_lsn` in ha_state_t | `repl_wait_ack()` logic, `standby_conn_t`, `repl_start/complete_catchup`, WAL sync path | **Missing** / **Stub** | -| 6 — Persistent channel | Persistent TCP fd per standby; `ha_find_and_connect_to_primary()`; EPOLLRDHUP failover; extended OP_ctrlHello; OP_snapshotTransfer; OP_walCatchupEntry; WAL streaming loop; heartbeat vtask; three-tier discovery | iomgr EPOLLIN/EPOLLRDHUP, `repl_needs_snapshot()`, `repl_start/complete_catchup()` | **Missing** | -| 7 — Client retry | Retry loop in libxce; `xce_retry_config_t`; redirect fields in error reply | `ha_should_redirect()`, `ha_no_primary_known()` | **Missing** | -| 8 — Majority-commit config | `repl_wait_majority_ack()` (shared with Req 5); OP_upgradeConfigAck; `committed_config_tag`; heartbeat piggybacking | `push_config_to_standbys()` (fire-and-forget → upgrade) | **Missing** | -| 9 — State durability | `ha_persist_term()`; `ha-term.bin` read in `ha_init()`; `voted_for_term`/`voted_for_name` in ha_state_t | Atomic write+rename pattern (existing in config_repl.c) | **Missing** | -| 10 — Observability | Structured log statements throughout all new code paths | Existing INFO/WARN/ERROR macros (xcelog.h) | **Missing** (alongside) | - ---- - -## 3. Implementation Approach Options - -### Option A: Change-at-a-time (recommended) - -Implement changes in dependency order: D → A → B → C → E → H → F → G → (observability woven in). - -**Rationale for ordering**: -1. D first: `--vcpus N` removes vtaskmgr's config dependency, unblocking A. -2. A before B: listener must be open before election dispatch works. -3. B before C: config bootstrap needs the proven election winner from quorum election. -4. E and H together: both use `repl_broadcast()` + `repl_wait_majority_ack()`; implement infrastructure once. -5. F after E: persistent channel carries WAL entries; broadcast must send over that channel. -6. G last: retry makes sense only after the controller states it retries against are implemented. - -**Files touched per change**: -- D: `main.c`, `cluster.c` (remove vcpus from config), `ha.c` (`SSL_CTX_free`) -- A: `ha.h` (enum + semaphore + fields), `myapp.c` (startup reorder), `cluster.c` (extract listener), `ha.c` (wait function, dispatch gate) -- B: `ha.h` (new types), `ha.c` (replace `am_i_primary`), capnp schema, `cluster.c` (dispatch handler) -- C: `config_repl.c` (`bootstrap_config_if_needed`, `xce_probe_config_tag`), capnp schema, `myapp.c` (integration) -- E: `repl.c` (`repl_broadcast`, `repl_wait_majority_ack`), `persist.h`/`persist.c` (`committed_lsn`, `persist_truncate_wal_above`), `ha.h`/`ha.c` (`last_known_committed_lsn`) -- H: `config_repl.c` (upgrade majority-commit), capnp schema, `ha.h` (`committed_config_tag`) -- F: `ha.c`/`cluster.c` (channel connect, extended hello, streaming, EPOLLRDHUP) -- G: `src/lib/` (retry loop, xce_retry_config_t, redirect fields) - -**Trade-offs**: -- ✅ Each atomic commit is testable before the next starts -- ✅ Follows existing TDD convention (failing test first) -- ❌ Long sequential dependency chain; F and G can only start late - -### Option B: Infrastructure-first parallel tracks - -Track 1 (election+bootstrap): D → A → B → C -Track 2 (replication): E → H -Join (channel): F using Track 1 + Track 2 artifacts -Final: G - -**Trade-offs**: -- ✅ Track 1 and Track 2 are mostly independent until F -- ❌ Track 2 cannot be fully validated without Track 1 (no quorum without election) -- ❌ More coordination overhead between developers on shared ha.h - -### Option C: Stub-forward then wire - -Add all new stubs with `-ENOSYS` first, then wire each one. Matches the existing codebase pattern (many stubs already present). - -**Trade-offs**: -- ✅ Compiles throughout; no long broken-build windows -- ❌ Stub-first delays real testability; TDD mandate disfavors this - -**Recommendation**: Option A. The dependency chain is well-defined and single-developer serial implementation is the realistic mode for this C codebase. - ---- - -## 4. Effort and Risk Summary - -| Req | Effort | Risk | Justification | -|---|---|---|---| -| 1 — Listener before election | M (3–5d) | Medium | Refactor startup order; must remove vtaskmgr/config dependency first | -| 2 — Quorum election | XL (2+ wks) | High | Full algorithm replacement; new wire protocol; FSM extension; concurrent dispatch | -| 3 — Config bootstrap | M (3–5d) | Medium | New function + probe RPC; depends on do_one_config_fetch (wired) | -| 4 — Startup validation | S (1d) | Low | Additive CLI + validation; ASAN one-liner | -| 5 — Majority-commit WAL | L (1–2 wks) | High | `committed_lsn` cross-cuts persist+repl+ha; `repl_broadcast` must become real | -| 6 — Persistent channel | XL (2+ wks) | High | New connection lifecycle; full state sync protocol; 3-tier discovery | -| 7 — Client retry | M (3d) | Low | Well-understood retry pattern; additive libxce change | -| 8 — Majority-commit config | M (3d) | Medium | Shares Req 5 infrastructure; mainly wiring push_config_to_standbys | -| 9 — State durability | S (1d) | Low | File write+read; extends ha_init | -| 10 — Observability | S (alongside) | Low | Log statements woven into each change | - -**Total estimated effort**: 6–9 weeks for one engineer implementing with full TDD. - -**High-risk dependencies**: -- Req 2 must complete before Req 3 can be integrated end-to-end. -- Req 5 (`repl_broadcast` implementation) must complete before Req 8 and before Req 6 (WAL streaming over the channel). -- Req 4 (vcpus CLI) must complete before Req 1 (vtaskmgr before election). - ---- - -## 5. Key Reusable Assets - -| Asset | Location | Reuse in | -|---|---|---| -| `repl_add/remove_standby` | repl.c | Req 6 (channel registration) | -| `repl_start/complete_catchup` | repl.c | Req 6 (state sync protocol) | -| `repl_needs_snapshot` | repl.c | Req 6 (snapshot decision) | -| `repl_on_ack` | repl.c | Req 5 (majority-commit ack handler) | -| `repl_trim_lsn` | repl.c | Req 5 (WAL trim with committed_lsn) | -| `connect_ssl_raw()` | netio.c/config_repl.c | Req 3 (probe RPC) | -| `do_one_config_fetch()` | config_repl.c | Req 3 (deferred fetch) | -| `xorshift64_next + ha_seed_jitter` | ha.h / ha.c | Req 1, 2 (jitter) | -| `ha_do_promote / ha_do_demote` | ha.c | Req 2, 6 (role transitions) | -| `ha_should_redirect / ha_no_primary_known` | ha.c | Req 7 (retry redirect) | -| iomgr epoll + EPOLLIN/EPOLLRDHUP | iomgr (cluster.c) | Req 1, 6 | -| Atomic write+rename pattern | config_repl.c | Req 9 (ha-term.bin) | -| INFO/WARN/ERROR macros | xcelog.h | Req 10 | - ---- - -## 6. Constraints for Design Phase - -- **C gnu17, no C++, no new external dependencies**. -- ASAN build (`nix build .#xcecore_debug`) must pass after every change. -- **Static peer membership**: `--peers` list is fixed at startup; no dynamic join/leave of controller peers. Quorum denominator is fixed for cluster lifetime. -- **TDD mandatory**: failing test in `src/controller/testsuite/` before each implementation change. -- `make test` must pass without a live server at every commit. -- `commit_timeout_ms` must be configurable (default 5000ms). -- **Single-node mode** (no `--peers`): quorum=1 everywhere; all new majority-commit paths must short-circuit to self-only. -- **WAL write stall window** (operational constraint, not a bug): during standby catch-up after election, `repl_wait_majority_ack()` will time out on every write until at least one standby completes catch-up. This is correct CP behavior; operators must be warned in docs. - ---- - -## 7. Research Needed (Design Phase) - -- **vtaskmgr API for pre-election dispatch**: confirm that `vtaskmgr_init(vcpus)` can be called before `cluster_config` is loaded (after removing vcpus from config struct). Verify no hidden config dependency. -- **iomgr async connect pattern for vote requests**: confirm `iomgr_async_vote_request()` can be implemented using existing async connect infrastructure without new socket primitives. -- **capnp schema extension**: confirm the capnp code-generation toolchain can add `OP_requestVote`, `OP_probeConfigTag`, `OP_walAck`, `OP_upgradeConfigAck`, `OP_snapshotTransfer`, `OP_walCatchupEntry` without breaking existing decode paths. -- **`persist_truncate_wal_above()` safety**: verify WAL segment file structure supports efficient truncation at arbitrary LSN boundary without full segment rewrite. -- **libxce client API surface**: identify the exact function in src/lib/ that wraps xce_encode()+send+recv+xce_decode() to locate the correct insertion point for retry logic. - ---- - -# Gap Analysis (Updated — Brownfield Audit) - -> Analysis date: 2026-06-01. Branch: persistent_ha. Phase: tasks-generated. -> This section audits current implementation state vs. all 11 task groups. - -## Analysis Summary - -- **Scope**: 10 requirements across 11 task groups. 30 subtasks checked `[x]`; only section headers and integration tasks 11.1–11.5 remain `[ ]`. -- **Implementation status**: **~95% complete**. All core subtasks (1.1–10.2) are marked done and confirmed implemented: `CTRL_INIT`/`CTRL_ELECTING` enum, ha-term.bin durability, `repl_wait_majority_ack`, `persist_truncate_wal_above`, `xce_request_with_retry`, `xce_probe_config_tag`, `repl_broadcast` (real, not stub). -- **Key architectural shift since original research**: The `election-iomgr-refactor` spec superseded parts of this spec. `bootstrap_config_if_needed` and `ha_find_and_connect_to_primary` are no longer called from `myapp_run` — standby connection is inbound (primary dials standby via `on_standby_connected`). `ha_find_and_connect_to_primary` exists only as a test hook reference in ha.h. -- **Remaining work**: Tasks 11.1–11.5 are manual/integration tests documented in `manual-tests.md`. The build passes with all unit tests green. -- **Effort remaining**: S (1–2 days for manual integration test execution + result recording). Risk: Low. - -## Requirement-to-Asset Map (Current State) - -| Req | Status | Key Evidence | -|-----|--------|-------------| -| 1 — Listener before election | ✅ Done | `CTRL_INIT`=2, `CTRL_ELECTING`=3 in ha.h; `cluster_open_listener()` called before election in myapp.c; `ha_wait_for_heartbeat_or_timeout()` implemented | -| 2 — Quorum election | ✅ Done | `am_i_primary_quorum` → superseded by `ha_start_election_async` + raft library (election-iomgr-refactor); `handle_request_vote`, `ha_elect_evaluate_and_signal` implemented | -| 3 — Config bootstrap | ✅ Done (changed arch) | `xce_probe_config_tag()` implemented in config_repl.c:1140; `bootstrap_config_if_needed` removed from myapp_run per election-iomgr-refactor task 5.3; `defer_config_fetch` field exists in ha_state_t; post-election config push via `ha_send_ctrlhello_to_all_standbys` covers deferred fetch | -| 4 — Startup validation | ✅ Done | `startup_validation.c` exists; cert pre-flight, peer count validation, vcpus CLI implemented; `SSL_CTX_free` ASAN fix in ha_fini | -| 5 — Majority-commit WAL | ✅ Done | `repl_broadcast()` real implementation (repl.c:193+); `repl_wait_majority_ack()` implemented (repl.c:338+); `committed_lsn` in persist_t (persist.c:1029); `persist_truncate_wal_above()` implemented (persist.c:1293+) | -| 6 — Persistent control channel | ✅ Done (changed arch) | `primary_ctrl_fd` in ha_state_t; `ha_on_primary_ctrl_fd_rdhup` triggers re-election; inbound channel via `on_standby_connected` in ha.c; `ha_arm_heartbeat_timeout_vtask` for timeout detection; `ha_find_and_connect_to_primary` is a hook-only reference now (CTRL_STANDBY never dials out) | -| 7 — Client retry | ✅ Done | `xce_request_with_retry()` + `xce_retry_config_t` in request.c/request.h (request.c:1664+) | -| 8 — Majority-commit config | ✅ Done | `OP_upgradeConfigAck` handler; `committed_config_tag` in ha_state_t; `push_config_to_standbys` upgraded; `repl_wait_majority_ack` with `OP_upgradeConfigAck` discriminant | -| 9 — State durability | ✅ Done | `ha_persist_term()` / `ha_restore_term()` in ha.h:317-340; ha-term.bin read in `cluster_init_phase1` (cluster.c:809); `voted_for_name` field in ha_state_t | -| 10 — Observability | ✅ Done | Structured INFO/WARN log statements throughout all paths | - -## Gaps: Tasks 11.1–11.5 (Manual Integration Tests) - -### Gap 1: Task 11.1 — Concurrent startup and config-fresh election -- **What's needed**: 3-node cluster starts simultaneously; verify only one elects primary; config-fresh node wins. -- **How to run**: See `manual-tests.md` — Test 1 (3-node clean start, verify ctrl-1 primary). -- **Infrastructure**: `deploy-ha.sh` + `start.sh` scripts; log grep for `CTRL_PRIMARY`. -- **Effort**: ~30 min. - -### Gap 2: Task 11.2 — Failover and WAL durability -- **What's needed**: Kill primary; verify standby detects via heartbeat timeout → promotes; committed WAL survives. -- **How to run**: `manual-tests.md` Test 2. -- **Effort**: ~30 min. - -### Gap 3: Task 11.3 — Unconfigured node bootstrap -- **What's needed**: Start node with `config_tag=0`; verify it receives config via primary's ctrlhello push after election. -- **How to run**: `manual-tests.md` Test 3. -- **Note**: Architecture changed — the deferred fetch now happens via `ha_send_ctrlhello_to_all_standbys` (primary pushes), not `do_one_config_fetch` (standby pulls). This is simpler and already works by default. -- **Effort**: ~30 min. - -### Gap 4: Task 11.4 — Config upgrade majority-commit and term durability -- **What's needed**: Config upgrade, partition, timeout; restart with persisted term. -- **How to run**: `manual-tests.md` Test 4. -- **Effort**: ~30 min. - -### Gap 5: Task 11.5 — Full ASAN build verification -- **What's needed**: `nix build .#xcecore_debug` clean. -- **Status**: Unit tests pass under `/xce-build controller` (which uses debug+ASAN). Full nix build is the final check. -- **Effort**: ~15 min build time. - -## Constraints Carried Forward - -- **`ha_find_and_connect_to_primary` is dead code**: Declared in ha.h with test hook but never called in production. CTRL_STANDBY waits for primary to dial in. Safe to delete in cleanup pass. -- **`bootstrap_config_if_needed` removed from myapp_run**: Covered by primary-push model in election-iomgr-refactor. Tasks 5.2–5.3 `[x]` reflect the original design; the implementation pivoted to push-on-ctrlhello. -- **Single-node path**: Verified via test_ha_election.c single-node shortcircuit. - -## Implementation Approach Recommendation - -No new code needed. Proceed directly to manual integration test execution: - -1. Run `/xce-build` to confirm full build clean. -2. Execute `manual-tests.md` tests 1–5 in order. -3. Record results in the Results Template at the bottom of `manual-tests.md`. -4. Mark tasks 11.1–11.5 `[x]` on pass. -5. Run `nix build .#xcecore_debug` for 11.5 ASAN verification. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -## Scope and Method - -Validation pass against the working tree on branch `persistent_ha` (including uncommitted changes to `ha.c`, `ha.h`, `myapp.c`, `raft.c`). All 50 acceptance criteria across the 10 requirements were checked against on-disk code. Note that the architecture has evolved past this spec's original design via four successor specs (`raft-library`, `election-iomgr-refactor`, `ha-heartbeat-liveness`, `ha-repl-hardening`, `ha-peer-connection-reuse`): the blocking election (`am_i_primary_quorum` + semaphore wait) was replaced by a fully async raft-library-driven election inside iomgr, and the standby-pull config bootstrap was replaced by a primary-push model. Several "gaps" below are therefore *sanctioned deviations*, not defects — they need requirements-doc reconciliation, not code. - -All checkboxes in `tasks.md` (1.1–11.5) are marked `[x]`, including 9.3+, and manual integration tests 11.1–11.4 are recorded as executed 2026-06-03. The memory note "9.3+ remaining" is stale. - -## 1. Requirement-to-Asset Map - -Status legend: **Implemented** / **Partial** / **Missing** / **Unknown** / **Constraint** (sanctioned architectural deviation needing spec reconciliation). - -### Requirement 1 — Listener and Connection Framework Before Election - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 1.1 listener bound before election | Implemented | `myapp.c:113` `cluster_open_listener()` before `ha_start_election_async()`; `cluster_register_listener` at `myapp.c:138` | -| 1.2 iomgr/vtaskmgr before election | Implemented | `myapp.c:122-131` iomgr+vtaskmgr init/start precede election (Step 3/4) | -| 1.3 not-ready rejection in INIT/ELECTING | Implemented | `ha_dispatch_gate()` `ha.c:1412` returns `-XCE_ERR_NOT_READY`; wired at vtask dispatch entry `cluster.c:226` | -| 1.4 standby error includes redirect host/port | **Partial** | `XCE_ERR_STANDBY` / `E_XCE_REDIRECT` returned (`ha.c:1481`, `join.c:241`), `ha_should_redirect()` exists, capnp fields `redirectHost/redirectPort` exist (`controller.capnp:565-566`) — **but no reply path ever populates `redirect_host/redirect_port`**; replies are sent with `data_which = nodata` | - -### Requirement 2 — Quorum-Based Controller Election - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 2.1 strict-majority quorum | Implemented | `raft.c:223-234` quorum = `n_peers/2 + 1`; `ha_elect_evaluate_and_signal` `ha.c:1995` | -| 2.2 lower-term deny | Implemented | `raft_on_vote_request` Step 1 (`raft.c:110`) | -| 2.3 higher-term update + clear vote | Implemented | `raft.c:121-130` (also steps a live leader down to follower) | -| 2.4 freshness gate deny | Implemented | `raft_cb_vote_eligible` `ha.c:202`; `handle_request_vote` `ha.c:1791` | -| 2.5 catch-up standby vote denied | Implemented | `in_catch_up` check `ha.c:218`, `ha.c:1845` | -| 2.6 one vote per term | Implemented | `raft.c:157-164` (with idempotent re-grant to same candidate — safe extension) | -| 2.7 yield to fresher peer | Implemented | yield gate `ha.c:2017-2031` (`max_resp_tag/max_resp_lsn`) | -| 2.8 tag-zero candidate vs configured peer → fatal exit | **Constraint** | Implemented as *voter-only standby*, not fatal exit: `ha.c:3133-3145` (tag==0 && n_peers>1 → CTRL_STANDBY, wait for push). Strictly safer for ops (node self-heals via config push) but deviates from the EARS text "exit with a fatal error" | -| 2.9 single-node immediate self-elect | Implemented | `raft.c:83-88` `RAFT_ELECTED_IMMEDIATELY`; zero-peer guard `ha.c:1985` | -| 2.10/2.11 split vote → one retry w/ jitter+term++, then standby | **Constraint** | raft library becomes FOLLOWER on no-quorum/timeout (`raft.c:237-244`, `raft_on_election_timeout`); retries are driven indefinitely by the standby liveness watchdog → `raft_on_leader_lost` → delayed `raft_check_and_start_election` (`raft.c:268-290`). Net behavior: unbounded jittered retry loop instead of exactly-once retry. Liveness outcome equivalent or better; spec text outdated | -| 2.12 identical-freshness tiebreak by jitter, not priority | **Partial** | Winner choice never uses priority values; but election *timing* uses deterministic priority-based stagger (`myapp.c:229-239` `priority * election_priority_step_ms`; `raft.c:272-273` `priority * election_timeout_ms`). Priority is repurposed as stagger index, not vote weight — spec wording should be reconciled | - -### Requirement 3 — Config Bootstrap for Unconfigured Nodes - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 3.1 probe all peers' config tag before candidacy | **Missing (as specified)** | `xce_probe_config_tag()` fully implemented (`config_repl.c:1140`, single-RTT SSL probe) and unit-tested, but **never called in production** — `bootstrap_config_if_needed()` was removed from `myapp_run` (see Constraints in prior pass). Replacement: unconditional voter-only gate when `my_config_tag==0 && n_peers>1` (`ha.c:3133`) | -| 3.2 voter-only participation, deferred fetch | Implemented (pivoted) | voter-only path `ha.c:3133-3145`; config arrives via primary push (`push_config_to_standbys` from `raft_cb_become_leader`/COMMITTING driver) | -| 3.3 all peers tag-zero → normal candidacy | **Partial / Unknown** | No probe means a tag-0 node cannot distinguish "peers have config" from "all-zero cold start"; it always goes voter-only first. Recovery relies on the heartbeat watchdog firing → `ha_on_primary_ctrl_fd_rdhup` → `raft_on_leader_lost` re-election, which **bypasses** the tag-0 gate (`ha.c:3183-3291`). All-zero cold start thus converges, but only after `heartbeat_timeout_ms` (~10s) delay per round; not covered by an automated test | -| 3.4 exponential-backoff probe retry 500ms→10s | **Missing (as specified)** | No probe retry loop exists; the passive watchdog loop replaces it. Spec text obsolete under the push model | -| 3.5 deferred fetch from proven winner | Implemented (pivoted) | Primary pushes config + ctrlhello to all standbys on COMMITTING (`ha.c:2087-2103`); `fetch_config_from_primary`/`do_one_config_fetch` standby-pull removed by design (CLAUDE.md: standbys never dial out) | - -### Requirement 4 — Startup Validation and Safety Checks - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 4.1 even-N peer rejection | Implemented | `validate_peer_count()` `startup_validation.c:90`; double-enforced in `raft_init` (`raft.c:21`) | -| 4.2 cert/key pre-flight with full path | Implemented | `validate_cert_files()` `startup_validation.c:106` (-ENOENT, path in message), wired in `main.c` | -| 4.3 vcpus >= 1 | Implemented | `validate_vcpus()` `startup_validation.c:25`; `--vcpus` argp option in `main.c` | -| 4.4 config-file vcpus deprecation WARN | Implemented | `validate_startup_config()` `startup_validation.c:46`; warned at `myapp.c:80-86` | - -### Requirement 5 — Majority-Commit WAL Replication - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 5.1 job submit / state transitions / queue changes majority-committed | **Partial** | Only `WAL_JOB_SUBMIT` routes through `persist_wal_sync_append()` (`persist.c:1118`). Job state transitions (`WAL_JOB_START/FINISH/FAIL/...`, `persist.c:1163`) and queue changes (`WAL_QUEUE_CTRL`, `persist.c:1194`) use `persist_wal_async()` — no majority wait | -| 5.2 host state / checkpoint refs async | Implemented | worker/host events → `persist_wal_async`; checkpoint markers never replicated (`persist.c:395`) | -| 5.3 standby durable write + ACK with LSN | Implemented | `OPCode_walAck` flow, `repl_on_ack` (`repl.c:178`), `acked_lsn` tracking | -| 5.4 lag eviction + catch-up trigger | Implemented | `pending_ack_count > max` eviction with inline catch-up start, `repl.c:214-222` | -| 5.5 commit timeout → error to caller | **Constraint** | Deliberate *degraded mode*: `persist_wal_sync_append` reports `-ETIMEDOUT/-ENODEV` via `sync_rc_out` but returns 0 and the operation proceeds with local-only durability (`persist.c:1003-1043`; hook comment `persist.c:1114`). This is an AP-leaning deviation from the spec's CP statement — documented in code, not in requirements | -| 5.6 truncate above committed LSN on promotion | Implemented | `ha_compute_safe_committed_lsn()` from voter replies (`ha.c:373-437`) → `persist_truncate_wal_above(safe_lsn)` in `raft_cb_become_leader` (`ha.c:471`); per controller CLAUDE.md this is the verified-against-TLA+ design | -| 5.7 truncate on catch-up reconnect | **Partial** | RDHUP/watchdog failover path truncates to `last_known_committed_lsn` before re-election (`ha.c:3204-3207`). A standby that *restarts* (rather than detecting RDHUP) replays its full WAL including uncommitted tail; no truncation occurs before it rejoins and receives catch-up entries — divergent-tail handling on the restart path is unverified | -| 5.8 single-node local write satisfies majority | **Partial** | `repl_wait_majority_ack` returns 0 when `n_active == 0` (`repl.c:384`), but `persist_wal_sync_append` short-circuits *before* calling it when `nstandbys == 0` and sets `sync_err = -ENODEV` **without advancing `committed_lsn`** (`persist.c:1023-1031`). Single-node `committed_lsn` appears to advance only via the election-time safe-LSN path — runtime advancement per entry is not evident. Needs verification (interacts with WAL trim guard, which uses `committed_lsn` as threshold) | - -### Requirement 6 — Persistent Standby-to-Primary Control Channel - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 6.1 standby establishes persistent TLS connection to primary | **Constraint** | Direction inverted by deliberate design (controller CLAUDE.md "Inbound vs. Outbound Connection Ownership"): the **primary dials out** to standbys; standby-side `ha_find_and_connect_to_primary()` is dead code (declared `ha.h:741`, never called in production). The functional goal (persistent channel, no per-RPC connections) is achieved | -| 6.2 disconnect → immediate election | Implemented | EPOLLRDHUP on `primary_ctrl_fd` → `ha_on_primary_ctrl_fd_rdhup()` (`ha.c:3183`) | -| 6.3 heartbeat resets timeout | Implemented | `ha_on_heartbeat_received()` on every NOP/ctrlhello (`cluster.c:249, 283`) | -| 6.4 heartbeat timeout + jitter → election | Implemented | `ha_arm_heartbeat_timeout_vtask` liveness poll; manual test 11.2 measured ~11s detection (configured timeout, not sub-second — sub-second applies to the RDHUP path) | -| 6.5 sync missing state before admitting to replication set | Implemented | heartbeat-reply-driven driver: standby `stateReport` → `ha_repl_decide()` (`ha.c:1157-1176`) → stream WAL / snapshot / config; `in_catch_up` gates quorum membership (`repl.c:367`) | -| 6.6 snapshot when standby older than oldest WAL segment | Implemented | `standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT` (`ha.c:1176`), `ha_send_snapshot_to_standby()` chunked transfer (`config_repl.c:1578`), standby handler `cluster.c:182` | -| 6.7 config push after catch-up on tag mismatch | Implemented | same driver pushes config on tag divergence; `push_config_to_standbys` | -| 6.8 winner closes outbound, accepts inbound | Implemented | role flip handled in `raft_cb_become_leader` / RDHUP path; surviving vote eps promoted to `standby_endpoints`; late joiners get fresh outbound connects from `primary_heartbeat_fn` | - -### Requirement 7 — Client-Side Transparent Retry (libxce) - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 7.1 NOT_READY → backoff retry | Implemented | `xce_request_with_retry()` `request.c:1664`; tests `test_libxce_retry.c` | -| 7.2 STANDBY + redirect → follow redirect | **Partial** | Client logic implemented and unit-tested via hook, but **no controller reply ever carries `redirect_host/port`** (see 1.4) — end-to-end the redirect-follow path can never trigger; falls back to backoff | -| 7.3 TCP refused/timeout → reconnect retry | Implemented | `request.c` retry classification | -| 7.4 non-retriable surfaced immediately | Implemented | AUTH/PERM/INVALID/NOENT surfaced without retry | -| 7.5 configurable params + defaults | Implemented | `xce_retry_config_t`, `XCE_RETRY_DEFAULTS` (`request.h:140-150`) | -| 7.6 thundering-herd jitter | Implemented | jittered exponential backoff; 50-client concurrency test | -| (adoption) | **Missing** | Zero production call sites: worker, scheduler, and apiserver backend never call `xce_request_with_retry` — the wrapper is shipped but unused | - -### Requirement 8 — Majority-Commit Config Replication - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 8.1 majority commit for config push | Implemented | `push_config_to_standbys` → `repl_wait_majority_ack(..., OPCode_upgradeConfigAck)` (`config_repl.c:905`) → `committed_config_tag` advanced (`config_repl.c:908`) | -| 8.2 standby durable write + ACK | Implemented | `handle_repl_config_push` + `OP_upgradeConfigAck`; `acked_config_lsn` tracked separately from WAL acks (`repl.c:375-380`) | -| 8.3 timeout → error | Implemented | `-ETIMEDOUT` from `repl_wait_majority_ack`, WARN logged | -| 8.4 promoted standby serves only committed config tag | **Partial** | `last_known_committed_config_tag` captured from CtrlHelloReply (`cluster.c:261`) and `committedConfigTag` piggybacked on heartbeat (capnp); `committed_config_tag` consulted at `ha.c:2514-2521` and safe-tag computed on promotion (`ha.c:485`). Enforcement at *serve time* (refusing to serve a fresher-but-uncommitted config) not explicitly verified | -| 8.5 single-node local write = majority | Implemented | safe-tag path on become-leader; `committed_config_tag` advanced on push without standbys | - -### Requirement 9 — Election State Durability - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 9.1 persist term+vote before reply | Implemented | raft `on_persist` fires before any grant/RPC (`raft.c:77, 176`); `ha_persist_term()` atomic write+rename with magic+CRC32 (`ha.c:1647-1681`) | -| 9.2 restore on startup | Implemented | `ha_restore_term()` (`ha.c:1683+`); manual test 11.4 confirmed `restored term=2` on restart | -| 9.3 missing file → term 0, no vote | Implemented | missing/short-read/bad-magic/CRC-mismatch all degrade safely to term 0 with WARN (`ha.c:1715-1737`) | - -### Requirement 10 — Observability - -| AC | Status | Assets / Evidence | -|----|--------|-------------------| -| 10.1 election outcome: term + votes granted + quorum size | **Partial** | FSM transition INFO logs carry term (`ha.c:507, 659, 870, 881, 2080, 2158`); majority-ack log carries acks+quorum (`ha.c:2929`); but no single election-complete record logs *votes granted* and *quorum size* together | -| 10.2 bootstrap probe finding | **Partial** | voter-only decision logged (`ha.c:3134`); per-peer probe logs exist in `xce_probe_config_tag` but that path is unused in production | -| 10.3 WAL quorum INFO / timeout WARN | Implemented | `repl.c:408-410` | -| 10.4 config quorum INFO / timeout WARN | Implemented | `config_repl.c` push path logs both | -| 10.5 FSM transition INFO | Implemented | all transitions logged, including RDHUP (`ha.c:3199`) | - -### Summary Tally (by requirement) - -| Req | Verdict | -|-----|---------| -| R1 | Implemented except redirect payload (1.4 Partial) | -| R2 | Implemented; 2.8/2.10/2.11/2.12 are sanctioned deviations needing spec reconciliation | -| R3 | Pivoted to primary-push; 3.1/3.4 Missing-as-specified, 3.3 Unknown (cold-start convergence untested) | -| R4 | Implemented (4/4) | -| R5 | Core implemented; 5.1 coverage gap, 5.5 CP→degraded-mode deviation, 5.7 restart-path gap, 5.8 needs verification | -| R6 | Implemented; 6.1 direction inverted by design (Constraint) | -| R7 | Library implemented; redirect payload + production adoption Missing | -| R8 | Implemented; 8.4 serve-time enforcement Partial | -| R9 | Implemented (3/3) | -| R10 | Implemented except consolidated election-outcome record (10.1) | - -**Net: ~40 of 50 acceptance criteria fully implemented; 7 partial; 3 missing-as-specified (all three superseded or deliberately pivoted).** No requirement is wholly unimplemented. - -## 2. Remaining Gaps — Concrete List - -1. **G1 — Redirect payload never populated** (R1.4, R7.2): capnp fields exist; standby replies send `nodata`. Client redirect-follow logic is dead end-to-end. -2. **G2 — WAL majority-commit covers only job submission** (R5.1): state transitions and queue changes are async-only; a failover can lose acknowledged state changes. -3. **G3 — Commit timeout is degraded-mode, not an error** (R5.5): spec promises CP (timeout error to caller); code chooses availability. Needs an explicit, documented decision. -4. **G4 — Single-node `committed_lsn` runtime advancement unverified** (R5.8): `-ENODEV` short-circuit skips the advance; interacts with the WAL trim guard. -5. **G5 — Standby restart path skips divergent-tail truncation** (R5.7): truncation happens only in the RDHUP/watchdog path. -6. **G6 — Config probe path unwired** (R3.1/3.3/3.4): `xce_probe_config_tag` is production-dead; all-zero cold-start converges only via watchdog timeout and lacks a test. -7. **G7 — libxce retry has zero production adopters** (R7 objective): worker/scheduler/apiserver still have whatever ad-hoc behavior they had. -8. **G8 — Observability**: no consolidated election-outcome log with vote tally + quorum size (R10.1). -9. **G9 — Spec/code reconciliation debt**: 2.8 (standby instead of fatal exit), 2.10/2.11 (watchdog retry loop instead of retry-once), 2.12 (priority-based stagger), 3.x (push model), 5.5 (degraded mode), 6.1 (inverted channel direction) — all stable code behavior contradicting EARS text. -10. **G10 — Cleanup**: `ha_find_and_connect_to_primary` dead code; bounded per-RDHUP `elect_ctx` leak (`ha.c:3252-3258`, documented UAF-avoidance). - -## 3. Implementation Approach Options - -### Option A — Extend existing components in place (code-first) -Close G1–G8 directly in `ha.c` / `persist.c` / `repl.c` / `join.c` / worker+scheduler clients; patch requirements.md text afterward. -- Files: `join.c`+`reply.c` (G1), `persist.c` hook routing (G2/G4), `cluster.c` startup replay (G5), `myapp.c`+`config_repl.c` (G6), worker/scheduler request sites (G7), `ha.c` logging (G8). -- ✅ No new modules; follows established patterns (every gap has an existing seam and a testsuite file). -- ✅ Fastest path to closing the spec. -- ❌ G2/G3 are *policy* changes with latency and availability consequences — landing them silently inside this aging spec hides a significant decision. -- ❌ `ha.c` is already 3,818 lines; further growth degrades maintainability. - -### Option B — Spec-first reconciliation, then targeted follow-up spec -Amend requirements.md/design.md to match the as-built architecture (push bootstrap, inverted channel, watchdog retry, degraded mode), close only the unambiguous small defects (G1, G4, G8, G10) here, and move G2/G3/G5/G7 into the active `ha-repl-hardening` spec (or a new `ha-cp-commit` spec) with their own requirements review. -- ✅ Honest record: the EARS text stops contradicting shipped behavior; the CP-vs-degraded decision gets explicit operator-facing review. -- ✅ Keeps this spec closeable; G2/G3 get proper TDD treatment in a spec that already owns repl hardening. -- ❌ More process; durability gap (G2) stays open until the follow-up lands. -- ❌ Risk that "deferred" becomes "dropped" without tracking. - -### Option C — Hybrid (recommended) -Phase 1 (this spec, S-size code): G1 redirect payload, G4 single-node commit advance verification/fix, G8 election-outcome log, G10 cleanup, plus requirements.md deviation annotations (G9) and a cold-start integration test for G6's watchdog fallback. -Phase 2 (`ha-repl-hardening` spec): G2 (extend sync-append coverage to state transitions/queue changes — likely with group-commit batching to bound latency), G3 (explicit CP-vs-degraded decision + config knob), G5 (restart-path truncation), G7 (retry adoption across daemons). -- ✅ Ships every low-risk fix now; routes every policy-bearing change through review. -- ✅ Matches the existing spec topology (repl-hardening already owns quorum/commit semantics). -- ❌ Requires coordinating two specs; G6's probe-vs-watchdog question must be answered during reconciliation (recommend: delete the probe path, document the watchdog fallback, keep the test). - -## 4. Effort & Risk (remaining work) - -| Gap | Effort | Risk | Justification | -|-----|--------|------|---------------| -| G1 redirect payload | S | Low | Fields exist in schema/typemap; populate in 2 reply sites + 1 e2e test | -| G2 sync-append coverage | M | Medium | Mechanical routing change but puts majority-RTT on hot job-state path; needs batching design + latency measurement | -| G3 CP-vs-degraded decision | S (code) / M (decision) | High | One-line behavior flip, but it trades availability for consistency cluster-wide; needs operator sign-off and failure-mode tests | -| G4 single-node committed_lsn | S | Medium | Small fix, but wrong answer corrupts WAL trim guard invariant (`committed_lsn <= applied_lsn`) | -| G5 restart-path truncation | M | Medium | Touches startup replay ordering; must honor TLA+ safe-commit-point rule (never truncate via stale ctrlhello value) | -| G6 cold-start test + probe removal | S | Low | Behavior already converges via watchdog; work is test + dead-code deletion + doc | -| G7 retry adoption | M | Medium | Multi-daemon change; behavioral shift in error surfacing for worker/scheduler/apiserver | -| G8 election-outcome log | S | Low | Data already in `ha_elect_ctx_t`/raft state at decision points | -| G9 spec reconciliation | S | Low | Documentation only; six identified deltas | -| G10 cleanup | S | Low | Dead-code deletion + bounded-leak fix behind existing tests | - -Aggregate remaining: **~M total for Phase 1 (all S items), ~L for Phase 2** (G2+G3+G5+G7 under ha-repl-hardening). - -## 5. Recommendations - -1. **Adopt Option C.** Close this spec with Phase 1; transfer G2/G3/G5/G7 into `ha-repl-hardening` as explicit requirements so they get their own approval gate. -2. **Reconcile requirements.md now** (G9): rewrite AC 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 to describe the as-built push/watchdog/outbound-primary architecture. The controller CLAUDE.md and TLA+ mapping are already correct — the spec is the lagging artifact. -3. **Decide G3 deliberately.** The spec's headline promise is "zero data loss on failover" (CP); the code ships degraded-mode availability. Either add a `commit_strict` config knob defaulting per the decision, or amend the spec's CP claim. Do not leave the contradiction implicit. -4. **Fix G1 immediately** — it is the cheapest fix with end-to-end user impact (failover redirect for clients), and it unblocks the already-tested libxce retry path. -5. **Delete dead code** (`ha_find_and_connect_to_primary`, production `xce_probe_config_tag` path if reconciliation confirms the push model) rather than leaving two competing bootstrap narratives in the tree. - -### Research Needed - -- **RN1**: Does single-node `committed_lsn` ever advance at runtime (G4)? Trace `persist_wal_sync_append` + WAL trim guard in a single-node soak; check whether `ha-repl-hardening` task 6.5 already addressed this. -- **RN2**: Latency budget for extending majority-commit to job state transitions (G2) — measure per-entry RTT on 3-node LAN; evaluate group commit / pipelined acks before committing to a design. -- **RN3**: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether `apply_wal_entry` overwrite semantics mask or corrupt the divergent tail. -- **RN4**: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary. -- **RN5**: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above `committed_config_tag` in any window between promotion and first quorum push? - ---- - -## Gap Re-Audit (2026-06-18) - -> Trigger: post-implementation gap audit. Tasks 46/46 `[x]`, `phase=implemented`. -> Method: re-verified the 2026-06-09 gap list (G1–G10) against the current working -> tree on `persistent_ha`. Since the prior pass, two successor specs landed and -> closed several of the gaps this spec recorded: **`libxce-failover`** (client-side -> transparent failover/redirect) and **`ha-commit-durability`** (bug-661, landed -> 2026-06-18). The earlier majority-commit model was also superseded by a -> `wal_classify()` WAIT/RETAIN scheme from `ha-commit-barrier-rollout`. Line numbers -> below reflect the current tree (ha.c is now 5,338 lines; persist.c 2,587). - -### Headline change vs 2026-06-09 - -The 2026-06-09 pass found ~40/50 ACs fully implemented with 10 open gaps (G1–G10). -**Five of those gaps are now CLOSED** by successor-spec work; the residual set is -smaller and almost entirely *spec-text reconciliation debt* rather than missing code. - -### Requirement-to-Asset Map — Delta Since 2026-06-09 - -| Gap (2026-06-09) | Req | New Status (2026-06-18) | Evidence | -|---|---|---|---| -| **G1** Redirect payload never populated | R1.4, R7.2 | **CLOSED** | `fill_redirect_reply()` populates `reply->redirect_host/redirect_port` from `ha->primary_host/port` (`join.c:28-32`); called on the `-E_XCE_REDIRECT` path (`join.c:282`); covered by `test_join_redirect.c`. Comment cites `libxce-failover Req 8.1`. | -| **G7** libxce retry has zero adopters | R7 obj. | **CLOSED (re-homed)** | Orphaned no-op `xce_request_with_retry()` + `xce_retry_send_hook` were **removed** (`request.c:1923-1936`); superseded by the `xcecli_*` failover loop in `src/lib/client.c` + `src/lib/client_failover.c`. `XCE_RETRY_DEFAULTS` retained as the shared tunable. Production redirect-follow is now live end-to-end (G1 + client_failover). | -| **G10** `ha_find_and_connect_to_primary` dead code | cleanup | **CLOSED** | No production `.c` reference remains (grep over `src/controller/*.c` returns nothing). Dead-code deletion done. | -| **G2** WAL majority-commit covers only job submit | R5.1 | **SUPERSEDED (model changed)** | Replaced by `wal_classify()` WAIT/RETAIN scheme (`persist.c:2004`). `WAL_JOB_SUBMIT` and `WAL_QUEUE_CTRL` are `WAL_MODE_WAIT` (commit-gated before requester reply); `WAL_WORKER_CTRL` JOIN/UPDATE are WAIT; job state transitions (`WAL_JOB_START/FINISH/FAIL/...`) are `WAL_MODE_RETAIN` — replicated and retained-until-committed but not blocking the worker-fact report. The request-shaped side of dual-origin events is realized by the handler barrier in `do_jobcontrol_op`. The original "everything sync-blocks" framing was deliberately not adopted (latency); coverage is now classification-driven, owned by `ha-commit-barrier-rollout` / `ha-commit-durability`. | -| **G3** Commit timeout degraded-mode vs CP | R5.5 | **SUPERSEDED (bug-661 / ha-commit-durability)** | Durability invariant now GUARANTEED: acknowledged ⇒ `fdatasync`'d on a fault-tolerant majority. `majority_candidate_locked` caps the primary self-contribution at `persist_durable_lsn`; `fsync_worker` group-commit flusher advances `durable_lsn` only on `fdatasync` success (fail-loud rc-gate); standby `fdatasync`s before WalAck. Verified by `test_commit_durability.c`, the ack-before-fdatasync E2E, and the rolling-failover soak (zero acked-job loss). Controller CLAUDE.md checklist item flipped to GUARANTEED 2026-06-18. The old "silent local-only degrade" path is gone. | - -### Residual Gaps (still open after 2026-06-18) - -| ID | Req | Status | Detail | -|---|---|---|---| -| **R1 (was G6)** Config probe path unwired | R3.1, R3.3, R3.4 | **Missing-as-specified (stable)** | `xce_probe_config_tag()` is fully implemented + unit-tested (`config_repl.c:1219`) but **still not called in production** — `myapp.c` has no probe call (grep confirms). The tag-0 voter-only gate (`ha.c:4495`) uses static `my_config_tag`/`n_peers`, not probe results. The comment at `ha.c:4493` ("xce_probe_config_tag path already called in myapp_run") is **stale/incorrect** — no such call exists. All-zero cold-start converges only via watchdog-driven re-election (~heartbeat_timeout), and that path lacks an automated convergence test (prior RN4). | -| **R2 (was G9)** Spec/code reconciliation debt | 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 | **Open (docs only)** | requirements.md EARS text still describes: fatal-exit for tag-0 candidate (code: voter-only standby), exactly-once split-vote retry (code: unbounded watchdog retry loop), exponential-backoff probe (code: passive watchdog), standby-dials-primary (code: primary dials out — inverted by design per controller CLAUDE.md), CP-timeout-error (code: now GUARANTEED durability, but worded as the old model). spec.json `note` still lists the now-closed G1/G3 framings. The TLA+ mapping + controller CLAUDE.md are authoritative and correct; the spec is the lagging artifact. | -| **R3 (was part of G4)** Single-node `committed_lsn` runtime advance | R5.8 | **Likely closed — verify** | `test_single_node_committed_lsn_advances` (`test_p2_integration.c:251`) and `test_legacy_sync_append_holds_committed_at_durable` (`test_commit_durability.c:660`) now exist, suggesting ha-commit-durability addressed the prior `-ENODEV` short-circuit concern. Recommend a quick read of those tests to confirm RN1 is retired before closing. | -| **R4 (was G5)** Standby restart-path divergent-tail truncation | R5.7 | **Re-scoped to recovery** | The RDHUP/watchdog failover path truncates to `last_known_committed_lsn`. The *restart* path is now governed by `ha-commit-durability` recovery (`recover_verdict_t` clean-vs-corruption, durable-floor re-derivation, pre-promotion fence). Whether a restarted standby with an uncommitted tail is correctly fenced/caught-up before rejoin is owned by that spec's recovery tests, not this one. Verify via `ha-commit-durability` rather than re-opening here. | -| **R5 (was G8)** Consolidated election-outcome log | R10.1 | **Open (minor)** | FSM-transition INFO logs carry term, and majority-ack logs carry acks+quorum, but there is still no single election-complete record emitting *votes-granted AND quorum-size* together. Data is available in `ha_elect_ctx_t`/raft state at the decision point. Low value, low effort. | - -### Residual Tally - -- **Closed since 2026-06-09:** G1, G7, G10 (outright), G2, G3 (superseded by a better model in successor specs). -- **Still open:** R1 (probe unwired + stale comment), R2 (spec-text reconciliation), R5 (consolidated election log). R3 likely-closed-pending-verify, R4 re-homed to ha-commit-durability recovery. -- **Net AC posture:** the durability and redirect deltas mean the *functional* CP promise of this spec is now met or exceeded; the open items are documentation honesty (R2), one unwired-but-replaced bootstrap path (R1), and one cosmetic log (R5). - -### Implementation Approach Options (for the residual work) - -#### Option A — Close-and-reconcile in place (recommended) -Phase-1-only, all S-size, no successor spec needed: -1. **R2** rewrite requirements.md AC 2.8 / 2.10–2.12 / 3.1–3.5 / 5.5 / 6.1 to describe the as-built push/watchdog/outbound-primary/guaranteed-durability architecture; refresh the spec.json `note` (G1/G3 are no longer "known deviations"). Docs only. -2. **R1** delete the stale `ha.c:4493` comment; either (a) delete the unused `xce_probe_config_tag` production path and document the static-gate + watchdog-convergence model, or (b) add a cold-start convergence integration test (3-node, all configs absent) so the watchdog fallback is covered. Recommend (a)+test. -3. **R5** add a single INFO "election complete: term=T votes=G/Q role=…" record at the become-leader / become-follower decision point. -4. **R3** read the two new single-node durability tests; if they cover runtime advance, retire RN1 and mark R5.8 closed. -- ✅ Closes the spec honestly with no policy decisions left implicit; smallest surface. -- ✅ Every item has an existing seam + testsuite file. -- ❌ Touches requirements.md (aging artifact) — but that is the point. -- **Effort: S total. Risk: Low** (docs + dead-code delete + one log line + one test). - -#### Option B — Defer reconciliation, close only code defects -Fix R1 comment + R5 log now; leave requirements.md as-is with an expanded spec.json deviation note pointing at controller CLAUDE.md as the source of truth. -- ✅ Even smaller; no requirements.md churn. -- ❌ Leaves EARS text contradicting shipped behavior — the recurring complaint across every prior pass. Reconciliation debt compounds. -- **Effort: XS. Risk: Low (but accrues doc debt).** - -#### Option C — Fold residuals into successor specs and close this one verbatim -Mark ha-config-bootstrap closed; track R1/R2/R5 under `ha-commit-durability` (R3/R4 already there) or a small `ha-spec-reconcile` doc spec. -- ✅ This spec stops being touched; clean lineage. -- ❌ R2 spans this spec's own EARS text — odd to reconcile it from another spec. R5 is trivially this spec's. -- **Effort: S (mostly bookkeeping). Risk: Low; tracking-loss risk medium.** - -### Effort & Risk — Residual - -| Item | Effort | Risk | Notes | -|---|---|---|---| -| R1 probe path + stale comment + cold-start test | S | Low | Behavior already converges via watchdog; delete dead path or add test | -| R2 requirements.md / spec.json reconciliation | S | Low | Docs only; 6 EARS deltas; CLAUDE.md is the correct source | -| R3 single-node commit verify | XS | Low | Read 2 existing tests; likely already closed | -| R4 restart-path truncation | — | — | Re-homed to ha-commit-durability recovery; verify there, not here | -| R5 consolidated election log | S | Low | One INFO line; data already in elect_ctx | - -**Recommendation:** Option A. The heavy lifting (durability, redirect, retry adoption, dead-code) is done by successor specs; the only thing standing between this spec and an honest `implemented`/closed state is requirements-text reconciliation (R2), one unwired-but-superseded path (R1, delete + document), and a cosmetic log (R5). All S/XS, all Low risk. diff --git a/.kiro/specs/ha-config-bootstrap/spec.json b/.kiro/specs/ha-config-bootstrap/spec.json deleted file mode 100644 index 86f5e0ec..00000000 --- a/.kiro/specs/ha-config-bootstrap/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "ha-config-bootstrap", - "description": "Replace LSF-style priority election with Raft-style quorum voting, add majority-commit WAL and config replication, establish persistent control channels, and harden the controller startup sequence for full CP high-availability.", - "phase": "implemented", - "created_at": "2026-05-27T01:24:00-07:00", - "updated_at": "2026-08-16T04:30:00-07:00", - "language": "en", - "status": "closed", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "note": "CLOSED 2026-08-16. All 3 known deviations from the 2026-06-09 gap analysis resolved or deliberately superseded, confirmed by this spec's own 2026-06-18 re-audit (research.md) plus a fresh cross-check: (1) redirect fields never populated -- closed by libxce-failover's fill_redirect_reply() (join.c:28), validate-impl GO 2026-07-18, explicitly closing this spec's Req 7 gap. Caveat: OP_upgradeConfig replies still lack redirect wiring, a narrower still-open gap found by admin-config-tool task 6.3 (2026-08-16) -- NOT a reopening of this deviation, since Req 7 only ever covered libxce's generic retry path, never OP_upgradeConfig specifically; track it under whichever spec next touches xceadmin/config-push. (2) WAL majority-commit covers WAL_JOB_SUBMIT only -- superseded by the deliberate wal_classify() WAIT/RETAIN scheme (ha-commit-barrier-rollout/ha-consistency-invariants, complete 2026-07-13); not a coverage gap, an intentional latency-driven design, not ha-repl-hardening as originally noted. (3) commit timeout silently degrades -- resolved as bug-661 via ha-commit-durability (17/17 tasks, validate-impl GO, re-verified live 2026-07-18: 8/8 rolling+cold-restart soak rounds, zero job loss)." -} diff --git a/.kiro/specs/ha-config-bootstrap/tasks.md b/.kiro/specs/ha-config-bootstrap/tasks.md deleted file mode 100644 index 455f49e2..00000000 --- a/.kiro/specs/ha-config-bootstrap/tasks.md +++ /dev/null @@ -1,293 +0,0 @@ -# Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election - ---- - -## Task Overview - -10 requirements → 8 implementation changes (D→A→B→C→E→H→F→G) + observability woven throughout. -Dependency order: startup validation (D) unblocks vtaskmgr before election (A); listener before election (A) unblocks quorum FSM (B); quorum FSM (B) unblocks config bootstrap (C); majority-commit WAL infrastructure (E) unblocks majority-commit config (H) and persistent channel (F); libxce retry (G) is independent of all controller changes. - ---- - -- [x] 1. Foundation: Wire protocol extensions and shared infrastructure -- [x] 1.1 Add 6 new opcodes and message structs to controller.capnp - - Append `requestVote @32`, `probeConfigTag @33`, `walAck @34`, `upgradeConfigAck @35`, `snapshotTransfer @36`, `walCatchupEntry @37` after existing `upgradeConfig @31` - - Add structs: `RequestVoteRequest`, `RequestVoteReply`, `ProbeConfigTagRequest`, `ProbeConfigTagReply`, `WalAckRequest`, `WalAckReply`, `UpgradeConfigAck`, `SnapshotTransfer`, `WalCatchupEntry` - - Extend `CtrlHeartbeat`: add `committedLsn :UInt64` and `committedConfigTag :UInt64` fields - - Extend `RequestReply`: add `redirectHost :Text` and `redirectPort :UInt16` fields for standby redirect - - Regenerate capnp bindings (`nix build .#xcecore_debug` must compile clean) - - `make test` passes; existing decode paths unaffected - - _Requirements: 2, 3, 5, 6, 7, 8_ - - _Boundary: Wire Protocol (controller.capnp)_ - -- [x] 1.2 Add new enum values and types to ha.h - - Add `CTRL_INIT=2`, `CTRL_ELECTING=3` to `ctrl_role_t` (values above existing 0/1) - - Add `ha_vote_entry_t` struct with fields: `peer`, `my_term`, `my_tag`, `my_lsn`, `vote_granted`, `replied`, `peer_tag`, `peer_lsn` - - Add `ha_elect_ctx_t` struct with C99 flexible array `ha_vote_entry_t votes[]`; allocation comment: `calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))` - - Add 8 new `ha_state_t` fields: `voted_for_term`, `voted_for_name[XCE_MAXNAME_LEN]`, `primary_ctrl_fd` (int, -1=none), `defer_config_fetch` (bool), `elect_ctx` (_Atomic void*), `last_known_committed_lsn` (_Atomic uint64_t), `heartbeat_seen` (sem_t), `last_known_committed_config_tag` (uint64_t) - - Declare: `ha_persist_term()`, `bootstrap_config_if_needed()`, `xce_probe_config_tag()` - - `make test` passes; no new functions yet — declarations only - - _Requirements: 1, 2, 3, 5, 6, 9_ - - _Boundary: Controller FSM (ha.h)_ - -- [x] 2. Change D: Startup validation and ASAN fix -- [x] 2.1 Write failing tests for startup validation - - In `src/controller/testsuite/test_ha_startup.c`: test even-N peer count returns `-EINVAL`; test cert missing returns `-ENOENT` with full path in message; test `--vcpus 0` returns `-EINVAL`; test `vcpus` key in config file logs WARN but does not fail - - `make test` fails (functions under test do not exist yet) - - _Requirements: 4.1, 4.2, 4.3, 4.4_ - -- [x] 2.2 Implement --vcpus CLI arg and remove vcpus from cluster config - - Add `--vcpus N` argp option in `main.c`; store in a new startup parameter struct passed to `myapp_run()` - - Remove `vcpus @7` field from `ClusterConfig` in `cluster.h` / `cluster.c` - - Update `vtaskmgr_init()` call site to use the CLI-provided vcpus value directly - - Log WARN when config file contains `vcpus` key (4.4) - - `make test` GREEN on vcpus tests - - _Requirements: 4.3, 4.4_ - - _Boundary: Startup Validation (main.c, cluster.h, cluster.c)_ - -- [x] 2.3 Add peer count validation and cert pre-flight checks - - In `main.c` after argument parsing: if `n_peers % 2 == 0 && n_peers > 1` → `fprintf(stderr, ...)` + `exit(1)` with message explaining odd-N requirement - - Call `access(cert_path, R_OK)` and `access(key_path, R_OK)` in `main.c` before any SSL init; fatal exit with full path on failure - - Fix ASAN leak: add `SSL_CTX_free(ha->ssl_ctx)` in `ha_fini()` - - `make test` GREEN on all startup validation tests - - _Requirements: 4.1, 4.2_ - - _Boundary: Startup Validation (main.c, ha.c)_ - -- [x] 3. Change A: Listener and iomgr/vtaskmgr before election -- [x] 3.1 Write failing tests for dispatch gate and FSM init state - - In `src/controller/testsuite/test_ha_startup.c` (extend): test that requests other than `OP_requestVote` and `OP_probeConfigTag` receive `XCE_ERR_NOT_READY` when role is `CTRL_INIT`; test `CTRL_STANDBY` returns `XCE_ERR_STANDBY` for primary-only ops; test `heartbeat_seen` semaphore initialized in `ha_init()` - - `make test` fails - - _Requirements: 1.1, 1.2, 1.3, 1.4_ - -- [x] 3.2 Extract cluster_open_listener() from cluster_init_phase2() - - Move `bind()+listen()` + socket setup into new `cluster_open_listener(cluster_t *c)` in `cluster.c` - - `cluster_init_phase2()` calls `cluster_open_listener()` for compatibility - - `myapp.c` startup sequence calls `cluster_open_listener()` before election (after `cluster_init_phase1()`) - - `iomgr_init()` and `vtaskmgr_init(vcpus)` also moved before election - - `make test` GREEN; listener is open in startup sequence before any election probe - - _Requirements: 1.1, 1.2_ - - _Boundary: Startup Sequence (cluster.c, myapp.c)_ - -- [x] 3.3 Implement dispatch gate for CTRL_INIT and CTRL_ELECTING states - - Initialize `ha.heartbeat_seen` semaphore in `ha_init()`; initialize `ha.primary_ctrl_fd = -1` - - Add role-check at vtaskmgr dispatch entry: `CTRL_INIT` and `CTRL_ELECTING` pass only `OP_requestVote` and `OP_probeConfigTag`; all others return `XCE_ERR_NOT_READY` - - `CTRL_STANDBY` returns `XCE_ERR_STANDBY` with `redirect_host/port` for primary-only ops - - `make test` GREEN on all dispatch gate tests - - _Requirements: 1.3, 1.4_ - - _Boundary: Controller FSM (ha.c, cluster.c)_ - -- [x] 3.4 Implement ha_wait_for_heartbeat_or_timeout() - - `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + xorshift64_jitter()` - - Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`; receiving a heartbeat transitions to `CTRL_STANDBY` and returns - - Election timeout fires → returns without setting standby; caller proceeds to `am_i_primary_quorum()` - - Called from `myapp.c` after `bootstrap_config_if_needed()` (see task 5) - - `make test` GREEN - - _Requirements: 1.1, 1.2_ - - _Boundary: Controller FSM (ha.c)_ - -- [x] 4. Change B: Quorum-based controller election (replaces am_i_primary) -- [x] 4.1 Write failing tests for quorum election logic - - In `src/controller/testsuite/test_ha_election.c`: test 2/3 votes → `CTRL_PRIMARY`; test only 1/3 vote → `CTRL_STANDBY`; test `voted_for` enforcement (deny second candidate in same term); test freshness gate (higher peer tag yields to peer → `CTRL_STANDBY`); test fresher candidate wins over stale peer; test split vote triggers retry with incremented term; test single-node self-elects immediately without network; test `in_catch_up=1` standby votes are denied - - `make test` fails - - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12_ - -- [x] 4.2 Implement handle_request_vote() and ha_persist_term() - - `ha_persist_term()`: write `ha_term_file_t` (magic + term + voted_for_term + voted_for_name + CRC32) atomically to `{workdir}/ha-term.bin` via write+rename - - `handle_request_vote()`: if `req->term < my_term` → deny; if `req->term > my_term` → update term, clear voted_for, persist; if already voted for different candidate → deny; if `req->config_tag < my_tag || (eq_tag && req->last_lsn < my_lsn)` → deny; otherwise → grant, set `voted_for_name`, persist, reply - - `handle_request_vote_reply()`: update `ha_elect_ctx_t` entry; post `ctx->done` when quorum reached or all replied - - `make test` GREEN on all vote handler sub-cases - - _Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1_ - - _Boundary: Controller FSM (ha.c)_ - -- [x] 4.3 Implement ha_elect_evaluate_and_signal() and am_i_primary_quorum() - - `ha_elect_evaluate_and_signal()`: count votes_granted; check freshness gate (any responding peer has higher config_tag or equal tag + higher LSN → yield); if `votes_granted >= quorum && !fresher_peer` → `CTRL_PRIMARY`; else → `CTRL_STANDBY` - - `am_i_primary_quorum()`: single-node shortcircuit (n_peers==1 → `CTRL_PRIMARY`); allocate `ha_elect_ctx_t`; spawn per-peer pthreads that call `connect_ssl_raw()` + send `OP_requestVote`; `sem_wait(&ctx->done)` for result; handle split-vote: one retry with incremented term + `xorshift64_jitter()` delay; second split → `CTRL_STANDBY` - - Replace `am_i_primary()` call in `myapp.c` with `am_i_primary_quorum()` - - Delete `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()` - - Log INFO: election outcome, term, votes granted, quorum size (10.1); log INFO: FSM transitions (10.5) - - `make test` GREEN on all election tests - - _Requirements: 2.1, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12, 10.1, 10.5_ - - _Boundary: Controller FSM (ha.c, myapp.c)_ - -- [x] 5. Change C: Config bootstrap for unconfigured nodes -- [x] 5.1 Write failing tests for config bootstrap - - In `src/controller/testsuite/test_ha_bootstrap.c`: test `config_tag != 0` → returns immediately; test all-zero peers → proceed to election; test any peer has `tag > 0` → `defer_config_fetch = true`; test no peer reachable → retries with exponential backoff (mock clock); test deferred fetch called when election winner known and `defer_config_fetch` is set; test `xce_probe_config_tag()` returns 0 on success, `-ECONNREFUSED` on unreachable - - `make test` fails - - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_ - -- [x] 5.2 Implement xce_probe_config_tag() - - Single-RTT SSL probe: `connect_ssl_raw()` + xce encode `OP_probeConfigTag` + decode `ProbeConfigTagReply` - - Returns 0 on success with `*tag_out` set; returns `-ECONNREFUSED` / `-ETIMEDOUT` on unreachable peer - - Peer dispatch gate exempts `OP_probeConfigTag` in `CTRL_INIT` and `CTRL_ELECTING` states - - Log INFO: probe outcome per peer (10.2) - - `make test` GREEN on probe unit tests - - _Requirements: 3.1, 1.3_ - - _Boundary: Config Bootstrap (config_repl.c)_ - -- [x] 5.3 Implement bootstrap_config_if_needed() and wire into startup - - Returns immediately if `config_tag != 0` - - Probes all peers; three outcomes: all-zero → proceed; any non-zero → set `defer_config_fetch`, return; no peer reachable → retry with backoff 500ms→10ms→...→10s cap; log WARN each failed attempt - - Integrate into `myapp.c`: call after `cluster_open_listener()` + `iomgr_init()` + `vtaskmgr_init()`, before `ha_wait_for_heartbeat_or_timeout()` - - After election: if `defer_config_fetch && role == CTRL_STANDBY` → `do_one_config_fetch()` from proven winner - - Log INFO: bootstrap outcome (10.2) - - `make test` GREEN on all bootstrap tests - - _Requirements: 3.2, 3.3, 3.4, 3.5, 10.2_ - - _Boundary: Config Bootstrap (config_repl.c, myapp.c)_ - -- [x] 6. Change 9: Election state durability (ha-term.bin) -- [x] 6.1 Write failing tests for ha-term.bin persist and restore - - In `src/controller/testsuite/test_ha_durability.c`: test `ha_persist_term()` writes file; test `ha_init()` restores term and voted_for from file; test CRC mismatch → zeroes state safely with WARN log; test missing file → term=0, no prior vote; test write+rename atomicity (partial write leaves old file intact) - - `make test` fails - - _Requirements: 9.1, 9.2, 9.3_ - -- [x] 6.2 Implement ha_init() restore path for ha-term.bin - - Read `{workdir}/ha-term.bin` in `ha_init()` at startup; verify magic and CRC32; on match → restore `ha.term` and `ha.voted_for_*`; on missing → term=0 (3rd boot); on CRC mismatch → log WARN, initialize to term=0 - - `ha_persist_term()` already implemented in task 4.2; this task only adds the read-path to `ha_init()` - - `make test` GREEN on all durability tests - - _Requirements: 9.1, 9.2, 9.3_ - - _Boundary: Term Durability (ha.c)_ - -- [x] 7. Change E: Majority-commit WAL replication -- [x] 7.1 Write failing tests for repl_broadcast and repl_wait_majority_ack - - In `src/controller/testsuite/test_repl_majority.c`: test 2/3 ACKs → `repl_wait_majority_ack` returns 0; test timeout returns `-ETIMEDOUT`; test single-node (`n_active == 0`) returns 0 immediately; test `in_catch_up=1` standby excluded from quorum denominator; test `pending_ack_count > max` removes standby from active set; test `OP_walAck` vs `OP_upgradeConfigAck` dispatched separately via `ack_op` - - `make test` fails - - _Requirements: 5.1, 5.4, 5.5, 5.8, 8.2_ - -- [x] 7.2 Implement repl_broadcast() (replace stub) - - Add `pending_ack_count` field to `standby_conn_t` - - `repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len)`: iterate active standbys; skip `in_catch_up==1`; if `pending_ack_count > max_pending_acks` → remove from active set, trigger catch-up reconnect, log WARN; else send payload over `standby.ctrl_fd`, increment `pending_ack_count`; returns 0 after fanout initiated - - `make test` GREEN on broadcast fanout tests - - _Requirements: 5.1, 5.4_ - - _Boundary: Replication Layer (repl.c, repl.h)_ - -- [x] 7.3 Implement repl_wait_majority_ack() - - `repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)`: `cond_timedwait` until `acked_count + 1 >= (n_active + 1) / 2`; single-node shortcircuit; `ack_op` disambiguates WAL vs config ACKs; returns 0 on quorum, `-ETIMEDOUT` on timeout - - `repl_on_ack()` extended to decrement `pending_ack_count` and signal condition - - Log INFO on quorum (10.3); log WARN on timeout (10.3) - - `make test` GREEN on all majority-ack tests - - _Requirements: 5.1, 5.5, 5.8, 10.3_ - - _Boundary: Replication Layer (repl.c)_ - -- [x] 7.4 Write failing tests for persist_truncate_wal_above() - - In `src/controller/testsuite/test_persist_truncate.c`: test truncation at segment boundary deletes whole segment; test truncation mid-segment scans sequentially to correct byte offset; test idempotency (calling twice with same `cutoff_lsn` is safe); test `applied_lsn` updated atomically; test no-op when `cutoff_lsn == applied_lsn` - - `make test` fails - - _Requirements: 5.6, 5.7_ - -- [x] 7.5 Implement committed_lsn in persist_t and persist_truncate_wal_above() - - Add `_Atomic uint64_t committed_lsn` to `persist_t` (invariant: `committed_lsn <= applied_lsn`) - - `persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)`: scan segment files named `seg-{LSN_hex}.wal`; find segment containing `cutoff_lsn`; sequential scan of entries until LSN > cutoff_lsn; truncate at that byte offset; delete later segments; atomically update `p->applied_lsn = cutoff_lsn`; idempotent - - Update WAL trim guard: use `committed_lsn` not `applied_lsn` as threshold - - `make test` GREEN on all truncation tests - - _Requirements: 5.6, 5.7_ - - _Boundary: Persistence Layer (persist.c, persist.h)_ - -- [x] 7.6 Wire majority-commit into persist_wal_sync_append() - - In `persist_wal_sync_append()`: after local write + fsync for `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE` → call `repl_broadcast(c, OP_walAck, payload, len)` + `repl_wait_majority_ack(c, commit_timeout_ms, OP_walAck)` → on quorum advance `committed_lsn` - - `WAL_HOST_STATE`, `WAL_CHECKPOINT_REF` bypass to `persist_wal_async()` (unchanged) - - `make test` passes; integration test verifies committed WAL entries survive failover - - _Requirements: 5.1, 5.2, 5.3_ - - _Boundary: Persistence Layer (persist.c), Replication Layer (repl.c)_ - -- [x] 8. Change H: Majority-commit config replication (P) -- [x] 8.1 Implement OP_upgradeConfigAck handler and committed_config_tag - - Add `committed_config_tag` field to cluster config state (ha_state_t via `last_known_committed_config_tag`); piggybacked on `CtrlHeartbeat` via `committedConfigTag` - - Standby handler for `OP_upgradeConfig`: write config blob locally + reply `OP_upgradeConfigAck` with `ackedTag` - - `push_config_to_standbys()` upgraded: write locally → `repl_broadcast(c, OP_upgradeConfig, blob, len)` → `repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck)` → on quorum `atomic_store(&committed_config_tag, tag)` → return; on timeout return `-ETIMEDOUT` - - Log INFO on quorum (10.4); log WARN on timeout (10.4) - - `make test` GREEN using `test_repl_majority.c` with `OP_upgradeConfigAck` - - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 10.4_ - - _Boundary: Config Replication (config_repl.c)_ - - _Depends: 7.3_ - -- [x] 9. Change F: Persistent standby-to-primary control channel -- [x] 9.1 Extract apply_wal_entry() prerequisite - - Extract the WAL replay loop body in the startup path into standalone `apply_wal_entry(cluster_t *c, wal_entry_t *e)` function; existing startup replay calls it in a loop; behavior unchanged - - `make test` GREEN; this is a preparatory refactor enabling standby catch-up streaming - - _Requirements: 6.5_ - - _Boundary: Persistence Layer (persist.c) / WAL replay_ - -- [x] 9.2 Implement ha_find_and_connect_to_primary() with three-tier discovery - - Tier 1: if `ha.voted_for_name` set → direct connect; on `XCE_ERR_NOT_READY` → backoff, retry Tier 1 - - Tier 2: probe remaining peers; follow `XCE_ERR_STANDBY` `redirect_host/port` immediately - - Tier 3: blind sequential scan with 100ms→5s exponential backoff - - Sets `ha.primary_ctrl_fd` on success; registers fd with iomgr `EPOLLIN | EPOLLRDHUP` - - Called after standby wins no election; also called by re-election loser - - `make test` GREEN (mock endpoints for each tier) - - _Requirements: 6.1_ - - _Boundary: Persistent Control Channel (ha.c)_ - -- [x] 9.3 Implement extended OP_ctrlHello handler and state sync sequence - - Extend `CtrlHelloRequest` with `applied_lsn` and `config_tag`; extend `CtrlHelloReply` with `committed_lsn`, `needs_snapshot`, `config_tag` - - Primary side `handle_ctrlhello()`: `repl_add_standby()` → `repl_start_catchup()`; reply; if `needs_snapshot` → `persist_checkpoint()` + send `OP_snapshotTransfer`; stream `OP_walCatchupEntry` from `[standby.applied_lsn+1 → committed_lsn]`; on `OP_walAck` reaching `committed_lsn` → `repl_complete_catchup()`; if config_tag mismatch → call `push_config_to_standbys()` - - Standby side: store `last_known_committed_lsn`; call `persist_truncate_wal_above(primary_committed_lsn)`; set `in_catch_up=1`; apply `OP_snapshotTransfer` if present; apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each; on catch-up end: `in_catch_up=0` - - `make test` GREEN; integration test: fresh standby joins and reaches parity with primary committed_lsn - - _Requirements: 6.5, 6.6, 6.7_ - - _Boundary: Persistent Control Channel (ha.c, cluster.c), Replication Layer (repl.c)_ - - _Depends: 7.5, 9.1_ - -- [x] 9.4 Implement EPOLLRDHUP failover and heartbeat timeout vtask - - `EPOLLRDHUP` event on `primary_ctrl_fd` → close fd, set `primary_ctrl_fd = -1`, transition to `CTRL_ELECTING`, call `persist_truncate_wal_above(last_known_committed_lsn)`, fire `am_i_primary_quorum()` - - Heartbeat timeout vtask: armed on each heartbeat receipt; fires after `election_timeout_ms + rand_jitter` of silence; same action as EPOLLRDHUP path - - On `CTRL_STANDBY` after new election → close outbound primary fd; on `CTRL_PRIMARY` after new election → accept inbound `OP_ctrlHello` connections; reuse per-standby fd table - - Log INFO: FSM transitions (10.5) - - `make test` GREEN; integration test: TCP disconnect → sub-second election start - - _Requirements: 6.2, 6.3, 6.4, 6.8, 10.5_ - - _Boundary: Persistent Control Channel (ha.c), Controller FSM (ha.c)_ - - _Depends: 9.2_ - -- [x] 10. Change G: libxce client-side transparent retry (P) - - **NOTE (2026-06-16):** only the standalone primitive (10.2 `xce_request_with_retry()`) + - its unit tests (10.1) were delivered. The primitive is **not wired into any production - request path** — `xce_*_req` wrappers still call plain `xce_request()` — and it lacks - multi-master rotation to discover a new primary after the connected one is killed (it only - follows an explicit redirect). Production integration + master-list failover are owned by - the **`libxce-failover`** spec (see `.kiro/specs/libxce-failover/brief.md`). Req 7 is - therefore satisfied at the primitive level only; transparent failover is not yet live. -- [x] 10.1 Write failing tests for xce_request_with_retry() - - In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation) - - `make test` fails - - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_ - -- [x] 10.2 Implement xce_retry_config_t and xce_request_with_retry() - - Add `xce_retry_config_t` to `request.h` with defaults: `max_retries=20`, `initial_retry_ms=50`, `max_retry_ms=5000`, `backoff_multiplier=2.0` - - `xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg)`: NULL cfg → use defaults; `XCE_ERR_NOT_READY` → backoff, retry; `XCE_ERR_STANDBY` with redirect → reconnect, retry; `ECONNREFUSED`/`ETIMEDOUT` → reconnect, retry; `XCE_ERR_AUTH`/`XCE_ERR_PERM`/`XCE_ERR_INVALID`/`XCE_ERR_NOENT` → surface immediately; exhausted → surface last error - - Jitter via `rand_r()` to prevent thundering herd (7.6) - - `make test` GREEN on all retry tests; 50-client concurrency test passes - - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_ - - _Boundary: Client Retry (request.c, request.h)_ - -- [x] 11. Integration validation and cleanup -- [x] 11.1 End-to-end: concurrent startup and config-fresh election - - Manual test (2026-06-03): 3-node cluster deployed via `/xce-deploy-ha` - - ctrl-1 wins election → CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY - - Config pushed to ctrl-2 and ctrl-3 (tag=4986624fd309142c, 2/2 standbys) - - All config files verified on standbys - - _Requirements: 1.1, 2.1, 2.7_ ✅ - - _Depends: 4.3, 9.4_ - -- [x] 11.2 End-to-end: failover and WAL durability - - Manual test (2026-06-03): liveness poll detects primary loss within ~11s - - Both standbys correctly transition CTRL_ELECTING via ha_on_primary_ctrl_fd_rdhup - - Note: With configs preserved on disk, failover election succeeds - - _Requirements: 5.6, 6.2, 5.5_ ✅ - - _Depends: 7.6, 9.4_ - -- [x] 11.3 End-to-end: unconfigured node bootstrap - - Manual test (2026-06-03): ctrl-2/ctrl-3 start with config_tag=0 - - Participate as voters only; receive config via handle_repl_config_push from primary - - Config files appear in standby conf/ directories after push - - _Requirements: 3.2, 3.5_ ✅ - - _Depends: 5.3, 9.3_ - -- [x] 11.4 End-to-end: config upgrade majority-commit and term durability - - Manual test (2026-06-03): ha-term.bin verified on all 3 nodes - - Magic bytes (0xDEADBEEFCAFEBABE), term, and voted_for_name all correct - - Restore confirmed: `HA: restored term=2 from ha-term.bin` on ctrl-1 restart - - _Requirements: 8.3, 9.1, 9.2_ ✅ - - _Depends: 8.1, 6.2_ - -- [x] 11.5 Full ASAN build verification - - `meson test --suite controller` clean (1/1 PASS) - - No new ASAN errors; pre-existing LeakSanitizer from test_worker_join only - - _Requirements: 4_ ✅ - - _Depends: 2.3, 4.3_ diff --git a/.kiro/specs/ha-consistency-invariants/brief.md b/.kiro/specs/ha-consistency-invariants/brief.md deleted file mode 100644 index 6b96fc26..00000000 --- a/.kiro/specs/ha-consistency-invariants/brief.md +++ /dev/null @@ -1,102 +0,0 @@ -# Brief: ha-consistency-invariants - -## Governing Model (decision: 2026-06-10) - -XCE HA adopts **Raft as the consensus model and CP as the CAP stance**. This supersedes the earlier "No Raft" steering decision (the single-writer constraint is retained — the controller remains the sole log writer — but log commitment now follows Raft semantics, not just election). All consistency requirements in this spec are **deductions from the Raft paper + CP**, not ad-hoc rules: - -| Deduced rule | Raft / CP source | -|---|---| -| 1. No distinct "primary committing" state | Raft has exactly three server states: Follower, Candidate, Leader. A candidate winning an election becomes Leader immediately. Raft's answer to "new leader can't yet serve safely" is not a fourth state — it is the commit rule: a leader may only count own-term entries toward commitment (§5.4.2), conventionally satisfied by committing a no-op entry at term start. The commit barrier subsumes the transitional state. | -| 2. Majority ACK is the hard commit barrier; reject otherwise | Raft commit rule (§5.3): an entry is committed only when the leader has replicated it on a majority; `commitIndex` advances only on majority match, and the state machine applies entries only once committed. CP: a leader in a minority partition cannot commit — the correct behavior is reject/error to the caller, never local-apply-and-hope. | -| 3. Full-consistency best-effort while running | Raft leaders maintain per-follower progress (nextIndex/matchIndex) and continuously retry AppendEntries to ALL followers (§5.3); majority is the commit threshold, not the fan-out target. Lagging followers catch up via continuous log streaming or InstallSnapshot — driven by the leader, not gated on heartbeat round-trips. | - -XCE mapping: `committed_lsn` ≈ Raft `commitIndex`; WAL entries ≈ log entries; standby ACK ≈ AppendEntries success; snapshot transfer ≈ InstallSnapshot; `CTRL_PRIMARY_COMMITTING` ≈ a state Raft does not have. - -## Problem - -The HA controller's consistency model is inconsistent across operation types. Job submit goes through a majority-ACK commit barrier, but job state changes, queue control, and worker control are applied locally and replicated fire-and-forget. A primary that has lost quorum can still have in-flight writes "commit" locally, and a newly elected primary force-promotes after a 60s timeout even with zero converged standbys. The `ha-config-bootstrap` spec explicitly deferred this as the "CP-vs-availability operator decision"; the decision is now made by adopting CP: **reject, never degrade**. - -## Current State - -Audit of `src/controller/` (2026-06-10) found these violations of the Raft+CP deductions: - -### Rule 1 — drop the PRIMARY_COMMITTING / PRIMARY distinction -- `CTRL_PRIMARY_COMMITTING` (ha.h:59-65) is a transitional leader state set at `ha.c:505` (raft callback) and `ha.c:2248` (legacy path), checked in ~7 places. Raft has no such state. -- `ha_is_primary()` (ha.c:917) returns 0 in COMMITTING; `ha_should_redirect()` (ha.c:1655) redirects clients to a primary that does not exist yet → redirect loop during the committing window. -- `committing_timeout_fn` (ha.c:3092-3121) force-promotes to PRIMARY after 60s **even with zero converged standbys** — accepting writes with no replication (violates the Raft commit rule outright). - -### Rule 2 — majority-ACK is the hard commit barrier; reject otherwise -- Job state changes (start/finish/fail/stop/resume/update/ckpt/rusage) use `persist_wal_async` only — local apply + fire-and-forget broadcast, no majority wait (persist.c:1169). Same for queue control (persist.c:1199) and worker control (persist.c:1241). Only WAL_JOB_SUBMIT uses `persist_wal_sync_append` + `repl_wait_majority_ack` (persist.c:1003-1049). -- `ha_has_write_quorum` (ha.c:1507-1525) counts connected standbys including those in catch-up mode, but `repl_broadcast` skips catch-up standbys — the gate can pass while the write cannot actually reach quorum. (Raft equivalent: matchIndex, not connection count, is what commitment consults.) -- Config push timeout: `push_config_to_standbys` (config_repl.c:916-928) logs WARN on majority-ACK timeout but returns the ack count; caller (config_repl.c:1057) ignores it — config upgrade proceeds without confirmed majority. -- TOCTOU: quorum checked at dispatch gate only; `repl_wait_majority_ack` (repl.c:430-434) short-circuits to success when `n_active == 0`, so a write admitted before mass disconnect commits with primary-only durability. -- Quorum loss while PRIMARY blocks writes at the gate (correct CP direction) but combined with the n_active==0 short-circuit this leaves a hole. -- **Apply-before-commit (to verify in requirements phase)**: Raft applies entries to the state machine only after commit. Audit confirmed `committed_lsn` advances only on majority ACK for job submit, but whether the in-memory state mutation precedes the ACK on each path needs per-path verification; local WAL append before ACK is fine (that is the Raft log), in-memory apply before commit is not. - -### Rule 3 — full consistency best-effort while running -- `repl_broadcast` (repl.c:210-282) already fans out to ALL active standbys — matches Raft. -- Lagging standbys evicted to catch-up mode (pending_ack_count > MAX_PENDING_ACKS) only re-enter via the 3s heartbeat-driven decision loop (`ha_dispatch_repl_action_for_standby`, ha.c:1373) — catch-up is heartbeat-gated, not leader-driven continuous streaming as Raft prescribes; a busy primary lets catch-up standbys fall arbitrarily behind. - -## Desired Outcome - -1. Single leader state: winning an election transitions directly to `CTRL_PRIMARY`. `CTRL_PRIMARY_COMMITTING`, its whitelist branch, and `committing_timeout_fn` are removed. Post-election write safety comes from the Raft own-term commit rule (the universal commit barrier; optionally an explicit no-op entry committed at term start), not a special state. -2. Every cluster-state-changing operation (job submit, job state change, queue control, worker control, config push) commits only after majority ACK. On timeout or quorum loss the operation is **rejected with an error to the caller**. `committed_lsn` / `committed_config_tag` only advance on real majority — they are XCE's `commitIndex` and must behave like it. -3. Quorum accounting is honest and ACK-based: `ha_has_write_quorum` counts only standbys actually reachable for the write; majority is verified at ACK time (matchIndex-style), not just at the dispatch gate; `repl_wait_majority_ack` never succeeds with zero active standbys when `n_peers > 1`. -4. Full-consistency best-effort: broadcast remains all-standbys; catch-up streaming is leader-driven and continuous (no heartbeat round-trip gating), with snapshot fallback for standbys behind the oldest retained WAL. - -## Approach - -Chosen: **new single spec layered on ha-wal-replication's ACK machinery, with the Raft paper (§5.3, §5.4) as the requirements authority**. - -The recently built WalAckReply / majority-ack / committed_lsn infrastructure (ha-wal-replication tasks 3.x-5.x) is the enforcement mechanism; this spec generalizes it from WAL_JOB_SUBMIT to all state-changing WAL types, makes rejection (not degradation) the universal timeout behavior, and then deletes the now-redundant COMMITTING state. - -Order matters: implement the universal commit barrier (Rule 2) FIRST, then merge the states (Rule 1) — merging first would open a window where a fresh primary accepts unreplicated writes, violating the own-term commit rule. - -Rejected alternatives: -- Extending ha-wal-replication in place — scope creep on a mid-implementation spec; state-machine surgery is a different boundary than WAL transport. -- Multi-spec split (state merge / write barrier / catch-up as 3 specs) — the pieces are safety-interlocked (see ordering above); splitting risks landing Rule 1 without Rule 2. - -## Scope - -- **In**: - - Route WAL_JOB_* state changes, WAL_QUEUE_CTRL, WAL_WORKER_CTRL through the sync majority-ACK path; reject on timeout. - - Verify (and fix where violated) apply-after-commit ordering on each write path. - - Config push: error on majority-ACK timeout, propagate to caller, fail the upgrade request. - - Fix `ha_has_write_quorum` catch-up accounting; verify quorum at ACK completion; remove the n_active==0 success short-circuit for n_peers > 1. - - Remove `CTRL_PRIMARY_COMMITTING`: direct ELECTING→PRIMARY transition, delete committing whitelist branch and `committing_timeout_fn`, fix `ha_is_primary` / `ha_should_redirect` semantics; evaluate explicit no-op own-term commit at promotion. - - Leader-driven continuous catch-up streaming (decouple from heartbeat round-trip). - - Tests: per-WAL-type commit-barrier tests, quorum-loss rejection tests, election→write-availability convergence test, catch-up progress test. -- **Out**: WAL transport/encoding (ha-wal-replication owns it); election/vote logic (raft-library owns it — already Raft-conformant); single-node (`n_peers <= 1`) behavior — self-quorum stays as-is (a 1-node Raft cluster's majority is 1); checkpoint replication (deliberately local-only — snapshots are per-node in Raft too); automatic primary step-down on quorum loss (Raft leaders in a minority partition also stay leader but cannot commit — blocking writes satisfies CP; step-down-on-silence is a possible follow-on optimization). - -## Boundary Candidates - -- Commit-barrier generalization (persist.c sync/async split, repl.c ack wait) -- Quorum accounting + TOCTOU closure (ha.c gate, repl.c ack path) -- State-machine simplification (ha.c enum, transitions, timeout removal) -- Leader-driven catch-up (config_repl.c streaming, ha.c decision loop) - -## Out of Boundary - -- Raft vote/election correctness (raft-library) -- WAL wire format and standby apply path (ha-wal-replication) -- Worker/scheduler reconnect-on-failover (worker-join-refactor / P2.5) -- Read consistency / leader leases (Raft §8 read-only optimization — future spec) -- Performance tuning of sync-commit latency (acceptable cost of CP; measure, don't optimize, in this spec) - -## Upstream / Downstream - -- **Upstream**: ha-wal-replication (ACK machinery, committed_lsn — must complete its remaining tasks first), ha-repl-hardening (payload data plane), raft-library (become_leader callback signature). -- **Downstream**: read-consistency / lease work; operator-facing degraded-mode metrics; possible primary step-down-on-quorum-loss spec; steering update (product.md decision log — "No Raft" superseded, done at discovery time). - -## Existing Spec Touchpoints - -- **Extends behavior introduced by**: ha-wal-replication (generalizes its majority-ack from job-submit to all WAL types), ha-config-bootstrap (resolves its documented "commit timeout silently degrades" deviation as REJECT per CP). -- **Adjacent**: ha-heartbeat-liveness (heartbeat period interacts with catch-up loop), raft-library (election→PRIMARY transition point changes). - -## Constraints - -- C11, vtask async I/O — commit barrier must not block vtask handlers; reuse the existing delaymgr-based `repl_wait_majority_ack` pattern. -- TDD mandatory: testsuite/ tests before C changes; existing convergence tests in `test_ha_connection_reuse.c` and ha-wal-replication tests must stay green. -- Single-writer model preserved: the controller remains the sole log writer (XCE's simplification of Raft — no concurrent client proposals); this spec narrows when the single writer may commit, it does not add writers. -- Semantics for `n_peers <= 1` (standalone) unchanged: self-quorum, no barrier (consistent with single-node Raft). -- Where this brief and the Raft paper disagree, the Raft paper wins; deviations must be documented in the design with justification. diff --git a/.kiro/specs/ha-consistency-invariants/design.md b/.kiro/specs/ha-consistency-invariants/design.md deleted file mode 100644 index 5756f695..00000000 --- a/.kiro/specs/ha-consistency-invariants/design.md +++ /dev/null @@ -1,425 +0,0 @@ -# Design: ha-consistency-invariants - -## Overview - -**Purpose**: This feature gives operators of multi-node XCE controller clusters a uniform Raft+CP consistency contract: every cluster-state-changing operation is durable on a majority before it is confirmed, operations that cannot reach majority are rejected (requests) or retained until committed (facts), and leadership is a single role with no weaker transitional state. - -**Users**: Cluster operators gain predictable failover semantics (no acknowledged change is ever lost); client/API consumers gain honest confirmations; the scheduler is protected from acting on uncommitted state. - -**Impact**: Removes `CTRL_PRIMARY_COMMITTING` from the HA state machine; generalizes the majority-ACK commit barrier from WAL_JOB_SUBMIT to all state-changing WAL types via a new commit engine; fixes quorum accounting (per-entry ACK verification, catch-up exclusion, TOCTOU closure); adds a commit-watermark egress filter to dataex; converts demotion into truncate-and-self-restart; makes catch-up ACK-driven. - -### Goals -- Majority ACK as the hard commit barrier for all state-changing operations (`committed_lsn` ≈ Raft commitIndex) -- Single leader role; post-election write safety via own-term no-op commit (Raft §5.4.2) -- Facts (worker-reported transitions) retained and retried until committed — never dropped while the primary runs and leads -- Cluster-wide single-outcome convergence for uncommitted suffixes (truncate + self-restart on demotion) -- Full fan-out with leader-driven, ACK-continued catch-up - -### Non-Goals -- Election/vote correctness (raft-library), WAL wire format (ha-wal-replication) -- Read consistency / leases — status queries may observe provisional in-memory state (documented deviation) -- Worker rejoin job-state re-sync (roadmap backlog; designated backstop for facts lost across failover) -- Entity-bearing snapshots; primary step-down on quorum loss; checkpoint replication -- Performance tuning of commit latency (measure only) - -## Boundary Commitments - -### This Spec Owns -- Commit semantics: when `committed_lsn` / `committed_config_tag` advance, and what confirmation means for every state-changing operation on the primary -- The commit engine (`commit.c`): per-LSN ACK evaluation, waiter completion, own-term gate, admission cap -- Operation classification (request-shaped vs fact-shaped) and the per-WAL-type write path -- HA leader-role lifecycle simplification (promotion no-op, COMMITTING removal, demotion truncate+restart) -- Quorum accounting used by the dispatch gate and commit evaluation -- The commit-watermark egress rule for scheduler-bound exports -- Quorum-loss observability logging - -### Out of Boundary -- Vote RPCs, term management, freshness gates (raft-library); WalAckReply wire format and standby apply mechanics (ha-wal-replication) -- Worker protocol changes of any kind (rejoin re-sync is a future spec) -- Snapshot file format (sentinel stays as-is); dual-zarray COW wiring -- apiserver/Go-tier behavior (error codes pass through existing envelopes) - -### Allowed Dependencies -- `repl.c` standby registry and `acked_lsn` per-standby tracking (ha-wal-replication machinery) — the commit engine consumes ACK events, never manages connections -- `persist.c` WAL append/replay/truncation primitives; `persist_recover` boot path (reused by self-restart) -- iomgr/vtaskmgr/delaymgr threading rules per `ha-async-io.md` (ACK processing stays on iomgr; blocking waits only on vtask threads) -- Dependency direction: `include/commit.h` depends only on libc/stdatomic types. `commit.c` is called BY `persist.c` (append), `repl.c` (ACK events), and `ha.c` (promotion/demotion); it calls into none of them except via registered callbacks. - -### Revalidation Triggers -- Any change to `acked_lsn` semantics or WalAckReply payload (ha-wal-replication) → re-verify commit evaluation -- Worker-rejoin-resync spec landing → re-verify idempotency guards and fact replay ordering -- Entity-bearing snapshots landing → re-evaluate demotion self-restart vs snapshot-based rebuild -- Any new state-changing opcode/WAL type → must be classified request/fact and routed through the commit engine - -## Architecture - -### Existing Architecture Analysis -- Single-writer model: only the primary appends WAL entries; standbys ACK (`acked_lsn` = matchIndex equivalent already exists per `standby_conn_t`) -- Hooks fire on `HOOK_AFTER_EVENT` after in-memory mutation — retained as-is under the provisional-state model (in-memory state is a provisional view; the committed log is authoritative; commit gates confirmation and irrevocable egress, not mutation) -- ACKs are decoded on the iomgr thread (`peer_ctrl_msg_handler` → `repl_on_ack`) — blocking commit-waiters can always be woken; pool exhaustion is throughput-only -- `repl_wait_majority_ack` currently counts `acked_lsn > 0` (any ACK ever) — replaced by per-entry evaluation in the commit engine -- Technical debt addressed: TOCTOU between dispatch gate and ACK; catch-up standbys counted toward write quorum; config push ACK result ignored; `committing_timeout_fn` force-promotion - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph clients - API[apiserver client] - WK[worker notify] - SCHED[scheduler] - end - subgraph controller_primary - GATE[ha dispatch gate] - HOOKS[persist hooks] - WAL[persist WAL append] - CE[commit engine] - REPL[repl broadcast] - DX[dataex egress filter] - HA[ha role lifecycle] - end - STBY[standbys] - - API --> GATE - WK --> GATE - GATE --> HOOKS - HOOKS --> WAL - WAL --> CE - WAL --> REPL - REPL --> STBY - STBY -- WalAck on iomgr --> CE - CE -- committed_lsn --> DX - DX --> SCHED - HA -- promote noop / demote restart --> CE -``` - -**Architecture Integration**: -- Selected pattern: single commit engine as the authority for `committed_lsn` advancement; all write paths converge on it with one of two completion modes -- Existing patterns preserved: hook-based persistence dispatch, full fan-out broadcast, heartbeat-driven repl decision loop (extended with ACK-driven continuation), `errnox` early-return error style -- New component rationale: `commit.c` exists because three requirements families (2.x, 3.x, 5.x) are one capability; placing it in `repl.c` would bloat connection management with consensus logic -- Steering compliance: never block iomgr; blocking waits only on vtask threads with admission cap relative to `n_vcpus`; TDD with testsuite/ tests first - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), pthreads, stdatomic | commit engine, HA lifecycle | no new dependencies | -| Data / Storage | existing WAL segments (persist.c) | log = authoritative state + fact queue | new WAL_NOOP entry type, no schema change | -| Messaging / Events | existing Cap'n Proto WalAckReply | ACK transport | no .capnp changes | - -## File Structure Plan - -### Directory Structure -``` -src/controller/ -├── commit.c # NEW — commit engine: per-LSN ACK evaluation, waiter registry, -│ # own-term gate, admission cap, commit-index advancement -├── include/commit.h # NEW — commit engine public API (controller-internal) -├── ha.c # MODIFIED — promotion/demotion lifecycle, gate, quorum accounting -├── ha.h # MODIFIED — remove CTRL_PRIMARY_COMMITTING, add own-term fields -├── persist.c # MODIFIED — write-path classification, WAL_NOOP, commit-engine calls -├── repl.c # MODIFIED — repl_on_ack feeds commit engine; legacy wait removed -├── repl.h # MODIFIED — ack callback signature -├── config_repl.c # MODIFIED — config barrier error propagation; ACK-driven catch-up -├── job.c # MODIFIED — submit_lsn tagging, mark_job_* idempotency guards -├── dataex.c # MODIFIED — egress watermark filter in collect path -│ # (Note: as-built, the filter predicate/wrapper live in a -│ # new file egress.c instead — see ha-convergence-recovery) -└── testsuite/ - ├── test_commit_engine.c # NEW — per-LSN majority, own-term gate, admission cap, monotonicity - ├── test_ha_role_lifecycle.c # NEW — direct promotion, no-op commit, demotion truncate+restart seam - └── test_egress_filter.c # NEW — uncommitted submit excluded from collect -src/include/persist.h # MODIFIED — WAL_NOOP type, classification enum -``` - -### Modified Files (what and why) -- `ha.c/ha.h` — delete COMMITTING state, watchdog (`committing_timeout_fn`), `ha_check_majority_convergence`, `repl_ack_count`; promotion appends no-op; `ha_has_write_quorum` excludes catch-up standbys; demotion path truncates + self-restarts; quorum-loss transition logging -- `persist.c` / `include/persist.h` — per-WAL-type classification table drives sync-wait vs fact-register; `WAL_NOOP` append/replay (replay = no-op); truncation unchanged -- `repl.c/repl.h` — `repl_on_ack` forwards `(ep, acked_lsn)` to `commit_on_ack`; `repl_wait_majority_ack`'s counting logic retired in favor of commit-engine evaluation (function shell may remain as adapter) -- `config_repl.c` — `push_config_to_standbys` returns `-ETIMEDOUT` on missed majority; caller fails the upgrade request; `ha_stream_wal_to_standby` continuation on walAck instead of next heartbeat -- `job.c` — `job_t` gains `submit_lsn`; `mark_job_*` gain state-transition guards (no DONE→RUN, idempotent re-apply) -- `dataex.c` — `collect_jobs` (scheduler export path only) skips jobs with `submit_lsn > committed_lsn` (as-built: this predicate/wrapper live in new file `egress.c` instead — see the Egress Filter section note below) - -## System Flows - -### Request-shaped write (e.g., job submit) - -```mermaid -sequenceDiagram - participant C as client - participant V as vtask thread - participant CE as commit engine - participant R as repl broadcast - participant S as standby - participant IO as iomgr thread - C->>V: state-changing request - V->>V: gate check incl write quorum - V->>V: mutate in-memory (provisional) - V->>CE: append WAL entry lsn=N, mode=WAIT - CE->>CE: admission cap check - V->>R: broadcast entry N - R->>S: wal entry N - V->>CE: block on commit(N) with timeout - S->>IO: WalAck lsn>=N - IO->>CE: commit_on_ack - CE->>CE: majority for N? advance committed_lsn - CE-->>V: wake waiter (committed | timeout) - alt committed - V->>C: success - else timeout or quorum lost - V->>C: error (operation failed) - end -``` - -Key decisions: admission cap rejects (NOT_READY) before blocking when `inflight_waits >= n_vcpus - COMMIT_WAIT_RESERVED`; majority is evaluated against the entry's LSN (`acked_lsn >= N` per standby, plus self), re-verified at ACK time — gate-time connectivity never substitutes (4.2, 4.3). - -### Fact-shaped write (worker-reported transition) — log-is-the-queue -Same path until the commit engine: mode=RETAIN. No blocking wait, no timeout; the vtask returns immediately after append+broadcast. The uncommitted suffix is re-delivered to lagging standbys by catch-up streaming; `committed_lsn` advances as ACKs arrive. The fact is "pending" purely by virtue of `lsn > committed_lsn` (2.8). Idempotency guards in `mark_job_*` make replay/duplicate application safe. - -### Role lifecycle (after this spec) - -```mermaid -stateDiagram-v2 - [*] --> INIT - INIT --> STANDBY: heartbeat seen - INIT --> ELECTING: timeout - ELECTING --> PRIMARY: quorum won, truncate to safe_lsn, append noop - ELECTING --> STANDBY: lost or higher term - PRIMARY --> RESTART: higher term seen, truncate above committed_lsn - STANDBY --> ELECTING: primary lost - RESTART --> INIT: process self restart via boot recovery -``` - -Promotion: `raft_cb_become_leader` sets `CTRL_PRIMARY` directly, registers persistence hooks, appends a `WAL_NOOP` entry of the current term, and broadcasts it. Request-shaped writes are accepted immediately and simply wait behind the contiguous commit index (the no-op commits first); status queries report primary (1.4). No wall-clock promotion/force path exists (1.5, 3.3). (Note: promotion is no longer unconditional — `raft_cb_become_leader` now runs several eligibility/fencing gates (recovery-truncated, apply-divergent, config serve-fence, etc.) that can refuse promotion and leave the node alive, non-promotable, and retrying instead of becoming primary; added by `ha-commit-durability` and `ha-term-safety`, see those specs' design.md.) - -Demotion: on observing a higher term while PRIMARY, the node truncates its WAL above `committed_lsn`, logs the event, and self-restarts; boot recovery (`persist_recover` + replay of the committed prefix) rebuilds in-memory state, discarding provisional entries (2.5, 2.7). Crash without truncation: the restarted node replays its full local WAL. (Note: the "rejoin as standby ... truncates + self-restarts once" rejoin-reconciliation mechanism described here is superseded by `ha-term-safety` (task 3.2) — `ha_rejoin_reconcile` is now a no-op stub; a rejoining node's divergent suffix is instead truncated precisely via the raft library's term-aware `on_truncate_from` driven off real AppendEntries `(term,lsn)` matching, not an LSN-watermark self-restart. See `ha-term-safety/design.md`.) - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1, 1.2 | direct PRIMARY, single leader role | HA Role Lifecycle | role lifecycle | -| 1.3 | no redirect from primary | HA Role Lifecycle (`ha_should_redirect`) | — | -| 1.4 | status reports primary pre-own-term-commit | HA Role Lifecycle | role lifecycle | -| 1.5 | no wall-clock leadership change | HA Role Lifecycle (watchdog deleted) | — | -| 2.1 | confirm only after majority ACK | Commit Engine, Write-Path Classification | request write | -| 2.2 | requester ops fail on commit timeout | Commit Engine (WAIT mode), Write-Path Classification | request write | -| 2.3 | no commit-index advance without majority | Commit Engine | request write | -| 2.4 | provisional state: no confirm, no irrevocable export | Commit Engine, Egress Filter | request write | -| 2.5 | single outcome convergence incl. memory | Demotion Convergence | role lifecycle | -| 2.6 | monotonic commit index | Commit Engine | — | -| 2.7 | state reconstructible from committed log | Demotion Convergence (boot recovery reuse) | role lifecycle | -| 2.8 | facts retained+retried until committed | Commit Engine (RETAIN mode), Idempotency Guards | fact write | -| 3.1 | no confirmation before own-term entry commits | Commit Engine own-term gate, no-op at promotion | role lifecycle | -| 3.2 | pre-own-term requests: barrier or NOT_READY | Commit Engine own-term gate | request write | -| 3.3 | no timeout bypassing 3.1 | HA Role Lifecycle (watchdog deleted) | — | -| 4.1 | quorum counts only members able to receive | Quorum Accounting | — | -| 4.2 | commit requires actual per-entry ACKs | Commit Engine | request write | -| 4.3 | post-acceptance quorum shrink → fail | Commit Engine (ACK-time evaluation, timeout) | request write | -| 4.4 | never zero-peer majority when n_peers>1 | Commit Engine | — | -| 5.1–5.3 | config change commit barrier | Config Barrier | — | -| 6.1 | replicate to all reachable standbys | existing `repl_broadcast` (unchanged, verified by test) | — | -| 6.2 | catch-up without per-step heartbeat | ACK-Driven Catch-up | — | -| 6.3, 6.4 | snapshot fallback, re-include on catch-up | existing paths (unchanged, verified by test) | — | -| 6.5 | lag never blocks majority confirmation | Commit Engine (majority threshold only) | — | -| 7.1 | reject state changes without quorum | Quorum Accounting + Commit Engine | — | -| 7.2 | log quorum-loss enter/exit | Quorum Observability | — | -| 7.3 | serve status + replication while degraded | dispatch gate (existing allowlist, verified by test) | — | -| 7.4 | auto-resume on quorum restore | Quorum Accounting (stateless gate re-evaluation) | — | -| 7.5 | standalone self-quorum unchanged | Commit Engine (n_peers<=1 short-circuit preserved) | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| Commit Engine | controller/commit.c | authority for commit-index advancement and completion | 2.1–2.4, 2.6, 2.8, 3.1, 3.2, 4.2–4.4, 6.5, 7.5 | repl ACK events (P0), persist append (P0) | Service, State | -| Write-Path Classification | controller/persist.c | route each WAL type to WAIT or RETAIN mode | 2.1, 2.2, 2.8 | Commit Engine (P0), hooks (P0) | Service | -| HA Role Lifecycle | controller/ha.c | direct promotion + no-op, COMMITTING removal, redirect fix | 1.1–1.5, 3.1, 3.3 | raft-library callbacks (P0), Commit Engine (P0) | State | -| Quorum Accounting | controller/ha.c | honest gate-time quorum, catch-up exclusion | 4.1, 7.1, 7.4 | repl registry (P0) | Service | -| Config Barrier | controller/config_repl.c | config majority-ACK with error propagation | 5.1–5.3 | Commit Engine pattern (P1) | Service | -| Egress Filter | controller/egress.c, job.c | exclude uncommitted submits from scheduler export | 2.4 | committed_lsn (P0) | Service | -| Demotion Convergence | controller/ha.c | truncate + self-restart; rejoin reconciliation | 2.5, 2.7 | persist truncation + recovery (P0) | State | -| ACK-Driven Catch-up | controller/config_repl.c | continue streaming on walAck | 6.2 | repl/iomgr (P0) | Event | -| Quorum Observability | controller/ha.c | log degraded-state transitions | 7.2 | — | — | -| Idempotency Guards | controller/job.c | safe duplicate/replay of fact transitions | 2.8 (supports 2.5, future resync) | — | Service | - -### Commit Engine (`commit.c` / `include/commit.h`) - -| Field | Detail | -|-------|--------| -| Intent | Single authority for per-entry commit evaluation, `committed_lsn` advancement, and waiter completion | -| Requirements | 2.1–2.4, 2.6, 2.8, 3.1, 3.2, 4.2–4.4, 6.5, 7.5 | - -**Responsibilities & Constraints** -- Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`; only the commit engine stores to it after init/truncation paths) -- Commit rule (Raft §5.3): entry N is committed when `count(self + standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.6). (Note: superseded by `ha-commit-durability` — the primary's own self-contribution is now capped at `persist_durable_lsn` (fdatasync'd locally), not the appended tip, so an entry only in the page cache cannot count self toward majority; see that spec's design.md.) -- Own-term gate: `writable_lsn` = LSN of the promotion no-op; completion (confirmation) is withheld for any entry until `committed_lsn >= writable_lsn` (3.1, 3.2) -- Admission cap: at most `n_vcpus - COMMIT_WAIT_RESERVED` (default reserved = 2) concurrent WAIT-mode blockers; excess requests fail fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`). (Note: superseded in part by `raft-drive-async` — when a vtaskmgr pool is wired via `commit_set_pool`, admission folds onto the pool's single `VTASK_LIVENESS_RESERVED` reservation instead of the local `COMMIT_WAIT_RESERVED` constant; the local cap is the fallback only when no pool is wired. See that spec's design.md.) -- `n_peers <= 1`: self-ACK constitutes majority; WAIT mode returns immediately (7.5). For `n_peers > 1`, zero peer ACKs never commit (4.4) — the legacy `n_active == 0 → success` short-circuit is removed - -**Dependencies** -- Inbound: `repl_on_ack` (iomgr thread) — ACK events (P0); `persist_wal_*` append paths — entry registration (P0); `ha.c` — promotion no-op LSN, demotion reset (P0) -- Outbound: none (callbacks only); signals waiters via condvar broadcast with per-LSN predicates - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -/* include/commit.h — all functions thread-safe; _on_ack callable from iomgr (non-blocking) */ -typedef enum { COMMIT_MODE_WAIT, COMMIT_MODE_RETAIN } commit_mode_t; - -int commit_engine_init(commit_engine_t *ce, persist_t *p, int n_peers, int n_vcpus); -void commit_engine_fini(commit_engine_t *ce); - -/* Register entry at append time. RETAIN returns 0 immediately. - * WAIT blocks up to timeout_ms: 0 = committed; -ETIMEDOUT = not committed - * (entry remains in log; outcome resolves per 2.5); -XCE_ERR_NOT_READY = - * admission cap exceeded or own-term gate unmet at entry. */ -int commit_submit(commit_engine_t *ce, uint64_t lsn, commit_mode_t mode, - uint32_t timeout_ms); - -/* iomgr path: record standby ACK, evaluate majority, advance committed_lsn - * contiguously, wake satisfied waiters. Never blocks. */ -void commit_on_ack(commit_engine_t *ce, int standby_idx, uint64_t acked_lsn); - -/* Promotion: set own-term gate to the no-op entry's LSN. */ -void commit_set_writable_lsn(commit_engine_t *ce, uint64_t noop_lsn); - -/* Quorum-membership change (standby add/remove/catch-up transition): - * re-evaluate pending entries; standbys in catch-up are excluded. */ -void commit_on_membership_change(commit_engine_t *ce); -``` -- Preconditions: entries submitted in LSN order (single-writer guarantees this) -- Postconditions: `committed_lsn` never decreases; never exceeds highest majority-acked contiguous LSN -- Invariants: no waiter completes successfully while `committed_lsn < its lsn` or `committed_lsn < writable_lsn` - -##### State Management -- State: per-standby view delegated to `standby_conn_t.acked_lsn` (no duplication); internal: sorted matchIndex scratch array, waiter list (lsn, condvar shared, mode), `writable_lsn`, inflight-wait counter -- Concurrency: one mutex + one condvar (broadcast on `committed_lsn` advance; waiters check per-LSN predicate); `committed_lsn` read lock-free via existing atomic - -**Implementation Notes** -- Integration: `repl_wait_majority_ack` becomes a thin adapter calling `commit_submit(WAIT)` during migration, then call sites move to the engine directly -- Validation: assert monotonicity under `XCE_TESTING`; counter for retained-uncommitted entries exposed in logs -- Risks: condvar broadcast storms under high write concurrency — acceptable at current scale; revisit with async continuation (out of scope) - -### Write-Path Classification (`persist.c`) - -| Field | Detail | -|-------|--------| -| Intent | Route every state-changing WAL type to its commit mode at the existing hook seam | -| Requirements | 2.1, 2.2, 2.8 | - -Classification (verified against call sites during implementation; table is the contract): - -> **SUPERSEDED IN PART (2026-06-11, cross-spec review)**: call-site verification during sub-spec generation found WAL_JOB_STOP/RESUME/UPDATE/FAIL originate from worker notify (fact-shaped, RETAIN) — the client-control intent of stop/resume is realized by the handler-level barrier in `do_jobcontrol_op` per the Reply-Ordering Contract below, which stands unchanged. The verified controlling table lives in `ha-commit-barrier-rollout/design.md` §Classification (see also its research.md §2). Rows below marked with the original assumption are retained for history: - -| WAL type | Shape | Mode | Failure behavior | -|----------|-------|------|------------------| -| WAL_JOB_SUBMIT | request (client) | WAIT | error to client (existing path) | -| WAL_JOB_STOP / RESUME | request (client job control) | WAIT | error to client | -| WAL_QUEUE_CTRL | request (client) | WAIT | error to client | -| WAL_WORKER_CTRL join/update | request (worker, replied) | WAIT | error reply to worker | -| WAL_JOB_START / FINISH / FAIL / RUSAGE / CKPT / PREASON | fact (worker notify) | RETAIN | none — retained until committed | -| WAL_WORKER_CTRL dead | fact (controller-detected) | RETAIN | none | -| WAL_NOOP | internal (promotion) | RETAIN | none — gates writability when committed | -| WAL_CHECKPOINT | local-only | none | unchanged (not replicated, by design) | - -> **Note (rows added by later specs, not in the original table):** `WAL_JOB_ARRAY_SUBMIT` — request (client array submit), WAIT, single commit barrier per array regardless of size (`job-array-single-wal-record` task 1.1). `WAL_CONFIG_MARKER` — RETAIN, the consensus/ordering record for a config change appended to the same unified log (`ha-term-safety` task 4.2; see also the Config Barrier note below). Current full table verified against `wal_classify` (persist.c). - -**Reply-Ordering Contract (commit-first, then reply)** -- For every request-shaped operation (job control, queue control, worker join/update), the handler sequence is fixed: validate → mutate (provisional) → WAL append → `commit_submit(WAIT)` → **reply**. No success reply may be sent to the requester before `commit_submit` returns 0; on non-zero return the handler replies with the error (2.1, 2.2). -- Known reply-before-commit call sites to restructure: `do_jobcontrol_op` (job.c:2198-2223 — line numbers current as of this audit, job.c has grown substantially since spec close) and any queue/worker handlers following the same pattern. `handle_jobsubmit` (job.c:1412 onward — line numbers current as of this audit) is the reference implementation of the correct ordering. (Note: `do_jobcontrol_op` is now fixed — it calls `persist_commit_barrier` before replying, per `ha-commit-barrier-rollout` Req 1.1-1.4; this row is historical, not an open item.) -- `handle_jobdispatch` (job.c:1119-1131 — line numbers current as of this audit — replies status=0 to the scheduler before processing allocations) is classified during implementation: if its allocations produce request-shaped state changes, the early reply must move behind the barrier; if its outcomes are fact-shaped (per-job dispatch results delivered via `notify_dispatch`), the early reply is an ACK-of-receipt and may stand. Task-level verification required; the classification assert enforces the outcome. (Confirmed as-built: outcomes are fact-shaped via `notify_dispatch`, so the early reply stands unchanged.) - -**Implementation Notes** -- Integration: `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` consult the table; `persist_wal_async` survives only as the RETAIN fast path (append + broadcast + register) -- Validation: a debug assert rejects any unclassified WAL type (revalidation trigger for new opcodes); integration tests assert no success reply is observable before `committed_lsn` covers the operation's entry -- Risks: reply restructuring touches every control handler in job.c/queue paths — mechanical but must be swept exhaustively; a missed site silently violates 2.1 - -### HA Role Lifecycle (`ha.c` / `ha.h`) - -| Field | Detail | -|-------|--------| -| Intent | Direct ELECTING→PRIMARY with own-term no-op; remove the transitional state and its timeout | -| Requirements | 1.1–1.5, 3.1, 3.3 | - -**Responsibilities & Constraints** -- `raft_cb_become_leader`: truncate to `safe_committed_lsn` (existing), set `CTRL_PRIMARY`, register persistence hooks (moved from `ha_promote_to_primary`), append+broadcast WAL_NOOP, call `commit_set_writable_lsn(noop_lsn)` -- Deletions: `CTRL_PRIMARY_COMMITTING` enum value, dispatch-gate COMMITTING branch, `committing_timeout_fn` + watchdog arming, `ha_check_majority_convergence`, `repl_ack_count` / `ha_repl_ack_received`; `ha_is_primary` / `ha_should_redirect` / vote-deny checks reduce to the single PRIMARY role -- Heartbeat re-arm condition drops the COMMITTING term; standby state reports continue to drive the catch-up decision loop unchanged - -**Contracts**: State [x] — role transitions per the lifecycle diagram; no other component may observe COMMITTING after this spec - -**Implementation Notes** -- Integration: legacy/XCE_TESTING election path (`ha_elect_evaluate_and_signal`) updated symmetrically; `cluster_init_phase2` call point unchanged -- Validation: existing convergence tests retargeted from COMMITTING→PRIMARY transition to noop-commit gating -- Risks: test fixtures referencing CTRL_PRIMARY_COMMITTING across the suite — mechanical but broad; sweep required - -### Quorum Accounting (`ha.c`) -Summary-only: `ha_has_write_quorum` counts standbys that are connected AND not `in_catch_up` (4.1); evaluated at the dispatch gate for fast rejection (7.1) and re-evaluated authoritatively by the commit engine at ACK time (4.2, 4.3); stateless, so quorum restoration auto-resumes acceptance (7.4). Transition logging lives here (see Quorum Observability). - -### Config Barrier (`config_repl.c`) -Summary-only: `push_config_to_standbys` returns `-ETIMEDOUT` when majority ACK is not reached within the commit timeout; the `upgradeConfig` caller propagates the error to the requester and fails the upgrade (5.1–5.3). Uses the same majority arithmetic as the commit engine (shared helper). (Note: superseded in part by `ha-term-safety` task 4.2 — config is no longer a fully parallel, simpler track. `config_commit_change` now appends a `WAL_CONFIG_MARKER` consensus record to the same unified `(term,lsn)` log BEFORE the byte-delivery barrier; the derived `config_committed_marker` is the commit authority and `committed_config_tag` is retired to a non-authoritative cache. The byte-delivery majority-ack barrier mechanism itself (`push_config_to_standbys`, `-ETIMEDOUT` on failure) is unchanged. See `ha-term-safety/design.md`.) - -### Egress Filter (`dataex.c`, `job.c`) -Summary-only: `job_t` gains `uint64_t submit_lsn` (set when the submit entry is appended). The scheduler export path skips jobs with `submit_lsn > committed_lsn` (2.4 — prevents dispatch of a job whose entry may never commit). Status/API query paths are intentionally NOT filtered (documented deviation; read consistency out of scope). Replay/standby-restored jobs carry their entry LSN (≤ committed by definition). (Note: the predicate and scheduler-bound collect wrapper this design assigns to `dataex.c` were factored into a new file `egress.c` — `job_submit_committed()` + `collect_jobs_committed()` — by `ha-convergence-recovery`, kept out of `dataex.c` for controller-unit-test linkability. `dataex.c`'s plain `collect_jobs` remains unfiltered, as intended, for status/API callers. The file's own header comment calls this "the same egress-filter component the design assigns to dataex.c, factored out for testability" — a refinement, not a contradiction.) - -### Demotion Convergence (`ha.c`) -Summary-only: on higher-term observation while PRIMARY (`raft_cb_become_follower` from PRIMARY): disable persistence hooks, truncate WAL above `committed_lsn`, WARN-log with both LSNs, then self-restart the process (re-exec of saved argv; under `XCE_TESTING`, a restart hook seam is invoked instead) — this part still matches as-built (`ha_schedule_demotion_convergence`). Boot recovery replays only the surviving committed prefix → provisional in-memory state is discarded (2.5, 2.7). (Note: the "Rejoin reconciliation" sentence originally here — a standby truncating + self-restarting once on rejoin — is superseded by `ha-term-safety` task 3.2: `ha_rejoin_reconcile` is now a no-op stub retained only as the CtrlHelloReply hook that records the learned committed boundary; the actual divergent-suffix reconciliation is owned by the raft library's term-aware `on_truncate_from`, not a watermark-clamped self-restart. See `ha-term-safety/design.md`.) - -### ACK-Driven Catch-up (`config_repl.c`) -Summary-only: catch-up streaming sends a bounded batch window; receipt of a walAck from a catch-up standby (iomgr) schedules the next batch immediately via a delaymgr task, instead of waiting for the next heartbeat reply (6.2). Heartbeat-driven evaluation remains as the fallback initiator and for snapshot decisions (6.3); `repl_complete_catchup` re-inclusion unchanged (6.4). - -### Quorum Observability + Idempotency Guards -Summary-only: quorum-loss/restore transitions WARN-logged once per transition with connected/required counts (7.2). `mark_job_start/finish/fail/...` gain transition guards: identical-state re-application is a logged no-op; backward transitions (terminal→running) are rejected with a WARN (supports 2.8 retry/replay and the future rejoin-resync). - -## Data Models - -### Domain Model (changes only) -- **WAL entry** (unchanged format; new type `WAL_NOOP` with empty payload — replay is a no-op, exists purely to commit a current-term entry) -- **job_t**: + `submit_lsn` (entry LSN of its submit record) — drives the egress filter; not persisted separately (recomputed at replay from the entry being applied) -- **commit_engine_t**: in-memory only (commit state is volatile, Raft-conformant — recovered via rejoin reconciliation, never persisted) -- Invariants: `committed_lsn` monotonic, contiguous; `writable_lsn` ≤ first confirmable LSN per term; an entry is confirmed ⇔ `committed_lsn ≥ lsn ≥` nothing pending below it - -## Error Handling - -### Error Strategy -- Request-shaped commit failure → `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` propagated via existing `errnox` early-return pattern to the client reply; no in-memory rollback (provisional model; outcome converges per 2.5) -- Admission-cap rejection and own-term-pending rejection both surface `-XCE_ERR_NOT_READY` (retriable; distinct log messages) -- Quorum loss: dispatch gate rejects request-shaped ops; fact-shaped ops are still appended and retained (2.8); WARN on transition only (no log storms) -- Demotion/restart: WARN with truncation range before restart; restart failures (exec error) FATAL — the node must not continue as a divergent primary - -### Monitoring -- Log counters: retained-uncommitted entries (fact backlog depth), inflight WAIT count vs cap, commit latency of the promotion no-op. Metrics export is out of scope (downstream). - -## Testing Strategy - -### Unit Tests (testsuite/, XCE_TESTING) -- `test_commit_engine.c`: per-LSN majority (entry N not committed by ACKs for M<N); contiguous advancement with out-of-order ACKs; monotonicity; own-term gate (no completion before noop LSN commits); admission cap (`n_vcpus - reserved` enforced, NOT_READY beyond); `n_peers<=1` self-quorum; zero-peer-ACK never commits for n_peers>1 (4.4); membership change re-evaluation (4.3) -- `test_ha_role_lifecycle.c`: ELECTING→PRIMARY direct (no COMMITTING observable); noop appended at promotion; `ha_should_redirect`/`ha_is_primary` semantics (1.3, 1.4); no committing watchdog armed (1.5/3.3) -- Quorum accounting: catch-up standby excluded from `ha_has_write_quorum` (4.1) -- Idempotency guards: duplicate FINISH no-op; DONE→RUN rejected -- `test_egress_filter.c`: job with `submit_lsn > committed_lsn` absent from scheduler collect; present after commit - -### Integration Tests (testsuite/) -- Fact retention: simulate quorum loss → worker-fact WAL types still append (RETAIN), no error, committed after standby ACKs resume (2.8, 7.4) -- Request rejection: job submit and queue ctrl fail with error when majority ACK withheld past timeout; `committed_lsn` unmoved (2.2, 2.3) -- Config barrier: upgradeConfig fails to requester on withheld ACKs; `committed_config_tag` unmoved (5.1–5.3) -- Demotion seam: PRIMARY sees higher term → truncation above `committed_lsn` + restart hook invoked; replay after seam yields committed-only state (2.5, 2.7) -- ACK-driven catch-up: lagging standby receives successive batches without intervening heartbeats (6.2); re-inclusion at parity (6.4) - -### E2E (live cluster, extends ha-wal-replication task 5.4) -- 3-node: submit jobs → kill primary → promoted standby owns all confirmed jobs and none of the unconfirmed ones; demoted node rejoins converged (single-outcome check, 2.5) -- 3-node: isolate primary from both standbys → submits fail, status still served (7.1, 7.3); heal → submits succeed without intervention (7.4) - -## Performance & Scalability -- Commit latency target: unchanged from ha-wal-replication baseline (majority RTT + standby fsync); the no-op adds one entry per election -- Throughput under degradation bounded by admission cap — write rejection is deliberate (CP) and `n_vcpus`-relative; sizing guidance deferred to operator docs (research item 3(e)) - -## Migration Strategy -Phased landing matching the brief's hard ordering (barrier before state removal): -1. Commit engine + per-LSN evaluation behind existing job-submit path (behavioral fix only) -2. Classification rollout: queue/worker/job-control → WAIT; facts → RETAIN; config barrier error propagation -3. Promotion no-op + own-term gate; then delete COMMITTING (+ watchdog, convergence checker); redirect fix -4. Demotion truncate+restart, rejoin reconciliation, egress filter, ACK-driven catch-up, observability -Rollback trigger per phase: controller testsuite regression or live-cluster E2E failure; each phase leaves the system no less consistent than before it. diff --git a/.kiro/specs/ha-consistency-invariants/requirements.md b/.kiro/specs/ha-consistency-invariants/requirements.md deleted file mode 100644 index 492db016..00000000 --- a/.kiro/specs/ha-consistency-invariants/requirements.md +++ /dev/null @@ -1,100 +0,0 @@ -# Requirements Document - -## Project Description (Input) - -XCE HA adopts Raft as the consensus model and CP as the CAP stance (decision 2026-06-10, supersedes the earlier "No Raft" steering entry; single-writer log retained as XCE's simplification). Operators of multi-node XCE controller clusters currently get inconsistent durability guarantees: only WAL_JOB_SUBMIT has a majority-ACK commit barrier, while job state changes, queue control, worker control, and config push apply locally and replicate fire-and-forget; a newly elected primary force-promotes after a 60s timeout even with zero converged standbys; and quorum accounting has gate-only (TOCTOU) and catch-up-counting holes. This feature enforces the consistency invariants deduced from Raft + CP across the controller: (1) remove the CTRL_PRIMARY_COMMITTING state — election winners become CTRL_PRIMARY directly, with post-election write safety provided by the own-term commit rule (Raft §5.4.2) instead of a transitional state; (2) make majority ACK the hard commit barrier for ALL cluster-state-changing operations — reject with an error on timeout or quorum loss, never apply-and-degrade, with committed_lsn/committed_config_tag behaving as Raft commitIndex and state-machine apply only after commit; (3) keep full-consistency replication as best-effort — fan out to all standbys with leader-driven continuous catch-up streaming, majority being only the commit threshold. The universal commit barrier must be implemented before the COMMITTING state is removed. Full audit findings, file:line references, scope boundaries, and the Raft-rule mapping table are in brief.md; the Raft paper is the requirements authority and deviations must be documented with justification. - -## Introduction - -The XCE controller cluster shall provide a uniform consistency contract governed by Raft consensus semantics with CP (consistency over availability) behavior. A cluster-state-changing operation is any operation that alters cluster state observable by clients or surviving a failover: job submission, job state transitions, queue control, worker membership changes, and configuration changes. The contract is: an operation is confirmed to its requester only after its log entry is durable on a majority of cluster members; an operation that cannot reach majority is reported as failed, never silently applied; and replication continuously targets all members, with majority as the commit threshold rather than the distribution target. Leadership is a single role — there is no transitional leader state with weaker guarantees. In-memory state is a provisional materialized view of the log: the persisted, majority-committed log (plus snapshots) is the authoritative cluster state, in-memory state must always be reconstructible from it, and anything externally consequential — confirmations to requesters and exports that drive external actions — is gated on commit, not on the in-memory view. The Raft paper (§5.3 commit rule, §5.4.2 own-term commit rule, Log Matching property) is the authority for these requirements; any deviation must be documented and justified in design. - -## Boundary Context - -- **In scope**: Commit/confirmation semantics for all cluster-state-changing operations on the primary; leadership role unification (no transitional leader state); quorum accounting for accept and commit decisions; quorum-loss and standalone-mode behavior; leader-driven catch-up of lagging standbys. -- **Out of scope**: Vote and election correctness (owned by raft-library); replication wire format and standby-side apply mechanics (owned by ha-wal-replication); read consistency and leader leases (future work — reads may be stale during partition, and status queries may observe provisional in-memory state that is not yet committed; this is a documented deviation, while irrevocable-action exports remain commit-gated per criterion 2.4); automatic primary step-down on quorum loss (blocking writes satisfies CP; step-down is a possible follow-on); checkpoint replication (checkpoints are per-node by design, as Raft snapshots are); worker/scheduler reconnection after failover (worker-join-refactor). -- **Adjacent expectations**: This feature relies on the replication acknowledgment machinery introduced by ha-wal-replication and the heartbeat cadence defined by ha-heartbeat-liveness. Retained reported facts (criterion 2.8) cover facts the primary has received; facts lost before reaching the controller — or lost with the controller across a failover — are recovered by worker re-synchronization at rejoin (worker-rejoin-resync, future spec on the roadmap), which this feature does not own but is the designated backstop. The HA steering document (`ha-async-io.md`) describes the current two-tier model including the transitional leader state and must be synchronized after implementation. - -## Requirements - -### Requirement 1: Unified Primary Role - -**Objective:** As a cluster operator, I want an election winner to act as primary immediately under a single leader role, so that clients are never redirected to a primary that does not exist and the cluster has no transitional mode with weaker consistency guarantees. - -#### Acceptance Criteria - -1. When a controller wins an election, the controller shall assume the primary role directly, without entering an intermediate transitional leader role. -2. The controller shall represent leadership as a single primary role; it shall not define a leader sub-state that accepts a different set of operations than the primary role. -3. While a controller is primary, when a client request requiring the primary arrives, the controller shall process the request subject to the commit barrier (Requirements 2 and 3) rather than redirecting the client to another node. -4. While a controller is primary, the controller shall report itself as the primary in status queries, regardless of whether it has yet committed an entry in its current term. -5. The controller shall not assume or change leadership based on a fixed wall-clock timeout that bypasses majority acknowledgment. - -### Requirement 2: Majority Commit Barrier for State-Changing Operations - -**Objective:** As a cluster operator, I want every cluster-state-changing operation to be durable on a majority of members before it is confirmed, so that no single node failure or failover can lose an acknowledged change. - -#### Acceptance Criteria - -1. When the primary receives a cluster-state-changing operation (job submission, job state transition, queue control, or worker membership change), the primary shall confirm the operation to its requester only after the corresponding log entry has been acknowledged by a majority of cluster members, counting itself. -2. When a state-changing operation has a requester awaiting confirmation (such as a job submission or queue control request) and majority acknowledgment is not obtained within the configured commit timeout, the primary shall report the operation as failed to that requester. -3. If an operation has not achieved majority acknowledgment, the primary shall not advance its commit index past that operation's entry. -4. While a state-changing operation's log entry is not yet committed, the primary shall treat the resulting in-memory state as provisional: the primary shall not confirm the operation to its requester and shall not export the provisional state to consumers whose resulting actions cannot be revoked (such as scheduling and dispatch decisions). -5. If an operation was reported as failed due to commit timeout, the cluster shall converge on a single outcome for its log entry — committed on all members or discarded on all members — the primary shall never retroactively report that operation as successful to the original requester, and each member's in-memory state shall converge to that same outcome. -6. The controller shall advance its commit index monotonically; the commit index shall never decrease. -7. The controller shall be able to reconstruct its in-memory state from its persisted committed state (snapshot plus committed log entries), discarding any provisional state not present in the committed log. -8. When the primary receives a reported fact with no requester awaiting confirmation (such as a worker-reported job state transition), the primary shall retain the fact and continue attempting commitment — including across quorum-loss periods — until the fact's log entry is committed; while the primary remains running and holds leadership, it shall not discard an uncommitted reported fact. -9. **(Facts-Apply invariant — added 2026-06-25 via `/kiro-discovery`.)** When the primary applies a committed reported fact, the state-machine mutation and any externally-consequential reaction it triggers (worker slot release, scheduler dispatch, client/apiserver notification) shall occur only after the fact's log entry is committed, through the same post-commit apply seam used for requester-driven operations; the primary shall not perform a fact's externally-consequential reactions on receipt ahead of commit. This refines criterion 8 (which governs the *durability/retention* of facts) by governing the *timing of their apply*, and aligns facts with the Introduction principle that "anything externally consequential … is gated on commit, not on the in-memory view." **Documented deviation (per the requirements authority's deviation rule):** the current implementation applies `WAL_MODE_RETAIN` facts and fires their reactions on receipt (before commit). Reconciling the implementation with this criterion is owned by the `raft-apply-decoupling` spec — candidate plan and step list in `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` — with the quorum-loss reaction policy tracked there as open decision **D1** (block-but-durable vs a local-apply allowlist). - -### Requirement 3: Post-Election Write Safety - -**Objective:** As a cluster operator, I want a newly elected primary to confirm operations only after an entry of its own term is committed, so that a leadership change never exposes uncommitted or divergent state from prior terms. - -#### Acceptance Criteria - -1. When a controller becomes primary, the primary shall not confirm any cluster-state-changing operation until at least one log entry created in its current term has been acknowledged by a majority of cluster members. -2. While the own-term commit condition of criterion 3.1 is unmet, if a state-changing request arrives, the primary shall either process it through the commit barrier (its entry satisfying criterion 3.1 upon majority acknowledgment) or reject it with an error indicating the cluster is not yet ready, and shall not confirm it by any other path. -3. While the own-term commit condition of criterion 3.1 is unmet, the primary shall not be subject to any timeout that confirms operations or advances the commit index without majority acknowledgment. - -### Requirement 4: Quorum Accounting Integrity - -**Objective:** As a cluster operator, I want accept and commit decisions based on members that can actually receive and acknowledge entries, so that a write is never admitted or confirmed against a quorum that does not exist. - -#### Acceptance Criteria - -1. When the primary evaluates whether to accept a state-changing operation, it shall count toward quorum only cluster members currently able to receive new log entries; members that are disconnected or undergoing catch-up shall not be counted. -2. When the primary determines that an entry is committed, it shall require actual acknowledgments from a majority of cluster members; connectivity observed at acceptance time shall not substitute for acknowledgment. -3. If the set of reachable members shrinks below a majority after an operation is accepted but before majority acknowledgment is obtained, the primary shall report the operation as failed rather than confirm it. -4. While the cluster is configured with more than one member, the primary shall never treat zero peer acknowledgments as constituting a majority. - -### Requirement 5: Configuration Change Commit Barrier - -**Objective:** As a cluster operator, I want configuration changes to follow the same majority-commit contract as other state changes, so that a failover never reverts a configuration upgrade that was reported as successful. - -#### Acceptance Criteria - -1. When the primary distributes a configuration change, the primary shall confirm the change to its requester only after a majority of cluster members have acknowledged it. -2. If majority acknowledgment for a configuration change is not obtained within the configured commit timeout, the primary shall report the configuration change as failed to its requester. -3. If a configuration change has not achieved majority acknowledgment, the primary shall not mark that configuration as committed. - -### Requirement 6: Full Replication and Leader-Driven Catch-Up - -**Objective:** As a cluster operator, I want the primary to continuously bring all standbys current, so that the cluster trends toward full consistency and any standby can win a future election without losing committed state. - -#### Acceptance Criteria - -1. While holding leadership, the primary shall replicate new entries to all reachable standbys, not only to a majority subset. -2. When a standby falls behind the live replication stream, the primary shall stream the missing entries to that standby continuously, without requiring a heartbeat exchange to drive each replication step. -3. If a standby is behind the oldest retained log entry, the primary shall bring it current by transferring a snapshot followed by subsequent log entries. -4. When a lagging standby catches up to the primary's commit index, the primary shall re-include it in live replication. -5. While one or more standbys lag, the primary shall continue to confirm operations that achieve majority acknowledgment; full replication is best-effort and shall not become a confirmation precondition. - -### Requirement 7: Quorum Loss and Standalone Behavior - -**Objective:** As a cluster operator, I want a primary that cannot reach majority to refuse state changes with a clear error and recover automatically, so that a partitioned cluster never diverges and no manual intervention is needed after the partition heals. - -#### Acceptance Criteria - -1. While the primary cannot reach a majority of cluster members, the primary shall reject cluster-state-changing operations with an error indicating that the cluster cannot currently commit. -2. When quorum loss occurs, the controller shall record the condition in its log so that operators can observe entry into and exit from the degraded state. -3. While quorum is lost, the primary shall continue to respond to status queries and replication-related traffic from returning members. -4. When majority connectivity is restored, the primary shall resume accepting state-changing operations without operator intervention. -5. While the cluster is configured with a single member, the controller shall treat its own acknowledgment as majority and confirm operations locally, preserving existing standalone behavior. diff --git a/.kiro/specs/ha-consistency-invariants/research.md b/.kiro/specs/ha-consistency-invariants/research.md deleted file mode 100644 index a22867df..00000000 --- a/.kiro/specs/ha-consistency-invariants/research.md +++ /dev/null @@ -1,268 +0,0 @@ -# Gap Analysis: ha-consistency-invariants - -Date: 2026-06-10. Sources: discovery audit (brief.md), code-explorer verification pass (persist.c, repl.c, ha.c, job.c, cluster.c, config_repl.c, ha-wal-replication tasks.md). - -## 1. Current State Investigation - -### Existing assets directly reusable - -| Asset | Location | Relevance | -|---|---|---| -| Per-standby acked LSN (≈ Raft matchIndex) | `standby_conn_t.acked_lsn` (repl.h), updated by `repl_on_ack` (repl.c:187) | Already exists — the bookkeeping needed for honest per-entry commit verification is in place, just unused by the wait logic | -| Per-standby join/applied LSN (≈ nextIndex) | `standby_conn_t.join_applied_lsn` | Catch-up positioning | -| Sync commit barrier (job submit only) | `persist_wal_sync_append` + `repl_wait_majority_ack` (persist.c:1003-1049, repl.c:386-461) | The pattern to generalize; pthread condvar wait (`ack_cond`) signaled by `repl_on_ack` | -| Full fan-out broadcast | `repl_broadcast` (repl.c:210-282) | Req 6.1 already satisfied | -| WAL streaming catch-up + snapshot fallback | `ha_stream_wal_to_standby` (config_repl.c:1450+), snapshot transfer | Req 6.3 satisfied; drive mechanism needs change for 6.2 | -| Hook-based persistence dispatch | `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1107/1199/1241) on HOOK_AFTER_EVENT | The single seam through which all state-changing writes flow | -| Quorum gate at dispatch | `ha_has_write_quorum` + `ha_dispatch_gate` (ha.c:1507-1612) | Req 7.1 partially satisfied (gate-time only) | -| ha-wal-replication ACK machinery | tasks 1.1–5.5 complete; only 5.4 (live 3-node integration test) pending | Upstream dependency effectively satisfied for design | - -### Verified behaviors (with evidence) - -1. **Apply-before-commit is universal — including job submit.** `job_submit()` (job.c:518) inserts the job into the in-memory table, THEN `xce_run_hooks(HOOK_AFTER_EVENT)` (job.c:553) fires the WAL write + majority wait. For state transitions, `mark_job_start/finish/...` mutate `job->job.state` between HOOK_BEFORE_EVENT and HOOK_AFTER_EVENT (e.g., job.c:585-604). Queue and worker state likewise mutate before their hooks fire. On every path, in-memory state precedes the log entry; what differs is only whether the *reply* waits (job submit: yes; everything else: no). Req 2.4 (apply only after commit) is violated everywhere today. -2. **Majority-ACK verification is not per-entry.** `repl_wait_majority_ack` (repl.c:408-443) counts standbys with `acked_lsn > 0` — i.e., "ACKed anything, ever" — not `acked_lsn >= this_entry_lsn`. Once a standby ACKs once, it counts toward every subsequent quorum until disconnect. Concurrent sync appends share one global `ack_cond` and would cross-satisfy each other. Req 4.2 needs per-LSN comparison; the `acked_lsn` field already supports it. -3. **Threading is permissive but finite.** _(Corrected 2026-06-11; original pass understated the pool and overstated the risk.)_ The sync wait is `pthread_cond_timedwait` on a vtask pool thread — default **8** threads, `--vcpus` CLI override (cluster.c:1082). Job submit already blocks one. Generalizing to all write types multiplies concurrent blocked workers; 8 simultaneous slow commits would stall the vtask pool. This is a **throughput risk, not a liveness risk**: WAL ACKs are decoded and waiters signaled on the iomgr thread (`peer_ctrl_msg_handler` → `repl_on_ack`, ha.c:2770-2792), so a fully blocked pool still receives its wakeups — requests queue until a commit completes or times out, but no deadlock is possible. Steering allows vtask handlers to block briefly ("return quickly"; the iomgr never-block rule doesn't apply). Mitigation inputs in Research Needed item 3 (commit-wait admission cap; async continuation as the durable fix). -4. **Job state transitions are reported facts, not requests.** `on_job_start/finish/fail` arrive via worker notifications (`do_notify`, cluster.c:118-156) with no client awaiting a reply. "Rejecting" a job-finish on commit timeout cannot un-finish the job on the worker; the entry must instead be held/retried until commit. The rejection semantics in Req 2.2 map cleanly to request-shaped operations (submit, queue ctrl, config) but need a hold-and-retry interpretation for notification-shaped facts. -5. **COMMITTING removal is concretely mapped.** `raft_cb_become_leader` (ha.c:505) sets COMMITTING; removal means: set CTRL_PRIMARY directly, move hook registration from `ha_promote_to_primary` (ha.c:1691-1698) into the become-leader path, delete the committing watchdog (ha.c:593-616, 3082-3125), make `ha_check_majority_convergence` (ha.c:1313) and `repl_ack_count`/`ha_repl_ack_received` obsolete, adjust the COMMITTING guard in the FSM path (ha.c:2967) and the dispatch-gate COMMITTING branch (ha.c:1594-1612). -6. **Catch-up is heartbeat-polled.** `ha_stream_wal_to_standby` sends all batches `[standby_lsn+1, committed_lsn]` in one call without per-batch ACK waits, but the NEXT round is only triggered by the next heartbeat reply (3s cadence) carrying an updated `applied_lsn` (ha.c:2734 → 1467). Req 6.2 needs ACK-driven (or WAL-append-driven) continuation. - -## 2. Requirement-to-Asset Map - -| Req | Existing asset | Gap | Tag | -|---|---|---|---| -| 1.1-1.2 (direct PRIMARY, single role) | become_leader callback | Remove COMMITTING state + watchdog; move hook registration | Missing (mapped) | -| 1.3 (no redirect from primary) | `ha_should_redirect` (ha.c:1655) | Returns redirect in COMMITTING; resolved by state removal | Missing | -| 1.5 / 3.3 (no timeout bypassing majority) | — | `committing_timeout_fn` force-promote must be deleted | Missing | -| 2.1 (majority barrier all ops) | sync path for WAL_JOB_SUBMIT | Generalize to JOB_* transitions, QUEUE_CTRL, WORKER_CTRL (persist.c:1169/1199/1241) | Missing | -| 2.2 (reject on timeout) | job submit error propagation (job.c:563-570) | Worker notify is pure fire-and-forget (no reply read, no retry, no rejoin resync) — controller MUST own durability via a pending-facts queue; rejection = permanent loss (already true today). See deep-dive §A | Missing (resolved — constraint identified) | -| 2.4 (provisional state, commit-gated egress) | Memory already rebuilt from checkpoint+WAL on restart (persist_recover); role-agnostic replay fns | DECIDED (2026-06-10): provisional-state model — mutation order unchanged; gate requester confirmation + irrevocable-action egress (dataex/scheduler export filters provisional entities by commit status); makes §C reconciliation mandatory core. See deep-dive §B | Missing (decided — egress gating + tagging) | -| 2.5 (single outcome for timed-out entries) | WAL truncation on promotion (ha.c:471) + on standby RDHUP (ha.c:3419); safe_committed_lsn majority-median derivation (ha.c:373-432) | FIVE convergence gaps: no truncation on demotion, no in-memory phantom reconciliation, snapshot is LSN-sentinel only (no entity state), committed_lsn not persisted (restart replays uncommitted suffix), no conflict detection in apply. See deep-dive §C | Missing (resolved — 5 gaps enumerated) | -| 3.1-3.2 (own-term commit) | — | No own-term tracking on promotion; no-op entry vs first-write decision | Missing | -| 4.1 (count only reachable) | `ha_has_write_quorum` (ha.c:1507) | Excludes neither catch-up nor per-write reachability | Missing | -| 4.2-4.4 (ACK-time per-entry verification) | `acked_lsn` per standby | Wait logic compares `> 0` not `>= entry_lsn`; no per-entry wait slots | Missing | -| 5.1-5.3 (config barrier) | `push_config_to_standbys` + `repl_wait_majority_ack` variant (config_repl.c:916-928) | Return code ignored by caller (config_repl.c:1057); error propagation | Missing (small) | -| 6.1 (full fan-out) | `repl_broadcast` | None | Done | -| 6.2 (leader-driven catch-up) | `ha_stream_wal_to_standby` | Continuation is heartbeat-polled, not ACK-driven | Missing | -| 6.3-6.4 (snapshot fallback, re-include) | snapshot transfer, `repl_complete_catchup` | None | Done | -| 6.5 (lag doesn't block confirmation) | majority threshold | Preserved by design | Done | -| 7.1 (reject on quorum loss) | dispatch gate | Gate-time only; ACK-time enforcement closes the TOCTOU | Partial | -| 7.2 (log quorum transitions) | — | Add observability logging | Missing (small) | -| 7.4 (auto-resume) | gate re-evaluates per request | None expected | Done (verify) | -| 7.5 (standalone self-quorum) | `n_peers <= 1` paths | Preserve unchanged | Constraint | - -## 3. Implementation Approach Options - -### Option A: Extend in place -Generalize `persist_wal_sync_append` to all WAL types at the existing hook sites; fix `repl_wait_majority_ack` to per-LSN comparison; delete COMMITTING inline. -- ✅ Smallest diff; all infrastructure exists; follows proven job-submit pattern. -- ❌ Blocking-condvar-per-write multiplies vtask pool throughput-starvation risk (no deadlock — ACKs arrive via iomgr; see §1 item 3); concurrent sync writes still share one condvar (broadcast wakeups checking per-LSN predicates works but is crude); apply-after-commit ordering not addressed (would be documented deviation). - -### Option B: New commit-barrier module -New `commit.c` owning Raft commitIndex logic: per-entry waiter registry (LSN → waiter list), matchIndex aggregation (`acked_lsn` scan on each ACK → advance committed_lsn → wake waiters at or below), explicit pending-entry queue for notification-shaped facts, hold/retry. Hooks call into it; repl.c feeds it ACKs. -- ✅ Clean Raft semantics (commitIndex = median of matchIndex); supports many in-flight entries; single place to test invariants; natural home for own-term tracking and apply-after-commit sequencing. -- ❌ Larger surface; new module must integrate with persist lock ordering and the fsync thread; more migration risk in one step. - -### Option C: Hybrid, phased (barrier-first ordering preserved) -- Phase 1 (repl.c surgical): per-LSN wait predicate (`acked_lsn >= entry_lsn` count ≥ majority), remove `n_active == 0` short-circuit for `n_peers > 1`, ACK-time quorum verification. Job submit immediately gets honest commits. -- Phase 2 (persist.c + small commit registry): generalize the barrier to QUEUE_CTRL, WORKER_CTRL, and request-shaped job ops; introduce a minimal pending-entry registry only for notification-shaped facts (hold/retry until committed); config push error propagation. -- Phase 3 (ha.c): own-term commit gate on promotion, then delete COMMITTING (+ watchdog, convergence checker, redirect fix). -- Phase 4 (config_repl.c): ACK-driven catch-up continuation; quorum-transition logging. -- ✅ Each phase independently testable and shippable; respects the brief's hard ordering (barrier before state removal); risk contained. -- ❌ Per-entry waiter logic lands in two steps; requires discipline to not stop halfway. - -## 4. Effort & Risk - -| Workstream | Effort | Risk | Justification | -|---|---|---|---| -| Per-LSN ACK verification + TOCTOU closure | S-M | Low | Fields exist; localized to repl.c wait/ack path | -| Barrier generalization (request-shaped ops) | M | Medium | Proven pattern, but vtask pool throughput stall needs mitigation (commit-wait admission cap or async continuation — see Research Needed item 3) | -| Notification-shaped facts (pending-facts queue) | M | Medium | Resolved: worker provides NO redelivery (verified), so a primary-side pending queue is the required mechanism; semantics now clear, implementation moderate | -| Provisional-state egress gating (replaces inversion) | M | Medium | DECIDED: mutation order unchanged; tag entities with WAL LSN, filter dataex/scheduler export to committed-only (closes dispatch-of-uncommitted race); smaller than inversion but couples to §C reconciliation being real | -| Demotion/restart convergence (Req 2.5 gaps) | M-L | **High** | NEW workstream: truncate-on-demotion, persist or reconcile committed_lsn at recovery, phantom in-memory state eviction (no teardown path exists; snapshot carries no entity state) | -| COMMITTING removal + own-term gate | S-M | Medium | Concretely mapped (6 sites); must land after barrier | -| Config push error propagation | S | Low | Two call sites | -| Leader-driven catch-up | M | Medium | Continuation trigger redesign in streaming path | - -Overall: **L–XL (2+ weeks across phases), Medium-High risk** — dominated by the two High-risk research items. - -## 5. Recommendations for Design Phase - -**Preferred approach: Option C** (phased hybrid), with Phase 2's pending-entry registry designed Raft-shaped from the start (commitIndex from matchIndex aggregation) so it can absorb Phase 1's logic rather than duplicate it. - -**Research Needed (carry into design):** -1. ~~Apply-after-commit strategy~~ **DECIDED** (user, 2026-06-10): provisional-state model — in-memory state is a provisional view, persisted majority-committed log is authoritative. Mutation order unchanged; gate confirmation + irrevocable-action egress on commit. Remaining design details: entity commit-tagging mechanism (per-entity WAL LSN vs per-type watermark), which egress points are "irrevocable" (dataex/scheduler export: yes; status queries: no, documented deviation), and the egress filter's interaction with array jobs sharing one submit entry. - - **Mechanism direction (discussed 2026-06-10, STM evaluation — REFINED same day)**: full STM rejected — transaction duration mismatch (commit point is a majority network ACK; STM transactions can't span syscalls), C ecosystem dead (libitm unmaintained, TSX disabled), single-writer + per-object RTTI locks already cover the concurrency. An ARIES-style undo log was briefly considered, then **dropped after the recovery-oriented reframing (user)**: this is a failover/recovery problem, not a transaction problem. Structural fact: a single-writer leader never truncates its own log while leader (Raft leaders only append), so a provisional entry on a live primary either commits later or the node is demoted — **discard events coincide exactly with role/lifecycle events** (demotion, restart, rejoin), never with normal operation. Therefore: no per-entry undo; discard is handled by recovery mechanisms — truncate divergent suffix + coarse memory rebuild via the existing `persist_recover`/`replay_wal` path at demotion/rejoin (rare, node not serving writes, cost acceptable). The only piece retained from the versioning discussion is the **watermark egress filter** (entity WAL-LSN tag, export only `lsn <= committed_lsn` to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch of a job the post-failover cluster never heard of), not as transaction isolation. - - **xzarray shadow evaluated (2026-06-11)**: the dual-zarray COW (`xzarray_freeze`/`xzarray_thaw`, zarray.c:695-757 — built and unit-tested, zero production callers; kvmgr vfreeze NULL "wired in P0-2") is slot-level STRUCTURAL COW: slots hold pointers, so in-place field mutations (job state transitions) are visible through both views. Not suitable for the committed-view egress (reads are merged active-over-shadow, single generation would collide with the future checkpoint consumer, point-in-time vs advancing watermark) nor for provisional rollback (only covers inserts, not mutations; demotion-time eviction doesn't need lock-free semantics anyway). Its real home is the entity-bearing-snapshots backlog item (its original P0-2 purpose), which indirectly strengthens §C eviction option (iii). Watermark filter stands as the egress mechanism. -2. ~~Notification-shaped facts~~ **DECIDED** (user, 2026-06-11): **log-is-the-queue, Raft-pure** — fact entries are appended to the WAL immediately and replicated/retried until committed with NO commit timeout (Raft leaders never time out entries; retry across quorum-loss periods per Req 2.8). No separate side-queue structure; the "pending facts" are simply the uncommitted log suffix under continuous retry. Crash-loss of uncommitted facts is accepted until worker-rejoin-resync (roadmap backstop) lands — strictly better than today's behavior. Remaining design details: idempotency guards in `mark_job_*` (needed for replay safety) and retry/backoff cadence. -3. **Wait mechanism under load**: blocking condvar per vtask worker vs async continuation (queue reply closure, complete on commit). Measure vtask pool headroom; consider dedicated commit-wait sizing. - - **Verified inputs (2026-06-11)**: (a) vtask pool default is ALREADY 8 with `--vcpus` override (cluster.c:1082; the "default 4" in ha-async-io.md was stale — fixed). (b) **No wakeup deadlock**: WAL ACKs are processed on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, ha.c:2770-2792, 2748) — blocked vtask workers always get woken; all-workers-blocked costs throughput, not liveness. (c) **Pool size should NOT scale with n_peers** (user question, analyzed): blocked-thread count ≈ in-flight write ops × commit latency (Little's law); ACKs gather in parallel so peer count barely moves latency. Write concurrency and cores are the sizing inputs. (d) Recommended interim mitigation: commit-wait admission cap — at most `n_vcpus - K` concurrent blocking waits (reserve K lanes for non-blocking traffic), excess write ops fail fast with NOT_READY (CP-correct under overload). Async continuation remains the durable fix. (e) **Sizing ownership (user, 2026-06-11)**: `--vcpus` is a tunable operator parameter — the right value is a deployment decision driven by host capability, cluster size, and workload; 8 is only the fallback default. Design constraint: all commit-wait correctness mechanisms must be expressed relative to `n_vcpus` (never absolute counts) so behavior is correct at any tuning; sizing guidance belongs in operator/tuning docs, not in spec-fixed constants. -4. ~~Own-term commit vehicle~~ **DECIDED** (user, 2026-06-11): **explicit no-op WAL entry at promotion** — bounded post-election write-unavailability independent of traffic; an idle cluster's new primary becomes writable as soon as the no-op commits. -5. **Multi-entry in-flight semantics**: confirm `committed_lsn` advancement is contiguous (no gaps) when multiple entries await ACKs. -6. ~~Req 2.5 verification~~ **RESOLVED into a workstream** (deep-dive §C): five convergence gaps enumerated. Design must choose the phantom-eviction strategy: (i) full in-memory rebuild on demotion (needs new teardown path for jobmgr/queues/workers, then `replay_wal` from checkpoint after truncation), (ii) process self-restart on demotion (crude, safe, CP-consistent), or (iii) entity-bearing snapshots (depends on the deferred "Task 11" snapshot format). Also: persist `committed_lsn` (or reconcile it from the cluster at recovery) so restart does not replay the uncommitted suffix. - -**Constraints to respect:** standalone `n_peers <= 1` self-quorum unchanged; iomgr thread never blocks (waits live on vtask/delaymgr threads only); single-writer model; ha-wal-replication task 5.4 (live integration test) still pending — its 3-node test should be extended to cover the new barrier rather than duplicated. - ---- - -# Design Synthesis Outcomes (2026-06-11, design phase) - -- **Generalization**: Requirements 2.x (op barrier), 3.x (own-term gate), and the per-LSN quorum fixes (4.2–4.4) are one capability — a single **commit engine** (`commit.c`) with two completion modes: WAIT (request-shaped, blocking with timeout + admission cap) and RETAIN (fact-shaped, log-is-the-queue, no timeout). Config (5.x) stays a parallel simpler track (one in-flight change, own tag) sharing the majority arithmetic helper. -- **Build vs adopt**: algorithm adopted from the Raft paper (§5.3 commit rule, §5.4.2 own-term no-op); machinery reuses existing assets — `standby_conn_t.acked_lsn` (matchIndex), `repl_broadcast`, catch-up streaming, `persist_recover` boot path (reused by demotion self-restart). No new libraries; no .capnp changes (WAL_NOOP is a controller-internal entry type). -- **Simplification** (locked by user decisions): no undo log, no MVCC/committed-view machinery, no side pending-facts queue (the uncommitted log suffix IS the queue), no persisted commitIndex (volatile per Raft; rejoin reconciliation instead), egress filter = one watermark comparison on `submit_lsn`, demotion eviction = truncate + process self-restart via the existing boot path. -- Design written to `design.md`; review gate passed (all 33 requirement IDs traced, no orphan components, boundary↔file alignment verified). - ---- - -# Deep-Dive Research: Requirements 2.2, 2.4, 2.5 (2026-06-10, second pass) - -Three parallel code-explorer passes resolving the Unknown tags. All findings verified with file:line evidence. - -## §A — Req 2.2: Notification-shaped facts have NO redelivery anywhere (worker → controller) - -**Verdict: the controller must own durability of worker-reported job state facts. A pending-facts queue on the primary is the only option that doesn't expand scope into the worker protocol.** - -**DECIDED (user, 2026-06-11)**: facts are retained and retried until committed — including across quorum-loss periods; the primary never discards an uncommitted fact while running and leading (Req 2.8). Worker-rejoin re-sync (roadmap backlog) is the designated backstop for facts that never reached the controller or were lost with it across failover. - -- **Send path is enqueue-and-forget**: worker job start/finish/fail → `xcecli_notify_*` (src/lib/client.c:289/373/392) → `xce_notify` (xceapi.c:270) → `xce_request` which enqueues and returns 0 immediately (request.c:309-331). No `cbuf_get` anywhere in the chain — the worker never reads a reply. -- **Errors are silently discarded twice**: (1) the return value of `xcecd_job_finished` is ignored at its call sites (xcecd_worker_utils.c:297-337, with a literal `/* FIXME: add error handling when msg send failure */`); (2) if the controller's dispatch gate rejects the notify with `-XCE_ERR_NOT_READY` (cluster.c:252-258), the error reply reaches the worker but is dropped by `cbuf_tryput` returning -EAGAIN with no consumer (xcectx.c:82-88, 304). -- **No resync on rejoin**: the worker rejoin path (`checking_connection`, controller_connector.c:176) re-issues only hello + join; `xce_join_req` carries only host resource attributes (request.c:1154-1181), no running-job state. Notifications lost during disconnect/failover are never re-delivered. -- **Controller side is also fire-and-forget**: `do_notify` (cluster.c:118-157) sends no reply at all once the gate passes. -- **No idempotency guards**: `mark_job_start/finish/fail` (job.c:583/638/696) unconditionally overwrite `job->job.state` — duplicate or out-of-order delivery (e.g., from a future retry/replay mechanism) is unguarded. -- **Scheduler-originated transitions differ**: `handle_jobdispatch` (job.c:459) replies synchronously (status 0 at job.c:471) BEFORE processing allocations; dispatch results flow back to the scheduler via `notify_dispatch` (job.c:440). Retry is owned by the scheduler side (none found in scheduler.c at the reviewed path). - -**Pre-existing bug exposed**: today, quorum-loss (or COMMITTING-window) rejection of `OPCode_notify` already loses job state transitions permanently — the worker never learns and never resends. The pending-facts queue fixes an existing silent-data-loss hole, not just a new-requirements need. - -**Design inputs**: queue must persist facts arriving while quorum is absent and drain them (through the commit barrier) when quorum returns; needs bounded size + eviction policy; `mark_job_*` need state-machine guards so queue drain/replay is idempotent and ordered. - -**Companion backlog item (2026-06-10)**: worker-rejoin-resync added to `.kiro/steering/roadmap.md` — long-planned, covers the disconnect/failover window this queue cannot (facts the worker generated while not connected). Out of scope for this spec; the idempotency guards designed here are a shared prerequisite. - -## §B — Req 2.4: Apply-after-commit inversion is FEASIBLE (and fixes an existing race) - -**Verdict (superseded 2026-06-10, user decision): the PROVISIONAL-STATE MODEL is adopted instead of inversion.** In-memory state is a provisional materialized view; the persisted majority-committed log is the authoritative state. Mutation order stays as-is; commit gating moves to (1) requester confirmation and (2) irrevocable-action egress points — the dataex/scheduler export must filter provisional entities (e.g., tag entities with their WAL LSN, export only `lsn <= committed_lsn`), closing the dispatch-of-uncommitted-job race noted below. This makes the §C reconciliation workstream mandatory core (provisional state must actually be discarded when the log disagrees). Requirements 2.4/2.7 amended accordingly. The inversion findings below are retained as evidence and fallback. - -**Original verdict: invert to log→commit→apply. All prerequisites verified; the rollback alternative is infeasible.** - -- **The BEFORE-hook seam exists and is free**: `xce_run_hooks` aborts the chain when a HOOK_BEFORE_EVENT hook returns negative (hook.c:137) — veto semantics are built in. No registered hook currently does anything on BEFORE for job/queue/worker events (sched.c:244, apiserver.c:134, persist.c:1099 all AFTER-only), and callers currently ignore the BEFORE return code — both sides of the seam are unused and available. -- **WAL payloads are constructible pre-mutation**: WAL_JOB_SUBMIT serializes the full `xce_job_t`, but every field is computable before table insertion — the job ID is assigned by the apiserver's `idgen()` before the RPC (idgen.c:22; `submit->jobid` pre-assigned, array jobs are `base_id + i`, job.c:34). State-transition payloads are just `{jobid, new_state}` where new_state is a per-transition constant (persist.c:1134-1169) — trivially known before mutation. -- **Replies don't need post-mutation state**: `handle_jobsubmit`'s reply carries only `jobid` (job.c:575-578), known pre-insertion. -- **Post-commit apply can reuse the replay functions**: the standby/recovery apply path (`apply_wal_entry` → `restore_job`/`restore_job_state`, persist.c:691-769, job.c:71-121) is role-agnostic with an EEXIST idempotency guard (jobj.c:696-699). Caveat: `apply_wal_entry` skips hooks by design (persist.c:686) — primary post-commit apply must either call restore fns directly or suppress hook double-fire (sched/apiserver AFTER hooks still need to run exactly once). -- **Rollback (the alternative) is infeasible**: there is no job deletion path — `job_obj_killer` is a stub (cluster.c:759-764), `key_del` is never called in src/controller/. Undo would be net-new, riskier code than inversion. -- **Inversion fixes an existing race**: today the submitted job is visible in `c->jobmgr` to concurrent vtask readers (dataex/`collect_jobs` snapshots, up to 8 vtask threads, cluster.c:1082) for the entire majority-wait window — no per-job write lock is held during the AFTER hook (job.c:553). An uncommitted job can leak to the scheduler via a racing dataex pull. Apply-after-commit closes this. -- **One trade-off**: state-transition callers hold the per-job WRITE lock across BEFORE/mutate/AFTER (job.c:629-631 etc.). Naively moving the commit wait inside that region blocks readers of that job for up to the commit timeout — design should release/restructure the lock around the wait (the pending-facts queue from §A naturally decouples this). - -## §C — Req 2.5: Convergence currently has FIVE gaps; safe_committed_lsn derivation is sound - -**Verdict: the "single outcome cluster-wide" requirement is NOT met today for the demoted-ex-primary and restart cases. New workstream required.** - -What works: -- New leader truncates above `safe_committed_lsn` on promotion (ha.c:471); standby truncates on RDHUP using `last_known_committed_lsn` (ha.c:3416-3420); `committed_lsn` advances only on majority ACK (persist.c:1035). -- `safe_committed_lsn` = majority-median of granted voters' committed LSNs, sorted descending, index `n_granted/2` (ha.c:373-432) — conservative and safe (never includes an uncommitted entry; may under-estimate and over-truncate, which is correct-but-lossy for uncommitted suffixes). - -The five gaps: -1. **No WAL truncation on demotion**: `raft_cb_become_follower` (ha.c:627-690) cancels timers, closes vote endpoints, sets STANDBY — but never calls `persist_truncate_wal_above`. The ex-primary's divergent uncommitted suffix survives on disk until its own future RDHUP event. (Also: this production path never disables persistence hooks — `ha_demote_to_standby` (ha.c:1712-1724) does, but is not called here.) -2. **No in-memory phantom eviction on demotion**: no path rebuilds or removes in-memory jobs/queues/workers applied from uncommitted entries. Phantoms persist indefinitely on the demoted node. -3. **Snapshots carry no entity state**: the snapshot format is a 16-byte sentinel (magic + checkpoint_lsn, persist.c:435-437, "Task 11" placeholder); `persist_load_snapshot` (persist.c:680-683) only moves the LSN watermark. Snapshot transfer can therefore NEVER evict phantoms — option (iii) below depends on implementing the real format. -4. **Restart resurrects phantoms**: `committed_lsn` is in-memory only (zero-init on boot, never persisted); `replay_wal` (persist.c:781-951) replays everything above `checkpoint_lsn` with no committed_lsn gate. A crashed ex-primary replays its own uncommitted suffix back into memory. -5. **No conflict detection in standby apply**: `apply_wal_entry` advances `current_lsn` and applies unconditionally by type (persist.c:720-721) — entries from the new primary are applied on top of phantom state with no divergence check. - -Eviction strategy options for design: (i) in-memory rebuild on demotion — new teardown for jobmgr/queues/workers, then truncate + `replay_wal` from checkpoint (most work, no restart); (ii) process self-restart on demotion after truncation — minimal code, CP-acceptable since the node is leaving leadership anyway; (iii) entity-bearing snapshot transfer from the new primary — depends on deferred Task 11 snapshot format. **DECIDED (user, 2026-06-11): option (ii) — process self-restart on demotion** (after truncating the divergent suffix). Simplest and battle-tested: recovery reuses the boot path; a demoted node's worker/scheduler connections migrate to the new primary anyway, so the restart's connection reset costs little. Rebuild-in-place (i) remains a future refinement only if restart proves operationally disruptive. For gap 4, note Raft treats commitIndex as VOLATILE (followers recover it from the leader): the Raft-conformant fix is leader-driven log reconciliation at rejoin (consistency check + truncate divergent suffix, then rebuild) rather than persisting `committed_lsn` locally — design should weigh reconciliation-at-rejoin vs local persistence; either must ensure the replayed-but-uncommitted suffix cannot leak through irrevocable egress before reconciliation completes. - ---- - -## Gap Re-Audit (2026-06-18) - -Roll-up gap audit of the UMBRELLA contract against the landed sub-specs. The umbrella -(`ha-consistency-invariants`) is design-generated and owns NO tasks of its own; it was -decomposed (2026-06-11) into five implementing sub-specs. This pass maps every umbrella -acceptance criterion to its owning sub-spec and verifies it against real code (file:line), -then enumerates the invariants still OPEN with their owning future specs. - -**Method**: codegraph + grep against `src/controller/{commit.c,persist.c,ha.c,ha_convergence.c,config_repl.c,egress.c,job.c}`; sub-spec `tasks.md` checkbox counts and `spec.json`; roadmap backlog (`.kiro/steering/roadmap.md`) for future-spec ownership. - -### Sub-spec landing status (verified against code, not just spec.json phase) - -> `spec.json.phase` is STALE for three sub-specs (still reads `tasks-generated`) and several -> `tasks.md` checkboxes were never re-ticked. The authoritative signal is the on-disk code + -> git history: the COMMITTING machinery is **fully gone**, the commit engine is live, demotion -> convergence is wired, and `ha-commit-durability` (23/23 tasks, all committed d9b43c2…1ca2d64) -> re-verified the whole protocol end-to-end. The audit below grades by CODE, flagging the -> stale-tracking discrepancy where it occurs. - -| Sub-spec | tasks.md ticks | Code verdict | Notes | -|---|---|---|---| -| ha-commit-engine | 12/18 ticked | **LANDED** | `commit.c` present & called from persist/ha/repl; per-LSN eval, self-quorum, own-term gate, admission cap all in code. Open boxes are stale tracking. | -| ha-commit-barrier-rollout | 23/23 | **LANDED** | `wal_classify` totality (persist.c:2004) + idempotency guards (job.c:627-797) + reply-after-commit. | -| ha-single-leader-role | 11/17 ticked | **LANDED** | Zero `CTRL_PRIMARY_COMMITTING`/`committing_timeout_fn`/`ha_check_majority_convergence`/`repl_ack_count` refs in src/controller (excl. testsuite). Open boxes are stale tracking. | -| ha-convergence-recovery | 18/19 | **LANDED (1 E2E open)** | `ha_convergence.c` present + demotion path wired (ha.c:1188-1232); egress filter (egress.c). Only open: task 8 = live 3-node E2E test. | -| ha-commit-durability | 23/23 | **LANDED** | Group-commit flusher + `durable_lsn`, commit≤durable invariant, durable-backed vote freshness, no-self-terminate fence. | - -### Requirement → sub-spec → code roll-up map - -| Umbrella criterion | Owning sub-spec | Code evidence (file:line) | Verdict | -|---|---|---|---| -| 1.1, 1.2 direct PRIMARY, single role | single-leader-role | `ha_promote_to_primary` ha.c:2298; no COMMITTING enum anywhere | MET | -| 1.3 no redirect from primary | single-leader-role | `ha_should_redirect` ha.c:2187 (standby-only redirect) | MET | -| 1.4 status reports primary pre-own-term | single-leader-role | role set CTRL_PRIMARY directly at promotion (ha.c:2298-2315) | MET | -| 1.5 / 3.3 no wall-clock leadership change | single-leader-role | `committing_timeout_fn` deleted (zero refs); `ha_check_priority_stepdown` deleted (ha.c:1472) | MET | -| 2.1 confirm only after majority ACK | commit-engine + barrier-rollout | `commit_submit` WAIT commit.c:284; reply-after-commit per classification | MET | -| 2.2 requester ops fail on commit timeout | commit-engine | `-ETIMEDOUT` from `commit_submit` commit.c:340-359 | MET | -| 2.3 no commit-index advance without majority | commit-engine | `commit_advance_locked` (median of acked_lsn); per-LSN eval | MET | -| 2.4 provisional state, irrevocable egress gated | convergence-recovery | `job_submit_committed`/`collect_jobs_committed` egress.c:47-109 | MET | -| 2.5 single-outcome convergence incl. memory | convergence-recovery | demotion truncate-above-committed + self-restart ha.c:1188-1232, ha_convergence.c | MET | -| 2.6 monotonic commit index | commit-engine | monotonic max store + XCE_TESTING assert (commit.c) | MET | -| 2.7 reconstruct from committed log | convergence-recovery + durability | boot recovery replay; `recovery_truncated` verdict (durability task 3.1) | MET | -| 2.8 facts retained+retried until committed | commit-engine + barrier-rollout | RETAIN mode commit.c:296-302; log-is-the-queue; idempotency guards job.c:627-797 | MET | -| 3.1 no confirm before own-term entry commits | single-leader-role + commit-engine | `ha_arm_own_term_gate` ha.c:2259 + `commit_set_writable_lsn` commit.c:383 (`writable_lsn` gate) | MET | -| 3.2 pre-own-term: barrier or NOT_READY | commit-engine | own-term gate withholds WAIT completion; admission cap → NOT_READY | MET | -| 4.1 quorum counts only members able to receive | commit-engine | `in_catch_up` exclusion (ha.c:325; commit eval excludes catch-up) | MET | -| 4.2 commit requires per-entry ACKs | commit-engine | `commit_on_ack` reads registry `acked_lsn`, per-LSN median commit.c:366 | MET | -| 4.3 post-acceptance quorum shrink → fail | commit-engine | ACK-time re-eval + WAIT timeout; `commit_on_membership_change` commit.c:397 | MET | -| 4.4 never zero-peer majority (n_peers>1) | commit-engine | legacy `n_active==0→success` removed; n_peers≤1 short-circuit only (commit.c:330) | MET | -| 5.1–5.3 config change commit barrier | barrier-rollout (config track) | `push_config_to_standbys` returns `-ETIMEDOUT`; caller forwards (config_repl.c:941,1115-1120); `committed_config_tag` gated (config_repl.c:963-969) | MET | -| 6.1 replicate to all reachable standbys | (upstream ha-wal-replication, verified) | `repl_broadcast` full fan-out (unchanged) | MET | -| 6.2 catch-up without per-step heartbeat | convergence-recovery | ack-driven continuation `ha_catchup_continuation_fn` config_repl.c:1732 (off-iomgr delaymgr) | MET | -| 6.3, 6.4 snapshot fallback, re-include | (upstream, verified) | snapshot transfer + `repl_complete_catchup` (unchanged) | MET | -| 6.5 lag never blocks majority confirmation | commit-engine | majority-median threshold only | MET | -| 7.1 reject state changes without quorum | commit-engine + ha gate | dispatch gate + ACK-time enforcement (TOCTOU closed) | MET | -| 7.2 log quorum-loss enter/exit | single-leader-role | `ha_log_quorum_transition` (ha.c:1085, 2076, 3969, 5054) | MET | -| 7.3 serve status while degraded | (gate allowlist, verified) | dispatch-gate allowlist (status/replication) | MET | -| 7.4 auto-resume on quorum restore | commit-engine + ha gate | stateless gate re-eval; restore edge logged ha.c:5054 | MET | -| 7.5 standalone self-quorum | commit-engine | `n_peers <= 1` self-quorum commit.c:100,330; ha.c:2037 | MET | - -**Umbrella criteria covered: 33/33 acceptance criteria MET in code.** One sub-task remains -unticked within scope — `ha-convergence-recovery` task 8 (live 3-node E2E convergence test). -This is test-evidence, not a missing invariant: the convergence MECHANISM is implemented and -unit/integration-tested; the standing 3-node E2E is the same pending live test inherited from -ha-wal-replication task 5.4. Tag: **MET (impl) / E2E-evidence pending.** - -### Still-OPEN invariants (Deferred — owned by FUTURE specs, NOT umbrella scope) - -These were declared out-of-scope by the umbrella (Boundary Context / Non-Goals) and remain -unimplemented. They are **Deferred**, not regressions — the umbrella never promised them; it -named the owning backstop. - -| Open invariant | Why still open | Owning future spec (roadmap) | Effort | Risk | -|---|---|---|---|---| -| Worker-rejoin job-state resync (facts lost across disconnect/failover, never reached controller) | Primary-side RETAIN queue only covers facts the controller already received; worker notify is fire-and-forget with zero redelivery (bug-396). Designated backstop in umbrella Req 2.8 / boundary. | **worker-rejoin-resync** (roadmap, planned, no effort yet) | M | Medium — touches worker join/hello protocol + reconcile-through-barrier; shares the already-landed `mark_job_*` idempotency guards | -| Automatic primary step-down on quorum loss | Umbrella satisfies CP by BLOCKING writes (Req 7.1); step-down is an availability/clarity optimization, explicitly out of scope. | **primary step-down-on-quorum-loss** (roadmap, optional CP follow-on) | S–M | Low — writes already blocked; step-down is additive role transition | -| Read consistency / leader leases | Status queries may observe provisional/stale in-memory state (documented deviation); irrevocable-action egress already commit-gated (2.4). | **read-consistency / leader-leases** (roadmap, Raft §8) | M | Medium — read-only lease protocol; must not regress write path | -| Entity-bearing snapshots ("Task 11" format) | Snapshot is a 16-byte LSN sentinel; demotion eviction uses self-restart instead. Not required by any umbrella criterion; strengthens §C option (iii) and faster recovery. | **entity-bearing-snapshots** (roadmap; dual-zarray COW already built/dormant) | M | Medium — serialization format + freeze/thaw wiring | -| Job-array single-WAL-record | Each array element = one WAL/commit/push today; amplifies commit-engine load linearly. Correctness-neutral (each element still commits); performance only. | **job-array-single-wal-record** (roadmap) | M | Medium — atomic array commit + scheduler local expansion | - -### Roll-up conclusion - -The umbrella contract is **substantively SATISFIED**: all 33 acceptance criteria are MET in -landed code across the four core sub-specs (commit-engine, barrier-rollout, single-leader-role, -convergence-recovery) plus the durability sub-spec, which closed bug-661 (ACK≠durable) and the -no-self-terminate promotion fence. The COMMITTING transitional state is fully removed; majority -ACK (now durable-majority) is the hard commit barrier for all state-changing ops; facts are -retained-and-retried; egress is commit-gated; demotion converges via truncate+self-restart. - -Status: **PARTIAL** only in the umbrella-as-program sense — the four explicitly-deferred -invariants above remain owned by named future specs, and one in-scope live 3-node E2E test -(convergence-recovery task 8) is still pending. No in-scope invariant is unimplemented. - -**Re-validation triggers still standing**: worker-rejoin-resync landing → re-verify idempotency -guards + fact replay ordering; entity-bearing snapshots landing → re-evaluate demotion -self-restart vs snapshot rebuild; any new state-changing opcode/WAL type → must be classified -(the `wal_classify` debug abort at persist.c:2046 enforces this). diff --git a/.kiro/specs/ha-consistency-invariants/spec.json b/.kiro/specs/ha-consistency-invariants/spec.json deleted file mode 100644 index c75a4caa..00000000 --- a/.kiro/specs/ha-consistency-invariants/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "ha-consistency-invariants", - "created_at": "2026-06-10T21:40:00-07:00", - "updated_at": "2026-06-11T09:30:00-07:00", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "note": "UMBRELLA CONTRACT spec (decomposed 2026-06-11): requirements.md/design.md/research.md are the authoritative contract for four implementation sub-specs — ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery (dependency order in steering/roadmap.md). Do NOT generate tasks for this spec; implement via the sub-specs. Design approved as contract; sub-spec designs inherit and slice it.", - "implementation_status": "implemented — all 4 sub-specs complete (ha-commit-engine 18/18, ha-commit-barrier-rollout 23/23, ha-single-leader-role 17/17, ha-convergence-recovery 19/19; phases reconciled 2026-07-03, all were mislabeled tasks-generated). Requirements 1-7 satisfied. Criterion 2.9 (Facts-Apply invariant, added 2026-06-25) is now SATISFIED — closed by raft-apply-decoupling task 4 (\"Fact apply-after-commit split\", 4.1/4.2) plus D1 (quorum-loss fact policy, resolved+implemented block-but-durable). WAL_MODE_RETAIN facts append-only on receipt; state mutation + reactions moved to the post-commit apply_committed_entry() seam (persist.c:2721) via apply_job_update_transition() for all 9 job-state fact types; the old apply-on-receipt path (apply_wal_entry()) has zero production call sites. Verified by task 6.3 (command/fact apply-notify timing), 7.2 (live-cluster kill-primary-mid-fact-stream soak), 7.4 (throughput regression). design.approved corrected true->matches the note's 'Design approved as contract' (was inconsistently false)." -} diff --git a/.kiro/specs/ha-convergence-recovery/brief.md b/.kiro/specs/ha-convergence-recovery/brief.md deleted file mode 100644 index 0e96faec..00000000 --- a/.kiro/specs/ha-convergence-recovery/brief.md +++ /dev/null @@ -1,24 +0,0 @@ -# Brief: ha-convergence-recovery - -> Sub-spec 4 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> Authoritative requirements/design/research: `.kiro/specs/ha-consistency-invariants/` (design.md §Demotion Convergence + §Egress Filter + §ACK-Driven Catch-up + §Quorum Observability; research.md §C five convergence gaps). This brief scopes the slice; do not re-derive decisions. - -## Problem -Five verified convergence gaps: a demoted ex-primary never truncates its divergent WAL suffix; phantom provisional in-memory state is never evicted; restart replays the uncommitted suffix (committed_lsn is volatile, correctly per Raft, but nothing reconciles at rejoin); standby apply has no conflict detection; and uncommitted jobs leak to the scheduler via dataex during the commit window (orphan-dispatch risk). - -## Desired Outcome -Cluster-wide single-outcome convergence and commit-gated egress: demotion truncates above `committed_lsn` and self-restarts through the existing boot recovery (provisional state discarded); a rejoining standby whose WAL tip exceeds the primary's committed boundary truncates and restarts once; the scheduler export path never sees a job whose submit entry is uncommitted; catch-up streaming continues on walAck without waiting for heartbeats; quorum-loss transitions are operator-visible in logs. - -## Scope -- **In**: demotion path in `ha.c` (`raft_cb_become_follower` from PRIMARY: disable hooks, truncate above committed_lsn, WARN-log, self-restart; `XCE_TESTING` restart-hook seam); rejoin reconciliation (generalize RDHUP truncation to the rejoin/hello path); `job_t.submit_lsn` + scheduler-export filter in `dataex.c` (status/API queries intentionally unfiltered — documented deviation); ACK-driven catch-up continuation in `config_repl.c` (heartbeat path remains fallback + snapshot initiator); quorum enter/exit WARN logging; verification tests for unchanged behaviors (full fan-out 6.1, snapshot fallback 6.3, re-inclusion 6.4, degraded status service 7.3); 3-node E2E (extends ha-wal-replication task 5.4: confirmed jobs survive failover, unconfirmed jobs converge to absent). -- **Out**: entity-bearing snapshots (roadmap backlog — would later upgrade eviction to option iii); read-consistency filtering of status queries; primary step-down on quorum loss; worker rejoin job-state resync. - -## Requirements (umbrella IDs) -2.4, 2.5, 2.7, 6.1, 6.2, 6.3, 6.4, 7.2, 7.3. - -## Upstream / Downstream -- **Upstream**: ha-commit-engine (committed_lsn semantics). Parallel-safe with specs 2–3 except: full E2E requires ha-single-leader-role landed. -- **Downstream**: entity-bearing snapshots (eviction upgrade path); worker-rejoin-resync (recovers facts across the restart this spec introduces). - -## Constraints -Self-restart must FATAL rather than continue if exec fails (a divergent ex-primary must not keep serving); restart seam mockable under XCE_TESTING; truncation always precedes restart so boot replay yields committed-only state. diff --git a/.kiro/specs/ha-convergence-recovery/design.md b/.kiro/specs/ha-convergence-recovery/design.md deleted file mode 100644 index 6cb1d06b..00000000 --- a/.kiro/specs/ha-convergence-recovery/design.md +++ /dev/null @@ -1,437 +0,0 @@ -# Design: ha-convergence-recovery - -> Sub-spec 4 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **Demotion Convergence**, **Egress Filter**, **ACK-Driven Catch-up**, and **Quorum Observability** components plus the role-lifecycle flow (`.kiro/specs/ha-consistency-invariants/design.md`) — umbrella Migration Strategy phase 4. Discovery is complete via the umbrella's `research.md` (deep-dive §C five convergence gaps; locked decisions) plus this spec's anchor-verification pass (`research.md`, 2026-06-11). Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping). - -## Overview - -**Purpose**: This feature closes the five verified convergence gaps so the cluster converges to a single outcome for every log entry and the scheduler never acts on uncommitted state: a demoted ex-primary truncates its uncommitted WAL suffix and self-restarts through boot recovery; a rejoining node whose log runs ahead of the cluster's committed boundary reconciles by truncation and one restart; scheduler-bound job exports are gated on the commit watermark; catch-up streaming continues on WAL ACK instead of waiting for heartbeats; and quorum-loss/restore transitions are operator-visible. - -**Users**: Cluster operators gain single-outcome failover semantics (no phantom state on demoted/rejoined nodes) and observable degraded periods; the scheduler is protected from orphan dispatch; lagging standbys catch up at ACK speed. - -**Impact**: Extends `raft_cb_become_follower` (ha.c:2104-2247 — line drift only, was ha.c:627-690 at design time; currently neither truncates nor disables hooks, verified gap) with the demotion-convergence sequence; generalizes the RDHUP truncation (ha.c:3416-3420 at design time) to the rejoin path (Note: superseded by ha-term-safety — the rejoin/RDHUP watermark-truncate heuristic this spec built was retired and replaced by the raft library's (term,lsn)-driven `on_truncate_from`; see the Rejoin Reconciliation component below and that spec's design.md); adds `submit_lsn` to the `job_t` wrapper and a watermark predicate on the scheduler export seams; adds an ACK-driven continuation to catch-up streaming (`ha_stream_wal_to_standby`, now config_repl.c:2351 — today re-driven only by the next heartbeat reply); adds quorum-transition WARN logging. No commit semantics change: this spec consumes `committed_lsn` (ha-commit-engine) and never modifies `commit.c`. - -### Goals -- Demotion = disable hooks → truncate above `committed_lsn` → WARN-log range → process self-restart via existing boot recovery; exec failure is FATAL; `XCE_TESTING` restart seam (1.1–1.7) -- Rejoin reconciliation: local tip beyond the primary's committed boundary → clamped truncation + one restart (2.1–2.3); single-outcome convergence E2E-verified (2.4) -- Commit-watermark egress filter on scheduler-bound exports only; status/API reads untouched (3.1–3.4) -- ACK-driven catch-up continuation with single-flight guard; heartbeat path stays fallback + snapshot initiator; parity triggers re-inclusion (5.1–5.3, 7.1) -- Quorum-loss/restore transitions WARN-logged once per change (8.1–8.3) -- Regression tests pinning existing full fan-out, snapshot fallback, re-inclusion, degraded-mode service (4.1, 6.1, 7.1, 9.1–9.2) - -### Non-Goals -- Commit-index advancement, majority evaluation, admission cap (ha-commit-engine — upstream, complete) -- WAL-type classification, reply ordering, config barrier (ha-commit-barrier-rollout) -- Promotion no-op, own-term gate consumers, CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; this spec's E2E depends on it having landed) -- Entity-bearing snapshots (roadmap — would upgrade eviction to in-place rebuild); read-consistency filtering of status queries; primary step-down on quorum loss; worker rejoin job-state resync (roadmap backstop) -- Standby-apply conflict detection beyond the rejoin truncation rule (umbrella gap 5 is subsumed by reconciliation: divergent suffixes are truncated at role/lifecycle events, never patched in place) - -## Boundary Commitments - -### This Spec Owns -- The demotion path of the role lifecycle: everything `raft_cb_become_follower` does when leaving PRIMARY (hook disable, truncation, logging, self-restart) and the self-restart primitive (saved argv, exec, FATAL fallback, test seam) -- Rejoin reconciliation at the standby's `CtrlHelloReply` intake (truncate-above-boundary decision, clamp, once-guard) (Note: superseded by ha-term-safety — `ha_rejoin_reconcile` is retained only as the intake hook that records the learned boundary; the truncate-above-boundary decision, clamp, and once-guard it describes were retired in favor of the raft library's term-aware truncation, see Rejoin Reconciliation component below) -- `job_t.submit_lsn` tagging (live append + replay) and the commit-watermark predicate on scheduler-bound export seams -- The ACK-driven catch-up continuation trigger (walAck branch → scheduled streaming) including the single-flight guard and the parity call to `repl_complete_catchup` -- Quorum-loss/restore transition logging and its `quorum_lost` state flag -- Regression tests for umbrella 6.1/6.3/6.4/7.3 and the 3-node convergence E2E (extends ha-wal-replication task 5.4) - -### Out of Boundary -- `commit.c` / `include/commit.h` — read-only consumer of `committed_lsn`; never modified here -- `repl_wait_majority_ack`, write-path classification, reply ordering (ha-commit-barrier-rollout); promotion path of the lifecycle (`raft_cb_become_leader`) and COMMITTING removal (ha-single-leader-role) -- `ha_stream_wal_to_standby` internals and the WAL batch wire format (ha-wal-replication — the continuation calls it, unchanged) -- `persist_truncate_wal_above`, `persist_recover`, `replay_wal` internals — reused as-is -- Status/API read paths (apiserver.c:83, opget.c:173) — intentionally unfiltered (documented deviation) -- Worker/scheduler reconnection behavior after the restart this spec introduces (worker-join-refactor / worker-rejoin-resync) - -### Allowed Dependencies -- `persist_t.committed_lsn` atomic (lock-free reads) — the watermark and truncation boundary; ha-commit-engine owns advancement -- `persist_truncate_wal_above` (persist.c:1306), `persist_recover` boot path — truncation + rebuild primitives -- `ha_stream_wal_to_standby` (config_repl.c:1450) and `repl_start_catchup`/`repl_complete_catchup` (repl.c:507/521) — existing catch-up machinery; `repl_complete_catchup` already feeds `commit_on_membership_change` via ha-commit-engine wiring -- iomgr/vtaskmgr/delaymgr threading rules per `ha-async-io.md`: the walAck branch and `raft_cb_become_follower` may run on iomgr — file I/O (truncation, WAL streaming) and exec are always deferred to delaymgr/vtask tasks -- `xce_disable_hook` / hook registration guard (`hooks_registered`) — existing hook lifecycle -- Established `XCE_TESTING` function-pointer seam pattern (e.g., `ha_repl_stream_wal_hook`, `xce_wal_stream_request_hook`) - -### Revalidation Triggers -- ha-commit-engine changing `committed_lsn` ownership or making it persisted → re-verify rejoin reconciliation and the egress watermark -- ha-commit-barrier-rollout landing (fact allowlist) → re-verify the degraded-mode gate pin: the no-quorum rejected set becomes request-shaped opcodes only; fact-shaped opcodes (`OPCode_notify`/`update`/`setpreason`/`clrpreason`) are admitted -- ha-single-leader-role landing → re-anchor the quorum-transition log call site at promotion standby registration (ha.c:531), which lives inside `raft_cb_become_leader` (ha.c:445-617) — code that spec owns and rewrites wholesale. **(Resolved as-built: re-anchored to ha.c:2042 inside `raft_cb_become_leader`, now ha.c:1615-2103; the demotion insertion point re-verified at ha.c:2104-2247. Both current per the fact-check pass below.)** -- ha-single-leader-role changing the become-follower/become-leader callback contract → re-verify the demotion sequence insertion point -- Entity-bearing snapshots landing → re-evaluate restart-based eviction vs snapshot-based rebuild (umbrella revalidation trigger) -- Any new scheduler-bound export path → must adopt the watermark predicate (same rule as new WAL types must be classified) -- worker-rejoin-resync landing → re-verify restart interaction (facts lost across the restart are its recovery target) - -## Architecture - -### Existing Architecture Analysis -> Note: this subsection is a research-time (2026-06-11) snapshot of the PRE-implementation baseline — the "verified gap" facts are what the code lacked before this spec ran. Where a later spec has since changed the mechanism, a note is added; otherwise only stale `file:line` citations are corrected in place. -- `raft_cb_become_follower` (now ha.c:2104-2247, was ha.c:627-690 at design time) cancels timers, closes vote endpoints, sets CTRL_STANDBY — verified: no truncation, no hook disable (the disable logic exists only in the uncalled `ha_demote_to_standby`, now ha.c:5081-5094, was ha.c:1712-1726 at design time) -- The RDHUP failover path already truncates above the locally known committed boundary (ha.c:3416-3420 at design time) — the pattern this spec generalizes to demotion and rejoin. (Note: superseded by ha-term-safety — this ad hoc watermark-truncate pattern, and the rejoin generalization built on it, was itself replaced by the raft library's (term,lsn)-driven `on_truncate_from` / `ha_cb_truncate_from_stub`, ha.c:2638-2655, whose own doc comment states it "REPLACES the committed-boundary truncate heuristic on the library-driven path"; see that spec's design.md) -- The standby learns the primary's `committed_lsn` at the `CtrlHelloReply` intake (now cluster.c:411-419, was cluster.c:283-298 at design time, stored to `ha->last_known_committed_lsn`) — the rejoin-reconciliation evaluation point -- Catch-up is heartbeat-polled: the stateReport branch (now ha.c:6683, was ha.c:2734) drives `ha_dispatch_repl_action_for_standby` (now ha.c:4331-4466, was ha.c:1373-1478), one action per heartbeat cycle; the walAck branch (now ha.c:6764-6812, was ha.c:2737-2754) only records the ACK. Replication target is already derived as `max(committed_lsn, last_written)` (now via `ha_compute_repl_target_lsn`, called at ha.c:6759, was inline at ha.c:2727-2732) -- `repl_complete_catchup` (now repl.c:769, was repl.c:521 at design time) has no production caller (discovered gap — in_catch_up is set by broadcast-overflow eviction, repl.c:240, and never cleared); the ACK-driven continuation supplies the missing parity trigger -- Scheduler-bound egress has two seams: collect-based full push (`xcesched_push_jobs` now sched.c:214, was sched.c:169, on scheduler join) and the incremental JOB_NEW push (`sched_job_new_fun` now sched.c:286, was sched.c:236-248 → `xcesched_push_joblist`); both must apply the watermark. Read paths (apiserver.c:83, opget.c:173 — both confirmed current, unchanged) share `collect_jobs` (dataex.c:294 — confirmed current, unchanged) and stay unfiltered -- No saved argv / exec machinery exists anywhere in the controller — self-restart is net-new (main.c:112 at design time; landed at main.c:291-301 as-built) - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph role_lifecycle[role lifecycle - ha.c] - BF[raft_cb_become_follower<br/>from PRIMARY] - DC[demotion convergence task<br/>delaymgr] - SR[ha_self_restart<br/>exec or test seam] - end - subgraph rejoin[rejoin - cluster.c + ha.c] - CHR[CtrlHelloReply intake] - RR[rejoin reconciliation] - end - subgraph persist[persist.c - reused] - TR[persist_truncate_wal_above] - REC[persist_recover boot replay] - end - subgraph egress[egress - dataex.c, sched.c, job.c] - TAG[submit_lsn tagging] - PRED[job_submit_committed predicate] - PUSH[scheduler push paths] - end - subgraph catchup[catch-up - ha.c + config_repl.c] - ACK[walAck branch iomgr] - CONT[continuation task delaymgr] - STREAM[ha_stream_wal_to_standby] - RCC[repl_complete_catchup] - end - QO[quorum observability - ha.c] - CL[(committed_lsn<br/>ha-commit-engine owned)] - - BF --> DC --> TR - DC --> SR - CHR --> RR --> TR - RR --> SR - SR -. process restart .-> REC - TAG --> PRED - CL --> PRED --> PUSH - ACK -->|behind target| CONT --> STREAM - ACK -->|at parity| RCC - CL --> ACK - QO -. WARN once per transition .- CL -``` - -**Architecture Integration** (sliced from umbrella): -- Selected pattern: convergence by recovery, not by transaction — discard events coincide with role/lifecycle events (demotion, rejoin), handled by truncate + boot-recovery rebuild; the only steady-state mechanism is the one-comparison watermark filter (umbrella-locked) -- Existing patterns preserved: RDHUP truncation pattern, heartbeat decision loop (kept as fallback), enqueue-only streaming, `XCE_TESTING` seams, `errnox` early-return style -- New code rationale: each gap closes with the smallest insertion at a verified seam; no new module — all changes live in the files that own the respective seams -- Steering compliance: no file I/O or blocking on iomgr (convergence + continuation run on delaymgr tasks); TDD with testsuite/ tests first - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), pthreads, stdatomic | demotion/rejoin/egress/catch-up logic | no new dependencies, no .capnp changes | -| Data / Storage | existing WAL + snapshot sentinel (persist.c) | truncation boundary + boot rebuild | `submit_lsn` is in-memory only (recomputed from entry LSN at replay) | -| Process | execv re-exec of saved argv | demotion/rejoin eviction | FATAL on failure; seam under XCE_TESTING | -| Build / Test | meson, testsuite/ pattern | new test files in `controller_unit_test_src` | E2E via `/xce-deploy-ha` | - -## File Structure Plan - -### Directory Structure -``` -src/controller/ -├── ha.c # MODIFIED — demotion branch inline in raft_cb_become_follower, walAck -│ # continuation trigger, quorum transition call sites -├── ha_convergence.c # NEW (as-built, not in original plan) — ha_self_restart, -│ # ha_rejoin_reconcile, cluster_apply_ctrlhello_reply, -│ # ha_log_quorum_transition, ha_demotion_convergence_do_run -│ # all landed here instead of ha.c -├── ha.h # MODIFIED — quorum_lost flag, catchup_inflight tracking, restart-hook -│ # seam declaration, new function prototypes -├── main.c # MODIFIED — save argv for re-exec (controller_saved_argv accessor) -├── cluster.c # MODIFIED — CtrlHelloReply intake calls rejoin reconciliation -├── config_repl.c # MODIFIED — catch-up continuation task fn (wraps ha_stream_wal_to_standby) -├── egress.c # NEW (as-built, not in original plan) — job_submit_committed predicate + -│ # collect_jobs_committed, factored out of dataex.c for -│ # test-binary linkage (research D3 refinement; declared in -│ # include/cluster.h, not dataex.h) -├── sched.c # MODIFIED — scheduler push paths consume the predicate (full + JOB_NEW push) -├── job.c # MODIFIED — restore_job tags submit_lsn from the replayed entry -├── persist.c # MODIFIED — persist_job_hook_fun tags submit_lsn at live append; -│ # apply_wal_entry passes entry LSN to restore_job -└── testsuite/ - ├── test_demotion_convergence.c # NEW — demotion + rejoin reconciliation + restart seam - ├── test_egress_filter.c # NEW — watermark filter on scheduler export seams - └── test_convergence_verification.c # NEW — ACK-driven catch-up, quorum logging, and - # regression pins for 6.1/6.3/6.4/7.3 (umbrella) -src/include/xce/types.h # MODIFIED — job_t wrapper gains uint64_t submit_lsn -src/controller/meson.build # MODIFIED — register new test files -``` - -### Modified Files (what and why) -- `ha.c` / `ha.h` — demotion branch in `raft_cb_become_follower` (ha.c:2104-2247; detect prior role PRIMARY → disable hooks inline → schedule convergence task); walAck-branch (ha.c:6764-6812) continuation scheduling with per-standby single-flight guard and parity `repl_complete_catchup` call; `ha_log_quorum_transition()` called from the standby_endpoints mutation sites (now ha.c:2042, 7651, 9350) and the gate rejection backstop (ha.c:4617). As-built, `ha_self_restart()` and `ha_rejoin_reconcile()` landed in a new `ha_convergence.c` (see File Structure Plan) rather than `ha.c` directly. -- `main.c` — store argc/argv before parsing (main.c:291-301); expose `controller_saved_argv()` for re-exec -- `cluster.c` — `CtrlHelloReply` intake (now cluster.c:411-419) additionally invokes `ha_rejoin_reconcile(c, chrep->committed_lsn)` via `cluster_apply_ctrlhello_reply`. (Note: superseded by ha-term-safety — `ha_rejoin_reconcile` now only records the boundary; see Rejoin Reconciliation component.) -- `config_repl.c` — `ha_catchup_continuation_fn` (now config_repl.c:2839, delaymgr task body): re-derive target, call `ha_stream_wal_to_standby` (config_repl.c:2351), clear the in-flight flag -- `egress.c` (as-built; design originally assigned this to `dataex.c`, see File Structure Plan) — `job_submit_committed(cluster_t*, job_t*)` predicate; `collect_jobs_committed()` wrapper used only by scheduler-bound callers (read paths keep plain `collect_jobs`, dataex.c:294) -- `sched.c` — `xcesched_push_jobs` (sched.c:214) collects via the committed-only wrapper; `xcesched_push_joblist` (sched.c:70, JOB_NEW incremental push) filters its list with the same predicate -- `job.c` / `persist.c` — tagging: live path sets `submit_lsn = written_lsn` in `persist_job_hook_fun` (now persist.c:6509); replay path threads `hdr->lsn` from `apply_wal_entry` (now persist.c:3281) into `restore_job` (now job.c:274) -- `src/include/xce/types.h` — `job_t` wrapper (types.h:483-490) gains `uint64_t submit_lsn` (0 = unknown/pre-feature = exportable); wire type `xce_job_t` untouched - -## System Flows - -### Demotion convergence (role lifecycle, after this spec — umbrella flow sliced) - -```mermaid -sequenceDiagram - participant RL as raft library - participant BF as raft_cb_become_follower - participant DM as delaymgr task - participant P as persist.c - participant OS as process - RL->>BF: higher term seen while PRIMARY - BF->>BF: disable persistence hooks (inline, first) - BF->>BF: set CTRL_STANDBY, cancel timers (existing) - BF->>DM: schedule demotion convergence (one-shot) - DM->>P: persist_truncate_wal_above(committed_lsn) - DM->>DM: WARN log: truncated (committed_lsn, prior tip] - DM->>OS: ha_self_restart: execv(saved argv) - alt exec fails - OS->>OS: FATAL (abort) — never continue divergent - else XCE_TESTING - DM->>DM: ha_restart_hook seam invoked instead - end - OS-->>P: boot: persist_recover replays committed prefix only -``` - -Key decisions: hooks are disabled synchronously in the callback (no provisional entry may be produced after the higher term is observed — 1.1); truncation and exec are deferred off the callback thread (iomgr-safe — D1); truncation always precedes restart so boot replay yields committed-only state (1.2, 1.3, 1.7); crash before/during the task is safe — the rejoin reconciliation flow covers the untruncated-suffix case (Note: as of ha-term-safety, that coverage is via the raft library's term-aware truncation, not this spec's rejoin-reconcile truncate path — see Rejoin Reconciliation component). The "exec fails → FATAL (abort)" branch in the diagram above is also superseded: see the Self-Restart Primitive component and Error Handling section for the current fence-and-retry behavior. - -### Rejoin reconciliation (generalizes RDHUP truncation, ha.c:3416-3420 at design time) - -> **(Note: superseded by ha-term-safety, closed 2026-07-17.)** The flow and "Key decisions" below describe this spec's original clamped-truncate-and-restart mechanism, which is no longer live. `ha_rejoin_reconcile` (ha_convergence.c:467-493) is now a documented no-op: the truncate-above-boundary decision, clamp, and once-guard were retired in favor of the raft library's own (term,lsn) log-matching on incoming AppendEntries, which drives `on_truncate_from` to delete exactly the divergent suffix without a self-restart. The function is retained only as the `CtrlHelloReply` hook that records the learned committed boundary. See ha-term-safety's design.md for the current mechanism. - -```mermaid -sequenceDiagram - participant PRI as primary - participant CL as cluster.c intake - participant RR as ha_rejoin_reconcile - participant P as persist.c - PRI->>CL: CtrlHelloReply (committed_lsn) - CL->>CL: store last_known_committed_lsn (existing) - CL->>RR: evaluate(local tip vs boundary) - alt tip > boundary and not already reconciled - RR->>P: truncate above max(boundary, local committed_lsn) - RR->>RR: WARN log range, set once-guard - RR->>RR: ha_self_restart (same primitive) - else tip <= boundary - RR->>CL: no action — join proceeds (2.3) - end -``` - -Key decisions: the clamp `max(primary_reported, local committed_lsn)` never discards the locally known committed prefix (2.2); "once" is structural — after restart the tip equals the boundary, so the condition cannot re-fire (plus a defensive in-process flag) (2.1); together with demotion convergence and the never-retroactive-success rule this yields single-outcome convergence, asserted end-to-end (2.4). - -### Commit-gated scheduler egress - -Live submit: `persist_job_hook_fun` appends the submit entry and tags `j->submit_lsn = written_lsn`. Export time (scheduler-bound only — full push on scheduler join, JOB_NEW incremental push): `job_submit_committed` admits the job iff `submit_lsn == 0 || submit_lsn <= committed_lsn` (one atomic load, lock-free — 3.1, 3.2). Replay/standby apply tags from the entry LSN; on a promoted primary, replayed entries sit at or below the promotion truncation point, so restored jobs are exportable by construction (3.3). Status/API readers bypass the predicate entirely (3.4). Accepted edge (documented): a timeout-failed submit that later commits reaches the scheduler at the next scheduler (re)join full push — the requester keeps its failure answer (2.4 alignment). - -### ACK-driven catch-up continuation - -```mermaid -sequenceDiagram - participant S as standby - participant IO as iomgr (walAck branch) - participant DM as delaymgr continuation - participant CR as ha_stream_wal_to_standby - S->>IO: WalAckReply(acked_lsn) - IO->>IO: repl_on_ack (existing; feeds commit engine) - alt acked_lsn < target = max(committed, last_written) - IO->>DM: schedule continuation if not already in flight - DM->>CR: stream [acked_lsn+1, target] (enqueue-only) - DM->>DM: clear in-flight flag - CR-->>S: next batches - else acked_lsn >= target - IO->>IO: repl_complete_catchup → re-inclusion (7.1) - end -``` - -Key decisions: continuation is scheduled from the walAck branch (ha.c:2737-2754) but executes on delaymgr (streaming reads WAL files — never on iomgr); the per-standby single-flight flag bounds concurrency (5.2); target re-derivation each round naturally extends the stream to entries appended meanwhile (5.1); the heartbeat decision loop is untouched — it remains the fallback initiator and the only snapshot-decision owner (5.3, 6.1); parity wires the existing-but-untriggered `repl_complete_catchup` (7.1). - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1 | hooks disabled on demotion before anything else | Demotion Convergence | demotion | -| 1.2 | truncate above committed_lsn on demotion | Demotion Convergence | demotion | -| 1.3 | self-restart rebuilds via boot recovery | Demotion Convergence, Self-Restart Primitive | demotion | -| 1.4 | exec failure → FATAL | Self-Restart Primitive | demotion | -| 1.5 | WARN log with truncation range | Demotion Convergence | demotion | -| 1.6 | XCE_TESTING restart seam | Self-Restart Primitive | demotion | -| 1.7 | state reconstructible from committed log | Demotion Convergence (boot recovery reuse) | demotion | -| 2.1 | rejoin: tip > boundary → truncate + restart once | Rejoin Reconciliation | rejoin | -| 2.2 | never truncate own committed prefix | Rejoin Reconciliation (clamp) | rejoin | -| 2.3 | tip <= boundary → no action | Rejoin Reconciliation | rejoin | -| 2.4 | single outcome, never retroactive success | Demotion Convergence + Rejoin Reconciliation (E2E) | demotion, rejoin | -| 3.1 | uncommitted submit excluded from scheduler export | Egress Filter | egress | -| 3.2 | included once committed | Egress Filter | egress | -| 3.3 | submit_lsn tagging live + replay | Egress Filter (tagging) | egress | -| 3.4 | status/API reads unfiltered | Egress Filter (boundary) | egress | -| 4.1 | full fan-out to all reachable standbys | Convergence Verification Suite (existing `repl_broadcast`) | — | -| 5.1 | continuation on walAck, no heartbeat wait | ACK-Driven Catch-up | catch-up | -| 5.2 | single-flight per standby | ACK-Driven Catch-up | catch-up | -| 5.3 | heartbeat loop stays fallback + snapshot initiator | ACK-Driven Catch-up (boundary) | catch-up | -| 6.1 | snapshot fallback for far-behind standby | Convergence Verification Suite (existing path) | — | -| 7.1 | re-inclusion at parity | ACK-Driven Catch-up (parity trigger), Convergence Verification Suite | catch-up | -| 8.1 | quorum-loss logged with counts | Quorum Observability | — | -| 8.2 | quorum-restore logged | Quorum Observability | — | -| 8.3 | once per transition, no storms | Quorum Observability | — | -| 9.1 | status served while quorum lost | Convergence Verification Suite (existing gate allowlist) | — | -| 9.2 | replication traffic served while quorum lost | Convergence Verification Suite (existing gate allowlist) | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| Demotion Convergence | controller/ha.c | PRIMARY→follower: disable hooks, truncate, log, restart | 1.1–1.3, 1.5, 1.7, 2.4 | persist truncation + recovery (P0), delaymgr (P0), Self-Restart (P0) | State | -| Self-Restart Primitive | controller/ha.c, main.c | re-exec saved argv; FATAL on failure; test seam | 1.3, 1.4, 1.6 | saved argv (P0) | Service | -| Rejoin Reconciliation | controller/ha.c, cluster.c | clamped truncation + one restart at rejoin | 2.1–2.4 | CtrlHelloReply intake (P0), persist truncation (P0), Self-Restart (P0) | State | -| Egress Filter | controller/dataex.c, sched.c, job.c, persist.c, types.h | watermark gate on scheduler-bound exports | 3.1–3.4 | committed_lsn (P0, read-only) | Service | -| ACK-Driven Catch-up | controller/ha.c, config_repl.c | continuation on walAck; parity re-inclusion | 5.1–5.3, 7.1 | walAck branch (P0), ha_stream_wal_to_standby (P0), repl_complete_catchup (P0), delaymgr (P0) | Event | -| Quorum Observability | controller/ha.c, ha.h | WARN once per quorum transition | 8.1–8.3 | ha_has_write_quorum (P0) | Service | -| Convergence Verification Suite | testsuite/, E2E | pin existing 6.1/6.3/6.4/7.3 behaviors + 3-node convergence | 2.4, 4.1, 6.1, 7.1, 9.1, 9.2 | /xce-deploy-ha (E2E only; needs ha-single-leader-role) | — | - -### Demotion Convergence + Self-Restart Primitive (`ha.c`, `main.c`) - -| Field | Detail | -|-------|--------| -| Intent | Discard the demoted ex-primary's provisional state by truncation + boot-recovery restart | -| Requirements | 1.1–1.7, 2.4 | - -**Responsibilities & Constraints** -- `raft_cb_become_follower` gains a was-PRIMARY branch (prior role read before the existing role store): synchronously disable the three persistence hooks (reuse the `ha_demote_to_standby` disable sequence + `hooks_registered` reset, ha.c:1712-1726 — currently uncalled from this path), then schedule the one-shot convergence task; all existing behavior (timer cancel, vote-ep close, STANDBY store, watchdog arm) is preserved for the non-PRIMARY cases -- Convergence task (delaymgr): `persist_truncate_wal_above(&c->persist, committed_lsn)` where `committed_lsn` is read from `persist_t` at task time; WARN with both LSNs (boundary and prior tip); then `ha_self_restart(c)` -- Deliberate scoping: a demotion from `CTRL_PRIMARY_COMMITTING` (a state still extant if this spec lands before ha-single-leader-role removes it) takes the legacy become-follower path with no truncation/restart — acceptable because a COMMITTING node has no registered persistence hooks and has appended no new entries, so its exposure is identical to today's -- `ha_self_restart`: flush logs; `execv` of `controller_saved_argv()`; if exec returns → ERROR + `abort()` (FATAL — 1.4). Under `XCE_TESTING`, invoke `ha_restart_hook(c)` instead and return (1.6); a settable fatal seam lets tests assert the FATAL path without dying -- fd hygiene: before exec, verify/ensure listener and peer fds do not leak into the new image (CLOEXEC audit; boot path re-creates all endpoints) -- Crash safety: if the node dies before the task runs, its untruncated suffix is handled by Rejoin Reconciliation at the next join — the two components together guarantee 2.4 - -##### Service Interface -```c -/* ha.h — controller-internal */ -/* Saved-argv accessor (main.c populates before argp_parse). NULL-terminated. */ -char **controller_saved_argv(void); - -/* Truncate-and-restart primitive. Never returns in production (exec or abort). - * Under XCE_TESTING returns 0 after invoking ha_restart_hook. */ -int ha_self_restart(cluster_t *c); - -#ifdef XCE_TESTING -extern int (*ha_restart_hook)(cluster_t *c); /* invoked instead of execv */ -extern void (*ha_fatal_hook)(const char *msg); /* invoked instead of abort */ -#endif -``` -- Preconditions: persistence hooks already disabled; truncation completed before `ha_self_restart` is called -- Postconditions (production): process image replaced; next boot replays only the surviving prefix -- Invariants: no new WAL entry is appended by this node between hook-disable and exec - -### Rejoin Reconciliation (`ha.c` + `cluster.c` intake) - -| Field | Detail | -|-------|--------| -| Intent | Converge a rejoining node whose log runs ahead of the cluster's committed boundary | -| Requirements | 2.1–2.4 | - -**(Note: superseded by ha-term-safety, closed 2026-07-17 — the mechanism below describes this spec's original design/as-shipped behavior; see the current-state summary immediately after.)** - -Original design/as-shipped: new `int ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn)` called from the `CtrlHelloReply` intake (cluster.c:283-298 at design time, now cluster.c:411-419) after the existing `last_known_committed_lsn` store. Decision: `local_tip = persist.current_lsn - 1`; if `local_tip > primary_committed_lsn` and the once-guard is clear → cutoff = `max(primary_committed_lsn, local committed_lsn)` (clamp, 2.2) → `persist_truncate_wal_above(cutoff)` → WARN with range → set once-guard → `ha_self_restart`. Otherwise return 0 with no side effects (2.3). The truncation/restart work is deferred to a delaymgr task if the intake runs on iomgr (same rule as demotion). Generalizes the RDHUP truncation (ha.c:3416-3420 at design time), which the design assumed remains in place for the failover path. Once-guard is structural (post-restart tip equals the boundary) plus a defensive `_Atomic int rejoin_reconciled` for the process lifetime (2.1). - -Current state (ha_convergence.c:467-493): ha-term-safety task 3.2 retired this entire clamped-truncate-and-restart body. `ha_rejoin_reconcile` now ignores `primary_committed_lsn` and unconditionally returns 0 — divergent-suffix reconciliation is owned by the raft library instead: an incoming AppendEntries is routed through `raft_on_append_entries_request`, whose (term,lsn) log-matching drives `on_truncate_from` (`ha_cb_truncate_from_stub`, ha.c:2638-2655) to delete exactly the divergent suffix, never a committed-prefix entry, and never by an LSN watermark. A node whose tip runs ahead keeps its entries until the leader's AE proves the conflict by term; until reconciled the existing promotion-time fence keeps it non-promotable without self-restarting. The function is retained solely as the `CtrlHelloReply` hook (now wired via `cluster_apply_ctrlhello_reply`, ha_convergence.c:512-524) that still records the learned committed boundary. The Testing Strategy section's rejoin-reconcile test descriptions below are similarly stale; current tests (`test_rejoin_no_watermark_truncate_no_restart`, `test_rejoin_repeated_calls_remain_no_op`) pin the no-op contract instead. - -### Egress Filter (`dataex.c`, `sched.c`, `job.c`, `persist.c`, `types.h`) - -| Field | Detail | -|-------|--------| -| Intent | One-watermark commit gate on scheduler-bound job exports | -| Requirements | 3.1–3.4 | - -**Responsibilities & Constraints** -- `job_t` wrapper (types.h:483-485) gains `uint64_t submit_lsn`; `0` = unknown/pre-feature = exportable (legacy WALs and pre-tagging jobs never strand); wire type `xce_job_t` unchanged -- Tagging: live — `persist_job_hook_fun` (persist.c:1107-1125) sets `submit_lsn = written_lsn` on each submitted job after a successful append; replay/standby apply — `apply_wal_entry` (persist.c:729-731) passes `hdr->lsn` to `restore_job` (job.c:71), which tags the restored job (3.3) -- Predicate (dataex.c): `bool job_submit_committed(cluster_t *c, job_t *j)` → `j->submit_lsn == 0 || j->submit_lsn <= atomic_load(committed_lsn)`; lock-free, one comparison -- Consumers (scheduler-bound only): `collect_jobs_committed()` wrapper in dataex.c used by `xcesched_push_jobs` (sched.c:183); the JOB_NEW incremental push (`xcesched_push_joblist`, fed by `sched_job_new_fun` sched.c:236-248) filters its list with the same predicate. Both seams verified in the anchor pass — research D3 -- Non-consumers (3.4): apiserver.c:83 and opget.c:173 keep plain `collect_jobs` — status/API reads intentionally unfiltered (documented deviation; read consistency out of scope) - -##### Service Interface -```c -/* dataex.h additions — controller-internal */ -bool job_submit_committed(cluster_t *c, job_t *j); /* watermark predicate */ -void collect_jobs_committed(cluster_t *c, collect_check_fn_t fn, xlist_t *l); -/* identical contract to collect_jobs, additionally excluding jobs whose - * submit entry is uncommitted; for scheduler-bound callers ONLY */ -``` -- Invariant: a job with `submit_lsn > committed_lsn` is never present in a scheduler-bound export payload; the same job remains visible to status/API reads - -### ACK-Driven Catch-up (`ha.c` walAck branch + `config_repl.c` continuation) - -| Field | Detail | -|-------|--------| -| Intent | Continue catch-up streaming at ACK speed; re-include at parity | -| Requirements | 5.1–5.3, 7.1 | - -**Responsibilities & Constraints** -- Trigger (iomgr, ha.c:2737-2754): after the existing `repl_on_ack`, compare `acked_lsn` to `target = max(committed_lsn, last_written)` (same derivation as ha.c:2727-2732). Behind target → schedule the continuation task unless the standby's `catchup_inflight` flag is set (5.2); at/above target → `repl_complete_catchup(&c->persist.repl, ep)` (7.1 — wires the existing, currently untriggered re-inclusion; ha-commit-engine's wiring propagates `commit_on_membership_change`) -- Continuation task (delaymgr, config_repl.c): re-derive target, call `ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target)` (unchanged internals, enqueue-only), clear `catchup_inflight`. Endpoint validity is re-checked against the standby registry at task time (the ep may have RDHUPed between schedule and run) -- Per-standby state: `_Atomic int catchup_inflight` keyed by the standby slot (placed alongside the existing per-peer arrays in `ha_state_t` or `standby_conn_t` — slot identity must survive between schedule and completion) -- The heartbeat decision loop (`ha_dispatch_repl_action_for_standby`, ha.c:1373-1478) is byte-identical: fallback initiator for standbys that never ACK, and sole owner of the snapshot decision (5.3; snapshot fallback 6.1 of this spec verified, not modified) -- Never blocks iomgr: scheduling is O(1); all file I/O is inside the delaymgr task - -### Quorum Observability (`ha.c`, `ha.h`) - -| Field | Detail | -|-------|--------| -| Intent | Operator-visible quorum-loss/restore transitions | -| Requirements | 8.1–8.3 | - -Summary: `ha_state_t` gains `_Atomic int quorum_lost` (0 at init). New `ha_log_quorum_transition(ha_state_t *ha)`: evaluate `ha_has_write_quorum`; on edge, atomically flip the flag and WARN once — loss: `"HA quorum lost: connected=%d required=%d"`; restore: `"HA quorum restored: connected=%d required=%d"` (8.1, 8.2). Call sites: the three standby_endpoints mutation points (promotion registration ha.c:531, RDHUP removal ha.c:3160-3166, `on_standby_connected` ha.c:3764+) plus the `ha_dispatch_gate` quorum-rejection branch as a backstop (flag check makes repeated rejections silent — 8.3). CAS on the flag guarantees once-per-transition under concurrency. Note: the promotion-registration anchor (ha.c:531) lives inside `raft_cb_become_leader` (ha.c:445-617), leader-role promotion code that ha-single-leader-role owns and rewrites wholesale — this call site must be re-anchored after ha-single-leader-role lands (or coordinated with that spec if this one is implemented first); see Revalidation Triggers. - -### Convergence Verification Suite (testsuite/, E2E) - -Summary: regression pins for the four existing behaviors the umbrella marked "Done (verify)": full fan-out (`repl_broadcast` reaches every active standby — 4.1), snapshot-fallback decision (`standby_lsn < oldest_wal_lsn` → HA_REPL_SEND_SNAPSHOT — 6.1), re-inclusion mechanism (`repl_complete_catchup` clears `in_catch_up`, standby re-enters broadcast and quorum — 7.1), degraded-mode allowlist (gate admits status + replication opcodes without quorum — 9.1, 9.2). Plus the 3-node E2E extending ha-wal-replication task 5.4: confirmed jobs survive failover; a timeout-failed submit converges to a single outcome on all members; the demoted ex-primary restarts once and rejoins converged; status served during isolation (2.4 + live 9.1). E2E requires ha-single-leader-role landed and a live `/xce-deploy-ha` cluster. - -## Data Models - -### Domain Model (changes only) -- **job_t** (types.h wrapper): + `uint64_t submit_lsn` — entry LSN of the job's submit record; `0` = unknown/exportable; in-memory only (recomputed at replay from the applied entry; never serialized — wire type unchanged) -- **ha_state_t**: + `_Atomic int quorum_lost` (transition edge detector); + per-standby `_Atomic int catchup_inflight[MAX_HA_CANDIDATES]` (single-flight guard); + `_Atomic int rejoin_reconciled` (defensive once-guard) -- **process**: saved argv (main.c static, NULL-terminated) — the restart vehicle -- Invariants: scheduler-bound exports contain only jobs with `submit_lsn <= committed_lsn` (or 0); at most one continuation in flight per standby; at most one rejoin reconciliation per process lifetime; truncation always precedes restart - -## Error Handling - -### Error Strategy -- Self-restart exec failure → ERROR log + `abort()` (FATAL) — a divergent ex-primary must never continue serving (locked constraint); under XCE_TESTING the fatal seam records instead of aborting -- Truncation failure during demotion/rejoin → ERROR, then restart anyway: boot replay of an untruncated log is reconciled at the next rejoin (the reconciliation loop converges); never proceed as a serving node past a failed convergence step -- Continuation-task endpoint gone at run time → clear in-flight flag and return (the heartbeat fallback re-evaluates) — no error surfaced -- Egress predicate is infallible (atomic load + compare); tagging failures cannot occur on the success path (tag set only after a successful append) -- Quorum logging is observability-only: WARN once per transition, never an error path, never gates the operation outcome (the gate's existing `-XCE_ERR_NOT_READY` behavior is unchanged) - -### Monitoring -- WARN logs: demotion truncation range; rejoin reconciliation range; quorum lost/restored with counts. INFO: continuation scheduling at TRACE/INFO consistent with existing streaming logs. Metrics export out of scope (umbrella). - -## Testing Strategy - -### Unit Tests (testsuite/, XCE_TESTING) -- `test_demotion_convergence.c`: become-follower from PRIMARY → hooks disabled before role store, truncation called with `committed_lsn` cutoff, WARN logged, `ha_restart_hook` invoked (1.1–1.3, 1.5, 1.6); exec-failure path invokes the fatal seam (1.4); become-follower from non-PRIMARY roles → no truncation/restart (regression); rejoin reconcile: tip > boundary → clamped cutoff `max(boundary, local committed)` + restart-once (2.1, 2.2); tip <= boundary → no action (2.3); second invocation in one lifetime → no second restart (2.1) -- `test_egress_filter.c`: live submit tags `submit_lsn = written_lsn`; replayed WAL_JOB_SUBMIT tags from entry LSN (3.3); job with `submit_lsn > committed_lsn` absent from `collect_jobs_committed` and from the JOB_NEW push list, present after `committed_lsn` advances (3.1, 3.2); `submit_lsn == 0` exportable; plain `collect_jobs` (status path) returns the uncommitted job (3.4) -- `test_convergence_verification.c`: walAck behind target schedules exactly one continuation (hook-counted via `ha_repl_stream_wal_hook`/`xce_wal_stream_request_hook`), duplicate ACKs while in flight schedule none (5.1, 5.2); ACK at parity calls `repl_complete_catchup` and clears `in_catch_up` (7.1); heartbeat decision path byte-equivalent for snapshot decisions (5.3, 6.1 pin); `repl_broadcast` reaches all active standbys, skips only in_catch_up/empty (4.1 pin); quorum transition logged once across repeated gate rejections, restore logged on reconnection (8.1–8.3); dispatch gate admits status (`OPCode_get`) and replication opcodes without quorum, rejects request-shaped opcodes only (`OPCode_jobsub`/`jobcontrol`/`queuecontrol`/`hostcontrol`/`jobdispatch`/`join`) and admits fact-shaped opcodes (`OPCode_notify`/`update`/`setpreason`/`clrpreason`) per ha-commit-barrier-rollout's fact allowlist (9.1, 9.2 pin) - -### Integration Tests (testsuite/) -- Demotion seam end-to-end: simulated PRIMARY with appended uncommitted suffix sees higher term → truncation + restart hook; a fresh replay over the surviving log yields committed-only state (1.7, 2.4 in-process half) -- Rejoin reconciliation through the cluster.c intake: deliver a CtrlHelloReply with a boundary below the local tip → reconcile fires; boundary at/above tip → join proceeds untouched - -### E2E (live cluster, extends ha-wal-replication task 5.4 — REQUIRES ha-single-leader-role landed) -- 3-node via `/xce-deploy-ha`: submit confirmed jobs → kill primary → promoted standby owns all confirmed jobs; a submit forced to time out at the barrier converges to a single outcome on all members and is never retroactively reported successful; the demoted ex-primary restarts once and rejoins converged (2.4, 1.3, 2.1) -- 3-node: isolate primary → submits fail, status still served, quorum-loss WARN present; heal → quorum-restored WARN, submits succeed (8.1, 8.2, 9.1 live) - -## Performance & Scalability -- Egress predicate: one atomic load + compare per job per export — negligible against existing per-job serialization -- Continuation: at most one in-flight stream per standby; scheduling O(1) on iomgr; streaming cost unchanged (same `ha_stream_wal_to_standby`), just driven at ACK cadence instead of 3s heartbeats — catch-up latency improves by up to one heartbeat period per round -- Demotion/rejoin restart: process restart cost accepted by design (rare role/lifecycle events; connections migrate to the new primary anyway — umbrella-locked rationale) - -## Migration Strategy -This spec is phase 4 (final) of the umbrella's hard-ordered landing. It depends on ha-commit-engine (`committed_lsn` semantics) being live; it is parallel-safe with ha-commit-barrier-rollout and ha-single-leader-role except that the 3-node E2E requires ha-single-leader-role landed (direct promotion — the demotion path asserted here is the post-COMMITTING-removal one). Despite that parallel-safety, implementation in roadmap order is recommended regardless: all four sub-specs touch `ha_dispatch_gate` and `raft_cb_become_follower` in disjoint but adjacent regions, and in-order landing avoids anchor churn and merge coordination (per cross-spec review 2026-06-11). Landing order inside the spec: egress filter (pure additive) → demotion convergence + self-restart → rejoin reconciliation (reuses the primitive) → ACK-driven catch-up → quorum observability → verification suite → E2E. Rollback trigger per umbrella: controller testsuite regression or live-cluster E2E failure; each step leaves the system no less consistent than before it (filter defaults open via `submit_lsn == 0`; demotion/rejoin only ever discard above the committed boundary). diff --git a/.kiro/specs/ha-convergence-recovery/requirements.md b/.kiro/specs/ha-convergence-recovery/requirements.md deleted file mode 100644 index ba8b17cb..00000000 --- a/.kiro/specs/ha-convergence-recovery/requirements.md +++ /dev/null @@ -1,125 +0,0 @@ -# Requirements Document - -> Sub-spec 4 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.4, 2.5, 2.7, 6.1, 6.2, 6.3, 6.4, 7.2, 7.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Umbrella criteria 6.1, 6.3, 6.4, and 7.3 are verification-of-existing-behavior requirements: they pin behavior that already exists with regression tests and introduce no new production code. - -## Project Description (Input) - -Five verified convergence gaps undermine the umbrella's single-outcome guarantee (research deep-dive §C): a demoted ex-primary never truncates its divergent WAL suffix; phantom provisional in-memory state is never evicted; restart replays the uncommitted suffix (committed_lsn is volatile, correctly per Raft, but nothing reconciles at rejoin); standby apply has no conflict detection; and uncommitted jobs leak to the scheduler during the commit window (orphan-dispatch risk). Cluster operators need cluster-wide single-outcome convergence and commit-gated egress: demotion truncates above `committed_lsn` and self-restarts through the existing boot recovery (provisional state discarded); a rejoining standby whose WAL tip exceeds the primary's committed boundary truncates and restarts once; the scheduler export path never sees a job whose submit entry is uncommitted; catch-up streaming continues on WAL acknowledgment without waiting for heartbeats; and quorum-loss transitions are operator-visible in logs. - -## Introduction - -This spec delivers the convergence-and-recovery slice of the umbrella contract. After this spec, an operation that timed out at the commit barrier converges to a single cluster-wide outcome: a demoted primary discards its provisional state by truncating its uncommitted WAL suffix and restarting through boot recovery; a rejoining node whose log runs ahead of the cluster's committed boundary reconciles by truncation and a one-time restart; and the scheduler — the consumer whose actions cannot be revoked — never observes a job whose submit entry is uncommitted. Catch-up of lagging standbys becomes acknowledgment-driven rather than heartbeat-polled, and entry into and exit from the quorum-lost degraded state is recorded in the log for operators. Existing behaviors the umbrella relies on — full fan-out replication, snapshot fallback for far-behind standbys, re-inclusion of caught-up standbys, and degraded-mode status service — are pinned with regression tests. - -## Boundary Context - -- **In scope**: Demotion convergence on the primary (disable persistence hooks, truncate above committed LSN, log, process self-restart, with a test-mode restart seam); rejoin reconciliation for a standby whose WAL tip exceeds the primary's committed boundary; commit-watermark egress filtering of scheduler-bound job exports (job submit-LSN tagging); acknowledgment-driven catch-up streaming continuation; quorum-loss/restore transition logging; regression tests pinning full fan-out, snapshot fallback, re-inclusion, and degraded-mode status service; the 3-node E2E convergence test extending ha-wal-replication task 5.4. -- **Out of scope**: Commit-index advancement and majority evaluation (ha-commit-engine — this spec consumes `committed_lsn`, never modifies `commit.c`); operation classification and reply ordering (ha-commit-barrier-rollout); promotion no-op, own-term gate consumers, and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role); entity-bearing snapshots (roadmap backlog — would later upgrade the eviction strategy); read-consistency filtering of status/API queries (documented deviation, future work); primary step-down on quorum loss (roadmap); worker rejoin job-state resync (roadmap backstop for facts lost across the restart this spec introduces); election/vote correctness (raft-library); replication wire format (ha-wal-replication). -- **Adjacent expectations**: Upstream, this spec relies on ha-commit-engine's `committed_lsn` semantics (per-LSN majority, monotonic, contiguous — complete) and on ha-wal-replication's ACK machinery and catch-up streaming. The full 3-node E2E additionally requires ha-single-leader-role to have landed (direct promotion, no transitional state); E2E tasks are explicitly marked with that dependency. Downstream, entity-bearing snapshots may upgrade demotion eviction from restart to snapshot-based rebuild, and worker-rejoin-resync recovers worker-reported facts across the restart this spec introduces. - -## Requirements - -### Requirement 1: Demotion Convergence (Truncate and Self-Restart) - -**Objective:** As a cluster operator, I want a demoted ex-primary to discard its uncommitted, potentially divergent state and rejoin from its committed state only, so that a failed-over cluster converges to a single outcome and a demoted node never keeps serving phantom state. - -#### Acceptance Criteria - -1. When a primary observes a higher term and is demoted, the controller shall stop persisting new state changes before continuing demotion, so that no further provisional entries are produced. *(umbrella 2.5)* -2. When a primary is demoted, the controller shall remove from its log all entries above its committed LSN. *(umbrella 2.5)* -3. When demotion truncation completes, the controller shall restart its own process so that in-memory state is rebuilt through boot recovery from the persisted committed state only, discarding all provisional in-memory state. *(umbrella 2.5, 2.7)* -4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)* — **SUPERSEDED (2026-07-10):** "terminate fatally" (`abort()`) violated the project-wide never-self-terminate invariant (CLAUDE.md "Process Lifecycle") on an operational condition (`execv()` can fail on `ENOMEM`/`EMFILE`/a permissions change mid-upgrade — none of these are programming-invariant violations). The safety property this criterion actually protects — a divergent ex-primary never serves authoritatively — is preserved without dying: on `execv()` failure the controller now sets a sticky, never-live-cleared `ha->self_restart_pending` flag (checked unconditionally by `raft_cb_become_leader` / `promotion_fence_timeout_fn` alongside `apply_divergent`, `ha.c`) and retries the re-exec periodically via delaymgr (`ha_self_restart_enter_fence_and_retry`, `ha_convergence.c`). The node stays alive, permanently non-promotable, and self-heals the moment a retry succeeds — never requiring manual admin recovery. See `ha-async-io.md`'s own `SUPERSEDES (2026-06-25)` note for the precedent of revising a locked decision when a stronger mechanism (loud-fence-and-stay-alive) was established after the original spec closed. -5. When a demotion occurs, the controller shall log the event with the truncation boundary (committed LSN and prior log tip) before restarting, so that operators can observe the discarded range. *(umbrella 2.5)* -6. Where test mode is enabled, the controller shall invoke a substitutable restart seam in place of process re-execution. *(locked constraint — testability)* -7. The controller shall be able to reconstruct its in-memory state from its persisted committed state, discarding any provisional state not present in the committed log. *(umbrella 2.7)* - -### Requirement 2: Rejoin Reconciliation and Single-Outcome Convergence - -**Objective:** As a cluster operator, I want a rejoining node whose log runs ahead of the cluster's committed boundary to reconcile by truncation and a one-time restart, so that every member's state converges to the same outcome even after a crash that skipped demotion convergence. - -#### Acceptance Criteria - -1. When a standby learns the primary's committed boundary at (re)join and its local log tip exceeds that boundary, the standby shall truncate its log above that boundary and restart its process once. *(umbrella 2.5)* -2. The standby shall not remove log entries at or below its own locally known committed LSN during rejoin reconciliation. *(umbrella 2.5 — safety clamp; commit index never regresses)* -3. While a standby's local log tip does not exceed the primary's committed boundary at rejoin, the standby shall join without truncation or restart. *(umbrella 2.5 — no spurious restarts)* -4. If an operation was reported as failed due to commit timeout, the cluster shall converge on a single outcome for its log entry — committed on all members or discarded on all members — the primary shall never retroactively report that operation as successful to the original requester, and each member's in-memory state shall converge to that same outcome. *(umbrella 2.5 — verified end-to-end)* - -### Requirement 3: Commit-Gated Scheduler Egress - -**Objective:** As a cluster operator, I want the scheduler to never observe a job whose submit entry is uncommitted, so that no job can be dispatched whose existence the post-failover cluster may never have heard of (orphan-dispatch prevention). - -#### Acceptance Criteria - -1. While a job's submit entry is not yet committed, the controller shall exclude that job from data exports that drive scheduling and dispatch decisions. *(umbrella 2.4)* -2. When a job's submit entry becomes committed, the controller shall include that job in subsequent scheduler-bound exports. *(umbrella 2.4)* -3. When a job is created or restored from the persisted log, the controller shall associate the job with the log position of its submit entry so its commit status is decidable at export time. *(umbrella 2.4)* -4. While a job's submit entry is not yet committed, the controller shall continue to include the job in status and API query responses; read paths are intentionally not commit-filtered. *(umbrella 2.4 boundary — documented deviation; read consistency is out of scope)* - -### Requirement 4: Full Fan-Out Replication (Verification of Existing Behavior) - -**Objective:** As a cluster operator, I want the primary to replicate to every reachable standby, so that the cluster trends toward full consistency — pinned by regression test, since the umbrella relies on this existing behavior. - -#### Acceptance Criteria - -1. While holding leadership, the primary shall replicate new log entries to all reachable standbys, not only to a majority subset. *(umbrella 6.1 — existing behavior, regression test only)* - -### Requirement 5: Acknowledgment-Driven Catch-Up Continuation - -**Objective:** As a cluster operator, I want a lagging standby brought current as fast as it can acknowledge, so that catch-up progress is not throttled to the heartbeat cadence. - -#### Acceptance Criteria - -1. When the primary receives a WAL acknowledgment from a standby that remains behind the replication target, the primary shall continue streaming the missing entries to that standby without waiting for the next heartbeat exchange. *(umbrella 6.2)* -2. While a catch-up continuation for a standby is already in flight, the primary shall not start a duplicate continuation for that standby. *(umbrella 6.2 — bounded continuation, no streaming storms)* -3. The heartbeat-driven replication evaluation shall remain in place as the fallback initiator of catch-up and as the initiator of snapshot-transfer decisions. *(umbrella 6.2, 6.3 — locked design boundary)* - -### Requirement 6: Snapshot Fallback (Verification of Existing Behavior) - -**Objective:** As a cluster operator, I want a standby that has fallen behind the oldest retained log entry brought current via snapshot, so that log retention limits never strand a member — pinned by regression test. - -#### Acceptance Criteria - -1. If a standby is behind the oldest retained log entry, the primary shall bring it current by transferring a snapshot followed by subsequent log entries. *(umbrella 6.3 — existing behavior, regression test only)* - -### Requirement 7: Catch-Up Re-Inclusion (Verification of Existing Behavior) - -**Objective:** As a cluster operator, I want a standby that has caught up re-included in live replication, so that it counts toward future commits — pinned by regression test. - -#### Acceptance Criteria - -1. When a lagging standby catches up to the primary's replication boundary, the primary shall re-include it in live replication. *(umbrella 6.4 — existing re-inclusion mechanism, regression test; the acknowledgment-driven continuation of Requirement 5 provides the production trigger at parity)* - -### Requirement 8: Quorum Transition Observability - -**Objective:** As a cluster operator, I want entry into and exit from the quorum-lost degraded state recorded in the log, so that I can observe and correlate degraded periods without instrumentation. - -#### Acceptance Criteria - -1. When the primary loses write quorum, the controller shall record the transition in its log, including the connected and required member counts. *(umbrella 7.2)* -2. When write quorum is restored, the controller shall record the transition in its log. *(umbrella 7.2)* -3. While quorum remains lost, the controller shall not repeatedly log the loss condition for every rejected operation; the transition shall be logged once per state change. *(umbrella 7.2 + umbrella error strategy — no log storms)* - -### Requirement 9: Degraded-Mode Service Continuity (Verification of Existing Behavior) - -**Objective:** As a cluster operator, I want a quorum-lost primary to keep answering status queries and replication traffic, so that the degraded cluster remains observable and can heal itself — pinned by regression test. - -#### Acceptance Criteria - -1. While quorum is lost, the primary shall continue to respond to status queries. *(umbrella 7.3 — existing dispatch-gate allowlist, regression test only)* -2. While quorum is lost, the primary shall continue to process replication-related traffic from returning members. *(umbrella 7.3 — existing dispatch-gate allowlist, regression test only)* - -## Umbrella Traceability - -| This spec | Umbrella ID | Notes | -|-----------|-------------|-------| -| 1.1–1.7 | 2.5, 2.7 | Demotion convergence: hook disable, truncate above committed_lsn, self-restart via boot recovery; FATAL on restart failure; XCE_TESTING seam (locked decisions) | -| 2.1–2.3 | 2.5 | Rejoin reconciliation — generalizes the existing RDHUP truncation pattern to the rejoin/hello path; restart at most once | -| 2.4 | 2.5 | Single-outcome convergence incl. in-memory state; never retroactive success — E2E-verified | -| 3.1–3.3 | 2.4 | Provisional-state egress gate: watermark filter on the submit entry's LSN, scheduler export only (locked decision) | -| 3.4 | 2.4 boundary | Status/API queries intentionally unfiltered — documented deviation (read consistency out of scope) | -| 4.1 | 6.1 | Verification of existing full fan-out broadcast — regression test, no new code | -| 5.1–5.3 | 6.2 | ACK-driven catch-up continuation; heartbeat path remains fallback + snapshot initiator | -| 6.1 | 6.3 | Verification of existing snapshot fallback — regression test, no new code | -| 7.1 | 6.4 | Verification of existing re-inclusion — regression test; Requirement 5 supplies the parity trigger | -| 8.1–8.3 | 7.2 | Quorum-loss/restore WARN logging, once per transition | -| 9.1, 9.2 | 7.3 | Verification of existing degraded-mode allowlist — regression test, no new code | diff --git a/.kiro/specs/ha-convergence-recovery/research.md b/.kiro/specs/ha-convergence-recovery/research.md deleted file mode 100644 index 186d0455..00000000 --- a/.kiro/specs/ha-convergence-recovery/research.md +++ /dev/null @@ -1,218 +0,0 @@ -# Research & Design Decisions: ha-convergence-recovery - -Date: 2026-06-11. Discovery type: integration-focused (extension of existing HA subsystem); the umbrella's `research.md` (deep-dive §C — five convergence gaps) is the primary discovery source. This log records the slice-level anchor verification pass and the design decisions local to this sub-spec. Umbrella-locked decisions are restated, not re-derived. - -## Umbrella-Locked Decisions (inputs, not open questions) - -- Demotion eviction = **truncate above `committed_lsn` + process SELF-RESTART** via the existing boot recovery (`persist_recover` + `replay_wal`) — option (ii) of umbrella research §C, user-decided 2026-06-11. Restart-hook seam under `XCE_TESTING`; **exec failure = FATAL** (a divergent ex-primary must not keep serving). Truncation always precedes restart. -- `commitIndex` is **volatile** per Raft — rejoin reconciliation, NOT persisted `committed_lsn`. -- Provisional-state model with **watermark egress filter**: `job_t.submit_lsn`, scheduler export only; status/API queries intentionally unfiltered (documented deviation). No undo log, no MVCC, no committed-view machinery. -- ACK-driven catch-up continuation; heartbeat evaluation remains the fallback initiator and the snapshot-decision owner. -- Umbrella 6.1/6.3/6.4/7.3 are existing behaviors — verified by regression test, no new production code. - -## 1. Anchor Verification Pass (2026-06-11, this worktree) - -All file:line anchors from the umbrella audit re-verified by grep before being written into design/tasks. Drift from the umbrella's 2026-06-10 numbers was negligible. - -| Anchor | Verified location | Finding | -|---|---|---| -| Demotion path | `raft_cb_become_follower` ha.c:627-690 | Confirmed gap: cancels timers, closes vote eps, sets CTRL_STANDBY — **never truncates, never disables hooks** (comment at ha.c:904 confirms hook disable lives only in `ha_demote_to_standby`) | -| Hook disable (unused on this path) | `ha_demote_to_standby` ha.c:1712-1726 | Disables the three persistence hooks + resets `hooks_registered`; NOT called from `raft_cb_become_follower` | -| Hook registration | `ha_promote_to_primary` ha.c:1686-1707 | Idempotency guard `hooks_registered`; symmetric disable exists | -| Truncation primitive | `persist_truncate_wal_above` persist.c:1306-1444 | Reused as-is; stores `committed_lsn = cutoff` at persist.c:1438 | -| RDHUP truncation pattern (to generalize) | ha.c:3416-3420 in `ha_on_primary_ctrl_fd_rdhup` | Truncates above local `ha->last_known_committed_lsn` | -| Rejoin signal point | cluster.c:283-298 | Standby stores the primary-reported `committed_lsn` from `CtrlHelloReply` into `ha->last_known_committed_lsn` — the natural rejoin-reconciliation hook | -| handle_ctrlhello | config_repl.c:1368 | Connection bookkeeping only; decisions live in the heartbeat loop | -| Boot recovery (reused by restart) | `persist_recover` persist.c:957, `replay_wal` persist.c:781 | Replays everything above `checkpoint_lsn` — no committed gate (umbrella gap 4; rejoin reconciliation is the fix, by design) | -| Heartbeat-driven repl decision loop | `ha_dispatch_repl_action_for_standby` ha.c:1373-1478; invoked from stateReport branch at ha.c:2734 | One action per heartbeat cycle per standby — confirms 6.2 gap | -| WAL ACK ingress (continuation hook point) | `repl_ack_decode_and_dispatch` ha.c:2632, walAck branch ha.c:2737-2754 → `repl_on_ack` repl.c:176 | Runs on iomgr (`peer_ctrl_msg_handler` ha.c:2777); replication target already computed as `max(committed_lsn, last_written)` at ha.c:2727-2732 | -| Catch-up streaming | `ha_stream_wal_to_standby` config_repl.c:1450-1583 | Streams `[standby_lsn+1, target]` in one call, enqueue-only; `xce_wal_stream_request_hook` test seam exists | -| Catch-up state | `repl_start_catchup` repl.c:507, `repl_complete_catchup` repl.c:521 | **Discovered gap**: `repl_complete_catchup` has NO production caller (in_catch_up set by eviction inline repl.c:240; documented protocol step 5 in config_repl.h:173 never wired). The ACK-driven continuation supplies the missing parity trigger (Req 7.1) | -| Quorum gate | `ha_has_write_quorum` ha.c:1507-1525, `ha_dispatch_gate` ha.c:1527+ | No transition logging anywhere; rejection is silent `-XCE_ERR_NOT_READY`. Allowlist for degraded mode at ha.c:1538-1558 (Req 9 regression target) | -| standby_endpoints mutations (quorum-change points) | ha.c:531 (promotion), ha.c:3160-3166 (RDHUP removal), `on_standby_connected` ha.c:3764+ | The three production sites where the quorum count changes | -| Scheduler export seams | `collect_jobs` dataex.c:294; scheduler full push `xcesched_push_jobs` sched.c:169/183 (on join, sched.c:609); **incremental push** `sched_job_new_fun` sched.c:236-248 → `xcesched_push_joblist` sched.c:14 | Two scheduler-bound egress seams, not one — see decision D3 | -| Unfiltered read paths (stay unfiltered) | apiserver.c:83, opget.c:173 (`OPCode_get`) | Both use `collect_jobs`; must NOT gain the filter (Req 3.4) | -| job_t | `src/include/xce/types.h:483-485` — `typedef struct { xce_job_t job; } job_t;` | Controller wrapper around the wire type; `submit_lsn` goes on the wrapper, wire format untouched | -| Replay restore | `apply_wal_entry` persist.c:691 → `restore_job` job.c:71 (WAL_JOB_SUBMIT at persist.c:729-731) | `restore_job` does not currently receive the entry LSN — tagging at replay needs it passed through | -| Submit-path append | `persist_job_hook_fun` persist.c:1107-1125, `persist_wal_sync_append` persist.c:1003+ (`written_lsn` at 1018-1020) | Tagging point for live submits | -| Controller entry | `main` main.c:112 | **No saved argv, no execv anywhere in src/controller or src/lib** — self-restart machinery is net-new | -| Hook chain semantics | `xce_run_hooks` hook.c:101-141 | AFTER hooks all run in registration order; rc = last hook's rc; sched hooks registered at scheduler join (sched.c:616), persist hooks at promotion (ha.c:1692) — relative order is timing-dependent (input to D3) | -| E2E baseline | ha-wal-replication tasks.md task 5.4 (line 111) | Live 3-node test, `/xce-deploy-ha`, still unimplemented (blocked-manual); this spec's E2E extends it | -| Test seam pattern | `ha_repl_stream_wal_hook` ha.c:2104, `ha_phase2_hook`, `xce_wal_stream_request_hook` | Established `XCE_TESTING` function-pointer seam style for the restart hook | - -## 2. Slice-Level Design Decisions - -**D1 — Demotion convergence runs off the callback thread.** `raft_cb_become_follower` can fire from vote handling (vtask) or peer message handling (iomgr). Truncation is file I/O and restart is process-fatal; per `ha-async-io.md` neither may run on iomgr. Decision: the become-follower callback detects PRIMARY→follower, disables persistence hooks inline (atomic, cheap — must happen before any further write), then schedules a one-shot demotion-convergence task (delaymgr, existing pattern) that truncates, WARN-logs the range, and self-restarts. The restart (exec) ends participation; any entry applied in the tiny window between truncation and exec is re-replayed at boot — harmless. - -**D2 — Self-restart mechanism.** `main()` saves `argv` (new `controller_saved_argv` accessor in main.c) before `argp_parse`. New `ha_self_restart()` in ha.c: flush logs, `execv(saved_argv[0], saved_argv)`; on exec failure log ERROR and `abort()` (FATAL — locked constraint). Under `XCE_TESTING`, `ha_restart_hook` function-pointer seam is invoked instead (and FATAL is asserted via a settable fatal hook rather than aborting the test binary). Implementation note: audit listener/connection fds for `FD_CLOEXEC` before exec (task bullet) so the re-exec'd image doesn't inherit live sockets. - -**D3 — Egress filter seams (refinement, not contradiction).** The umbrella names `collect_jobs` (dataex.c) as the scheduler export path. Anchor verification found TWO scheduler-bound egress seams: (a) the collect-based full push (`xcesched_push_jobs`, on scheduler join) and (b) the incremental JOB_NEW push (`sched_job_new_fun` → `xcesched_push_joblist`), whose ordering relative to the persist hook (and hence the commit wait) is registration-timing-dependent. Both are "exports that drive scheduling decisions" (umbrella 2.4), so the same watermark rule applies at both. Decision: one shared predicate `job_submit_committed(cluster_t*, job_t*)` (`submit_lsn == 0 || submit_lsn <= committed_lsn`; exported from dataex.c next to the collect machinery), consumed by (a) a scheduler-bound collect wrapper in dataex.c and (b) a filter pass in the JOB_NEW push list. Status/API readers (apiserver.c, opget.c) keep calling unfiltered `collect_jsons`/`collect_jobs` — Req 3.4. Known accepted edge (documented deviation, matches umbrella 2.5 semantics): a submit that times out at the barrier but commits later is visible to the scheduler at the next scheduler (re)join full push — the requester was already told "failed"; never retroactively reported successful. - -**D4 — submit_lsn semantics.** `0` = pre-feature/unknown → treated as committed (export allowed) so replayed legacy WALs and standby-restored jobs do not strand. Live submit: `persist_job_hook_fun` sets `j->submit_lsn = written_lsn` after append. Replay/standby apply: `apply_wal_entry` passes `hdr->lsn` through to `restore_job`, which tags the restored job. On the primary, replayed entries are ≤ the promotion truncation point (safe_lsn → `committed_lsn`), so restored jobs pass the filter by construction — matching the umbrella's "replay-restored jobs carry their entry LSN (≤ committed by definition)". - -**D5 — Rejoin reconciliation trigger and clamp.** The reconciliation evaluates at the standby's `CtrlHelloReply` intake (cluster.c:283-298) — the moment the rejoining node learns the primary's committed boundary. Condition: local WAL tip (`persist.current_lsn - 1`) > primary-reported `committed_lsn`. Action: truncate above `max(primary_reported_committed, local committed_lsn)` (clamp — never truncate the locally known committed prefix; Req 2.2), WARN-log, self-restart once (reuses D2). "Once" is structural: after restart the tip equals the truncation boundary, so the condition cannot re-fire for the same divergence; a defensive in-process flag prevents re-entry within one process lifetime. - -**D6 — ACK-driven continuation with single-flight guard.** Hook point: walAck branch of `repl_ack_decode_and_dispatch` (ha.c:2737-2754, iomgr). After `repl_on_ack`, if `acked_lsn < target` (target = `max(committed_lsn, last_written)`, same derivation as ha.c:2727-2732), schedule a continuation task (delaymgr — streaming reads WAL files, must leave iomgr) that calls the existing `ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target)`. Per-standby `catchup_inflight` flag (set when scheduled, cleared when the task completes) prevents duplicate continuations (Req 5.2). At parity (`acked_lsn >= target`): call `repl_complete_catchup` — wiring the discovered missing production trigger for re-inclusion (Req 7.1); ha-commit-engine's wiring makes that call feed `commit_on_membership_change`. Heartbeat loop untouched as fallback + snapshot initiator (Req 5.3, 6.1). - -**D7 — Quorum transition logging.** New helper `ha_log_quorum_transition(ha)` compares `ha_has_write_quorum` against an `_Atomic int quorum_lost` flag on `ha_state_t` and WARN-logs only on change ("HA quorum lost: connected=X required=Y" / "HA quorum restored: connected=X required=Y"). Called from the three standby_endpoints mutation sites (promotion registration, RDHUP removal, on_standby_connected) plus the dispatch-gate rejection path as a backstop. No per-rejection logging (Req 8.3). - -## 3. Verification-of-Existing Findings (Reqs 4, 6, 7, 9) - -- 6.1 full fan-out: `repl_broadcast` (repl.c:210+) iterates all active standbys, skipping only empty slots and in_catch_up — behavior present; regression test pins it. -- 6.3 snapshot fallback: `HA_REPL_SEND_SNAPSHOT` decision + `ha_send_snapshot_to_standby` (ha.c:1420-1443) present, including fresh-checkpoint-then-send; regression test pins the decision boundary (`standby_lsn < oldest_wal_lsn`). -- 6.4 re-inclusion: mechanism (`repl_complete_catchup`) present but trigger missing in production (see anchor table) — the regression test pins the mechanism; D6 supplies the trigger. This is the one "verification" item that needed a production wiring change, delivered inside the Req 5 component rather than as new scope. -- 7.3 degraded service: dispatch-gate quorum-loss allowlist (ha.c:1538-1558) admits heartbeat/replication opcodes + `OPCode_get`/probe; regression test pins the allowlist behavior for status + replication ops. - -## 4. Risks - -| Risk | Mitigation | -|---|---| -| Self-restart loops (restart → rejoin → truncate → restart …) | Structural once-guard (D5); E2E asserts a single restart per divergence event | -| exec inherits live fds | CLOEXEC audit task; listeners re-created by boot path | -| Demotion convergence racing standby apply before restart | Hooks disabled first; truncate+exec in one task; post-truncation appends replay at boot (D1) | -| Egress filter starving the scheduler after primary boot | `submit_lsn` ≤ promotion truncation point for replayed jobs (D4); filter passes by construction once promoted | -| Continuation storms under bursty ACKs | Single-flight per-standby guard (D6); streaming remains enqueue-only | - -## 5. Design Synthesis Outcomes - -- **Generalization**: demotion truncation and rejoin reconciliation are one capability ("converge local log to the cluster's committed boundary, then rebuild via boot recovery") with two triggers; both reuse `persist_truncate_wal_above` + one new `ha_self_restart` primitive. -- **Build vs adopt**: zero new libraries; all five workstreams reuse existing primitives (truncation, boot recovery, streaming, dispatch-gate allowlist, hook seam pattern). Net-new machinery is limited to saved-argv re-exec, the watermark predicate, the continuation scheduler, and the transition-logging helper. -- **Simplification (locked)**: no in-place memory teardown (restart instead), no persisted commit index, no per-entry undo, no filtering of read paths. -- Design review gate: all 26 requirement IDs traced; no orphan components; boundary ↔ file-structure alignment verified. - ---- - -## Gap Re-Audit (2026-06-18) - -**Framing**: POST-IMPLEMENTATION GAP AUDIT. Tasks 1-7 (18 sub-tasks) are complete and code-verified; one task remains open (`[ ]` Task 8 — live 3-node E2E). This pass re-verifies every EARS requirement against the on-disk code (codegraph + grep, 2026-06-18 worktree) and characterizes the single open task. - -**Verdict**: PARTIAL — all 9 requirements (26 acceptance criteria) are IMPLEMENTED and unit/integration-tested in code. The only residual gap is the environment-dependent end-to-end verification of the cross-node behaviors (umbrella 2.4/8.1/8.2/9.1/9.2 over a real cluster), which is Task 8. - -### Requirement-to-Asset Map (re-verified 2026-06-18) - -| Req (criteria) | Asset (file:line) | Status | Notes | -|---|---|---|---| -| **1.1** hooks disabled on demotion before anything else | `raft_cb_become_follower` ha.c:1202-1232 (`prior_role == CTRL_PRIMARY && c != NULL` branch reads role before store, disables hooks inline) | MET | guard at ha.c:1205; demotion scheduled at ha.c:1232 | -| **1.2** truncate above committed_lsn on demotion | `ha_demotion_convergence_fn` ha_convergence.c:144-181 → `persist_truncate_wal_above(&c->persist, committed)` ha_convergence.c:163 | MET | committed read via atomic acquire ha_convergence.c:153 | -| **1.3 / 1.7** self-restart rebuilds via boot recovery | `ha_self_restart` ha_convergence.c:82-120 (`execv(argv[0], argv)` :111); boot path `persist_recover`/`replay_wal` reused | MET | truncation precedes restart (1.2 before :180) | -| **1.4** exec failure → FATAL | ha_convergence.c:115-117 `ERROR + abort()`; no-saved-argv → abort ha_convergence.c:101-105 | MET | controller never self-terminates on operational condition — this is crash-equivalent fatal, per locked constraint | -| **1.5** WARN log with truncation range | `ha_demotion_convergence_fn` WARN with committed_lsn + prior_tip, ha_convergence.c:170-173 | MET | prior_tip read under persist lock :157-161 | -| **1.6** XCE_TESTING restart seam | `ha_restart_hook` / `ha_fatal_hook` ha_convergence.c:84-96 | MET | tests drive via these seams | -| **2.1** rejoin tip>boundary → truncate+restart once | `ha_rejoin_reconcile` ha_convergence.c:271+ → `ha_rejoin_reconcile_fn` :240-269; `_Atomic rejoin_reconciled` once-guard | MET | test_rejoin_clamp_…_restarts_once, test_rejoin_second_call_no_second_restart | -| **2.2** never truncate own committed prefix (clamp) | cutoff = `max(primary_committed, local committed_lsn)` (test asserts cutoff=6 when boundary=4, local committed=6) | MET | test_demotion_convergence.c:599-635 | -| **2.3** tip<=boundary → no action | `ha_rejoin_reconcile` early-return path, returns 0 no side effects | MET | test_rejoin_tip_at_or_below_boundary_no_action :642-665 | -| **2.4** single outcome, never retroactive success | demotion + rejoin convergence (in-process halves tested: append committed prefix + uncommitted suffix → demote seam → replay → committed-only). Cross-node end-to-end = Task 8 | PARTIAL (in-process MET; live E2E = Task 8) | the only criterion split across unit + E2E | -| **3.1 / 3.2** uncommitted submit excluded from scheduler export / included once committed | `job_submit_committed` egress.c:47-56 (`submit_lsn==0 \|\| submit_lsn<=committed_lsn`); `collect_jobs_committed` egress.c:76-109 | MET | sched.c:218 (full push) + sched.c:38 (JOB_NEW push) consume it | -| **3.3** submit_lsn tagging live + replay | live: `persist_job_hook_fun` persist.c:2202-2220 (`j->submit_lsn = submit_lsn`); replay: `restore_job(entry_lsn)` from `apply_wal_entry` (hdr->lsn) | MET | tag set on local-write success incl. -ETIMEDOUT/-NOT_READY | -| **3.4** status/API reads unfiltered | apiserver.c:83 + opget.c:173 use plain `collect_jobs` | MET | grep-confirmed: no read path calls the committed-only wrapper | -| **4.1** full fan-out to all reachable standbys | `repl_broadcast` regression pin in test_convergence_verification.c (existing behavior) | MET | verification-only, no new production code | -| **5.1** continuation on walAck, no heartbeat wait | walAck branch → `ha_schedule_catchup_continuation` config_repl.c:1784; body `ha_catchup_continuation_fn` :1739 streams `[acked+1, target]` on delaymgr | MET | re-derives target each round (extends to entries appended meanwhile) | -| **5.2** single-flight per standby | `_Atomic catchup_inflight[]` CAS expected=0→1 config_repl.c:1839; cleared on every exit path :1772/:1791/:1811 | MET | duplicate ACK while in flight schedules nothing | -| **5.3** heartbeat loop stays fallback + snapshot initiator | `ha_dispatch_repl_action_for_standby` byte-unchanged (heartbeat path) | MET | snapshot decision ownership preserved | -| **6.1** snapshot fallback for far-behind standby | `repl_needs_snapshot` repl.c:630 + HA_REPL_SEND_SNAPSHOT decision pin | MET | verification-only | -| **7.1** re-inclusion at parity | `repl_complete_catchup` repl.c:602-628 (clears in_catch_up, calls commit_on_membership_change) — production caller now at parity branch config_repl.c:1831 | MET | closed the "no production caller" gap from §1 | -| **8.1 / 8.2 / 8.3** quorum loss/restore logged once per transition with counts | `ha_log_quorum_transition` ha_convergence.c:412-452 (CAS edge-detect on `quorum_lost`, WARN once); 5 call sites ha.c:1085, 2076, 3969, 5056, gate backstop | MET | mirrors `ha_has_write_quorum` body (connected=repl_active_standby_count, required=n_peers/2) | -| **9.1 / 9.2** status + replication served while quorum lost | `ha_dispatch_gate` degraded-mode allowlist regression pin (admits OPCode_get/heartbeat/repl/fact opcodes; rejects request-shaped) | MET | verification-only | - -**Boundary audits re-confirmed (per Task 7 completion notes)**: `commit.c`/`commit.h` UNTOUCHED by the spec; status/API read paths UNFILTERED; non-PRIMARY `raft_cb_become_follower` behavior byte-equivalent (guarded by `prior_role==CTRL_PRIMARY && c!=NULL`). New TUs `egress.c` and `ha_convergence.c` registered in both `controller_src` and `controller_unit_test_src` (meson.build:6-8, 137-138); three new test files registered (meson.build:130-132). - -### The One Open Task — Task 8 (3-node E2E convergence test) - -**Nature**: ENVIRONMENT-DEPENDENT INTEGRATION TEST — not code work. No production code or unit-test gap remains; every in-process half of Task 8's assertions is already covered by tasks 1-6 (single-outcome demotion+replay in test_demotion_convergence; rejoin clamp/once-guard; commit-gated egress in test_egress_filter; quorum-loss/restore WARN-once in test_convergence_verification). Task 8 exercises the cross-node, cross-process behaviors that a single-process unit binary structurally cannot: -1. Kill primary → promoted standby owns all confirmed jobs. -2. Force a submit to time out at the barrier (withhold standby ACKs) → entry committed on all members OR absent on all members; requester's failure answer never retroactively changed; scheduler never dispatched the unconfirmed job (2.4 end-to-end). -3. Restart the killed ex-primary → exactly one self-restart/reconciliation cycle, converged rejoin, no phantom jobs (1.3 + 2.1 over a real re-exec). -4. Isolate primary → submits fail, joblist/status still served, quorum-loss WARN; heal → quorum-restored WARN, submits succeed (8.1/8.2/9.1/9.2 live). - -**Prerequisites**: -- A running 3-node cluster via `/xce-deploy-ha` (e.g. `localhost:9991 9992 9993`). -- **ha-single-leader-role landed** (direct promotion, CTRL_PRIMARY_COMMITTING removed) — confirmed satisfied in this worktree: `raft_cb_become_leader` ha.c:1026-1036 enters CTRL_PRIMARY directly ("ha-single-leader-role (Req 1.1): enter CTRL_PRIMARY directly"). So the dependency is already met; only the live infra is missing. -- Operator-observed assertions (log inspection for WARN transitions; joblist/status probes during isolation). - -**Effort**: LOW-MEDIUM. The scenario script (deploy → kill/isolate/heal → assert) is mechanical; the cost is standing up and observing a live cluster and authoring the convergence assertions. Estimated a few hours of hands-on cluster time, no design or code uncertainty. Shares the exact blocked-manual status as ha-wal-replication task 5.4. - -**Risk**: LOW for the spec's correctness claims (all logic is unit-verified). Residual risk is integration-surface only — re-exec inheriting an unexpected fd, scheduler/worker reconnection after the restart (explicitly out of scope here; worker-rejoin-resync roadmap backstop), or a timing artifact in the live barrier-timeout path. These are discoverable ONLY on real infra, which is precisely why Task 8 exists. - -### Options for closing Task 8 - -- **Option A — Run the live E2E now** (recommended when infra is available): execute `/xce-deploy-ha` 3-node, run the four scenarios, record converged-outcome + WARN-log assertions in completion notes. Closes 2.4/8.1/8.2/9.1/9.2 end-to-end and marks the spec fully complete. Effort LOW-MEDIUM, Risk LOW. Gated only on cluster availability + ha-single-leader-role (already landed). -- **Option B — Accept as blocked-manual and ship the spec on unit coverage** (current de-facto state): document Task 8 as deferred-to-infra, identical to ha-wal-replication 5.4. All production code is complete and regression-pinned; the umbrella's single-outcome guarantee is established in-process. Effort ZERO, Risk LOW-MEDIUM (integration-surface bugs would surface later, in a deploy/soak rather than in CI). -- **Option C — Invest in automated multi-process E2E harness** (out of current scope): build a scripted kill/isolate/heal driver around `/xce-deploy-ha` so this and ha-wal-replication 5.4 run in CI. Effort HIGH (new test infrastructure), Risk LOW. Pays off across the HA spec family but is a separate workstream, not part of this spec's task list. - -**Recommendation**: Option A if a cluster can be stood up this session; otherwise Option B with an explicit deferral note (consistent with how the sibling ha-wal-replication 5.4 is tracked). Option C is the right long-term move for the HA family but belongs in a dedicated test-infra effort. - ---- - -## Live finding (2026-06-20): cluster "won't hold a primary" — candidate manifestation for the deferred Task 8 E2E - -Surfaced during controller-status `ctrlstatus` live validation on a 3-node `/xce-deploy-ha` cluster. Logged here (not as a new spec) because the convergence-recovery **production code is already complete** (per the Options above) — so the symptom is a *runtime* instability to diagnose against this spec's domain, not missing implementation. - -### Observation -- `wait-primary` named `ctrl-2` as primary in 42 ms, but seconds later `ctrlstatus` answered by `ctrl-3` (standby) reported `Primary: unknown`, `term: 3`, then client failover **exhausted** ("cluster not ready after 21 attempts", `E_XCE_NOT_READY`). The cluster elected a primary then lost it / could not keep one serving. -- Matches the **original /kiro-discovery logs** (the motivating logs for the controller-status spec): repeated PreVote rounds, term bumps, "PreVote round timed out before a decision — treating as LOSS", "no primary served within 30000 ms" — recurring election churn, not a single clean failover. -- The controller unit suite (this session) shows the convergence machinery active and looping in-process: repeated `HA quorum lost: connected=0 required=1`, `PRIMARY demoted (term=7) — scheduling demotion convergence (truncate above committed_lsn + self-restart)`, and `HA rejoin reconcile: local tip exceeded committed boundary; truncated ... restarting once`. The demotion path's **self-restart** is exactly the action that, if re-triggered in a loop, produces primary churn. - -### Differential (root cause NOT yet confirmed — needs live diagnosis) -1. **Demotion self-restart churn (this spec's domain):** quorum-loss → demote → truncate-above-committed → self-restart → rejoin → quorum-loss → … If a transient quorum flap re-enters demotion convergence repeatedly, the cluster never stabilizes a primary. Owned here. -2. **PreVote / election thrash (ha-prevote / ha-single-leader-role):** PreVote rounds timing out as LOSS and re-arming, term bumping without a stable leader. Owned upstream, not here. -3. **Environment (fsync):** on an overlayfs sandbox, `fdatasync errno=5` blocks the durability/commit path (bug-795), so a primary may never advance committed and never stabilize. May be a **sandbox artifact** — the user's real-fs `/tmp/xce-ha-deploy` could behave differently. MUST be ruled out first by reproducing on a real fs. - -### Recommended next step -Run this spec's deferred **Task 8** (Option A) on a real-fs cluster with instrumentation capturing, per node and per term: quorum enter/exit transitions, demotion-convergence entries, PreVote round outcomes, and committed-LSN advance. That evidence tells us whether the churn is convergence self-restart (close it here), election thrash (route to ha-prevote / ha-single-leader-role), or an fsync/env artifact (no code owner). Until that diagnosis runs, do NOT assume this is a convergence-recovery code defect — the code is present and unit-green; this is an integration/runtime observation. - -### Related -- controller-status `ctrlstatus` is the diagnostic tool here (answers in every role incl. fenced/no-quorum; term/commit/apply visible; responder timestamp for log correlation) — use it to watch the churn live. -- The failover-exhaustion error rendering crashed (`xce_strerror_r` global-buffer-overflow, bug-827, fixed in libxce `utils.c`) — rebuild before re-observing so the client prints a clean `not-ready` instead of aborting. - ---- - -## Gap Analysis (re-validation) — 2026-06-24 - -**Framing**: Third re-validation pass, verified against the on-disk code in this worktree (`raft_tla`). Primary question: is **bug-1028** (post-failover availability — new primary stuck in catch-up loop, not serving) addressed in code or still open, and has the spec's status changed since the 2026-06-18/06-20 audits. Spec phase = `tasks-generated`, all three approval gates green, `ready_for_implementation=true`; roadmap.md:30 marks the spec `[x]` done. - -**Verdict**: STILL PARTIAL on the spec's own task list (Task 8 live E2E remains the lone open item), but the **bug-1028 availability concern is RESOLVED in code** — fixed 2026-06-23 in the term-safety domain, NOT in this spec's self-restart machinery. Two of the three differentials raised in the 2026-06-20 live finding have since been closed by sibling specs. The convergence-recovery production code is intact and unchanged in substance; one anchor evolved materially (rejoin self-restart retired by ha-term-safety). - -### bug-1028 disposition (the headline question) - -bug-1028 = "after a clean failover the new primary never becomes available to serve clients (wait-primary times out); standby[0] never converges." This is **FIXED (2026-06-23)** and the fix is live: - -- **Root cause** (per buglog bug-1028): `ha_stream_wal_to_standby` (config_repl.c) stamped each streamed entry's term and the AppendEntries `prev_log_term` from the raw on-disk `hdr.term`, while the standby's `logOk` (`raft_on_append_entries_request`) and the library send path source terms from the authoritative per-LSN index (`persist_entry_term`). The term mismatch made `logOk` reject the catch-up stream forever → standby stuck in catch-up → new primary never reaches commit quorum → never serves. This is a **term-safety / catch-up-correctness** defect, distinct from bug-828 (data safety) and bug-854 (client discovery). -- **Fix verified in code**: `ha_stream_wal_to_standby` now takes `const persist_t *p` (config_repl.h:394-395, config_repl.c:2032-2033); all three former `hdr.term` source sites use `p ? persist_entry_term(p, hdr.lsn) : hdr.term` (config_repl.c:2125, 2143, 2197). Both production callers pass `&c->persist`: the heartbeat-driven streamer (ha.c:2734-2739, explicit "bug-1028" comment) and the ACK-driven continuation (config_repl.c:2283, this spec's Req 5 component). bug-1067 layered on top (leader-current-term AE header @8) for the cross-term catch-up case. -- **Bearing on this spec**: bug-1028's "stuck not serving" is the live manifestation that the 2026-06-20 finding flagged as possibly owned here (differential #1: demotion self-restart churn). It was NOT a convergence self-restart defect — it was the streamer's term source. This spec's Req 5 continuation **consumed the same fix** (its config_repl.c streamer call passes `&c->persist`), so the catch-up component is correct, but ownership of the bug sits with ha-term-safety / raft-log-replication. - -### Material drift since the 2026-06-18 audit (Constraint) - -The 2026-06-18 map listed Req 2.1-2.3 rejoin reconciliation as MET via `ha_rejoin_reconcile_fn` (a delaymgr truncate+self-restart task). **ha-term-safety task 3.2 RETIRED that** (ha_convergence.c:254-280): `ha_rejoin_reconcile` is now a no-op that only retains the CtrlHelloReply hook for recording the learned boundary. Divergent-suffix reconciliation is now owned by the raft library's term-aware `on_truncate_from` (driven by `raft_on_append_entries_request`'s (term,lsn) log-matching), which truncates exactly the divergent suffix and never a committed-prefix entry — replacing the old term-blind LSN-watermark truncate+restart. Until reconciled, the existing promotion-time fence keeps the member non-promotable WITHOUT self-restarting. This **removes** the rejoin self-restart loop (differential #1 / #2 of the 2026-06-20 live finding) as a churn source. - -The **demotion** convergence half (Req 1.x) is UNCHANGED and still active: `raft_cb_become_follower` reads prior role (ha.c:1274), and on PRIMARY→follower disables hooks inline + WARN-logs + calls `ha_schedule_demotion_convergence` (ha.c:1277-1304) → `ha_demotion_convergence_fn` truncates above committed then `ha_self_restart` execv's (ha_convergence.c:144-180, 82-120). So a *demoted ex-primary* still self-restarts; a *rejoining standby* no longer does. - -### Requirement-to-Asset Map (delta only — full map in the 2026-06-18 section) - -| Req (criteria) | Status now | Note / gap tag | -|---|---|---| -| 1.1-1.7 demotion truncate+self-restart | MET (unchanged) | ha_convergence.c:82-211; scheduled ha.c:1277-1304 | -| 2.1-2.3 rejoin reconcile | MET-by-different-mechanism | **Constraint**: self-restart path retired by ha-term-safety 3.2; now library `on_truncate_from`. Acceptance intent (converge a log running ahead, never lose committed prefix) still satisfied, by a stronger term-aware primitive | -| 2.4 single outcome, never retroactive | PARTIAL | in-process MET; cross-node = Task 8 | -| 3.1-3.4 egress watermark filter | MET (unchanged) | egress.c:47-109; sched.c consumers | -| 5.1-5.3 ACK-driven continuation | MET; **also carries the bug-1028 fix** | config_repl.c:2032+ streamer now term-correct via `persist_entry_term`; single-flight `catchup_inflight[]` ha.h:204 | -| 7.1 re-inclusion at parity | MET (unchanged) | `repl_complete_catchup` | -| 8.1-8.3 quorum transition logging | MET (unchanged) | `ha_log_quorum_transition` ha_convergence.c:340+ | -| 9.1-9.2 degraded service | MET (unchanged) | dispatch-gate allowlist | -| Task 8 live 3-node E2E | OPEN (Missing — env-dependent) | only residual gap | - -### Options A/B/C (for closing the residual gap) - -- **Option A — Run Task 8 live E2E now** (recommended). Both former E2E blockers are now satisfied: ha-single-leader-role landed (direct CTRL_PRIMARY) AND bug-1028 (the very symptom Task 8 would have hit) is fixed, so a clean failover should now serve. Execute `/xce-deploy-ha` 3-node, run the four scenarios, record converged-outcome + WARN-log assertions. Closes 2.4/8.1/8.2/9.1/9.2 end-to-end. Effort LOW-MEDIUM, Risk LOW. Pairs naturally with `/xce-test-ha` soak which already exists in this worktree. -- **Option B — Ship on unit coverage, defer Task 8 to infra** (current de-facto state). All production code complete + regression-pinned; bug-1028 closed independently. Effort ZERO, Risk LOW-MEDIUM (integration-surface bugs surface in deploy/soak not CI). Consistent with sibling ha-wal-replication 5.4. -- **Option C — Automated multi-process E2E harness in CI**. Out of this spec's scope; the `/xce-test-ha` soak skill is the seed. Effort HIGH, Risk LOW; pays off across the HA family. - -### Effort / Risk - -- **Closing the spec**: Effort **S** (Task 8 is mechanical scenario scripting, no code/design uncertainty), Risk **Low** (all logic unit-green; bug-1028 — the integration-surface risk that mattered — is already fixed). -- **No new production work is indicated by this re-validation.** The one "change" since 2026-06-18 (rejoin self-restart retirement) was delivered by ha-term-safety, not owed here, and strictly de-risks the availability story. - -### Recommendations for design phase - -- **Preferred**: Option A. With bug-1028 fixed and ha-single-leader-role landed, run the deferred Task 8 against a real-fs `/xce-deploy-ha` cluster (or piggyback on `/xce-test-ha`) to convert 2.4/8.x/9.x from in-process MET to end-to-end MET, then mark the spec fully complete and the roadmap `[x]` is fully earned. -- **Update spec text (low priority docs)**: annotate Req 2.1-2.3 in requirements/design to reflect that rejoin reconciliation is now the library's term-aware `on_truncate_from` (ha-term-safety 3.2), not a self-restart — so a future reader does not re-derive the retired delaymgr path. The acceptance intent is unchanged; only the mechanism moved. -- **Research-Needed (carry forward)**: when Task 8 runs, capture per-node/per-term quorum transitions, demotion-convergence entries, and committed-LSN advance to confirm (a) no demotion self-restart churn under transient quorum flaps and (b) the rule-out of the overlayfs `fdatasync errno=5` artifact (bug-795) on real fs. These were the open differentials from the 2026-06-20 live finding; bug-1028's fix retires the catch-up-loop differential, leaving only demotion-churn and fsync-env to confirm empirically. diff --git a/.kiro/specs/ha-convergence-recovery/spec.json b/.kiro/specs/ha-convergence-recovery/spec.json deleted file mode 100644 index 58b6b6a1..00000000 --- a/.kiro/specs/ha-convergence-recovery/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "ha-convergence-recovery", - "created_at": "2026-06-11T09:23:10Z", - "updated_at": "2026-06-11T09:23:10Z", - "language": "en", - "phase": "implemented", - "umbrella": "ha-consistency-invariants", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation_status": "implemented — 19/19 tasks complete, 8-round live-cluster E2E validated (task 8). Phase reconciled 2026-07-03 (was mislabeled tasks-generated)." -} diff --git a/.kiro/specs/ha-convergence-recovery/tasks.md b/.kiro/specs/ha-convergence-recovery/tasks.md deleted file mode 100644 index 3a00fd57..00000000 --- a/.kiro/specs/ha-convergence-recovery/tasks.md +++ /dev/null @@ -1,112 +0,0 @@ -# Implementation Plan: ha-convergence-recovery - -> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. No `(P)` markers: groups 2–5 all modify `ha.c`/`ha.h` and group 1's tagging touches `persist.c` (shared with the demotion-path file set via the truncation/recovery seams), so the plan is sequential — the one nominally independent group (egress filter) is placed first instead of parallelized. -> Verify exact line numbers with grep before editing (this spec's research.md §1 has the 2026-06-11 verified anchors; files drift). -> Task 8 (E2E) additionally requires **ha-single-leader-role** to have landed and a live `/xce-deploy-ha` cluster. - -- [x] 1. Egress filter: submit_lsn tagging and commit-watermark gate on scheduler exports -- [x] 1.1 Write RED tests for tagging and the scheduler-export watermark - - Create `testsuite/test_egress_filter.c` (XCE_TESTING, AAA structure) and register it in `controller_unit_test_src` (meson.build); add `uint64_t submit_lsn` to the `job_t` wrapper (types.h:483-485 anchor) and declare `job_submit_committed` / `collect_jobs_committed` in the dataex header so tests compile - - Tests: a live-submitted job is tagged with its WAL entry LSN; a job restored via `apply_wal_entry`/`restore_job` replay is tagged with the replayed entry's LSN; a job with `submit_lsn > committed_lsn` is absent from `collect_jobs_committed` results and from the JOB_NEW incremental push list; the same job appears once `committed_lsn` advances to cover it; `submit_lsn == 0` is always exportable; plain `collect_jobs` (status/API path) still returns the uncommitted job - - Observable completion: build succeeds with the new field/header declarations; all new test cases FAIL (red) against unimplemented predicate/tagging - - _Requirements: 3.1, 3.2, 3.3, 3.4_ - -- [x] 1.2 Implement tagging and the watermark gate to green - - Tagging: `persist_job_hook_fun` (persist.c:1107-1125 anchor) sets `submit_lsn = written_lsn` on each submitted job after a successful append; `apply_wal_entry` (persist.c:729-731 anchor) passes `hdr->lsn` through to `restore_job` (job.c:71), which tags the restored job - - Predicate + wrapper in dataex.c: `job_submit_committed` = `submit_lsn == 0 || submit_lsn <= committed_lsn` (one atomic load, lock-free); `collect_jobs_committed` wraps `collect_jobs` with the predicate - - Scheduler-bound consumers only: `xcesched_push_jobs` (sched.c:183 anchor) collects via the committed-only wrapper; `xcesched_push_joblist` (JOB_NEW push fed by `sched_job_new_fun`, sched.c:236-248 anchor) filters its list with the predicate; apiserver.c:83 and opget.c:173 remain on unfiltered `collect_jobs` - - Observable completion: all task 1.1 tests PASS; grep confirms no status/API read path calls the committed-only wrapper - - _Requirements: 3.1, 3.2, 3.3, 3.4_ - -- [x] 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path -- [x] 2.1 Write RED tests for the self-restart primitive and demotion sequence - - Create `testsuite/test_demotion_convergence.c` and register it in meson; declare `controller_saved_argv`, `ha_self_restart`, and the `XCE_TESTING` seams (`ha_restart_hook`, `ha_fatal_hook`) in ha.h so tests compile - - Tests: become-follower invoked while role is CTRL_PRIMARY → persistence hooks disabled (verify via `hooks_registered` reset and a hook-counter probe) BEFORE further events are persisted, `persist_truncate_wal_above` called with the `committed_lsn` cutoff, WARN with both LSNs emitted, `ha_restart_hook` invoked exactly once; `ha_self_restart` with a failing restart hook path → `ha_fatal_hook` invoked (FATAL contract); become-follower from STANDBY/ELECTING/INIT → no truncation, no restart (regression); integration: append a committed prefix + uncommitted suffix, run the demotion seam, then replay the surviving WAL into a fresh context → committed-only state (provisional entries absent) - - Observable completion: new cases FAIL (red) — current `raft_cb_become_follower` (ha.c:627-690 anchor) neither truncates nor disables hooks - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.4_ - -- [x] 2.2 Implement the self-restart primitive and demotion convergence to green - - main.c: save the NULL-terminated argv before `argp_parse` (main.c:112 anchor); expose `controller_saved_argv()` - - ha.c: `ha_self_restart(c)` — flush logs, `execv(saved_argv[0], saved_argv)`; on exec return → ERROR + `abort()`; under XCE_TESTING invoke `ha_restart_hook` (and `ha_fatal_hook` instead of abort); audit/ensure listener and peer fds are CLOEXEC so the re-exec'd image inherits no live sockets - - `raft_cb_become_follower`: read prior role before the role store; was-PRIMARY branch disables the three persistence hooks inline (reuse the `ha_demote_to_standby` disable sequence, ha.c:1712-1726 anchor) then schedules a one-shot delaymgr convergence task: truncate above `committed_lsn` → WARN range → `ha_self_restart` (file I/O and exec never on the callback thread) - - Observable completion: all task 2.1 tests PASS; non-PRIMARY demotion behavior byte-equivalent to before - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7_ - -- [x] 3. Rejoin reconciliation at the CtrlHelloReply intake -- [x] 3.1 Write RED tests for clamped rejoin truncation with once-guard - - Tests in `test_demotion_convergence.c`: `ha_rejoin_reconcile` with local tip > primary boundary → truncation at `max(primary_committed, local committed_lsn)` (clamp verified with a local committed_lsn ABOVE the reported boundary) and `ha_restart_hook` invoked once; tip <= boundary → no truncation, no restart, returns 0; second reconcile call in the same process lifetime → no second restart (defensive once-guard); intake integration: delivering a CtrlHelloReply through the cluster.c handler path stores `last_known_committed_lsn` (existing) AND triggers reconciliation - - Observable completion: new cases FAIL (red) — no reconciliation exists at the intake (cluster.c:283-298 anchor) - - _Requirements: 2.1, 2.2, 2.3_ - -- [x] 3.2 Implement rejoin reconciliation to green - - ha.c: `ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn)` — evaluate `local_tip = persist.current_lsn - 1` vs boundary; clamped cutoff; WARN range; `_Atomic int rejoin_reconciled` once-guard; truncation + restart deferred to a delaymgr task (intake may run on iomgr); reuses `ha_self_restart` from task 2.2 - - cluster.c: CtrlHelloReply intake calls `ha_rejoin_reconcile` after the existing `last_known_committed_lsn` store; RDHUP truncation path (ha.c:3416-3420 anchor) left unchanged - - Observable completion: all task 3.1 tests PASS; a node whose tip matches the boundary joins with zero side effects - - _Depends: 2.2_ - - _Requirements: 2.1, 2.2, 2.3_ - -- [x] 4. ACK-driven catch-up continuation and re-inclusion trigger -- [x] 4.1 Write RED tests for continuation, single-flight, and parity re-inclusion - - Create `testsuite/test_convergence_verification.c` and register it in meson - - Tests (use the existing `ha_repl_stream_wal_hook` / `xce_wal_stream_request_hook` seams to count streaming invocations): a walAck with `acked_lsn < max(committed_lsn, last_written)` schedules exactly one continuation that streams `[acked_lsn+1, target]` WITHOUT any intervening heartbeat/stateReport dispatch; a second walAck while the continuation is in flight schedules nothing (single-flight); after the continuation completes, the next lagging walAck schedules again; a walAck with `acked_lsn >= target` calls `repl_complete_catchup` and the standby's `in_catch_up` clears; the heartbeat decision path (`ha_dispatch_repl_action_for_standby`) still produces HA_REPL_SEND_SNAPSHOT / HA_REPL_STREAM_WAL decisions unchanged (fallback + snapshot ownership pin) - - Observable completion: new cases FAIL (red) — current walAck branch (ha.c:2737-2754 anchor) only records the ACK; `repl_complete_catchup` (repl.c:521) has no production caller - - _Requirements: 5.1, 5.2, 5.3, 7.1_ - -- [x] 4.2 Implement the continuation trigger and parity re-inclusion to green - - ha.c walAck branch: after `repl_on_ack`, derive target as at ha.c:2727-2732; behind target → schedule the continuation via delaymgr unless the standby's `catchup_inflight` flag is set; at parity → `repl_complete_catchup` (ha-commit-engine wiring propagates the membership re-evaluation) - - config_repl.c: continuation task body — re-validate the endpoint against the registry, re-derive target, call `ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target)` (internals untouched), clear `catchup_inflight` on all exit paths - - ha.h: per-standby `_Atomic int catchup_inflight` storage with slot identity stable across schedule/run - - Observable completion: all task 4.1 tests PASS; heartbeat decision loop diff-clean (fallback + snapshot initiator preserved) - - _Requirements: 5.1, 5.2, 5.3, 7.1_ - -- [x] 5. Quorum transition observability -- [x] 5.1 Write RED tests for once-per-transition quorum logging - - Tests in `test_convergence_verification.c` (log-capture or counter-hook pattern): dropping standbys below majority logs the loss WARN exactly once with connected/required counts; repeated dispatch-gate rejections while quorum stays lost add no further transition logs; restoring a standby logs the restore WARN once; a second loss/restore cycle logs again (edge detector, not a latch) - - Observable completion: new cases FAIL (red) — no transition logging exists (ha.c:1507-1525 anchor) - - _Requirements: 8.1, 8.2, 8.3_ - -- [x] 5.2 Implement quorum transition logging to green - - ha.h: `_Atomic int quorum_lost` on `ha_state_t`; ha.c: `ha_log_quorum_transition(ha)` — CAS edge detection around `ha_has_write_quorum`, WARN with connected/required counts on each edge only - - Call sites: promotion standby registration (ha.c:531 anchor — NOTE: this anchor is inside `raft_cb_become_leader` (ha.c:445-617), promotion code owned and rewritten wholesale by ha-single-leader-role; re-anchor after that spec lands, or coordinate with it if this task is implemented first), RDHUP removal (ha.c:3160-3166 anchor), `on_standby_connected` (ha.c:3764+ anchor), and the `ha_dispatch_gate` quorum-rejection branch as backstop - - Observable completion: all task 5.1 tests PASS; no log line is emitted per rejected operation - - _Requirements: 8.1, 8.2, 8.3_ - -- [x] 6. Verification of existing behaviors (regression pins — no production code) -- [x] 6.1 Write regression tests pinning full fan-out, snapshot fallback, and degraded-mode service - - Tests in `test_convergence_verification.c`: `repl_broadcast` enqueues to every registered active standby and skips only empty slots and `in_catch_up` (umbrella 6.1 pin); `ha_evaluate_repl_action` returns HA_REPL_SEND_SNAPSHOT when `standby_lsn < oldest_wal_lsn` and the snapshot path takes a fresh checkpoint before sending (umbrella 6.3 pin, decision boundary only); `ha_dispatch_gate` without write quorum admits `OPCode_get`, heartbeat, the replication opcode allowlist, and the fact-shaped opcodes (`OPCode_notify`/`update`/`setpreason`/`clrpreason` — admitted without quorum per ha-commit-barrier-rollout's fact allowlist) while rejecting request-shaped opcodes only (`OPCode_jobsub`/`jobcontrol`/`queuecontrol`/`hostcontrol`/`jobdispatch`/`join`) with `-XCE_ERR_NOT_READY` (umbrella 7.3 pin); `repl_complete_catchup` clears `in_catch_up` and the standby re-enters broadcast iteration (umbrella 6.4 mechanism pin, complements task 4's trigger test) - - Observable completion: all pins PASS against current code without production changes (these encode behavior that already exists; any failure is a real regression or a wrong assumption to escalate) - - _Requirements: 4.1, 6.1, 7.1, 9.1, 9.2_ - -- [x] 7. Full controller regression under ASAN - - Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix any regression from the become-follower changes, the collect-path wrapper, or the walAck branch additions - - Audit: grep confirms status/API read paths are unfiltered, `commit.c` is untouched by this spec, and `raft_cb_become_follower`'s non-PRIMARY behavior is unchanged - - Observable completion: full testsuite green under ASAN; audit results recorded in completion notes - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 4.1, 5.1, 5.2, 5.3, 6.1, 7.1, 8.1, 8.2, 8.3, 9.1, 9.2_ - -- [x] 8. 3-node E2E convergence test (extends ha-wal-replication task 5.4) - - _Blocked: manual live-cluster integration test — requires a running `/xce-deploy-ha` 3-node cluster with kill-primary / isolate / heal scenarios and operator-observed convergence assertions; not automatable in the unit suite (same blocked-manual status as ha-wal-replication task 5.4). Unit-testable in-process halves of these assertions are already covered: single-outcome demotion+replay (test_demotion_convergence), rejoin reconciliation clamp/once-guard, commit-gated egress (test_egress_filter), quorum-loss/restore WARN once (test_convergence_verification). Run on real infra to close 2.4/8.1/8.2/9.1/9.2 end-to-end._ - - **DEPENDS on ha-single-leader-role having landed** (direct promotion, COMMITTING removed — the demotion/rejoin path asserted here is the post-removal one); requires a live `/xce-deploy-ha` cluster — manual integration test, not automatable in the unit suite (same constraint as ha-wal-replication 5.4) - - Deploy 3 nodes; submit confirmed jobs → kill primary → promoted standby owns all confirmed jobs; force a submit to time out at the barrier (withhold standby ACKs) → verify single-outcome convergence: the entry is committed on all members or absent on all members, the requester's failure answer is never retroactively changed, and the scheduler never dispatched the unconfirmed job - - Restart the killed ex-primary → verify exactly one self-restart/reconciliation cycle and a converged rejoin (WAL tip at the cluster's committed boundary; no phantom jobs in joblist) - - Isolate the primary from both standbys → submits fail, `joblist`/status still served, quorum-loss WARN in the log; heal → quorum-restored WARN, submits succeed without intervention - - Observable completion: all E2E assertions hold on a live `/xce-deploy-ha` cluster; results recorded in completion notes - - _Depends: 7_ - - _Requirements: 2.4, 9.1, 9.2, 8.1, 8.2, 3.1_ - - **DONE 2026-06-24 (live `/xce-test-ha`, 8 rounds):** `PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)` (exit 0). Live-covered E2E bullets: kill-primary → promoted standby owns **all** confirmed jobs (zero loss across 8 failovers, jobs 3→30, every round `expect-job` ok — Req 9.1/9.2/3.1); restart killed ex-primary → converged rejoin, 3-of-3 reconverged each round, **no phantom jobs** in `joblist` (exactly N present, never N+extra — Req 2.4 single-outcome on the survive-and-rejoin side); plus deterministic leadership coverage (ctrl-3 forced to lead via the RequestVote-freshness fence maneuver). Build note: required a controller rebuild first — the deployed binaries predated the bug-1077 libxce fix; the soak then also served as the live regression for bug-1077 (~40 failover client round-trips, no SEGV). NOT exercised by this live soak (remain covered by the in-process unit halves, tasks 2/4/5): the withhold-ACK barrier-timeout single-outcome (test_demotion_convergence) and the isolate-from-both-standbys quorum-loss/restore WARN (test_convergence_verification) — both need network partition / ACK injection the soak driver does not drive. Optional future live validation: a partition/heal scenario for Req 8.1/8.2 end-to-end. - -## Implementation Notes - -- **Task 1.2 — predicate placed in new `egress.c`, not `dataex.c`** (design D3 "refinement, not contradiction"): `dataex.c` defines `notify()` (dataex.c:384) and `collect_confvars()`, both already stubbed in the unit-test binary (test_commit_barrier_reply.c, test_worker_join.c). Linking `dataex.c` into `controller_unit_test_src` is a duplicate-symbol error, so `job_submit_committed`/`collect_jobs_committed` live in a dedicated TU `src/controller/egress.c`, registered in BOTH `controller_src` and `controller_unit_test_src`. Header decls remain in `cluster.h` (the controller's "dataex header" — there is no `dataex.h`). Any future scheduler-bound export path adopts the predicate from there. -- **Test-link rule**: a production symbol that the unit-test suite already stubs cannot have its owning `.c` added to `controller_unit_test_src`. Factor newly-testable logic into its own TU instead. -- **`restore_job` gained an `entry_lsn` parameter** (replay tagging). Callers updated: cluster.h decl, job.c def, persist.c extern + apply_wal_entry caller, test_repl_apply.c stub. Any new caller must pass the WAL entry header LSN (`hdr->lsn`), or `0` if unknown (0 = exportable). -- **Build note**: the full meson controller suite SIGTERMs at the 300s wall-clock cap on the slow commit-barrier timeout suites (pre-existing). For per-task GREEN confirmation, run `controller_unit_test -t <suite>` directly with `ASAN_OPTIONS=detect_leaks=0` (harness teardown leaks are a pre-existing accepted pattern; the meson gate sets the same option). -- **Task 2.1 RED scaffold + anchor drift (post ha-single-leader-role)**: throwaway stubs live in new `src/controller/ha_convergence.c` (`controller_saved_argv`→NULL, `ha_self_restart`→return 0, seam pointers `ha_restart_hook`/`ha_fatal_hook`=NULL) — **task 2.2 moves the real `controller_saved_argv` to main.c and `ha_self_restart` + the demotion branch to ha.c** per design file-layout. A test-only seam `raft_cb_become_follower_with_cluster_for_test` (preserves `raft_c=c`) was added to ha.c. Verified anchors: `raft_cb_become_follower` ha.c:622; `ha_demote_to_standby` ha.c:1711 (hook-disable sequence to reuse); `persist_truncate_wal_above` persist.c:1545; role enum CTRL_PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 (ha.h:59-64); `committed_lsn`/`current_lsn` on `persist_t` in src/include/persist.h; MAX_STANDBYS=3. -- **Follow-up for task 2.2**: add a `CTRL_INIT` non-PRIMARY regression case to `test_demotion_convergence.c` (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8). [DONE in 2.2: INIT case added, comment corrected.] -- **Task 2.2 — latent persist bug fixed**: `persist_truncate_wal_above` (persist.c) had a no-op early-return for `cutoff_seg_idx < 0` (cutoff strictly below the oldest retained segment — the first real caller is the demotion path, where a demoted ex-primary's committed boundary can sit below its entire retained log). Now deletes all segments, sets `current_lsn = cutoff+1`, `committed_lsn = cutoff` — correct truncate-above semantics; reviewed safe for the checkpoint path (P0 `load_snapshot` restores only LSN counters, no entity state; entity-bearing snapshots out of scope) and the promotion-truncation caller (no data loss; the discriminant cannot conflate "below oldest" with "above newest"). **Follow-up (non-blocking, for task 6.1 or 7): add a targeted `persist_truncate_wal_above` test exercising the `cutoff_seg_idx < 0` branch** — currently established correct by analysis only, no direct assertion. -- **CLOEXEC**: controller sockets are already `FD_CLOEXEC` at creation (`socket_create` netio.c:219, `socket_accept` netio.c:308) — the re-exec'd image inherits no live sockets; no netio.c change was needed. -- **Disk space**: heavy nix builds can fill the root overlay with stale `/tmp/nix-shell.*` AND `/tmp/xce_*` test dirs (reviewers reclaimed ~16GB nix-shell + ~570GB stale xce test dirs). If a build/test fails on "No space left on device", clear stale `/tmp/nix-shell.*` and `/tmp/xce_*` and retry — not a code failure. -- **Task 3.2 — delaymgr one-shot helper + once-guard test strength**: `ha_rejoin_reconcile` lives in `ha_convergence.c` (alongside `ha_self_restart`/demotion task, unit-linkable) and defers truncate→WARN→restart to a delaymgr one-shot, reusing the 2.2 pattern. The test helper `dc_run_pending_task` was fixed to DEQUEUE (`ordq_get`+free) mirroring the real delaymgr worker (delay.c:65-90) — the prior peek-without-dequeue caused a test-only heap-UAF (re-running an already-freed one-shot); production never re-runs a one-shot, so the body-frees-own-ctx pattern is safe (bug-343). **Follow-up (non-blocking, for task 6.1 or 7): add a once-guard test that calls `ha_rejoin_reconcile` TWICE before draining the delaymgr (tip unchanged) and asserts exactly one task scheduled** — the current second-call test runs truncation synchronously between calls, so call 2 exits at the 2.3 early-return and exercises only the structural guard, not the defensive CAS (CAS is present + correct, needed for a duplicate CtrlHelloReply arriving before the deferred truncation runs). -- **Task 5.1 done (RED) + 5.2 notes**: counter-hook seam `ha_quorum_transition_log_hook(int lost,int connected,int required)` is the observable (no log-capture harness exists); `_Atomic int quorum_lost` added to ha_state_t; no-op stub `ha_log_quorum_transition` in ha_convergence.c. **5.2 must**: (a) report `connected = repl_active_standby_count(repl)` and `required = n_peers/2` to match the `ha_has_write_quorum` BODY (ha.c:~1488-1498) — its header comment says "ceil(n_peers/2)+1" but that's a pre-existing stale comment; do NOT change the quorum math, just mirror the body so the 5.1 tests pass; (b) re-grep the promotion standby-registration call site inside `raft_cb_become_leader` (rewritten by ha-single-leader-role) — RDHUP removal (~ha.c:3160), `on_standby_connected` (~ha.c:3764), and the gate-rejection backstop (~ha.c:1511) are verified present; (c) invoke the hook once per WARN emitted (CAS edge-detect on `quorum_lost`). -- **Task 5.2 done (GREEN) — quorum-decision duplication note**: `ha_log_quorum_transition` (ha_convergence.c) re-derives `connected = repl_active_standby_count(repl)` / `required = n_peers/2` / single-node short-circuit as a FAITHFUL same-input mirror of the file-static `ha_has_write_quorum` (ha.c) — it cannot report a state inconsistent with the gate. Non-blocking maintainability risk: if `ha_has_write_quorum`'s decision ever changes, this mirror must change too. **Follow-up (optional cleanup): expose `ha_has_write_quorum` via a non-static accessor and call it from `ha_log_quorum_transition`** instead of duplicating. Call sites wired (post ha-single-leader-role rewrite): promotion ha.c:567, RDHUP removal ha.c:3180, on_standby_connected ha.c:3837, gate backstop ha.c:1521. -- **Task 7 done (GO) — full ASAN regression + boundary audit**: full controller suite green under ASAN, established suite-by-suite (50/50 suites, 524 tests, 0 failures, 0 ASAN memory errors) because the aggregate meson `controller test` SIGTERMs at the pre-existing 300s cap (slow commit-barrier timeout suites serialized). Audits PASS: (1) status/API read paths UNFILTERED — `collect_jobs_committed` sole caller is `sched.c:218` (full push) + the `job_submit_committed` predicate filter at the JOB_NEW push; `apiserver.c:83` + `opget.c:173` stay on plain `collect_jobs`; (2) `commit.c`/`commit.h` UNTOUCHED across the whole spec (`git diff --stat e558a62..HEAD` empty for both); (3) `raft_cb_become_follower` demotion additions guarded by `prior_role == CTRL_PRIMARY && c != NULL` (ha.c:687, :711) — STANDBY/ELECTING/INIT byte-equivalent to pre-spec. Production files touched match the design File Structure Plan (with dataex.c→egress.c per D3). No production change needed in task 7. -- **Task 8 (live 3-node E2E) — BLOCKED (manual)**: requires a running `/xce-deploy-ha` cluster + operator-observed kill/isolate/heal convergence; not automatable in the unit suite (same status as ha-wal-replication 5.4). All unit-testable in-process halves are covered by tasks 1-6. To close: run on real infra and record results. diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/brief.md b/.kiro/specs/ha-elect-ctx-lifecycle/brief.md deleted file mode 100644 index 1f051420..00000000 --- a/.kiro/specs/ha-elect-ctx-lifecycle/brief.md +++ /dev/null @@ -1,62 +0,0 @@ -# Discovery Brief: ha-elect-ctx-lifecycle - -_Filed 2026-06-16 as a tracked follow-up from the ha-prevote feature. Not yet specced — run `/kiro-spec-requirements ha-elect-ctx-lifecycle` to begin._ - -## Problem - -The HA election **carrier** (`ha_elect_ctx_t`, allocated as `mc` in both -`ha_start_election_async` and `ha_start_prevote_async`, stored in -`ha->elect_ctx`) is **never freed**. Each failover/election/PreVote round -`calloc`s a fresh carrier and the prior one is deliberately leaked: - -```c -/* ha.c ~3746 */ -/* Do NOT free a prior ctx (live vote endpoints may hold a bare - * pointer to it — same UAF caution as the rdhup path). */ -atomic_store_explicit(&ha->elect_ctx, mc, memory_order_release); -``` - -Root cause: per-peer vote endpoints hold a bare pointer to the carrier -(`vctx->ctx = ectx`). Those `vctx` are freed asynchronously by `endpoint_fini` -(FREEUP) on the iomgr thread when the vote endpoint closes — at a time the -round-starting thread cannot cheaply observe. There is **no actual -`ha_election_cleanup()`** (only an aspirational comment at ha.c:3504). So -freeing the carrier when a new round starts would risk a **use-after-free** if -an endpoint callback still references it. - -## Why it's not urgent - -- **Pre-existing**: the real-election path (`ha_start_election_async`) has the - identical leak; ha-prevote reused the same carrier pattern and did **not** - worsen the per-failover rate (a suppressed disruptive rejoin now leaks *one* - PreVote carrier where it previously caused an election storm leaking several). -- **Bounded**: one small struct (`sizeof(ha_elect_ctx_t) + n_peers*votes[]`, - hundreds of bytes for a 3-node cluster) per failover/PreVote event. Failover - is a rare event, not steady-state — no unbounded growth under normal operation. -- Surfaced as ~174KB / ~1548 allocs under ASAN during the full controller test - suite (many simulated rounds); `detect_leaks=0` is used in the test command. - -## Scope (when specced) - -- Reference-count the carrier (increment per `vctx` that captures it, decrement - on each `vctx` free, free at zero) **or** a teardown-ordered deferred free - tied to confirmed vote-endpoint release. -- Apply to **both** the real-election and PreVote carrier paths — they share the - `ha->elect_ctx` field and the same UAF hazard; fixing one without the other - leaves the hazard. -- Must not regress failover correctness (bug-339/bug-340) or introduce a UAF on - the async endpoint-teardown path. Validate under ASAN with leak detection - **enabled** (the success criterion is zero carrier leaks across many rounds). - -## Out of scope - -- The PreVote protocol itself (shipped and validated in `ha-prevote`). -- Any change to election/voting *semantics* — this is purely memory lifetime. - -## Pointers - -- `src/controller/ha.c`: `ha_start_election_async`, `ha_start_prevote_async` - (carrier alloc + the "do NOT free prior ctx" comments ~3503, ~3746), - `raft_cb_send_vote_request` / `ha_send_pre_vote_request` (set `vctx->ctx`), - endpoint teardown via `endpoint_fini` (frees `vctx`). -- `src/controller/ha.h`: `ha_elect_ctx_t`, `vote_ep_ctx_t`. diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/design.md b/.kiro/specs/ha-elect-ctx-lifecycle/design.md deleted file mode 100644 index 61259e58..00000000 --- a/.kiro/specs/ha-elect-ctx-lifecycle/design.md +++ /dev/null @@ -1,183 +0,0 @@ -# Design Document: ha-elect-ctx-lifecycle - -## Overview - -**Purpose**: Give the HA election carrier (`ha_elect_ctx_t`) a real, safe lifetime instead of the current deliberate per-round leak, without touching election/PreVote/failover semantics. - -**Users**: Controller maintainers (leak is invisible to operators today — ASAN's `detect_leaks=0` default hides it — but it accumulates one carrier per failover/election/PreVote round for the life of the process). - -**Impact**: Adds a reference count to `ha_elect_ctx_t` and one new generic completion hook to `endpoint_t` (`on_fini`, mirroring the existing `on_connected` fire-once hook). No change to when elections start, how votes are tallied, or how endpoints are torn down — this design only observes the moment a vote endpoint's teardown actually completes and releases the carrier reference that endpoint held. - -### Goals -- Carrier is freed exactly once, exactly when no vote endpoint can still reference it (Req 1). -- Freeing can never race a live endpoint's read of the carrier (Req 2). -- One mechanism, applied identically to the real-election path (`raft_cb_send_vote_request` / `ha_start_election_async`) and the PreVote path (`raft_cb_send_pre_vote_request` / `ha_start_prevote_async`) (Req 3). -- Zero observable change to vote tallying, freshness gates, or failover timing (Req 4). - -### Non-Goals -- Changing `endpoint_fini`, `vote_ep_teardown_async`, or any teardown *ordering/timing* — this design adds an observation point, not a new teardown path. -- Changing the PreVote protocol, `ha_elect_ctx_t`'s field contents, or vote-tally logic. -- A general-purpose object-lifetime framework. The mechanism here is specific to this one carrier; it is not offered as reusable infrastructure for other structs. - -## Boundary Commitments - -### This Spec Owns -- The reference count field on `ha_elect_ctx_t` and its increment/decrement discipline. -- The two capture sites that must increment (`raft_cb_send_vote_request`, `raft_cb_send_pre_vote_request`). -- The one release path that must decrement (vote-endpoint fini completion) and the one release path that must decrement on round supersession (`ha_start_election_async`, `ha_start_prevote_async` overwriting `ha->elect_ctx`). -- The new `endpoint_t.on_fini` hook's existence and its firing point inside `endpoint_fini` (one field, one conditional call — not a new subsystem). - -### Out of Boundary -- The PreVote protocol itself (`ha-prevote`, shipped). -- Any change to election, vote-granting, or failover semantics. -- `endpoint_fini`'s existing teardown mechanics (`FREEUP(ep->data)`, socket/SSL/packet cleanup) — untouched, just preceded by one optional hook call. -- The async endpoint-teardown *ordering* itself (`vote_ep_teardown_async`, `endpoint_mark_tearing_down`, the bug-1197 deferred-vtask path) — this design fires after that ordering completes, whatever path got there. -- The bug-1298 endpoint `refcnt` mechanism — a different lifetime (the `endpoint_t` itself) from the one this spec adds (the `ha_elect_ctx_t` carrier a vote endpoint's `ep->data` points into). The two refcounts are independent and must not be conflated. - -### Allowed Dependencies -- `endpoint_t` (`src/include/netio.h`, `src/lib/netio.c`) — may add one field + one call site, may not change existing field semantics. -- `ha_elect_ctx_t`, `vote_ep_ctx_t` (`src/controller/ha.h`, `src/controller/ha.c`) — internal to `ha.c`/`ha.h`, this spec's primary work area. -- Existing atomics discipline on `ha->elect_ctx` (`atomic_store_explicit` / `atomic_load_explicit`, `memory_order_release` / `memory_order_acquire`) — extended (one site moves to `atomic_exchange_explicit` to retrieve the superseded pointer), not replaced. - -### Revalidation Triggers -- Any new site that assigns `ep->data = <vote_ep_ctx_t*>` and wires up a vote endpoint outside the two existing capture sites (would need the same increment + hook wiring). -- Any change to `endpoint_fini`'s call sequence that could reorder `FREEUP(ep->data)` relative to the new `on_fini` hook. -- Any future feature that gives `endpoint_t.on_fini` a second consumer (must confirm the two consumers cannot conflict on the same endpoint). -- Any change to the single-node / no-local-config fast-exit paths in `ha_start_election_async` that currently `free(ctx)` directly without ever creating a vote endpoint (they must keep doing so — see Architecture). - -## Architecture - -### Existing Architecture Analysis - -Two sibling capture functions allocate a fresh `ha_elect_ctx_t` each round and publish it via `atomic_store_explicit(&ha->elect_ctx, ..., memory_order_release)`: - -- `ha_start_election_async` (real election) has **two fast-exit paths that already free the ctx correctly**: no-local-config bail-out and the single-node case (`n_peers == 1`, `raft_start_election` resolves synchronously via `on_become_leader`). Neither path ever creates a vote endpoint, so the ctx is never shared — a direct `free(ctx)` is correct today and stays correct; this design does not touch them. -- The **multi-node path** is where the leak lives: `raft_start_election` fires `on_send_vote_request` per peer, each call routing through `raft_cb_send_vote_request`, which `calloc`s a `vote_ep_ctx_t` (`vctx`), sets `vctx->ctx = ectx`, attaches it as `ep->data`, and hands the endpoint to iomgr. `ha_start_prevote_async` → `raft_cb_send_pre_vote_request` is the structurally identical PreVote sibling. -- `vctx` is freed exactly once, generically, inside `endpoint_fini` (`src/lib/netio.c:1171`, `FREEUP(ep->data)`) — reached via `endpoint_unref` (bug-1298 refcount hitting 0) on every teardown path: connect failure, `iomgr_monitor` failure, the flag-OFF loser-path loop (`ha.c:~2142`), the flag-ON deferred-vtask path (`vote_ep_teardown_async` → `vote_ep_teardown_vtask_entry`), and RDHUP/err-handler paths. This is deliberately the **single** place `vctx` memory is reclaimed; the design must not duplicate that free, only observe it. (Line numbers as of current source; `g_drive_async_peerctrl_dispatch_enabled` now defaults ON per raft-drive-async, so the flag-ON deferred-vtask path is the live default today — both paths still funnel through `endpoint_fini` exactly once, so the mechanism is unaffected.) -- The carrier itself is currently **never** freed for the multi-node case: the next round's `atomic_store_explicit(&ha->elect_ctx, mc, ...)` (both the election and PreVote sites) silently discards the previous pointer, with a comment explicitly documenting the leak as deliberate (UAF avoidance). - -### Mechanism - -Reference-count the carrier. One bias reference represents "this is (or was, pending drain) the published `ha->elect_ctx` value"; one reference per live `vote_ep_ctx_t` that captured it. - -``` -ha_elect_ctx_t.refcount (new, _Atomic int, starts at 1) - -+1 at capture: raft_cb_send_vote_request / raft_cb_send_pre_vote_request, - at the point ep->data = vctx is committed (i.e. only for vctx instances - that actually reach a live endpoint — the pre-endpoint_get() alloc-fail - bailouts never increment, so they need no matching decrement) - --1 at vote-endpoint completion: new endpoint_t.on_fini(ep) hook, fired by - endpoint_fini() immediately before FREEUP(ep->data), for every teardown - path (connect-fail, iomgr_monitor-fail, loser-path unref, deferred - vtask teardown, RDHUP/err-handler) — all of them funnel through - endpoint_fini exactly once (bug-1298 invariant), so this is the single - correct observation point regardless of which path got there - --1 at supersession: ha_start_election_async / ha_start_prevote_async, at - the atomic_store_explicit(&ha->elect_ctx, new_ctx, ...) site — changed - to atomic_exchange_explicit to retrieve the outgoing pointer, then - release it (decrement; free at 0) instead of discarding it - -free when refcount reaches 0 (pthread_mutex_destroy + sem_destroy + free), -regardless of which of the two release kinds above is the last one to fire -``` - -Why a bias reference is required (not pure per-endpoint counting): between `atomic_store_explicit(&ha->elect_ctx, ctx, ...)` and the first `raft_cb_send_vote_request` callback actually running, the carrier has zero vote-endpoint references yet but is already live and readable (e.g. `ha.c:1638-1639`, `2008-2009` inside `raft_cb_become_leader` read `ha->elect_ctx` without assuming any endpoint exists — e.g. single-vote-outstanding or mid-fan-out windows). A pure "count of attached endpoints" would allow a spurious free at refcount 0 before the round is superseded. The bias reference removes that window: the carrier is only reclaimed once it is *both* superseded *and* drained of endpoint references, in whichever order those two events occur. - -Why a generic `endpoint_t.on_fini` hook (not per-call-site decrements at every `endpoint_unref`/teardown call): a `vctx` is only *actually freed* inside `endpoint_fini`, but `endpoint_unref` (which may or may not trigger `endpoint_fini` synchronously, depending on whether a pool vtask still holds a ref) is called from roughly a dozen sites in `ha.c` across connect-failure, iomgr-failure, the loser-path loop, and the bug-1197 deferred-teardown-vtask path. Decrementing at each call site would require knowing, at every one of those dozen sites, whether this call is the one that actually drops the count to zero — which duplicates `endpoint_fini`'s own refcount logic and is exactly the kind of coupling Req 2.3/2.4 (uniform coverage of every free path, "including error and abort paths") is trying to make robust against drift. A single hook fired at the one true free point is correct by construction for every current and future teardown path, mirroring the existing `on_connected` fire-once hook already on `endpoint_t`. - -### Architecture Pattern & Boundary Map - -```mermaid -sequenceDiagram - participant ES as ElectionStart (election or PreVote) - participant CB as SendVoteRequest callback - participant EP as VoteEndpoint - participant FI as endpoint_fini - - ES->>ES: allocate carrier, refcount = 1 - ES->>ES: exchange published carrier for new one, get old carrier - ES->>ES: release old carrier (decrement, free if zero) - ES->>CB: start election, fan out per peer - CB->>CB: record carrier on vctx, increment refcount - CB->>EP: attach vctx as endpoint data, set on_fini hook - Note over EP: connect, vote RPC, reply, or error -<br/>any teardown path funnels here - EP->>FI: endpoint_fini runs once refcount reaches zero - FI->>FI: call on_fini hook if set (decrement carrier refcount, free if zero) - FI->>FI: free endpoint data (existing behavior, unchanged) -``` - -**Selected pattern**: reference counting with a bias reference, observed via a generic completion hook — same shape as the existing bug-1298 `endpoint_t.refcnt` mechanism, applied one level up (the carrier a vote endpoint points into, not the endpoint itself). -**Domain boundaries**: carrier lifetime lives entirely in `ha.c`/`ha.h` (internal type, per research.md); the only cross-boundary touch is the one generic hook field added to `endpoint_t`. -**Existing patterns preserved**: `endpoint_fini` remains the single `vctx` free site; `atomic_store_explicit(..., memory_order_release)` / `atomic_load_explicit(..., memory_order_acquire)` discipline on `ha->elect_ctx` is preserved (the supersession site becomes `atomic_exchange_explicit`, same memory order semantics, just also yielding the old value instead of discarding it). -**New components rationale**: `on_fini` hook — needed because no existing hook fires at endpoint-teardown completion; `refcount` field — needed because the leak is a multi-referrer lifetime problem, not a single-owner one. -**Steering compliance**: no thread-role change — the hook fires on whatever thread already calls `endpoint_fini` today (iomgr thread or pool worker, per bug-1197's deferred-teardown-vtask path); the decrement/free is O(1) non-blocking (atomic decrement + conditional `free()`), so it introduces no new blocking work on any thread class. - -## File Structure Plan - -### Modified Files -- `src/include/netio.h` — add `void (*on_fini)(struct endpoint_t *ep);` field to `endpoint_t`, adjacent to the existing `on_connected` field, with a doc comment cross-referencing it as the completion-side counterpart. -- `src/lib/netio.c` — in `endpoint_fini`, add `if (ep->on_fini) { ep->on_fini(ep); }` immediately before the existing `FREEUP(ep->data)` line (as-built: the hook is invoked unconditionally when set, not gated on `ep->data`; `vote_ep_on_fini` itself null-checks `ep->data`). No other change to the function. -- `src/controller/ha.h` — add `_Atomic int refcount;` to `ha_elect_ctx_t` (next to the existing `done_posted` guard field, same struct region). -- `src/controller/ha.c`: - - New static helper `ha_elect_ctx_release(ha_elect_ctx_t *ctx)` — decrement; on reaching 0, `pthread_mutex_destroy` + `sem_destroy` + `free`. Single choke point for both release kinds (endpoint completion, supersession). - - New static hook function `vote_ep_on_fini(endpoint_t *ep)` — reads `vctx = ep->data`, calls `ha_elect_ctx_release(vctx->ctx)`. Wired as `ep->on_fini` at both capture sites, same place `endpoint_set_err_handler` is already called. - - `raft_cb_send_vote_request`: initialize `ectx->refcount` is NOT done here (ctx already exists, allocated earlier) — this site only increments (`atomic_fetch_add_explicit(&ectx->refcount, 1, memory_order_relaxed)`) at the point `ep->data = vctx` is committed, and sets `ep->on_fini`. - - `raft_cb_send_pre_vote_request`: identical increment + hook wiring at its `ep->data = vctx` site. - - `ha_start_election_async`: at carrier allocation, initialize `ctx->refcount = 1`. At the multi-node publish site, change `atomic_store_explicit(&ha->elect_ctx, ctx, memory_order_release)` to `atomic_exchange_explicit(&ha->elect_ctx, ctx, memory_order_acq_rel)` capturing the return as `old_ctx`; if `old_ctx` non-NULL, call `ha_elect_ctx_release(old_ctx)`. The two existing fast-exit `free(ctx)` paths (no-local-config, single-node) are unchanged — they never share the ctx, so a direct free at refcount 1→destroy is still correct (or, for consistency and to avoid a second free idiom, route them through `ha_elect_ctx_release(ctx)` too — same result, one less code path to reason about). - - `ha_start_prevote_async`: identical `refcount = 1` init at allocation; identical `atomic_exchange_explicit` + `ha_elect_ctx_release(old_ctx)` at its publish site (the former "Do NOT free a prior ctx" comment site, now `ha.c:~8042`), comment updated to describe the new mechanism instead of documenting the leak as deliberate. - -### No New Files -This is a localized extension to two existing files (`ha.c`/`ha.h`) plus a minimal, generic addition to the shared endpoint abstraction (`netio.h`/`netio.c`). No new translation units. - -## System Flows - -See the sequence diagram in Architecture above — it is the complete flow for both the real-election and PreVote paths (structurally identical, differing only in which capture function runs). - -State transitions for `ha_elect_ctx_t.refcount` (per instance, monotonically decreasing after peak): - -```mermaid -stateDiagram-v2 - [*] --> Live: calloc, refcount=1 (bias) - Live --> Live: vote-ep capture (+1) — 0..n_peers-1 times - Live --> Live: vote-ep endpoint_fini completes (-1) — any order, any thread - Live --> Superseded: next round's atomic_exchange (-1, bias released) - Live --> Freed: refcount reaches 0 while still current (all endpoints drained before next round starts) - Superseded --> Freed: last outstanding vote-ep drains (-1 reaches 0) - Freed --> [*] -``` - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | Reclaim once all referencing vote-eps freed | `ha_elect_ctx_release`, `refcount` field | — | refcount state diagram | -| 1.2 | No bounded wait/timeout — driven by actual release | `vote_ep_on_fini` hook fired from `endpoint_fini` | `endpoint_t.on_fini` | sequence diagram | -| 1.3 | Zero ASAN leak over many rounds under leak detection | full mechanism (all sites below) | — | validation: `--detect_leaks=1` soak, new test | -| 2.1 | Never free while a vctx holds a reference | refcount discipline (+1 capture / -1 completion) | — | sequence diagram | -| 2.2 | Carrier still valid at `endpoint_fini` read time | hook fires from inside `endpoint_fini`, before `FREEUP(ep->data)` — vctx itself (thus `vctx->ctx`) still valid at hook-call time | `endpoint_t.on_fini` firing order | sequence diagram | -| 2.3 | Same mechanism at both capture sites | `raft_cb_send_vote_request`, `raft_cb_send_pre_vote_request` both increment + wire hook | — | Modified Files | -| 2.4 | Same mechanism on every vctx free path, incl. error/abort | single hook fires from the one true `vctx` free site (`endpoint_fini`) regardless of which teardown path reached it | `endpoint_t.on_fini` | Architecture §Mechanism rationale | -| 3.1 | Same lifetime mechanism for election and PreVote | identical `refcount`/`ha_elect_ctx_release`/`vote_ep_on_fini` reused by both paths | — | Modified Files | -| 3.2 | Superseded PreVote round's carrier reclaimable like election's | `ha_start_prevote_async`'s publish site uses the same `atomic_exchange_explicit` + release pattern | — | state diagram | -| 4.1 | No change to vote-tally/quorum/freshness behavior | mechanism touches only allocation/free bookkeeping, never `votes[]` contents or tally logic | — | — | -| 4.2 | No change to election/PreVote/failover timing or outcome | hook/refcount work is O(1) non-blocking, off the liveness-critical send/reply path | — | Architecture §Steering compliance | -| 4.3 | Full HA soak shows no new loss/divergence/stall | validation plan (Testing Strategy) | — | — | - -## Testing Strategy - -- **New unit test — leak closure**: run N consecutive multi-node election (and separately, PreVote) rounds in the existing controller test harness under ASAN with leak detection **enabled** (distinct from the suite's current `detect_leaks=0` default, per research.md) and assert zero `ha_elect_ctx_t` bytes leaked. This is Req 1.3's direct acceptance test. -- **New unit test — no premature free**: construct a round with an in-flight vote endpoint (captured, not yet torn down) and drive a supersession (start a new round); assert the carrier is still valid (not freed) while the outstanding endpoint's teardown has not yet completed, and becomes freed only after its `on_fini` fires. Directly tests Req 2.1/2.2. -- **New unit test — both paths covered**: parametrize (or duplicate) the above two tests for `ha_start_prevote_async` / `raft_cb_send_pre_vote_request`, confirming Req 3.1/3.2 — no path left on the old leak-forever behavior. -- **Regression — existing election/PreVote/failover suites**: run unchanged; must stay green with zero behavioral diff (Req 4.1/4.2). Any assertion change here would itself indicate a semantic regression. -- **Live — full HA soak**: `/xce-test-ha` sequential and chaotic regression post-merge (Req 4.3), watching specifically for job loss, divergence, or a liveness stall newly attributable to this change (bug-339/bug-340 vote-freshness behavior must be bit-for-bit unaffected, since the fix touches allocation/free timing, not vote-freshness computation). - -## Risks - -| Risk | Mitigation | -|------|------------| -| A future second consumer of `endpoint_t.on_fini` on the same endpoint could silently override this one | Not applicable today (vote endpoints have exactly one hook consumer); flagged as a Revalidation Trigger above so a future feature adding a second hook consumer on a vote endpoint is forced to notice the conflict. | -| Off-by-one in the increment/decrement pairing (e.g. incrementing at `vctx->ctx = ectx` instead of at `ep->data = vctx`) could reintroduce either a leak or a premature free | Increment is placed at the exact point the reference becomes "live" (attached to an endpoint that will reach `endpoint_fini`); the two pre-`endpoint_get()` alloc-fail bailouts never increment, matching that they never attach `ep->data`. Verified by the "no premature free" unit test above. | -| Relaxed memory order on the increment (`memory_order_relaxed`) racing the release of the bias reference | The increment happens on the same thread that is about to hand the endpoint to iomgr for connect — happens-before is already established by the existing `ep->data = vctx` write and subsequent `endpoint_connect`/`iomgr_monitor` calls, which themselves use acquire/release on the endpoint's own `refcnt`. The carrier's `refcount` only needs atomicity (concurrent increments from a fan-out of N peers, concurrent decrements from N `endpoint_fini` calls on different threads), not ordering relative to carrier *contents* — the carrier's field contents are fully populated before the round's `atomic_store`/`atomic_exchange` publish (`memory_order_release`), which is the actual publish barrier readers rely on. | diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/requirements.md b/.kiro/specs/ha-elect-ctx-lifecycle/requirements.md deleted file mode 100644 index 7ac762dc..00000000 --- a/.kiro/specs/ha-elect-ctx-lifecycle/requirements.md +++ /dev/null @@ -1,55 +0,0 @@ -# Requirements Document - -## Introduction - -The HA election carrier (`ha_elect_ctx_t`, allocated as `mc` in both `ha_start_election_async` and `ha_start_prevote_async`, stored in `ha->elect_ctx`) is never freed today. Each failover, real-election, or PreVote round allocates a fresh carrier and deliberately leaks the previous one, because per-peer vote endpoints (`vote_ep_ctx_t.ctx`) hold a bare pointer into the carrier and are torn down asynchronously by `endpoint_fini` on the iomgr thread — a time the round-starting thread cannot cheaply observe. A naive free at round-start would risk a use-after-free if a still-live endpoint callback later dereferences the superseded carrier. - -This feature gives the election carrier a real, safe lifetime: it is freed once no vote endpoint can still reference it, and never freed while one might. It applies uniformly to both the real-election and PreVote carrier paths, since they share `ha->elect_ctx` and the identical hazard — fixing one without the other leaves the UAF risk in place for the unfixed path. - -This is a memory-lifetime fix only. It must not change election, PreVote, or failover semantics, and must not weaken or alter any existing correctness guarantee (including the bug-339/bug-340 vote-freshness fixes). - -> Full discovery context (problem statement, code pointers, deferred-priority rationale, implementation options) is in `.kiro/specs/ha-elect-ctx-lifecycle/brief.md` and `.kiro/specs/ha-elect-ctx-lifecycle/research.md`. - -## Boundary Context - -- **In scope**: the lifetime of the `ha_elect_ctx_t` carrier allocated by `ha_start_election_async` and `ha_start_prevote_async`; safe reclamation once every `vote_ep_ctx_t` that references it (via `vctx->ctx`) has been freed; coverage of every capture site (`raft_cb_send_vote_request`, `raft_cb_send_pre_vote_request`) and every `vctx` free site (including `endpoint_fini` FREEUP and error/abort paths that never reach a live endpoint); the memory-order discipline between the reclamation mechanism and the existing `atomic_store(&ha->elect_ctx, ..., release)` publish / acquire reads. -- **Out of scope**: the PreVote protocol itself (shipped and validated in `ha-prevote`); any change to election, vote-granting, or failover *semantics*; the `ha_elect_ctx_t` struct's field contents or tally logic; the async endpoint-teardown ordering itself (`endpoint_fini`, `vote_ep_teardown_async`) beyond observing when it completes. -- **Adjacent expectations**: relies on the existing per-endpoint teardown path (`endpoint_fini` freeing `vctx`) continuing to run exactly as it does today — this feature observes that teardown, it does not change when or how an endpoint tears down. Must remain compatible with the existing "do NOT free a prior ctx" UAF-caution comments elsewhere in `ha.c` (the rdhup path, delaymgr teardown) — this feature is what finally makes one of those comments obsolete for the carrier specifically, without touching the others' own hazards. - -## Requirements - -### Requirement 1: Bounded carrier lifetime - -**Objective:** As a maintainer, I want the election carrier to be freed once it is no longer referenced, so that a long-running controller does not accumulate one leaked carrier per failover/election/PreVote round indefinitely. - -#### Acceptance Criteria -1. When every `vote_ep_ctx_t` that captured a reference to an `ha_elect_ctx_t` carrier has been freed, the Controller shall reclaim that carrier. -2. The Controller shall reclaim a superseded carrier without requiring a bounded wait, timeout, or external trigger — reclamation shall be driven by the actual release of the last referencing endpoint. -3. Under an ASAN build with leak detection enabled, running many consecutive election/PreVote rounds shall report zero leaked `ha_elect_ctx_t` allocations. - -### Requirement 2: No use-after-free on the endpoint-teardown path - -**Objective:** As a maintainer, I want carrier reclamation to never race a live endpoint's use of that carrier, so that fixing the leak cannot introduce a use-after-free. - -#### Acceptance Criteria -1. The Controller shall never free an `ha_elect_ctx_t` carrier while any `vote_ep_ctx_t` still holds an unfreed reference to it (`vctx->ctx`). -2. When a vote endpoint's asynchronous teardown (`endpoint_fini` FREEUP) reads or dereferences its captured carrier, that carrier shall still be valid at the time of that read, regardless of whether a newer election/PreVote round has since started. -3. The Controller shall apply the same safe-reclamation mechanism to every capture site of the carrier (`raft_cb_send_vote_request` for real elections, `raft_cb_send_pre_vote_request` for PreVote), so no capture site is left with the old leak-forever behavior once this feature lands. -4. The Controller shall apply the same safe-reclamation mechanism on every `vctx` free path, including error and abort paths that free a `vote_ep_ctx_t` without the endpoint ever reaching a live connected state. - -### Requirement 3: Uniform coverage across real-election and PreVote - -**Objective:** As a maintainer, I want the fix applied identically to both carrier-allocating paths, so that neither path is left with the original UAF-risk-driven leak. - -#### Acceptance Criteria -1. The Controller shall use the same carrier-lifetime mechanism for carriers allocated by `ha_start_election_async` (real election) and `ha_start_prevote_async` (PreVote). -2. When a PreVote round is superseded by a new round (e.g. a suppressed disruptive rejoin or a fresh liveness-triggered campaign), the superseded round's carrier shall become eligible for reclamation exactly as a superseded real-election carrier does. - -### Requirement 4: No regression to election, PreVote, or failover correctness - -**Objective:** As an operator, I want this memory-lifetime change to leave voting and failover behavior completely unchanged, so that fixing a leak cannot introduce a liveness or safety regression. - -#### Acceptance Criteria -1. The Controller shall preserve all existing vote-tally, quorum, and freshness-gate behavior (including the bug-339/bug-340 vote-freshness fixes) unchanged by this feature. -2. The Controller shall preserve all existing election, PreVote, and failover timing and outcome behavior unchanged by this feature — this feature shall not be observable from vote outcomes, election timing, or failover convergence. -3. A full HA soak (sequential and chaotic `/xce-test-ha` regression) run after this feature lands shall show no new job loss, divergence, or liveness stall attributable to this change. diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/research.md b/.kiro/specs/ha-elect-ctx-lifecycle/research.md deleted file mode 100644 index 4123ca53..00000000 --- a/.kiro/specs/ha-elect-ctx-lifecycle/research.md +++ /dev/null @@ -1,56 +0,0 @@ -# Research / Gap Analysis — ha-elect-ctx-lifecycle - -## Gap Re-Audit (2026-06-18) - -> **Prerequisite not met:** this spec has only `brief.md` — **no `requirements.md`** (phase=`initialized`, `ready_for_implementation=false`). A full requirements↔codebase gap map cannot be produced. Run `/kiro-spec-requirements ha-elect-ctx-lifecycle` to formalize EARS requirements, then re-run `/kiro-validate-gap ha-elect-ctx-lifecycle`. The note below is a **pre-requirements gap sketch** derived from `brief.md` + a current code check. - -### Current State (verified 2026-06-18) -The deliberate-leak pattern the brief describes is **still present and unfixed** in `src/controller/ha.c`: -- `ha.c:4229` — "Do NOT free a prior ctx (live vote endpoints may hold a bare pointer to it…)" immediately before `atomic_store(&ha->elect_ctx, mc)`. Each election/PreVote round `calloc`s a fresh `ha_elect_ctx_t` and intentionally leaks the previous one. -- `ha.c:3978`, `ha.c:4830` — sibling "Do NOT free … may still reference / may already be running" comments on the async endpoint / delaymgr teardown paths (the same UAF caution). -- No `ha_election_cleanup()` exists (only the aspirational comment the brief cites). Carrier read sites: `ha.c:123-124`, `234-235`, `903-904`. -- Carrier captured by per-peer vote endpoints (`vote_ep_ctx_t.ctx = ectx`), freed asynchronously by `endpoint_fini` (FREEUP) on the iomgr thread — the root cause of the UAF hazard that makes a naïve free unsafe. - -### Requirement-to-Asset Map (pre-requirements — tags: MET / Missing / Constraint) -| Intended capability (from brief) | Current code | Tag | -|----------------------------------|--------------|-----| -| Carrier (`ha_elect_ctx_t`) freed when no endpoint references it | leaked by design at `ha.c:4229` | **Missing** | -| Safe free vs async `endpoint_fini` (`vctx`) teardown | no refcount / no deferred-free ordering | **Missing** | -| Applies to BOTH real-election and PreVote carrier paths (shared `ha->elect_ctx`) | both paths leak identically | **Missing / Constraint** (fix one without the other leaves the hazard) | -| No regression of failover correctness (bug-339/bug-340) | current correctness intact; leak is memory-only | **Constraint** | - -### Severity / Why deferred -- **Pre-existing + bounded**: one small struct (`sizeof(ha_elect_ctx_t) + n_peers*votes[]`, hundreds of bytes for 3 nodes) per *failover/PreVote event* — a rare event, not steady-state. No unbounded growth under normal operation. -- Surfaces as ~174 KB / ~1548 allocs under ASAN across the full controller suite's many simulated rounds; the test command uses `detect_leaks=0`, so it does not fail CI today. - -### Implementation Approach Options (for the eventual design) -- **Option A — Reference-count the carrier.** Increment per `vctx` that captures it, decrement on each `vctx` free, free at zero. Clean, localized; needs an atomic refcount on `ha_elect_ctx_t` and disciplined inc/dec at every capture/free site. Effort **M**, Risk **Medium** (must cover every `vctx` lifecycle exit, incl. error paths). -- **Option B — Teardown-ordered deferred free.** Tie the free to confirmed vote-endpoint release (e.g. a generation/epoch the iomgr teardown signals back). Avoids per-capture bookkeeping but needs a reliable "all endpoints of round N released" signal. Effort **M–L**, Risk **Medium**. -- **Option C — Hybrid.** Refcount as the primitive, plus an epoch guard so a superseded round's carrier is reclaimed promptly once its endpoints drain. Effort **L**, Risk **Medium**. - -### Research-Needed (carry to design) -- Enumerate **all** `vctx->ctx = ectx` capture sites and **all** `vctx` free sites (incl. `endpoint_fini` FREEUP, error/abort paths, `raft_cb_send_vote_request`, `ha_send_pre_vote_request`). -- Confirm the memory-order discipline for the refcount vs the existing `atomic_store(&ha->elect_ctx, …, release)` / acquire reads. -- Validation must run ASAN with **leak detection ENABLED** (success = zero carrier leaks across many rounds) — distinct from the current `detect_leaks=0` suite default. - -### Effort / Risk (overall) -- Effort **M** (single-area C change in `ha.c`/`ha.h`), Risk **Medium** (async-teardown UAF is the real hazard; must not regress bug-339/bug-340 failover correctness). - -**Next step:** `/kiro-spec-requirements ha-elect-ctx-lifecycle` → `/kiro-spec-design` → implement under leak-detection-enabled ASAN. - -## Design Discovery (2026-07-11) - -Requirements approved; re-verified the code against the 2026-06-18 sketch above (unchanged in the intervening weeks) and traced every capture/free site precisely: - -- **Capture sites** (2, exactly as sketched): `raft_cb_send_vote_request` (`ha.c:~191`, `ep->data = vctx` at `~223`), `raft_cb_send_pre_vote_request` (`ha.c:~335`, `ep->data` assignment at its mirror site). -- **The single true `vctx` free site**: `endpoint_fini` (`src/lib/netio.c:1094`, `FREEUP(ep->data)`), reached via the existing bug-1298 `endpoint_t.refcnt` mechanism from every teardown path (connect-fail, `iomgr_monitor`-fail, the flag-OFF loser-path loop at `ha.c:~1553`, the flag-ON deferred-vtask path `vote_ep_teardown_async`, RDHUP/err-handler). Confirmed roughly a dozen `endpoint_unref`/`free(vctx)` call sites in `ha.c`, but only ~6 are vote-endpoint-related (the rest — `~8256`, `~8310`, `~8330`, `~8337`, `~8432`, `~8447` — are `reconnect_ctx_t`/`sctx` standby-reconnect endpoints, explicitly out of scope, confirmed by reading each in context: they null `ep->data` before unref and free their own `sctx` separately). -- **The supersession/leak point**: both `ha_start_election_async` (`ha.c:~7153`) and `ha_start_prevote_async` (`ha.c:~6899`, the exact "Do NOT free a prior ctx" comment cited in the 06-18 sketch) `atomic_store_explicit` a new carrier over the old one with no release. -- **Already-correct paths, confirmed out of scope**: `ha_start_election_async`'s no-local-config bail-out and single-node (`n_peers==1`) fast exits already `free(ctx)` directly and correctly — no vote endpoint is ever created in either case, so the carrier is never shared. These are not part of the leak and this design must not alter their correctness. - -### Design Decisions (synthesis) -- **Generalization considered and rejected**: could have generalized to a reusable "refcounted heap object" utility. Rejected — exactly one struct (`ha_elect_ctx_t`) needs this in the codebase today; a generic utility would be speculative abstraction (YAGNI). Instead generalized the *interface* one notch: `endpoint_t.on_fini`, a single generic completion hook mirroring the existing `on_connected` fire-once hook, which naturally supports this consumer and any future one without new machinery. -- **Build vs. adopt**: the codebase already has a directly analogous, proven mechanism — the bug-1298 `endpoint_t.refcnt` (reference-counted endpoint lifetime, freed at 0). Adopted the *pattern* (atomic refcount + release-to-zero-frees) rather than the mechanism itself, since the two lifetimes are genuinely different objects (the endpoint vs. the carrier a vote endpoint's `ep->data` points into) — conflating them was considered and rejected (Option B/teardown-signal from the 06-18 sketch) because it would require the carrier to know about individual endpoint teardown ordering, whereas a hook fired at each endpoint's own completion is simpler and matches Req 2.4's "every free path" requirement by construction. -- **Simplification**: settled on Option A (refcount) over Option B (deferred/epoch signal) and Option C (hybrid) from the 06-18 sketch — a plain atomic refcount with a bias reference for the "currently published" slot is sufficient; no epoch/generation counter is needed because the bias reference already disambiguates "still current" from "superseded but draining." -- **Memory-order note carried to implementation**: the increment on capture can use `memory_order_relaxed` (only atomicity is required, not ordering relative to carrier contents — the carrier's fields are fully populated before the publishing `atomic_store`/`atomic_exchange`, which is the real publish barrier). The supersession release moves the publish site from `atomic_store_explicit` to `atomic_exchange_explicit` to retrieve the outgoing pointer without a second load (avoids a TOCTOU gap between reading the old pointer and overwriting it). - -Full mechanism, file-by-file plan, and requirements traceability are in `design.md`. diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/spec.json b/.kiro/specs/ha-elect-ctx-lifecycle/spec.json deleted file mode 100644 index e146b1d6..00000000 --- a/.kiro/specs/ha-elect-ctx-lifecycle/spec.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "feature_name": "ha-elect-ctx-lifecycle", - "created_at": "2026-06-16T03:10:00Z", - "updated_at": "2026-07-18T09:56:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T09:56:00Z", - "note": "Cross-task/feature-level validation of a spec whose 13/13 tasks.md leaf checkboxes (1.1, 2.1-2.4, 3.1-3.5) had been complete since before 2026-07-11, but whose spec.json phase was never advanced past tasks-generated (same administrative-oversight pattern as several sibling specs validated earlier today: ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote). This is the LAST of 9 Group-1 validate-impl closure passes run this session. Ran immediately after ha-prevote (GO) -- extra weight was given to the fact that ha-prevote's own soak already independently exercised this spec's fix (commit 6529dcf) via test_many_prevote_rounds_no_carrier_leak, giving this pass a useful cross-check without treating it as sufficient on its own.\n\nCROSS-TASK CONSISTENCY / DESIGN ALIGNMENT: confirmed live on-disk source matches design.md's File Structure Plan exactly, field-for-field. src/include/netio.h:103/110 -- on_fini is declared immediately adjacent to on_connected with the documented cross-reference comment (as specified). src/lib/netio.c:1112-1113 -- endpoint_fini calls ep->on_fini(ep) immediately before FREEUP(ep->data) (line 1116), matching the designed firing point exactly. src/controller/ha.h:126 -- _Atomic int refcount added to ha_elect_ctx_t with a doc comment that reproduces the design's bias-reference rationale verbatim. src/controller/ha.c: ha_elect_ctx_release (line 150, decrement + pthread_mutex_destroy/sem_destroy/free at zero) and vote_ep_on_fini (line 180, reads vctx from ep->data, releases vctx->ctx) are the single choke points design.md specifies; both raft_cb_send_vote_request (line 282 increment, line 293 hook wire) and raft_cb_send_pre_vote_request (line 421 increment, line 432 hook wire) apply the identical increment+wire pattern -- Requirement 3's 'uniform coverage' claim holds by direct source inspection, not just test titles. Supersession release sites present at lines 7106-7108 and 7376-7378 (atomic_exchange_explicit retrieving the outgoing pointer, then ha_elect_ctx_release on it) matching the design's 'exchange instead of store' mechanism for both ha_start_election_async and ha_start_prevote_async. Two additional release call sites (7423, 7442) cover error/abort paths per Requirement 2.4. Dependency direction and Boundary Commitments honored: no change to endpoint_fini's existing teardown mechanics beyond the one conditional hook call, no change to PreVote/election/failover semantics, the unrelated bug-1298 endpoint refcnt mechanism is untouched and not conflated with the new carrier refcount. No Revalidation Trigger has fired (no second on_fini consumer exists, no new vote endpoint capture site was added outside the two documented ones).\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) -- zero compiler/linker errors, xce_controller/controller_unit_test/xce_test binaries confirmed with session-fresh mtimes. Standalone controller_unit_test binary run directly (authoritative over meson's aggregate line per this project's convention): 972 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- zero regressions across the full suite, including existing vote-tally/quorum/freshness-gate tests (bug-339/bug-340 vote-freshness fixes unaffected). LEAK-SPECIFIC CHECK (this spec's core deliverable is a leak fix, so this is the load-bearing evidence): re-ran test_ha_elect_ctx_lifecycle.c in isolation with ASAN_OPTIONS=detect_leaks=1 (leak detection is off by project-wide default per cerebrum.md convention; this spec's own tests are written to enable it explicitly for exactly this purpose) -- 5 Tests, 0 Failures, 0 Ignored, OK, exit 0: test_release_at_refcount_one_frees_no_crash, test_release_at_refcount_two_leaves_valid, test_many_election_rounds_no_carrier_leak (20 rounds x 2 votes), test_many_prevote_rounds_no_carrier_leak (20 rounds x 2 votes), test_carrier_survives_supersession_with_outstanding_endpoint -- zero leaked ha_elect_ctx_t allocations across 40 real-election + 40 PreVote vote endpoints, and no premature-free UAF. TBD/TODO/FIXME/HACK/XXX grep on the spec's boundary files (ha.c, ha.h, netio.h, netio.c, test_ha_elect_ctx_lifecycle.c): one match (ha.c:1844, 'TODO(ha-term-safety task 2.x)') -- belongs to a different spec's stub, not this one's boundary. Secrets grep: clean.\n\nRUNTIME LIVENESS: ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir. FIRST ATTEMPT FAILED at round 1: 'no new primary elected after kill (no serving primary at term>2)'. Root-caused via the controller logs, not assumed: cold-start election was unusually slow this run (ctrl-1 reached CTRL_PRIMARY term=2 only ~10.1s after process start), and the driver's round-1 kill fired ~1.7s after that -- before ctrl-1's just-started config-push handshake to the two standbys ('ctrlhello sent' at the same timestamp as promotion) could complete. Both standbys were still at tag=0 (no local config) at kill time, so per ha-config-bootstrap's own Req 2.8 guard they correctly refused to start PreVote/RDHUP re-election ('no local config (tag=0, n_peers=3) -- voter only, not starting PreVote/RDHUP re-election') rather than risk electing an unconfigured primary. This is a test-harness timing race (round 1 has no guard waiting for initial standby config receipt before killing the primary) -- not a code defect in this spec's boundary (no code this spec owns touches config propagation or the ha-config-bootstrap tag gate). RETRY: same command, fresh deploy dir, immediately after -- PASS: 10/10 rounds, 36 jobs submitted, zero job loss, zero committed-prefix divergence, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) served as primary at least once. Grepped both the failed and the passing run's driver + all per-node controller logs for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker) and other crash signatures (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches in either run. No leftover xce_controller processes after either run (driver's own cleanup on FAIL, and clean exit on PASS, both confirmed via ps).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory, not reproduced here. The round-1 config-push/kill-timing race is a test-harness gap in run-ha-soak.sh's sequential mode (no wait for initial 3-of-3 config convergence before the first kill), not owned by this spec -- flagged here as a known_limitation for whoever next touches the driver or ha-config-bootstrap, not adopted as a blocker for this spec's own GO. No new bugs found or fixed during this validation pass; no code changes were made -- this was a read/build/test/soak-only pass.", - "known_limitations": [ - "Test-harness timing race in .claude/skills/xce-test-ha/run-ha-soak.sh sequential mode: round 1's kill is not gated on the standbys having received their initial config push, so on a slow cold-start election (observed here: ~10s to CTRL_PRIMARY) round 1 can fire before config propagates, hitting ha-config-bootstrap's Req 2.8 voter-only guard and failing the round with 'no serving primary at term>2'. Reproduced once in this session, did not reproduce on an immediate retry (10/10 clean). Not owned by this spec (no code this spec touches affects config propagation or the tag=0 guard) -- flagged for whoever next touches run-ha-soak.sh or ha-config-bootstrap; a fix would add a short 3-of-3-config-received wait before round 1's first kill.", - "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker. Did not reproduce in either soak run in this pass (failed or passing). No action item for this spec.", - "Requirement 4.3 names both sequential AND chaotic /xce-test-ha regression modes. Per the explicit validation-scope instruction for this pass, only the sequential 10-round soak was (re-)run; chaotic mode was not independently re-run for this spec. This mirrors the evidence scope of the sibling ha-prevote validation (also sequential-only) and is a low-risk gap given this is a pure memory-lifetime fix with zero semantic surface on election/failover outcomes (chaotic mode's specific hazard class -- double-fault commit-prefix divergence timing -- is orthogonal to carrier refcounting)." - ] - } -} diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/tasks.md b/.kiro/specs/ha-elect-ctx-lifecycle/tasks.md deleted file mode 100644 index c2f2ef82..00000000 --- a/.kiro/specs/ha-elect-ctx-lifecycle/tasks.md +++ /dev/null @@ -1,79 +0,0 @@ -# Implementation Plan: ha-elect-ctx-lifecycle - -- [x] 1. Foundation: endpoint teardown-completion hook -- [x] 1.1 Add a generic completion hook to the endpoint abstraction, fired exactly once when an endpoint's teardown reaches its final free - - Add an `on_fini` function pointer field to `endpoint_t` (src/include/netio.h), mirroring the existing `on_connected` fire-once hook - - In `endpoint_fini` (src/lib/netio.c), call the hook (if set) immediately before the existing `FREEUP(ep->data)` line — no other change to the function's sequence - - New unit test: set `on_fini` on a test endpoint, drive it through teardown, assert the hook fires exactly once and that `ep->data` is still valid (non-freed) at the moment the hook runs - - Observable: the new unit test passes; every other existing endpoint (which never sets `on_fini`) tears down with zero behavioral change (hook is a no-op NULL check) - - _Requirements: 2.2_ - -- [x] 2. Core: election carrier reference-counted lifecycle -- [x] 2.1 Add a reference count to the election carrier and a single release helper - - Add an `_Atomic int refcount` field to `ha_elect_ctx_t` (src/controller/ha.h) - - Add a static helper in ha.c that decrements the refcount and, on reaching zero, destroys the mutex/semaphore and frees the carrier - - New unit test: call the helper once on a freshly allocated carrier (refcount=1) and assert it is freed; call it once on a carrier with refcount=2 and assert it is still valid with refcount=1 - - Observable: the helper is the only place `ha_elect_ctx_t` is destroyed/freed in ha.c - - _Requirements: 1.1_ - - _Boundary: ElectionCarrierLifecycle (ha.c/ha.h)_ - -- [x] 2.2 (P) Increment the carrier refcount and wire the completion hook at the real-election vote-request capture site - - In `raft_cb_send_vote_request`, at the point `ep->data = vctx` is committed (only for vctx instances that reach a live endpoint), atomically increment the carrier's refcount and set `ep->on_fini` to a new hook function that releases the carrier via the 2.1 helper - - The two pre-`endpoint_get()` alloc-fail bailouts in this function do not increment (they never attach `ep->data`) and need no matching release - - Observable: after a multi-node election round runs to completion and every vote endpoint's teardown finishes, the carrier's refcount reflects only the still-outstanding supersession bias reference (no endpoint-attributable count remains) - - _Requirements: 1.1, 2.1, 2.3, 2.4_ - - _Boundary: raft_cb_send_vote_request_ - - _Depends: 1.1, 2.1_ - -- [x] 2.3 (P) Increment the carrier refcount and wire the completion hook at the PreVote vote-request capture site - - Same mechanism as 2.2, applied to `raft_cb_send_pre_vote_request` - - Observable: identical post-round refcount behavior as 2.2, but for PreVote rounds - - _Requirements: 1.1, 2.1, 2.3, 2.4, 3.1_ - - _Boundary: raft_cb_send_pre_vote_request_ - - _Depends: 1.1, 2.1_ - -- [x] 2.4 Release the superseded carrier when a new election or PreVote round publishes its own - - In `ha_start_election_async` and `ha_start_prevote_async`, change the carrier-publish assignment from `atomic_store_explicit` to `atomic_exchange_explicit` to retrieve the outgoing carrier pointer, then release it via the 2.1 helper if non-NULL - - Route the two existing no-vote-endpoint fast-exit paths in `ha_start_election_async` (no-local-config bail-out, single-node case) through the same 2.1 helper instead of a direct `free()`, for one consistent destruction path - - Update the "Do NOT free a prior ctx" comment at the PreVote publish site to describe the new mechanism instead of documenting the leak as deliberate - - Observable: starting a second round while the first round's carrier already has zero endpoint references frees the first round's carrier immediately at supersession; starting a second round while the first still has an outstanding vote endpoint leaves the first carrier alive until that endpoint's teardown completes - - _Requirements: 1.1, 1.2, 3.1, 3.2_ - - _Boundary: ElectionCarrierLifecycle (ha.c publish sites)_ - - _Depends: 2.1, 2.2, 2.3_ - -- [x] 3. Validation: leak closure and premature-free safety -- [x] 3.1 (P) Prove zero carrier leaks across many consecutive real-election rounds under leak-detection-enabled ASAN - - New unit test drives N consecutive multi-node election rounds with ASAN leak detection enabled (distinct from the suite's current `detect_leaks=0` default) - - Temporarily revert the 2.2/2.4 changes to confirm this test fails (leak reported) pre-fix, then restore and confirm it passes (RED→GREEN) - - Observable: zero `ha_elect_ctx_t` bytes reported leaked across the run - - _Requirements: 1.3_ - - _Boundary: election test coverage_ - - _Depends: 2.2, 2.4_ - -- [x] 3.2 (P) Prove zero carrier leaks across many consecutive PreVote rounds under leak-detection-enabled ASAN - - Same shape as 3.1, driving PreVote rounds via `ha_start_prevote_async` instead - - Observable: zero `ha_elect_ctx_t` bytes reported leaked across the run - - _Requirements: 1.3, 3.1, 3.2_ - - _Boundary: PreVote test coverage_ - - _Depends: 2.3, 2.4_ - -- [x] 3.3 (P) Prove the carrier is never freed while an outstanding vote endpoint still references it across a round supersession - - Construct a round with one captured, not-yet-torn-down vote endpoint, then start a new round (triggering supersession) before that endpoint's teardown completes - - Assert the original carrier is still valid (not freed) immediately after supersession, and becomes freed only after the outstanding endpoint's `on_fini` fires - - Covers both the real-election and PreVote capture paths (parametrized or duplicated per design's Testing Strategy) - - Observable: a naive "free on supersession" implementation would fail this test (UAF or premature free caught by ASAN); the refcounted implementation passes it - - _Requirements: 2.1, 2.2, 2.3, 3.1, 3.2_ - - _Boundary: no-premature-free test coverage_ - - _Depends: 2.2, 2.3, 2.4_ - -- [x] 3.4 Run the full existing controller unit suite and confirm no behavioral change - - Run the complete `controller_unit_test` suite; assertions in pre-existing election, PreVote, and failover test files must be byte-for-byte unchanged - - Observable: full suite green, zero modified assertions in pre-existing election/prevote/failover tests - - _Requirements: 4.1, 4.2_ - - _Depends: 3.1, 3.2, 3.3_ - -- [x] 3.5 Run a full HA sequential and chaotic regression soak and confirm no new job loss, divergence, or liveness stall - - Run `/xce-test-ha` sequential and chaotic regression against a live 3-node cluster - - Observable: zero committed job loss, zero divergence, no liveness stall attributable to this change; bug-339/bug-340 vote-freshness behavior unaffected - - _Requirements: 4.3_ - - _Depends: 3.4_ diff --git a/.kiro/specs/ha-entity-snapshots/design.md b/.kiro/specs/ha-entity-snapshots/design.md deleted file mode 100644 index d7b42bd2..00000000 --- a/.kiro/specs/ha-entity-snapshots/design.md +++ /dev/null @@ -1,559 +0,0 @@ -# Design Document: ha-entity-snapshots - -## Overview - -**Purpose**: This feature makes the XCE controller's periodic persistence checkpoint an -actually-restorable copy of committed job and queue state, closing the permanent data gap a -rejoining or far-behind cluster member hits today (task 5.10a / raft-ha-core-safety; bug-795/ -828/851 family). Today `write_snapshot` records only a position marker; `load_snapshot` restores -nothing. This design adds the missing entity payload and restore step, and closes the one real -retention hazard it introduces (a snapshot file being rotated out from under an in-flight -transfer). - -**Users**: The XCE HA controller cluster itself is the direct consumer — every member that -rejoins empty or falls behind the retained WAL history. The observable beneficiary is the -cluster operator, who currently sees a silently incomplete member after such a rejoin. - -**Impact**: Modifies the existing checkpoint write/read/rotate logic in `persist.c` and adds one -small tracking array in `ha.c`. No wire protocol change, no new external dependency, no new -capnp schema struct — the existing `Job` schema and its generated codec already carry everything -needed for jobs. Queue *definitions* (name, attrs, priority, thresholds, ACLs) are config- -provisioned and already replicated independently via `push_config_to_standbys` -(`config_repl.c`) — out of this spec's boundary. Only a queue's runtime admin status -(open/closed/active/inactive), the one piece `WAL_QUEUE_CTRL` tracks and config replication does -not carry, is this spec's concern — restored via the existing `restore_queue_state(name, status)`, -no capnp involved on the queue side at all. (Note: the on-disk wire format was later migrated to a -capnp `QueueAdminState` frame — superseded by config-upgrade-snapshot, see that spec's design.md; -`restore_queue_state`'s own `(name, status)` call contract is unchanged.) - -### Goals -- A checkpoint written at LSN X contains every live/retained job and queue record as of X. -- Loading that checkpoint reconstructs those records before the member's position advances. -- History is never discarded before it is captured in a checkpoint whose entities are durable. -- A member that rejoins via checkpoint ends up bit-for-bit consistent with the rest of the - cluster — no permanent gap. -- A checkpoint produced by a prior version of the binary never crashes or corrupts a member - that reads it. - -### Non-Goals -- Lock-free, copy-on-write entity enumeration (zarray COW exists but is not wired into `kvmgr`; - reviving that wiring is a separate, larger effort — see `research.md` Option B). -- Any change to the wire transport/chunking protocol that ships a snapshot to a peer. -- Any change to the catch-up decision logic (`ha_evaluate_repl_action`) that chooses between a - snapshot transfer and incremental WAL streaming. -- Worker/compute-node state, cluster configuration document snapshotting, or election/term-safety - behavior. -- Queue *definition* fields (name, attrs, priority, thresholds, windows, ACLs) — these are - config-provisioned and already replicated independently (`push_config_to_standbys`); this spec - snapshots only the queue's runtime admin status. - -## Boundary Commitments - -### This Spec Owns -- The on-disk content format of a checkpoint (entity payload framing) and the write/read logic - that produces and consumes it (`write_snapshot`, `load_snapshot` internals). -- Point-in-time collection and encoding of live job and queue entities at checkpoint time. -- The entity-restore step on load and gating the member's committed-position advance on that - step's success. -- Distinguishing a legacy (pre-feature) position-only checkpoint from an entity-bearing one. -- Protecting an in-flight-transferring checkpoint file from premature rotation deletion. - -### Out of Boundary -- The wire opcode and chunk-reassembly protocol that delivers a checkpoint to a peer - (`OPCode_snapshotTransfer`, `ha_handle_snapshot_chunk`'s framing) — reused unchanged. -- The decision of *when* a member needs a checkpoint vs. incremental streaming - (`ha_evaluate_repl_action`) — unchanged. -- The WAL-segment trim floor's standby-acked-LSN computation (`repl_trim_lsn`) — unchanged; see - Architecture below for why no new floor input is needed there. -- Worker/join state, config-document snapshotting (`config-upgrade-snapshot`), election and - term-safety behavior (`raft-ha-core-safety`, `ha-term-safety`). - -### Allowed Dependencies -- Existing capnp codegen for `Job` (`encode_Job`/`write_Job`/`decode_Job`, already generated from - `src/schema/controller.capnp`). Queue entries need no capnp — `restore_queue_state`'s contract - is a plain `(name, status)` pair (see Architecture below). (Note: config-upgrade-snapshot later - added a capnp `QueueAdminState` frame as the on-disk encoding of that same pair — see Data - Models below.) -- `restore_job` (`job.c:274`) and `restore_queue_state` (`queue.c:208`) — existing, unmodified - entity-restore functions already used by WAL replay. -- The `collect_jobs`/`collect_queues` enumeration pattern (`dataex.c`) and the - `job_should_retain`/`worker_should_retain` retention predicates (`persist.c`). -- The existing tmp-file-then-atomic-rename pattern already used by `write_snapshot` and by - `ha_handle_snapshot_chunk`'s `snap.part` assembly. -- The existing per-peer `ha->snapshot_inflight[]` array shape (`ha.h`) as the pattern for the new - in-flight-LSN tracking. - -### Revalidation Triggers -- Any change to the `Job` capnp schema shape — re-verify encode/decode symmetry and compatibility - with `restore_job`. -- Any change to `restore_queue_state`'s `(name, status)` contract, or any future need to snapshot - more than status for queues — re-verify the queue-side format and the config-vs-WAL boundary - this design draws. -- Any change to `collect_jobs`/`collect_queues` locking semantics, or to the per-entity - `RTTI_LOCK_READ`/`WRITE` discipline (e.g. if `kvmgr` COW is wired later) — re-verify the - point-in-time consistency argument in Architecture below. -- Any change to `persist_checkpoint`'s `write_snapshot` → `trim_wal_segments` sequencing - (currently sequential, trim skipped on write failure) — re-verify Requirement 3. -- Any change to `ha_handle_snapshot_chunk`'s ordering (currently: `applied_lsn` only advances - after `persist_load_snapshot` returns success) — re-verify Requirement 4. - -## Architecture - -### Existing Architecture Analysis - -- `persist_checkpoint` (persist.c:2820) already sequences correctly for Requirement 3: it calls - `write_snapshot`, and only calls `trim_wal_segments` if `write_snapshot` returned 0 - (`if (rc != 0) goto done;`, skipping both trim and `rotate_snapshots`). **This means once - `write_snapshot` only returns 0 when the entity payload is fully and durably written, the - existing trim gating already satisfies "retention never outruns restorability" — no new floor - variable is needed in `trim_wal_segments`/`repl_trim_lsn`.** This is a refinement over the - initial gap analysis (`research.md`), which assumed a new trim-floor input was needed. -- The one retention hazard `write_snapshot`'s success does *not* already guard is - `rotate_snapshots` (persist.c:2677): it keeps only `snap_keep_n` newest snapshot *files* by - filename sort and deletes the rest, with no awareness of an in-flight transfer. If a checkpoint - cadence faster than a slow transfer rotates out the file a standby is still reading from a - chunked wire transfer, that transfer breaks. This is the one genuinely new guard this design - adds. -- `ha_handle_snapshot_chunk` (ha.c:3635-3794) already has the right shape for Requirement 4: it - only advances `ha->applied_lsn` after `persist_load_snapshot` returns success. Once - `persist_load_snapshot`/`load_snapshot` correctly gate their own success on entity restoration - (Requirement 2), this call site needs no change. -- `collect_jobs`/`collect_queues` (dataex.c) already hold `RTTI_LOCK_READ` on `c->jobmgr`/ - `c->qmgr` for the duration of their walk. **This table lock is a distinct lock from the - per-entity lock** — verified via `restore_queue_state` (queue.c:208), which takes - `lock(q, RTTI_LOCK_WRITE)` on the individual `queue_t*` after already releasing the table lock - from its own `key_find`, and via `job.c`'s state-transition sites (e.g. lines 1006, 1093, 1137), - which take `lock(j, RTTI_LOCK_WRITE)` on the individual `job_t*` (line numbers shift as job.c - grows; verify by grepping `lock(j, RTTI_LOCK_WRITE)` if re-checking this). The table lock only excludes - concurrent insert/delete of table entries; it does **not** exclude a concurrent per-entity - writer mutating an entity's fields mid-walk. Requirement 1.2's point-in-time consistency - therefore requires the entity-collection helper to additionally take each entity's own - `RTTI_LOCK_READ` (mirroring `restore_queue_state`'s own per-entity write-lock pattern) around - that entity's individual encode call, nested inside the table-level walk — this excludes a - concurrent per-entity writer for exactly the duration of that one entity's encode, without - needing the (currently unwired) zarray COW mechanism. - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TB - subgraph Primary["Primary — checkpoint write path"] - PC[persist_checkpoint] --> WS[write_snapshot] - WS --> CJ["collect+encode jobs\n(table READ lock + per-job READ lock)"] - WS --> CQ["collect name+status pairs\n(table READ lock + per-queue READ lock,\nnow capnp QueueAdminState --\nconfig-upgrade-snapshot)"] - CJ --> TMP[tmp file + fdatasync + rename] - CQ --> TMP - TMP -->|rc==0| TRIM[trim_wal_segments] - TMP -->|rc==0| ROT["rotate_snapshots\n(skips in-flight LSNs)"] - TMP -->|rc!=0| SKIP["skip trim + rotate\n(existing goto done)"] - end - - subgraph Send["Snapshot transfer (unchanged)"] - DISPATCH[ha_snapshot_send_dispatch] -->|sets snapshot_inflight_lsn for peer| WIRE[OPCode_snapshotTransfer chunks] - WIRE --> DONE[clears snapshot_inflight_lsn on completion] - end - - ROT -.reads.-> DISPATCH - - subgraph Standby["Standby — catch-up receive path"] - RECV[ha_handle_snapshot_chunk] --> RENAME[assemble + rename final file] - RENAME --> LS[persist_load_snapshot] - LS --> RJ["restore_job per entity"] - LS --> RQ["restore_queue_state per entity"] - RJ -->|all succeed| ADV["advance current_lsn / checkpoint_lsn"] - RQ -->|all succeed| ADV - RJ -->|any fails| FAIL["return error, position unchanged"] - ADV --> APPLIED["ha->applied_lsn advances\n(existing, unchanged)"] - end -``` - -**Architecture Integration**: -- Selected pattern: extend the existing checkpoint write/read pipeline in place (Option A from - `research.md`); no new subsystem, no new wire message. -- Domain boundaries: entity content and restoration stay inside `persist.c` - (write/load/rotate); the one cross-file touch is the new `snapshot_inflight_lsn[]` array in - `ha.h`/`ha.c`, read by `rotate_snapshots` and written by the existing send-dispatch path. -- Existing patterns preserved: tmp-file-then-rename atomicity, `RTTI_LOCK_READ`/`WRITE` table AND - per-entity locking (mirroring `restore_queue_state`'s own nested-lock pattern), the - `collect_jobs`/`collect_queues` iteration shape, the `capnp_encode_job` wrapper shape. -- New components rationale: a small entity-frame reader/writer pair (mirrors the existing WAL - entry length-prefixed framing style, so no new capnp schema struct or codegen is required — - jobs use the existing `Job` capnp codec, queues use a plain fixed `(name, status)` record since - that is the entirety of `restore_queue_state`'s contract); the in-flight-LSN guard in - `rotate_snapshots`. (Note: "no new capnp schema struct" for queues held at this spec's close but - was superseded shortly after by config-upgrade-snapshot, which added a `QueueAdminState` capnp - struct and migrated the queue frame to it — see that spec's design.md.) -- Steering compliance: no `fork()` used for the snapshot (per `product.md`'s "No fork() for - snapshots" constraint) — consistency comes from the table read-lock, not process duplication. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Serialization | Cap'n Proto (existing project-wide IDL) | Per-job `Job` encode/decode; queues use a plain fixed record (no capnp) | Codegen already exists; no schema change | -| Persistence | Flat file, tmp+rename (existing pattern) | Checkpoint file on disk | Same pattern as today's sentinel-only file | -| Concurrency | `RTTI_LOCK_READ`/`WRITE`, table AND per-entity (existing) | Point-in-time entity consistency during collection | zarray COW deferred (Non-Goal) | -| Transport | `OPCode_snapshotTransfer` (existing, unmodified) | Ships the (now larger) file to a standby | Out of boundary | - -## File Structure Plan - -### Modified Files -- `src/controller/persist.c` — `write_snapshot`: append entity payload after the existing - header — job frames (capnp `Job`) and queue frames (plain `(name, status)` records, no capnp). - (Note: queue frames were later migrated to capnp `QueueAdminState`, and host frames plus a - trailing config-bytes section were appended after them — superseded by config-upgrade-snapshot, - see Data Models below and that spec's design.md.) - `load_snapshot`: restore entities before advancing `current_lsn`/`checkpoint_lsn`; detect - legacy (position-only) files by remaining file size. `rotate_snapshots`: skip deleting any file - whose LSN is currently in-flight. New static helpers: a snapshot entity-frame writer/reader - pair, and an entity-collection helper built on `collect_jobs`/`collect_queues` that additionally - takes each entity's own per-entity `RTTI_LOCK_READ` around its individual encode/collect call - (see Architecture). -- `src/include/persist.h` — extend `persist_load_snapshot`'s signature with an out-parameter - reporting whether entities were restored (see Components below); no other public surface - changes. -- `src/controller/ha.h` — add `_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]` next - to the existing `snapshot_inflight[MAX_HA_CANDIDATES]` array. -- `src/controller/ha.c` — `ha_snapshot_send_dispatch`/`ha_snapshot_do_send`: record the LSN being - sent into `snapshot_inflight_lsn[peer_idx]` alongside the existing flag, clear it on completion - (same lifecycle as the existing flag). `ha_handle_snapshot_chunk`: read the new - `persist_load_snapshot` out-parameter; log (not fail) when entities were not restored, since - the caller-side catch-up-completeness signal (Requirement 4.3) is carried by existing - status/comparison tooling, not a new error path here. - -### New Test Files -- `src/controller/testsuite/test_persist_snapshot_entities.c` — unit coverage for - `write_snapshot`/`load_snapshot` entity round-trip, partial-failure discard, legacy-file - handling, and the `rotate_snapshots` in-flight guard (see Testing Strategy). - -> No new production files. This stays entirely inside the existing checkpoint machinery's file -> set plus one small, symmetric addition in `ha.h`/`ha.c` for the rotation guard. - -## System Flows - -See the Architecture Pattern & Boundary Map diagram above for the write/send/receive flow. One -additional sequence clarifies the failure-ordering contract that Requirement 2.3 depends on: - -```mermaid -sequenceDiagram - participant Standby - participant persist_load_snapshot - participant restore_job - participant restore_queue_state - - Standby->>persist_load_snapshot: load(path, lsn) - persist_load_snapshot->>persist_load_snapshot: read header, detect format - alt legacy (position-only) file - persist_load_snapshot-->>Standby: rc=0, entities_restored=false - else entity-bearing file - loop each job frame - persist_load_snapshot->>restore_job: decode + restore - restore_job-->>persist_load_snapshot: rc - alt rc != 0 - persist_load_snapshot-->>Standby: rc=error, position UNCHANGED - end - end - loop each queue frame - persist_load_snapshot->>restore_queue_state: restore(name, status) - restore_queue_state-->>persist_load_snapshot: rc - alt rc != 0 - persist_load_snapshot-->>Standby: rc=error, position UNCHANGED - end - end - persist_load_snapshot->>persist_load_snapshot: advance current_lsn/checkpoint_lsn - persist_load_snapshot-->>Standby: rc=0, entities_restored=true - end -``` - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|---|---|---|---|---| -| 1.1 | Checkpoint includes every live/retained job+queue record | `write_snapshot`, entity-collection helper | `collect_jobs`/`collect_queues` + `job_should_retain`; queue side is `(name, status)` only (definition is config-owned) | Write path (diagram 1) | -| 1.2 | Internally consistent point-in-time view | Entity-collection helper | Table `RTTI_LOCK_READ` + per-entity `RTTI_LOCK_READ` nested around each encode | Architecture: Existing Architecture Analysis | -| 1.3 | Discard incomplete checkpoint | `write_snapshot` | Existing tmp+rename pattern | Write path (diagram 1) | -| 1.4 | Record committed position | `write_snapshot` (existing header, unchanged) | — | — | -| 2.1/2.2 | Restore entities before advancing position | `load_snapshot` | `restore_job`, `restore_queue_state` | Sequence diagram | -| 2.3 | Failed reconstruction must not advance position | `load_snapshot` | rc propagation from `restore_job`/`restore_queue_state` | Sequence diagram | -| 2.4 | Legacy checkpoint treated as insufficient | `load_snapshot`, `persist_load_snapshot` | New `entities_restored` out-parameter | Sequence diagram | -| 3.1/3.2/3.3 | Retention never outruns restorability | `persist_checkpoint` (existing sequencing, unchanged) | `write_snapshot` rc gates `trim_wal_segments`/`rotate_snapshots` | Write path (diagram 1) | -| 4.1/4.2 | End-to-end catch-up correctness | `ha_handle_snapshot_chunk` (existing, unchanged ordering) | `persist_load_snapshot` | Receive path (diagram 1) | -| 4.3 | Catch-up completeness observable | Existing `walcompare`/status tooling (out of boundary; confirmed compatible) | — | — | -| 5.1/5.2 | Safe rollout against old checkpoints | `load_snapshot` format discriminator | File-size-based legacy detection | Sequence diagram | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|---|---|---|---|---|---| -| `write_snapshot` (extended) | persist.c | Serialize header + live entities to a durable checkpoint file | 1.1-1.4, 3.1-3.3 | `collect_jobs`/`collect_queues` (P0), `capnp_encode_job` (P0) | State | -| `load_snapshot` (extended) | persist.c | Deserialize and restore entities; gate position advance on success | 2.1-2.4, 5.1-5.2 | `restore_job`, `restore_queue_state` (P0) | State | -| `rotate_snapshots` (extended) | persist.c | Delete old checkpoint files, skipping any in-flight LSN | 3.1-3.3 | `ha->snapshot_inflight_lsn[]` (P0) | State | -| `snapshot_inflight_lsn[]` (new) | ha.h / ha.c | Track which checkpoint LSN each in-flight peer transfer references | 3.1-3.3 | Existing `snapshot_inflight[]` lifecycle (P0) | State | - -### Persistence Layer - -#### `write_snapshot` (extended) - -| Field | Detail | -|-------|--------| -| Intent | Produce a durable, entity-bearing checkpoint file at `checkpoint_lsn` | -| Requirements | 1.1, 1.2, 1.3, 1.4, 3.1, 3.2, 3.3 | - -**Responsibilities & Constraints** -- Writes the existing legacy header unchanged (magic, `checkpoint_lsn`, optional config-ref). -- Appends, in order: a format-version marker, a job count and per-job length-prefixed capnp `Job` - frames, a queue count and per-queue fixed `(name, status)` frames (no capnp — matches - `restore_queue_state`'s exact contract; queue definition is config-owned, out of boundary). - (Note: superseded by config-upgrade-snapshot — queue frames are now capnp `QueueAdminState`, - and host frames plus a config-bytes section were appended after the queue section; see Data - Models below.) -- Collects under `RTTI_LOCK_READ` on `c->jobmgr` (released before the queue pass begins); collects - under `RTTI_LOCK_READ` on `c->qmgr` for queues. The two table locks are never held - simultaneously (no new lock-ordering surface). **Within each table's held READ lock, also takes - each individual entity's own `RTTI_LOCK_READ`** (`lock(j, RTTI_LOCK_READ)` / - `lock(q, RTTI_LOCK_READ)`) around that one entity's encode/read call — this is required for - Requirement 1.2: the table lock alone does not exclude a concurrent per-entity writer (see - Architecture). -- Filters via the existing `job_should_retain` predicate so the checkpoint's notion of "live" - matches the existing GC retention policy (Requirement 1.1). Every queue present in `c->qmgr` is - included (a queue's mere table presence already means config says it exists; no retain filter - needed for a plain status word). -- Writes to the existing `.tmp` path, `fdatasync`s, then atomically renames to the final path — - identical pattern to today, just with more bytes. Any failure at any step (encode, write, - sync) leaves no file at the final path (Requirement 1.3); the tmp file is unlinked. - -**Contracts**: State [x] - -##### State Management -- State model: file on disk, atomically visible only at the final path once fully written. -- Persistence & consistency: `fdatasync` before rename, matching the existing sentinel-only - behavior — durability semantics are unchanged, only the payload grows. -- Concurrency strategy: two sequential table-read-lock sections (jobs, then queues), each with a - nested per-entity read lock around the individual encode/read call; no lock is held during the - disk write itself. - -**Implementation Notes** -- Integration: called from `persist_checkpoint`, unchanged call site. -- Validation: unit test asserts a checkpoint containing N jobs + M queues round-trips through - `load_snapshot` with all N+M records field-identical. -- Risks: enumeration+encode duration under `RTTI_LOCK_READ` blocks concurrent job/queue mutation - for that duration. Mitigated by checkpoints already running off the hot path on a configurable - `snap_interval_s` cadence; measure actual duration at a realistic job-table size during - implementation (see `research.md` Research Needed). - -#### `load_snapshot` (extended) - -| Field | Detail | -|-------|--------| -| Intent | Reconstruct entities from a checkpoint file and gate position advance on success | -| Requirements | 2.1, 2.2, 2.3, 2.4, 5.1, 5.2 | - -**Responsibilities & Constraints** -- Reads the existing legacy header exactly as today (unchanged). -- Determines format: if no bytes remain in the file after the legacy header, this is a - pre-feature checkpoint — do not advance `current_lsn`/`checkpoint_lsn` as "fully caught up"; - return success with `entities_restored=false` so the caller falls back to normal catch-up - (Requirement 2.4). This never crashes or misinterprets an old file (Requirement 5.1, 5.2) — a - short file is a valid, expected legacy shape, not a corruption. -- If entity frames are present, decode each job frame (capnp) and call `restore_job`; read each - queue frame's plain `(name, status)` pair and call `restore_queue_state(name, status)` directly - — in file order. On any decode or restore failure, abort immediately: return an error, leave - `current_lsn`/`checkpoint_lsn` unchanged (Requirement 2.3). -- Only after every entity restores successfully does the function advance - `current_lsn`/`checkpoint_lsn` to the checkpoint's position (Requirement 2.1, 2.2). - -**Contracts**: State [x] - -##### State Management -- State model: in-memory job/queue tables (`c->jobmgr`/`c->qmgr`), mutated in place by the - existing `restore_job`/`restore_queue_state` (unmodified — same functions WAL replay already - calls, so no new restore code path exists to diverge from replay's behavior). -- Persistence & consistency: `current_lsn`/`checkpoint_lsn` advance is the single commit point - for "this checkpoint is now fully applied" — atomic in the sense that no partial-entity state - is ever exposed as "caught up." -- Concurrency strategy: runs on the receiving member during catch-up, before it accepts write - traffic for those entities — no concurrent mutation expected at this point in the lifecycle - (same assumption WAL replay already makes at startup). - -**Public signature change** (`src/include/persist.h`): -```c -int persist_load_snapshot(persist_t* p, const char* path, uint64_t lsn, - bool* out_entities_restored); -``` -- Preconditions: `path` names a file written by `write_snapshot` (legacy or current format). -- Postconditions: on rc==0, `*out_entities_restored` reports whether this call actually restored - entity state (`true`) or only observed a legacy position marker (`false`); `current_lsn`/ - `checkpoint_lsn` reflect the checkpoint's position only when entities were fully restored or - the file was legacy-format (position-only, as today). On rc!=0, position is unchanged. -- Invariants: `*out_entities_restored` is only set when rc==0. - -**Implementation Notes** -- Integration: `ha_handle_snapshot_chunk` passes the new out-parameter through; on - `entities_restored==false` it logs (does not fail) since a legacy file is an expected rollout - state, not an error — the member simply continues through the existing incremental catch-up - path for the remaining history. -- Validation: unit test with a deliberately truncated/corrupted entity frame asserts - `current_lsn` unchanged and rc!=0; unit test with a legacy-sized file asserts rc==0 and - `entities_restored==false`. - -#### `rotate_snapshots` (extended) - -| Field | Detail | -|-------|--------| -| Intent | Delete old checkpoint files without breaking an in-flight transfer | -| Requirements | 3.1, 3.2, 3.3 | - -**Responsibilities & Constraints** -- Unchanged selection logic (keep `snap_keep_n` newest by filename/LSN). -- Before deleting a candidate file, checks whether its LSN matches any entry in - `ha->snapshot_inflight_lsn[]` (via `p->cluster->ha`, already reachable from `persist_t`); if so, - skip deletion for this rotation pass and retry on the next checkpoint cycle. - -**Contracts**: State [x] - -##### State Management -- State model: reads `ha_state_t`'s new array; does not itself own it (owned by the send-dispatch - path, see below). - -**Implementation Notes** -- Integration: minimal — one lookup added to the existing deletion loop. -- Validation: unit test simulates an in-flight LSN matching an otherwise-rotatable file and - asserts it survives rotation. - -### HA Replication Layer - -#### `snapshot_inflight_lsn[]` (new array) - -| Field | Detail | -|-------|--------| -| Intent | Record which checkpoint LSN each in-flight peer snapshot transfer references | -| Requirements | 3.1, 3.2, 3.3 | - -**Responsibilities & Constraints** -- `_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]` in `ha_state_t` (ha.h), declared - next to the existing `_Atomic int snapshot_inflight[MAX_HA_CANDIDATES]`. -- Set by `ha_snapshot_send_dispatch`/`ha_snapshot_do_send` at the same point the existing - `snapshot_inflight[peer_idx]` flag is set to 1 (before dispatch); cleared at the same points - the existing flag is cleared to 0 (on completion or CAS-release failure) — identical lifecycle, - one more field. - -**Contracts**: State [x] - -##### State Management -- State model: plain atomic array, same shape and lifecycle as the existing flag array it sits - beside. -- Concurrency strategy: no new locking — atomics only, matching the existing flag's discipline. - -**Implementation Notes** -- Integration: two write sites (set/clear), both already-existing call sites in `ha.c`. -- Validation: covered by the `rotate_snapshots` guard test above (integration-level: dispatch a - send, assert its LSN is protected, complete it, assert protection is released). - -## Data Models - -### Checkpoint File Layout (Physical) - -``` -[existing legacy header — unchanged, 16 or 57 bytes depending on config-ref presence] -[uint32_t format_version] -- absent (EOF here) => legacy, position-only file -[uint32_t job_count] -[job_count * { uint32_t payload_len; uint8_t capnp_Job_bytes[payload_len] }] -[uint32_t queue_count] -[queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }] -- (Note: this raw - -- shape is the - -- layout AS OF - -- this spec's - -- close; see below) -``` - -> **Note (superseded by config-upgrade-snapshot, see that spec's design.md):** the physical layout -> above is this spec's as-shipped shape only. config-upgrade-snapshot subsequently (1) migrated the -> queue frame from the raw `{name_len, name, status}` shape above to a length-prefixed capnp -> `QueueAdminState` frame (same shape as the job-frame loop, content unchanged: name + status), -> (2) appended a new host-frames section (`[uint32_t host_count][host_count * {payload_len, -> capnp HostAdminState bytes}]`) directly after the queue section, and (3) appended a trailing -> flat length-prefixed config-bytes blob after that (`[uint32_t cfg_len][cfg_len bytes]`, written -> on every checkpoint). `format_version` was bumped to `2` for this change. Current source: -> `write_snapshot`/`load_snapshot_entities` in `src/controller/persist.c`. - -- Job frames are length-prefixed capnp `Job` messages, matching the framing style already used - for WAL entries (`wal_entry_hdr_t` + payload) — no new schema/codegen for a one-off container. -- Queue frames carry only what `restore_queue_state` can consume: a name and a status word — no - capnp at all. A queue's *definition* (attrs, priority, thresholds, ACLs) is config-provisioned - and already replicated via `push_config_to_standbys`; encoding it here would duplicate data this - spec does not own and has no restore path for. (Note: the wire encoding of this same - name+status pair was later moved to capnp — see the note above.) -- A file ending exactly at the legacy header (no bytes remain) is unambiguously the old format — - the discriminator is "does more data exist," not a flag bit, so old files need zero - reinterpretation. -- `format_version` is present for forward evolution (a future layout bumps this value and - `load_snapshot` dispatches on it) without breaking this design's readers. (Note: superseded by - config-upgrade-snapshot's actual choice, see that spec's design.md — when it bumped - `format_version` to `2` it made `load_snapshot` hard-reject any non-matching version with - `-EINVAL` rather than dispatch, on the reasoning that a version mismatch must never be silently - masked as an expected legacy shape or real corruption could go undetected. The legacy - (no-`format_version`-present) detection this bullet's neighbor describes is unaffected.) - -### Consistency & Integrity -- Per-entity encode happens while both the enumerating table's `RTTI_LOCK_READ` AND that entity's - own nested `RTTI_LOCK_READ` are held (see Architecture) — the table lock alone is insufficient, - since entity mutation takes a separate per-entity lock; the nested per-entity lock is what - excludes a concurrent writer for that entity's encode, giving each entity a single consistent - snapshot of its own fields with no entity encoded mid-mutation. -- Cross-entity consistency (jobs relative to queues) is not required by Requirement 1.2, which - is scoped to "no record split between its own pre/post-checkpoint value" — jobs and queues are - independent aggregates for this feature's purposes, consistent with `collect_jobs`/ - `collect_queues` already treating them as separately-locked tables today. - -## Error Handling - -### Error Strategy -- **Write-side failure** (encode error, disk I/O error) at any point during `write_snapshot`: - unlink the tmp file, return a non-zero rc, which the existing `persist_checkpoint` caller - already treats as "skip trim and rotate this cycle" — no operational-condition self-termination - (per `src/controller/CLAUDE.md`'s controller-never-self-terminates invariant); the next - checkpoint cycle simply retries. -- **Read-side failure** (corrupt/truncated entity frame, decode error, restore error) during - `load_snapshot`: return a non-zero rc with position unchanged; the caller - (`ha_handle_snapshot_chunk`) logs the failure and does not advance `applied_lsn` — the member - stays fenced/behind and the existing catch-up retry mechanics (unchanged by this spec) bring it - current on a subsequent attempt. Consistent with the project's fail-loud-and-stay-alive - philosophy — never a self-terminate on this operational condition. -- **Legacy file encountered**: not an error. `entities_restored=false` with rc==0 is the expected, - handled outcome for a file predating this feature. - -### Monitoring -- Existing `WARN`/`ERROR` log lines at each new failure point (matching the codebase's existing - logging conventions in `persist.c`/`ha.c`); no new metric is introduced by this design, but - Requirement 4.3's "observable to an operator" is satisfied by the existing `walcompare`/ - `waldump` tooling already used this session to diagnose divergence — confirm during - implementation that it reports on entity-snapshot-derived state, not only WAL-visible LSNs. - -## Testing Strategy - -- **Unit — `write_snapshot`/`load_snapshot` round-trip**: a checkpoint with N jobs and M queues, - loaded back, yields N job records field-identical to the originals and M queue statuses - correctly reapplied via `restore_queue_state` (Requirement 1.1, 1.2, 2.1, 2.2). -- **Unit — queue status frame round-trip**: a queue's `(name, status)` pair written and read back - matches exactly, and `restore_queue_state` is invoked with those exact values (Requirement 1.1). -- **Unit — concurrent-mutation-during-snapshot consistency**: a job actively transitioning state - (e.g. via `mark_job_start`) on a concurrent thread while `write_snapshot`'s enumeration walks it - must be captured as either the full pre- or full post-transition value, never a torn mix of - fields — proves the nested per-entity lock, not just the table lock, is doing the work - (Requirement 1.2). -- **Unit — partial write discarded**: a simulated encode/write failure partway through leaves no - file at the final path (Requirement 1.3). -- **Unit — partial restore does not advance position**: a deliberately corrupted entity frame - causes `load_snapshot` to return an error with `current_lsn`/`checkpoint_lsn` unchanged - (Requirement 2.3). -- **Unit — legacy file handling**: a file sized to end exactly at the legacy header yields rc==0, - `entities_restored=false`, no crash (Requirement 2.4, 5.1, 5.2). -- **Unit — rotation guard**: a file matching an in-flight LSN survives `rotate_snapshots` even - when it would otherwise fall outside `snap_keep_n` (Requirement 3.1, 3.2, 3.3). -- **Integration — chaotic catch-up soak**: extend the existing `/xce-test-ha --chaotic` soak - scenario so at least one round forces a `SEND_SNAPSHOT` catch-up (empty or far-behind member); - assert `committed_divergence=0` and `committed_job_loss=0` post-round (Requirement 4.1, 4.2) — - this is the direct acceptance test for the residual soak failure this spec exists to close. -- **Integration — rolling upgrade simulation**: a pre-feature-format checkpoint file present on - disk, loaded by the new binary, does not crash and correctly falls back to full catch-up - rather than reporting false-positive completeness (Requirement 5.1, 5.2). diff --git a/.kiro/specs/ha-entity-snapshots/requirements.md b/.kiro/specs/ha-entity-snapshots/requirements.md deleted file mode 100644 index dd58310b..00000000 --- a/.kiro/specs/ha-entity-snapshots/requirements.md +++ /dev/null @@ -1,64 +0,0 @@ -# Requirements Document - -## Project Description (Input) -ha-entity-snapshots: Job/queue entity-bearing WAL snapshots so a rejoining or empty-restart node's catch-up gap (task 5.10a / raft-ha-core-safety, bug-795/828/851 family) closes without permanent committed-prefix loss. Today write_snapshot (persist.c ~1051) writes a 16-byte sentinel with ZERO job/queue entities and load_snapshot only advances the LSN watermark past entries it never received (restores nothing); trim_wal_segments deletes the WAL prefix below a checkpoint. A member that rejoins empty or behind a trimmed prefix is streamed only [watermark..committed] with no snapshot backfill for [1..watermark-1] — a permanent gap (observed live: 10-round chaotic soak shows committed_divergence>0 every round from this catch-up lag). Scope: write_snapshot Cap'n-Proto-serializes all live job/queue entities at checkpoint_lsn; load_snapshot deserializes + restores them BEFORE advancing current_lsn; trim_wal_segments must not trim below the LSN covered by a durable entity-bearing snapshot; install-snapshot-before-stream on the catch-up receive side. Full investigation at .kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md (VERDICT section) and .kiro/specs/raft-ha-core-safety/tasks.md task 5.10(a). - -## Introduction -XCE cluster members that rejoin the cluster empty, or that fall behind the oldest history the cluster still retains directly, currently receive no way to recover the committed job and queue history that existed before their position. The periodic checkpoint the cluster produces carries only a position marker, not the actual job/queue data, and older history is deleted once a checkpoint is taken. As a result, a member that catches up this way ends up reporting itself as caught up while permanently missing part of the committed record — a gap that persists indefinitely and that surfaces as data divergence across the cluster. This feature makes the checkpoint a complete, restorable copy of committed job and queue state, and ensures history is never discarded before that copy exists, so that every member — however it rejoins — ends up holding the same complete committed state as the rest of the cluster. - -## Boundary Context -- **In scope**: - - The content of a checkpoint: it must represent the actual job and queue records committed as of that point, not merely a position. - - Restoring a checkpoint: reconstructing the actual job and queue state it contains, not merely advancing a position. - - Coordinating history retention so that log data is never discarded before its content is captured in a checkpoint that has been confirmed complete and durable. - - The end-to-end outcome for a member that rejoins empty or significantly behind: it must reach the same complete committed job/queue state as an already-consistent member. - - Safe behavior when a member encounters a checkpoint produced before this capability existed (rolling upgrade). -- **Out of scope**: - - Worker/compute-node registration and state — workers re-establish their state via the existing join protocol, independent of this feature. - - The mechanics of transferring a checkpoint to a peer over the network — that transport already exists and is unchanged by this feature. - - Cluster configuration document snapshotting/versioning — owned by the `config-upgrade-snapshot` feature. - - Election, vote-freshness, and term-safety behavior — owned by `raft-ha-core-safety` / `ha-term-safety`. -- **Adjacent expectations**: This feature assumes the existing catch-up decision logic (choosing between an incremental replay and a full checkpoint transfer for a member whose position is below the cluster's retained history) continues to trigger correctly and is unchanged by this feature; what changes is that the checkpoint it delivers is now complete, and the member's outcome after catch-up is now correct. - -## Requirements - -### Requirement 1: Complete entity-bearing checkpoint content -**Objective:** As a cluster operator, I want a checkpoint to contain the full set of committed job and queue records as of that point, so that a member restoring it receives actual data rather than only a position marker. - -#### Acceptance Criteria -1. When the system produces a checkpoint at a committed position, the Persistence Service shall include in the checkpoint every job and queue record that is live or retained under the existing retention policy as of that position. -2. The Persistence Service shall produce a checkpoint that reflects a single, internally consistent point-in-time view of job and queue state, with no record split between its pre-checkpoint and post-checkpoint value. -3. If a checkpoint cannot be produced completely (for example, an I/O failure partway through), then the Persistence Service shall discard the incomplete checkpoint and shall not present it as a valid restore point. -4. The Persistence Service shall record, as part of the checkpoint, the committed position the contained entity state corresponds to. - -### Requirement 2: Full restoration from a checkpoint -**Objective:** As a rejoining or newly provisioned cluster member, I want loading a checkpoint to reconstruct the actual job and queue state it contains, so that I hold complete data rather than an empty state with an advanced position. - -#### Acceptance Criteria -1. When a member loads a checkpoint, the Persistence Service shall reconstruct every job and queue record contained in the checkpoint into the member's live state before advancing the member's committed position. -2. The Persistence Service shall consider the member's committed position advanced to the checkpoint's position only after all contained records have been reconstructed. -3. If reconstruction fails partway through, then the Persistence Service shall not advance the member's committed position and shall report the failure rather than silently proceeding with partial state. -4. Where a loaded checkpoint carries no entity state (a checkpoint produced before this capability existed), the Persistence Service shall treat that checkpoint as insufficient on its own and shall require the member to obtain the missing history through the normal catch-up path rather than treating the checkpoint as complete. - -### Requirement 3: Retention never outruns restorability -**Objective:** As a cluster operator, I want history retention (discarding older log data) to never outrun the point at which that history is fully captured in a restorable checkpoint, so that a rejoining member is never left with a permanent, unrecoverable gap. - -#### Acceptance Criteria -1. While a portion of history has not yet been captured in a checkpoint whose entity content is confirmed durable, the Persistence Service shall retain the corresponding log data. -2. The Persistence Service shall discard log data only when the history it covers is fully represented in a durable, entity-bearing checkpoint at or beyond that point. -3. If retention is requested for a position not yet covered by a durable, entity-bearing checkpoint, then the Persistence Service shall reject or defer that retention rather than discarding the data. - -### Requirement 4: End-to-end catch-up correctness for a behind or empty member -**Objective:** As a cluster operator, I want a member that rejoins empty or significantly behind to end up holding the same committed job and queue state as the rest of the cluster, so that a failover onto that member never serves incomplete or divergent data. - -#### Acceptance Criteria -1. When a member's position is behind the oldest history the cluster still retains directly, the system shall bring that member to full committed-consistent job and queue state through a checkpoint transfer followed by any remaining incremental history. -2. After a member completes catch-up, whether through a checkpoint, incremental replay, or both, the system shall hold the same committed job and queue records as an already-consistent member, with no permanent gap. -3. The system shall make the completeness, or incompleteness, of a member's catch-up observable to an operator through existing status and comparison tooling, rather than reporting a member as fully caught up when it is not. - -### Requirement 5: Safe rollout against existing deployments -**Objective:** As a cluster operator upgrading an existing deployment, I want the new checkpoint format to coexist safely with checkpoints produced by the prior version, so that upgrading never corrupts state or crashes a member. - -#### Acceptance Criteria -1. When a member reads a checkpoint file, the Persistence Service shall determine whether it is an entity-bearing checkpoint or a prior-format, position-only checkpoint before deciding how to use it. -2. The Persistence Service shall never crash, corrupt state, or fabricate job or queue records when reading a prior-format checkpoint. diff --git a/.kiro/specs/ha-entity-snapshots/research.md b/.kiro/specs/ha-entity-snapshots/research.md deleted file mode 100644 index 42478e5b..00000000 --- a/.kiro/specs/ha-entity-snapshots/research.md +++ /dev/null @@ -1,205 +0,0 @@ -# Gap Analysis: ha-entity-snapshots - -## Summary - -This feature is **substantially cheaper than it looks**. Every low-level primitive it needs -already exists in the codebase for an unrelated purpose (WAL replay reconstruction) or is -auto-generated by the existing Cap'n Proto schema. The work is almost entirely **wiring**: -enumerate live entities → serialize into the snapshot file → on load, deserialize and call the -existing restore functions → extend the existing trim-floor guard. No new capnp schema struct -is strictly required (a length-prefixed sequence of existing per-entity blobs works), though a -dedicated `Snapshot` struct would be cleaner. The main open design question is write-side -concurrency: the "proper" mechanism (zarray copy-on-write) exists and is tested, but was never -wired into the job/queue backing store (`kvmgr`) — a deliberate deferral ("wired in P0-2"), not -a bug. This is the one real judgment call for the design phase. - -## Requirement-to-Asset Map - -| Requirement | Existing Asset | Gap | -|---|---|---| -| R1.1 include every live/retained job+queue record | `collect_jobs`/`collect_queues` (dataex.c:215-330, lock READ → `iter_make`/`iter_next` → unlock); `job_should_retain`/`worker_should_retain` (persist.c:3013+) already implement the retention predicate | **Missing**: nothing enumerates for the *snapshot* purpose today — `write_snapshot` (persist.c ~1477-1519) writes only a 57-byte header. Wiring, not new logic. | -| R1.1 per-entity serialization | `capnp_encode_job` (persist.c:45), `capnp_encode_worker` (persist.c:68); capnp codegen for `QueueConf`/`xce_queue_t` — `encode_QueueConf`, `write_QueueConf`, `new_QueueConf`, `new_QueueConf_list` all already generated (controller.capnp.h) | **Missing**: a `capnp_encode_queue` wrapper — trivial, mirrors `capnp_encode_job` line-for-line (~15 lines). | -| R1.2 internally consistent point-in-time view | zarray COW (`src/lib/zarray.c` `xzarray_freeze`/`xzarray_thaw`, tested in `test_zarray_cow.c`) | **Constraint**: COW exists but is **NOT wired into `kvmgr`** (the hash table backing `c->jobmgr`/`c->qmgr`) — `kvmgr.c:297-298` explicitly stubs `vfreeze`/`vthaw` to NULL with a "wired in P0-2" comment. Consistency must come from a `RTTI_LOCK_READ` held across the full enumeration+encode instead (same lock `collect_jobs` already takes, just held longer). This is the one real design decision — see Options below. | -| R1.3 discard incomplete checkpoint | `write_snapshot`'s existing tmp-file + atomic `rename` pattern (persist.c ~1477+, same pattern as `ha_handle_snapshot_chunk`'s `snap.part` → final rename, ha.c:2589-2594) | **None** — extend the existing tmp+rename-on-success pattern to the entity payload; a partial write never reaches the final path. | -| R1.4 record committed position | Already present (`checkpoint_lsn` in the existing sentinel header, persist.c) | **None.** | -| R2.1/2.2 restore entities before advancing position | `restore_job` (job.c:96, called from WAL replay persist.c:1915) and `restore_queue_state` (queue.c:208, called persist.c:1945) — **both already exist and are proven** (used every restart via `apply_wal_entry`) | **Missing**: `load_snapshot` (persist.c:1811-1847) currently sets `current_lsn`/`checkpoint_lsn` directly (line ~1835) with no entity step. Insert a decode-and-`restore_job`/`restore_queue_state` loop **before** that assignment, reusing the exact WAL-replay call. | -| R2.3 failed reconstruction must not advance position | N/A — new logic, but trivial given restore functions already return rc | **Missing**: gate the `current_lsn`/`checkpoint_lsn` store on the restore loop's success (straightforward given R2.1/2.2's insert point). | -| R2.4 pre-feature checkpoint treated as insufficient | N/A | **Missing**: a format/version discriminator. Cheapest: if the file is exactly the legacy 57-byte sentinel size with no trailing entity payload, treat as position-only (do not claim completeness); a new format adds a marker/count after the existing header. | -| R3 retention never outruns restorability | `repl_trim_lsn` (repl.c:650-683) + the bug-1147 guard in `trim_wal_segments` (persist.c:1615, floor = min(checkpoint_lsn, every active standby's `acked_lsn`, disconnected-slot `last_disconnected_lsn`)) — **this is an EXISTING, working guard**, confirmed by its own comment: "does NOT fix the empty/far-behind join case... requires entity-bearing snapshots (separate spec)" — i.e. this spec is the one it names. | **Extension, not new**: add "a standby is currently mid-install of snapshot at LSN X" as another floor input into `repl_trim_lsn`'s `min_lsn` computation — same shape as the existing acked_lsn/disconnected-slot terms. | -| R4 end-to-end catch-up correctness | `ha_evaluate_repl_action` (ha.c:2644, unchanged by this spec) → `ha_snapshot_send_dispatch`/`ha_snapshot_do_send` (ha.c ~2788+, unchanged) → wire opcode `OPCode_snapshotTransfer` → `handle_snapshot_transfer` (cluster.c:197-229) → **`ha_handle_snapshot_chunk`** (ha.c:2529-2609): assembles `snap.part`, renames to final path, calls `persist_load_snapshot`, then sets `ha->applied_lsn`. | **Insertion point identified, not missing infra**: the entity-restore work belongs entirely inside `persist_load_snapshot`/`load_snapshot` (R2's gap). Once that returns correctly, `ha_handle_snapshot_chunk`'s existing "only advance `applied_lsn` after `persist_load_snapshot` succeeds" structure already gives R4's ordering for free. | -| R4.3 operator-observable catch-up completeness | `walcompare`/`waldump` tooling already used this session to diagnose divergence live | **None new required**; existing tooling already surfaces divergence. Confirm it also reports on entity-snapshot-derived state, not just WAL-visible LSNs (verify in design). | -| R5 safe rollout / old-format checkpoints | `rotate_snapshots` already discriminates by filename pattern (`*.capnp`) | **Missing**: the actual content-based old-vs-new format check (R2.4's gap, restated here for the read path). | - -## Implementation Approach Options - -### Option A: Extend existing components in place (RECOMMENDED) -- Extend `write_snapshot`/`load_snapshot` (persist.c) directly: after the existing sentinel - header, append a length-prefixed sequence of capnp-encoded `Job` and `QueueConf` blobs - (reusing `capnp_encode_job`, a new trivial `capnp_encode_queue`, and on load `decode_Job`/ - `decode_QueueConf` + `restore_job`/`restore_queue_state`). -- Write-side consistency: hold `RTTI_LOCK_READ` on `c->jobmgr`/`c->qmgr` across the full - enumerate+encode (mirrors `collect_jobs`'s existing lock scope, just held for the whole - snapshot instead of one collection pass). Do the actual disk write from an in-memory buffer - built while holding the lock, then release the lock before `fdatasync`+rename — so file I/O - latency does not extend the lock hold. -- Extend `repl_trim_lsn` with one more floor term (in-flight snapshot-install LSN per standby). -- ✅ Every serialization/restore primitive already exists and is proven (WAL replay uses the - same `restore_job`/`restore_queue_state`) — minimal new code, maximum reuse of tested paths. -- ✅ No new wire opcode, no new capnp schema struct strictly required. -- ❌ Holding a read lock across a full job-table walk blocks concurrent job mutations for the - checkpoint's duration — needs a bound (checkpoint interval is already configurable; - `snap_interval_s` exists) and a measurement of walk cost at realistic job-table sizes. - -### Option B: Wire zarray COW into kvmgr first, then build the snapshot on top -- Complete the deferred "P0-2" work (wire `xzarray_freeze`/`xzarray_thaw` into `kvmgr`), then - `write_snapshot` freezes a shadow, walks it lock-free, and thaws — no read-lock hold during - the walk. -- ✅ Removes the lock-hold concern entirely; likely benefits other future features too. -- ❌ Meaningfully larger and touches a shared, load-bearing data structure (`kvmgr` backs both - job and queue tables and possibly other managers) used cluster-wide — higher blast radius and - risk for a benefit (lock-free enumeration) that may not be needed if Option A's lock-hold - duration proves acceptable in practice. -- ❌ It was deliberately deferred once already; reviving it is its own scoped effort, arguably a - separate spec/prerequisite rather than embedded in this one. - -### Option C: Hybrid — ship Option A now, leave a documented seam for Option B later -- Build the snapshot feature entirely per Option A, but structure the enumeration behind a - single seam function (e.g. `persist_collect_snapshot_entities`) so that if kvmgr COW is wired - later, only that one function's internals change (lock+iterate → freeze+iterate+thaw), - nothing else in the snapshot read/write/trim/catch-up path needs to change. -- ✅ Ships the actual bug fix now; keeps the door open without committing to the larger - refactor's risk today. -- ❌ Slightly more upfront interface discipline than just inlining the lock+iterate call. - -## Effort & Risk - -- **Effort: M (3-7 days).** All hard primitives (encode/decode/restore) pre-exist and are - proven; the work is wiring four call sites (write, load, trim, and the format-discriminator) - plus tests. Design and TDD discipline (per `.kiro/steering/testing.md`) will take real time - given this is HA/persistence-critical code, but there is no unknown technology. -- **Risk: Medium.** Known patterns throughout (Option A), but this is exactly the kind of - code the `raft-ha-core-safety` family (bug-795/828/851/1306) has shown is easy to get subtly - wrong under concurrent chaos — the actual risk is in getting R2.3 (failed-restore-must-not- - advance-position) and R3 (trim-floor extension) correct under a live chaotic soak, not in - writing the serialization code itself. Recommend the same validation bar the recent session's - fixes used: unit RED/GREEN + a chaotic multi-round soak showing `committed_divergence=0`. - -## Research Needed (carry to design phase) - -1. **Snapshot file layout**: a single new capnp `Snapshot { jobs: List(Job); queues: - List(QueueConf); ... }` struct (cleaner, one `capn` init/write) vs. keeping the existing raw - header + hand-rolled length-prefixed blob sequence (smaller diff, matches the current - sentinel's hand-rolled style). Design should pick one; either is buildable on existing codegen. -2. **Lock-hold duration measurement**: before committing to Option A's approach, get an order- - of-magnitude estimate or measurement of `collect_jobs`+encode cost at a realistic job-table - size, to confirm the read-lock hold is acceptably short (checkpoints already run on a - `snap_interval_s` cadence via `delay_task`, off the hot request path). -3. **Format discriminator mechanics**: exact byte-level marker to distinguish old 57-byte - sentinel-only files from new entity-bearing files without breaking `rotate_snapshots`'s - existing filename-based selection. -4. **Trim-floor extension bookkeeping**: where to track "standby X mid-install of snapshot LSN - Y" — likely alongside the existing `standby_conn_t`/disconnected-slot state already used by - `repl_trim_lsn`, but the exact struct field and its lifecycle (set on `SEND_SNAPSHOT` - dispatch, cleared on the receiver's ack) needs design-phase detail. -5. **Interaction with `ha_snapshot_offload_enabled()` slow-lane path** (ha.c - `ha_snapshot_send_dispatch`): confirm the entity payload size doesn't change the offload - flag-ON vs flag-OFF dispatch decision or chunking assumptions already in place for - `ha_handle_snapshot_chunk`'s multi-chunk reassembly. - -## Recommendation - -**Option A** (extend `write_snapshot`/`load_snapshot`/`trim_wal_segments` in place, structured -per **Option C**'s seam discipline) for the design phase. Do not pursue Option B (kvmgr COW -wiring) as part of this spec — it is a legitimate future improvement but was already -deliberately deferred once, and this spec's actual requirements (R1-R5) do not need it: a -bounded read-lock hold during a periodic, off-hot-path checkpoint is a reasonable cost, and it -keeps this spec's blast radius to `persist.c`/`ha.c`/`repl.c`, matching the existing bug-1147 -guard's pattern rather than touching the shared kvmgr. - ---- - -## Design Phase — Synthesis & Refinements (2026-07-01) - -### Key refinement over the gap analysis -The gap analysis (above) assumed `trim_wal_segments`/`repl_trim_lsn` needed a new floor input -keyed on "a standby is currently mid-install of a snapshot." Closer reading of -`persist_checkpoint` (persist.c:1763) shows this is **already handled**: `write_snapshot` runs -before `trim_wal_segments`, and a non-zero `write_snapshot` return skips both `trim_wal_segments` -and `rotate_snapshots` via the existing `goto done`. Once `write_snapshot` only returns 0 when -the entity payload is fully and durably written (this spec's Requirement 1.3), that existing -sequencing alone satisfies Requirement 3 — **no new code needed in `trim_wal_segments`/ -`repl_trim_lsn`**. - -The one retention hazard that genuinely has no existing guard is **`rotate_snapshots`** deleting -a checkpoint *file* while a standby is still mid-transfer reading it over the wire (a checkpoint -cadence faster than a slow transfer). This is the actual new mechanism this design adds: a -`snapshot_inflight_lsn[]` array (mirroring the existing `snapshot_inflight[]` flag array in -`ha_state_t`) that `rotate_snapshots` consults before deleting a candidate file. - -### Synthesis: Generalization -Requirements 1 and 2 both need "enumerate a live entity of kind K, encode it, and (for restore) -decode+apply it." Rather than writing separate bespoke job-snapshot and queue-snapshot code -paths, the design generalizes the *interface* (a small per-kind {collect, encode, decode+restore} -triple) while keeping the *implementation* scoped to exactly the two kinds Requirement 1 names -(job, queue) — not built out for hypothetical future entity kinds (e.g. workers), consistent with -the requirement's explicit boundary. - -### Synthesis: Build vs. Adopt -Adopt, not build: the project's own Cap'n Proto schema and generated codecs for `Job`/ -`QueueConf` already exist and are already proven via WAL replay's `restore_job`/ -`restore_queue_state`. No external library, no new schema struct, no new wire protocol — this -spec is pure reuse of already-solved, already-tested primitives. - -### Synthesis: Simplification -Rejected a new top-level capnp `Snapshot` message struct (would require a schema/build change for -a one-off container) in favor of a hand-rolled length-prefixed frame sequence — matching the -project's own existing WAL entry framing style (`wal_entry_hdr_t` + payload) and avoiding a new -abstraction layer for something with exactly one caller. Rejected reviving the deferred zarray -COW wiring (Option B) as unnecessary for this spec's scope — a bounded `RTTI_LOCK_READ` hold -during an off-hot-path, periodic checkpoint is a reasonable cost that keeps this spec's blast -radius to `persist.c`/`ha.c`, matching the existing bug-1147 guard's precedent rather than -touching the shared `kvmgr`. - -### Design Decisions Log -- **On-disk format**: legacy header (unchanged) + `format_version` + `job_count` + length-prefixed - `Job` frames + `queue_count` + length-prefixed `QueueConf` frames. Legacy-file detection is - "no bytes remain after the legacy header" — no explicit flag bit needed. -- **Consistency boundary**: per-entity, not cross-entity — jobs and queues are independently - locked/encoded in two sequential `RTTI_LOCK_READ` sections, never held simultaneously. -- **API change**: `persist_load_snapshot` gains an `out_entities_restored` out-parameter so the - caller can distinguish "fully restored" from "legacy position-only" without treating the latter - as an error. -- **New guard location**: `rotate_snapshots` (not `trim_wal_segments`), backed by a new - `snapshot_inflight_lsn[MAX_HA_CANDIDATES]` array in `ha_state_t`, lifecycle-mirrored to the - existing `snapshot_inflight[]` flag. - ---- - -## Design Validation Pass — Corrections (2026-07-01) - -`/kiro-validate-design` found two real defects in the first design draft, both confirmed against -code and fixed in `design.md`: - -1. **Queue entity shape was wrong.** The draft planned to serialize a full `QueueConf` capnp - entity per queue, but `restore_queue_state(cluster_t*, const char* name, uint32_t status)` - (queue.c:208) — the only existing queue-restore function — takes just a name and status word; - its own comment states queues are config-built at startup and only admin status changes are - WAL-tracked. Confirmed independently: queue *definitions* are already replicated via - `push_config_to_standbys` (config_repl.c), a separate, already-committed, majority-acked - channel — entirely out of this spec's boundary (user confirmation: "the basic configuration - is not lived in cluster state snapshot as we have independent configuration"). **Fix**: the - queue-side snapshot payload is now a plain `(name, status)` record, no capnp, restored via the - unchanged `restore_queue_state`. This *simplifies* the design (no queue capnp encoder needed - at all). -2. **The table-lock-only consistency argument was false.** The original Architecture section - claimed `RTTI_LOCK_READ` on `c->jobmgr`/`c->qmgr` (the table lock) excludes concurrent entity - mutation. Verified false: `restore_queue_state` itself takes a *separate* per-entity - `lock(q, RTTI_LOCK_WRITE)`, and job state-transition sites (job.c:1006/1093/1137) take - `lock(j, RTTI_LOCK_WRITE)` — a distinct lock object from the table lock. **Fix**: the - entity-collection helper now nests each entity's own `RTTI_LOCK_READ` inside the table walk, - around that entity's individual encode call — mirroring `restore_queue_state`'s own - nested-lock pattern. Added a new test item (concurrent-mutation-during-snapshot consistency) - to prove this directly. - -Both fixes are local repairs to the write-side collection logic and the queue payload format — -the overall Option-A approach, boundary, and file structure plan are unchanged. diff --git a/.kiro/specs/ha-entity-snapshots/spec.json b/.kiro/specs/ha-entity-snapshots/spec.json deleted file mode 100644 index 85390fcd..00000000 --- a/.kiro/specs/ha-entity-snapshots/spec.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "feature_name": "ha-entity-snapshots", - "created_at": "2026-07-01T03:32:11Z", - "updated_at": "2026-07-18T00:35:32Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T00:35:32Z", - "note": "First post-implementation validation run -- all 21 tasks (1.1-8.1) were already [x] with no prior validate_impl record and phase stuck at tasks-generated (administrative oversight). Pre-check on task 7.1: tasks.md carries a 2026-07-01 'RESOLVED' note and git history (3fab291 block -> f3c86a6/9f4292d/b9ad018/1439e26 resolve) confirms the 2026-07-02-era 'pending upstream snapshot-boundary recon' blocker was genuinely closed, not just checkbox-checked -- it was owned by raft-ha-core-safety task 3 (Snapshot-boundary reconciliation, [x]) / task 5.10a (CLOSED), whose investigation found and fixed four real bugs (walcompare per-member checkpoint-floor false-positive, a delaymgr seconds/ms deadline-type-confusion starvation bug, a hardcoded 64MB WAL-segment-rotation stall, and a netio.h INVOKE_MSG_HANDLER UAF during concurrent snapshot-transfer+election) culminating in a live `--force-snapshot` soak: committed_divergence=0 committed_job_loss=0 result=PASS with a confirmed real 'HA: snapshot' install log line, plus a clean 3-round --chaotic regression. Proceeded to full validation on that basis. Full controller_unit_test suite: meson Ok:3/Fail:0; ANSI-stripped authoritative count (project convention, since meson's summary can mask Unity sub-group failures) = 962 PASS / 0 FAIL, including all 9 dedicated test_persist_snapshot_entities.c cases green. TBD/secrets grep clean (the one 'XXXXXX' hit is an mktemp template, not a TODO; ha.c:1844's stale TODO is a pre-existing, unrelated, already-self-flagged ha-term-safety item, not introduced here). File Structure Plan matches actual files exactly (persist.c, persist.h, ha.h, ha.c, test_persist_snapshot_entities.c) -- no drift, no boundary spillover into any Out-of-Boundary area (wire transport, catch-up decision logic, repl_trim_lsn, worker/join, config-doc snapshotting all untouched). Cross-task integration verified by direct code read: the `out_entities_restored` out-parameter flows persist.h's declaration -> load_snapshot's real implementation -> both consumers (ha.c:3112-3147 wire receive path / task 6.1, persist.c:4277-4285 local boot recovery / task 8.1); `snapshot_inflight_lsn[]` is wired ha.h declaration -> ha.c's 4 set/clear lifecycle sites -> persist.c:2157's rotate_snapshots consultation, matching design.md's Architecture Pattern & Boundary Map diagram exactly. `persist_checkpoint` (persist.c:2262) confirmed to gate `trim_wal_segments`/`rotate_snapshots` on `write_snapshot`'s rc, satisfying Requirement 3 without a new trim-floor variable exactly as design.md's Architecture Analysis argued. All 12 requirements.md line items (1.1-5.2) map through design.md's Requirements Traceability table to a completed task, no gaps. The pre-existing `restore_job` decode leak flagged in tasks.md's Implementation Notes (2.2) is confirmed fixed in task 5.2's commit (9fdb677): `free_Job(&j)` now runs on both the early-EEXIST-return and normal-completion paths in job.c.", - "known_limitations": [ - "Non-Goal (by design, documented in design.md): zarray copy-on-write entity enumeration is not wired into kvmgr; point-in-time checkpoint consistency instead relies on table-level + per-entity RTTI_LOCK_READ/WRITE nesting during the collection walk. Correct as implemented, just not the eventual lock-free approach research.md's Option B describes.", - "tasks.md Implementation Note (5.2): restore_queue_state's -ENOENT ('queue not yet built from config') aborts the ENTIRE checkpoint restore, including already-successfully-restored jobs earlier in the same file. This is design-compliant (Requirement 2.3 is unconditional: any restore failure leaves position unchanged) but is a real residual availability risk -- a benign config-lag on the destination member could make catch-up fail outright rather than skip-and-continue. Flagged as a WATCH-ITEM in the spec's own notes if a future catch-up round is observed to stall.", - "Test-only, non-production: a pre-existing ASan leak in the test fixture itself (snapent_make -> makeInstance/makeJob/makeQueue/xkvmgr_add, ~18KB/113 allocations across the whole test file), masked by this suite's standing ASAN_OPTIONS=detect_leaks=0 convention; and a minor verbatim-duplicated 'skip past job section' loop between snapent_read_checkpoint_jobs and snapent_read_checkpoint_queues. Neither blocks any requirement; both are called out in tasks.md's own Implementation Notes as candidates for a future fixture cleanup pass if the test file keeps growing." - ] - } -} diff --git a/.kiro/specs/ha-entity-snapshots/tasks.md b/.kiro/specs/ha-entity-snapshots/tasks.md deleted file mode 100644 index 3ed1074a..00000000 --- a/.kiro/specs/ha-entity-snapshots/tasks.md +++ /dev/null @@ -1,166 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: entity-snapshot test harness -- [x] 1.1 Build a reusable test fixture for a persist_t backed by a populated job/queue table - - Model on the existing `make_hook_test_persist` / `cbrp_setup` fixture patterns in the test - suite - - Fixture can seed N jobs (varying state) and M queues (varying status) into the cluster's - job/queue tables - - Observable: a new test file exists and compiles, exposing a helper that produces a ready - persist_t+cluster_t with seeded entities - - _Requirements: 1.1, 2.1_ - -- [x] 2. Core: entity-bearing checkpoint write path — jobs -- [x] 2.1 RED: job checkpoint content, consistency, and failure-discard tests - - Assert a checkpoint containing N jobs round-trips through load with all N job records - field-identical - - Assert a job actively transitioning state on a concurrent thread during the checkpoint walk - is captured as a single consistent pre- or post-transition value, never torn - - Assert a simulated write/encode failure partway through leaves no file at the final - checkpoint path — the same discard behavior that (per design) is what already makes the - existing trim/rotate sequencing satisfy the retention-safety requirement - - Observable: new tests compile and fail (RED) against the current sentinel-only write_snapshot - - _Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3_ - - _Boundary: Persistence Layer (write_snapshot)_ -- [x] 2.2 Collect and encode live jobs into the checkpoint under per-entity consistency - - Enumerate live jobs under the existing retention policy, take each job's own read lock - nested inside the table walk, encode via the existing job serializer, and append to the - checkpoint after the current header - - Any failure at any step discards the temp file and leaves no file at the final path - - Observable: the RED tests from 2.1 pass - - _Requirements: 1.1, 1.2, 1.3, 1.4_ - - _Boundary: Persistence Layer (write_snapshot)_ - -- [x] 3. Core: entity-bearing checkpoint write path — queues -- [x] 3.1 RED: queue admin-status round-trip test - - Assert a queue's (name, status) pair written into the checkpoint and read back matches - exactly - - Observable: new test compiles and fails (RED) - - _Requirements: 1.1_ - - _Boundary: Persistence Layer (write_snapshot)_ - - _Depends: 2.2_ -- [x] 3.2 Collect queue admin-status into the checkpoint - - Enumerate every queue present in the live table (its table presence already implies - config-provisioned existence), take each queue's own read lock nested inside the table walk, - append its name+status as a plain record (no serialization schema needed — matches the - existing restore function's exact contract) - - Observable: the RED test from 3.1 passes; a checkpoint containing M queues restores M - correct statuses - - _Requirements: 1.1, 1.2_ - - _Boundary: Persistence Layer (write_snapshot)_ - -- [x] 4. Core: protect an in-flight checkpoint transfer from rotation -- [x] 4.1 (P) Track which checkpoint position each in-flight peer transfer references - - Add a per-peer tracking array alongside the existing in-flight-transfer flag; set it when a - transfer begins, clear it when the transfer completes - - Observable: a unit test confirms the tracked position is set before a transfer's first byte - goes out and cleared after completion - - _Requirements: 3.1, 3.2, 3.3_ - - _Boundary: HA Replication Layer_ -- [x] 4.2 RED: rotation-guard test - - Assert a checkpoint file whose position matches an in-flight transfer survives file rotation - even when it would otherwise be rotated out - - Observable: new test compiles and fails (RED) against the current rotation logic - - _Requirements: 3.1, 3.2, 3.3_ - - _Depends: 4.1_ -- [x] 4.3 Skip rotating out a checkpoint file that is currently mid-transfer - - Extend the existing file-rotation pass to check the in-flight tracking before deleting a - candidate file - - Observable: the RED test from 4.2 passes - - _Requirements: 3.1, 3.2, 3.3_ - -- [x] 5. Core: entity restoration on load -- [x] 5.1 RED: restore-before-advance, failure-safety, and legacy-format tests - - Assert loading a checkpoint with N jobs + M queues restores all of them before the member's - position advances - - Assert a deliberately corrupted entity record causes the load to fail with the member's - position left unchanged - - Assert a checkpoint file predating this feature (position marker only) is accepted without - crashing, reports itself as position-only, and does not falsely claim full restoration - - Observable: new tests compile and fail (RED) against the current watermark-only load - - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_ - - _Boundary: Persistence Layer (load_snapshot)_ - - _Depends: 2.2, 3.2_ -- [x] 5.2 Restore entities before advancing the member's position, with legacy-format detection - - Detect whether a checkpoint carries entity content or is a legacy position-only file - - Restore every job then every queue status via the existing restore functions, in file order; - on any failure, stop and leave position unchanged - - Only after every entity restores successfully, advance the member's committed position; - extend the load function's contract with an indicator of whether entities were actually - restored - - Observable: the RED tests from 5.1 pass - - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_ - - _Boundary: Persistence Layer (load_snapshot)_ - -- [x] 6. Integration: wire the catch-up receive path to the new restore contract -- [x] 6.1 Consume the new restoration indicator in the snapshot-chunk receive handler - - The handler advances the member's applied position only after the load succeeds (unchanged - ordering); on a legacy (position-only) file it logs the fallback rather than treating it as - a failure - - Observable: given a legacy-format file, the member logs the fallback and continues through - normal catch-up; given an entity-bearing file, the member's applied position advances only - after restoration succeeds - - _Requirements: 2.4, 4.1, 4.2, 5.1, 5.2_ - - _Depends: 5.2_ - -- [x] 7. Validation: chaotic catch-up soak proves end-to-end correctness -- [x] 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round - - Configure or trigger a round where a member rejoins empty or far enough behind that it must - receive a full checkpoint rather than incremental history - - Observable: the soak verdict for that round reports zero committed-job loss and zero - committed-prefix divergence, and existing comparison tooling confirms the rejoined member's - job/queue state matches the rest of the cluster - - _Requirements: 4.1, 4.2, 4.3_ - - _RESOLVED 2026-07-01 (raft-ha-core-safety task 5.10a, same investigation — see that spec's - task 5.10 note for the full chain). The blocking `walcompare` per-member checkpoint-floor gap - was fixed (commit `f3c86a6`). The "specific soak run never actually exercised a full-checkpoint - catch-up at all" gap was closed by building a `--force-snapshot` soak mode (commit `9f4292d`) - plus a `--wal-segment-bytes` CLI override (this session) — the WAL segment physical size was a - hardcoded 64 MB compile-time constant, and `trim_wal_segments()` never deletes the currently- - open segment, so at soak job counts a segment never rotated and nothing was ever trimmable, - regardless of checkpoint correctness. Getting there also required fixing a real delaymgr - starvation bug (checkpoint scheduled but never executed, commit `b9ad018`) and a real - heap-use-after-free in `INVOKE_MSG_HANDLER` (`include/netio.h`) surfaced by the first-ever live - exercise of a snapshot transfer landing during an election. With all fixes in place: - `--force-snapshot` soak result = `committed_divergence=0 committed_job_loss=0 result=PASS`, - target node's log confirms a real `HA: snapshot` install (not just ordinary WAL catch-up), and - `verify_no_loss` confirms every submitted job (entity content, not raw WAL) is present and - correct post-restore on the snapshot-caught-up member. Full 217-test controller suite green - before/after; a 3-round `--chaotic` regression also clean._ - -- [x] 8. Validation: rolling-upgrade compatibility -- [x] 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live - cluster - - Plant a checkpoint file in the legacy (position-only) format, start a member on the upgraded - binary against it, and observe its behavior - - Observable: the member does not crash, logs the legacy-format fallback, and reaches full - committed consistency through normal catch-up rather than reporting false-positive - completeness - - _Requirements: 5.1, 5.2_ - - _Depends: 6.1_ - - (8.1) `persist_recover`'s local boot-time snapshot load (persist.c ~2790) passed `NULL` for - `out_entities_restored` -- it had no observability parity with the wire receive path's - (`ha_handle_snapshot_chunk`, task 6.1) legacy-vs-entity-bearing INFO log. Added the same - conditional INFO logging at this call site (small, additive, no behavior change -- WAL replay - in Step 3 still applies unconditionally either way). Live-verified: planted a hand-crafted - 16-byte legacy sentinel (`XCESNAP\0` + LE checkpoint_lsn=1, no entity frames) as ctrl-1's own - `snap-0000000000000001.capnp` in a 3-node cluster (reusing the ha-entity-snap soak deploy, - WAL segments left intact), restarted all 3. ctrl-1: no crash, logged - "persist_recover: ... is a legacy position-only checkpoint ... continuing via normal WAL - replay", was elected primary, served the full 37-job list, and `walcompare` reported all 3 - members' committed prefixes (LSN 1..49) byte-identical -- confirms full committed consistency - reached via ordinary WAL replay, no false-positive completeness. - -## Implementation Notes -- (1.1) Test fixture `snapent_make`/`snapent_teardown` lives in `test_persist_snapshot_entities.c`; seeded jobs currently carry only jobid+state+an all-zeroed submit. Task 2.1's "field-identical round-trip" assertion needs richer submit-payload seeding (e.g. non-zero cmdline/user/queue in `xce_job_submit_t`) to be a meaningful field-by-field comparison — extend `snapent_seed_jobs`, do not re-seed separately. -- (2.1) `write_snapshot`'s existing `fdatasync(fd)` call (persist.c:1505) does not check its return value at all — task 2.2 must fix this while adding entity encoding, since design.md's error-handling contract ("any failure at any step -- encode, write, sync -- leaves no file at the final path") requires it. -- (2.1) The write-failure-discard test (`test_snapent_checkpoint_write_failure_leaves_no_final_file`) only exercises the pre-existing `open()`-failure branch (via directory removal) and passes today without any entity-encode loop existing yet. Task 2.2 should add a genuine mid-encode-loop fault-injection test (failing to encode e.g. the Nth of N jobs) once the per-job encode loop exists — the current test cannot validate that path. -- (2.2) `restore_job` (job.c) decodes via `decode_Job` → `decode_JobSubmitData_ptr`/`decode_JobStartInfo_ptr` (STRING_DUP'd fields) into a local `xce_job_t j`; that decoded struct is never released via `free_Job(&j)` after `create_job` takes its own copy — a real, pre-existing leak on `restore_job`'s decode path. Not touched by task 2.2 (its `load_snapshot` excursion was fully reverted, see next note) — task 5.2 will reintroduce this call site and must fix the leak there. -- (2.2) First attempt ALSO touched `load_snapshot`/`persist.h` to make task 2.1's round-trip tests pass end-to-end, but review found it advanced `current_lsn`/`checkpoint_lsn` BEFORE the entity-decode loop ran -- a live silent-divergence hazard reachable through the unmodified `ha_handle_snapshot_chunk` production path (reachable today, not hypothetical, since `write_snapshot` now always emits entity-bearing checkpoints). REMEDIATED by fully reverting `load_snapshot`/`persist.h` to pre-2.2 state (byte-identical to commit `0f344da`) and rewriting the round-trip tests to parse `write_snapshot`'s raw on-disk bytes directly (new `snapent_read_checkpoint_jobs` helper using `read_Job`/`decode_Job`) instead of round-tripping through `persist_load_snapshot`. Task 5.2 must implement `load_snapshot`'s entity-decode from scratch per design.md's restore-before-advance-position contract -- do not assume any decode scaffolding exists yet. -- (3.1) `snapent_read_checkpoint_queues`'s job-section skip loop duplicates `snapent_read_checkpoint_jobs`'s (task 2.2) skip logic verbatim rather than sharing a helper -- minor, non-blocking; consider factoring a shared "skip past job section" reader if task 5.x's test coverage grows the file further. -- (3.2) Pre-existing ASan leak in the test fixture itself (`snapent_make` -> `makeInstance`/`makeJob`/`makeQueue`/`xkvmgr_add`, ~18KB/113 allocations), unrelated to any production code -- present since task 1.1, masked by the project's standing `ASAN_OPTIONS=detect_leaks=0` convention for this suite. Not blocking any task; worth a dedicated fixture-teardown cleanup pass if this test file keeps growing. -- (4.1) `snapshot_inflight_lsn[peer_idx]` clears to 0 BEFORE `snapshot_inflight[peer_idx]` clears to 0 in every path (completion + all 3 failure sites in `ha_snapshot_send_dispatch`). Task 4.3's `rotate_snapshots` consultation must therefore treat `snapshot_inflight_lsn[i] != 0` as the "protected" signal directly (checkpoint LSNs are never 0), not "check flag==1 first, then trust LSN" -- there's a narrow window where the flag is still 1 but the LSN has already cleared. -- (execution note, after 4.1) Remaining tasks split by risk given a mid-session spend-limit hit: low-risk/small/isolated tasks (4.2, 4.3, 7.1, 8.1) executed directly in the main context with self-review (build + test + boundary/TBD/secrets grep), no subagent dispatch. High-risk tasks (5.1/5.2 `load_snapshot` restore -- the exact function that caused task 2.2's rejection; 6.1 wiring into the live `ha_handle_snapshot_chunk` production path) keep the full implementer + independent-reviewer subagent cycle, since that adversarial review already caught a real live-hazard bug once this spec (task 2.2) and the remaining risk is concentrated there, not spread evenly. -- (5.1) `persist_load_snapshot`'s wrapper currently maps `*out_entities_restored = (rc==0)` -- a deliberately naive stopgap so the 3 RED tests fail for the right reasons. `load_snapshot`'s real body is completely untouched (still watermark-only, byte-identical to before this task). Task 5.2 must replace the wrapper's naive mapping wholesale with real format-detection + restore-then-gate-position logic; it owns turning `load_snapshot` from a no-op-on-entities function into the real restore path. -- (5.2) `restore_queue_state`'s `-ENOENT` ("queue not yet built from config") aborts the ENTIRE checkpoint restore, including already-successfully-restored jobs earlier in the same file -- this is design-compliant (design.md/Req 2.3 are unconditional: any restore failure leaves position unchanged) and not a defect, but it is a real residual availability risk: a benign config-lag on the destination member (a queue the source references not yet locally provisioned) would make catch-up fail outright rather than skip-and-continue. WATCH-ITEM for task 7.1's chaotic soak: if a catch-up round gets stuck/never completes, this exact code path (queue.c's restore_queue_state ENOENT -> persist.c's load_snapshot_entities abort) is the first place to look. -- (5.2) Test-only, non-blocking: the destination fixture's `snapent_teardown` does not release job_t objects added via `restore_job` (only the ones seeded at fixture creation) -- a pre-existing category of test-fixture-teardown gap (same class as tasks.md's (3.2) note), now slightly enlarged since load-path tests exercise restore_job against a destination fixture. Not production code; extend snapent_teardown if this file keeps growing. diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/brief.md b/.kiro/specs/ha-heartbeat-committed-propagation/brief.md deleted file mode 100644 index 01d24564..00000000 --- a/.kiro/specs/ha-heartbeat-committed-propagation/brief.md +++ /dev/null @@ -1,50 +0,0 @@ -# Brief: ha-heartbeat-committed-propagation - -> Opened 2026-06-20 from a controller-status (`ctrlstatus`) live finding. This is the **live-fresh off-primary committed** follow-up that controller-status Req 10 explicitly routed out of its own boundary. - -## Problem -A standby's knowledge of the cluster's committed position — `ha->last_known_committed_lsn` / `ha->last_known_committed_config_tag` — is refreshed **only at CtrlHello join / RDHUP-failover** (`cluster_apply_ctrlhello_reply`, `ha_convergence.c:375`) and an election safe-commit path. It is **not** refreshed on steady-state heartbeats. So between joins, a standby's view of committed lags the live primary, even as the standby keeps applying replicated entries. - -This surfaced live in `ctrlstatus` (controller-status Req 10): a standby reported `committedLsn: 0` while `appliedLsn: 10` — honest per Req 10.2 (it never learned a committed position) but not useful. Even after a join, the reported committed can sit stale between joins. controller-status Req 10 deliberately reports the honest best-known value and documented this staleness as an **out-of-boundary** caveat to be fixed here. - -The wire already anticipates this: `HeartbeatMsg` (`src/schema/controller.capnp:761`) **defines** `committedLsn @2` and `committedConfigTag @3`, but **no code populates or consumes them** — the `OP_nop` heartbeat carries no committed payload, and `ha_on_heartbeat_received` only updates liveness. - -## Current State -- `HeartbeatMsg{committedLsn, committedConfigTag}` fields exist in the schema but are dead (no producer, no consumer). -- `last_known_committed_lsn` (`ha.h:170`, `_Atomic`) + `last_known_committed_config_tag` (`ha.h:172`, plain) written only at join/RDHUP + election safe-commit. -- `OP_nop` heartbeat receive path (`cluster.c:300`) resets liveness only. -- controller-status `ctrlstatus_fill` already reports `last_known_committed_*` off-primary (controller-status task 6.2) — it will automatically reflect a fresher value once this spec lands. - -## Desired Outcome -The primary stamps its current `committed_lsn` / `committed_config_tag` into every heartbeat; each standby consumes them in the heartbeat receive path and **monotonically** advances `last_known_committed_lsn` / `last_known_committed_config_tag`. Off-primary committed then tracks the live primary within one heartbeat interval, and `ctrlstatus` on a standby shows a near-real-time committed position instead of a join-stale (or zero) value. - -## Approach -Populate the already-defined `HeartbeatMsg.committedLsn`/`committedConfigTag` on the primary's heartbeat sender; consume them on the standby's heartbeat receive path with a monotonic update of `last_known_committed_*`. No change to how committed is produced/replicated/persisted — this only propagates an already-committed marker as liveness-adjacent metadata. - -## Scope -- **In**: primary heartbeat sender populates `committedLsn`/`committedConfigTag`; standby heartbeat receive path consumes them → monotonic update of `last_known_committed_lsn`/`last_known_committed_config_tag`; unit + integration tests (standby's learned committed advances on heartbeat without a re-join); memory-ordering consistency for the non-atomic `last_known_committed_config_tag` vs the `ctrlstatus` reader. -- **Out**: how `committed_lsn` is produced/advanced (ha-commit-engine / commit durability); the `ctrlstatus` reporting itself (controller-status Req 10, already done); WAL catch-up triggering (stays ACK-driven per ha-convergence-recovery); election safe-commit derivation (unchanged — must NOT use heartbeat-borne committed for truncation, per the existing "never truncate from stale ctrlhello/heartbeat" rule). - -## Boundary Candidates -- Primary heartbeat sender (stamp committed into `HeartbeatMsg`). -- Standby heartbeat receive (consume + monotonic `last_known_*` update). -- The monotonicity + memory-ordering contract on `last_known_committed_*`. - -## Out of Boundary -- Committed-LSN production / quorum advance (commit engine). -- `ctrlstatus` rendering and field sourcing (controller-status). -- Any use of heartbeat-borne committed for WAL truncation or vote-freshness floor changes (explicitly forbidden — truncation/freshness must stay durable-backed; this value is advisory/reporting only). - -## Upstream / Downstream -- **Upstream**: ha-heartbeat-liveness (owns `HeartbeatMsg` + the heartbeat tick/receive path), ha-commit-engine (committed_lsn semantics). -- **Downstream**: controller-status Req 10 (off-primary committed becomes live-fresh; removes the documented staleness caveat); ha-vote-freshness-invariant (voter floor `max(durable_floor, last_known_committed_lsn)` benefits from a fresher `last_known_committed_lsn`, but freshness must remain correct — verify no safety regression). - -## Existing Spec Touchpoints -- **Extends**: ha-heartbeat-liveness (heartbeat content/behavior). -- **Adjacent**: ha-repl-hardening (standby replication state), controller-status (the consumer that motivated this), ha-convergence-recovery (catch-up stays ACK-driven, not heartbeat-driven). - -## Constraints -- `last_known_committed_*` updates must be **monotonic** (never regress on an out-of-order/stale heartbeat). -- Heartbeat path is hot — keep the added work to two field reads + two conditional stores. -- The heartbeat-borne committed is **advisory** (reporting/liveness): it must never feed WAL truncation or weaken the durable-backed vote-freshness floor. -- `last_known_committed_config_tag` is non-atomic (`ha.h:172`); reconcile the writer/reader memory ordering with the `ctrlstatus` best-effort read (consider promoting to `_Atomic`). diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/design.md b/.kiro/specs/ha-heartbeat-committed-propagation/design.md deleted file mode 100644 index 3dfc1dd4..00000000 --- a/.kiro/specs/ha-heartbeat-committed-propagation/design.md +++ /dev/null @@ -1,204 +0,0 @@ -# Design Document — ha-heartbeat-committed-propagation - -## Overview - -**Purpose:** keep a standby controller's knowledge of the cluster's committed position current from the primary's heartbeats, instead of only at a join/failover handshake. The serving primary attaches its committed position (committed LSN + committed configuration version) to every HA heartbeat; each standby consumes it and monotonically advances its `last_known_committed_*` state. - -**Users:** operators inspecting a standby via the controller-status `ctrlstatus` query (its off-primary committed becomes near-real-time instead of join-stale or zero), and the HA vote-freshness path (which already reads `last_known_committed_lsn` and gains a fresher, term-guarded value). - -**Impact:** the HA heartbeat is a bare `OP_nop` today. This wires up the already-defined-but-unused `HeartbeatMsg{committedLsn, committedConfigTag}` body on that heartbeat (producer + consumer), adds a reconcile-free monotonic updater, and promotes one field to atomic. No new opcode; no change to how committed is produced/replicated/persisted; no change to the vote-floor formula. - -### Goals -- The serving primary advertises its committed LSN + committed config version on every heartbeat (1.1). -- A standby refreshes its known committed position from heartbeats, monotonically, within one heartbeat interval, with no re-join (1.2, 1.3, 2.x). -- The propagated value is advisory: it never drives truncation and never weakens the vote-freshness floor (3.x). -- Existing heartbeat liveness/failover behavior is unchanged (4.x). - -### Non-Goals -- Changing how `committed_lsn` is produced, advanced, replicated, or persisted (commit engine / durability). -- The join/failover handshake (`cluster_apply_ctrlhello_reply`) that already seeds the standby's committed knowledge — left as-is (it remains the initial source and the only path that reconciles/truncates). -- Catch-up streaming triggering (stays ACK-driven, per ha-convergence-recovery). -- The `ctrlstatus` reporting/rendering of committed (owned by controller-status; it already reads `last_known_committed_*`). - -## Boundary Commitments - -### This Spec Owns -- The producer: attaching the primary's committed LSN + committed config tag (and its term) to the HA heartbeat request. -- The consumer: reading that payload on a standby and **monotonically** updating `last_known_committed_lsn` / `last_known_committed_config_tag`, gated by a term check, **without** reconciliation/truncation. -- The new reconcile-free monotonic updater `ha_observe_heartbeat_committed`. -- Promoting `last_known_committed_config_tag` to `_Atomic` and migrating its existing writers. - -### Out of Boundary -- The vote-freshness floor **formula** (`max(durable_floor, last_known_committed_lsn)`) — unchanged; this spec only makes its input fresher. -- The join handshake / RDHUP path and its truncation/reconcile (`cluster_apply_ctrlhello_reply`, `ha_rejoin_reconcile`). -- Committed production/replication/persistence, WAL truncation, catch-up streaming. -- The client keepalive `OPCode_heartbeat` / `xce_heartbeat` (a different message — must not be touched or reused). -- `ctrlstatus` field sourcing/rendering (controller-status). - -### Allowed Dependencies -- The Cap'n Proto schema + codegen: the existing `HeartbeatMsg` struct and the `heartbeatmsg @25` Request-body union member (`src/schema/controller.capnp`) — reused, no shape change. -- The existing HA heartbeat tick (`primary_heartbeat_fn`, `ha.c`) and the standby `OP_nop` receive path (`cluster.c`). -- Read access to the primary's `persist.committed_lsn` and `ha->committed_config_tag`; read/write of `ha->last_known_committed_*` and `ha->term`. -- The request/reply plumbing and a new opcode-less sender alongside `xce_nop_req`. - -### Revalidation Triggers -- Any change to `HeartbeatMsg`'s committed fields or the `heartbeatmsg` body ordinal → re-check producer + consumer. -- Any change to the voter-floor formula or to who writes `last_known_committed_*` → re-check the term-guard and monotonicity argument (Security section). -- Any change making the HA heartbeat use a different opcode/body → re-check the consumer's `body_which` branch. -- Changing `last_known_committed_config_tag`'s type → re-check all writers + the `ctrlstatus` reader. - -## Architecture - -### Existing Architecture Analysis -- **HA heartbeat = bare `OP_nop`** (pre-feature baseline, since replaced by this spec — see File Structure Plan): `primary_heartbeat_fn` (now `ha.c:9665`; line drifted from unrelated growth added by later specs elsewhere in the file) sent `xce_nop_req(ep)` with no body; the standby `OP_nop` handler (now `cluster.c:427`, same drift) replied a `StateReport` and reset liveness (`ha_on_heartbeat_received`). It read no request body. -- **Unused scaffolding:** `HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3}` and the `heartbeatmsg @25` Request-body member exist but have zero `.c` producers/consumers. (Verified current: schema unchanged.) -- **Committed knowledge at baseline:** `last_known_committed_lsn` (`_Atomic`, now `ha.h:183`) written at join (`cluster_apply_ctrlhello_reply` — a store that ALSO truncates via `ha_rejoin_reconcile`) and at the election safe-commit path (`ha.c:925` area, still current); `last_known_committed_config_tag` (plain pre-task-1.1, now `ha.h:185`) written only at join. Correction: this bullet originally implied a third writer of `last_known_committed_config_tag` at the election path — verified false; that path writes `last_known_committed_lsn` and the *separate* `committed_config_tag` field (Req 8.1, set on quorum ACK), not `last_known_committed_config_tag`. -- **Monotonic precedent:** `commit.c` advances `committed_lsn` with monotonic-max semantics — mirrored here. -- **Vote floor already consumes it:** `max(durable_floor, last_known_committed_lsn)`, computed in `ha_vote_self_freshness` (now `ha.c:5310`; citation corrected — durable_floor is supplied by the voter-path caller, e.g. `cluster.c:544`). -- **Opcode/body independence:** `OP_hello` already carries different bodies (`ctrlhellomsg` vs `replymsg`), so `OP_nop` + `heartbeatmsg` body is consistent with the existing pattern. - -### Architecture Pattern & Boundary Map -Extend-in-place along the existing heartbeat path. **Dependency direction (unchanged):** `schema → libxce (sender) → controller (primary tick + standby receive)`. No upward dependencies; no new opcode. - -```mermaid -graph LR - Tick[primary_heartbeat_fn tick] --> Sender[xce_ha_heartbeat_req nop plus HeartbeatMsg body] - Sender --> Wire[OP_nop request with heartbeatmsg body] - Wire --> Handler[cluster.c OP_nop handler] - Handler --> Guard[term guard plus monotonic] - Guard --> Observe[ha_observe_heartbeat_committed] - Observe --> State[last_known_committed_lsn and config_tag] - State --> Floor[vote freshness floor unchanged] - State --> Status[ctrlstatus off primary committed] -``` - -Key decisions: the primary stamps committed onto the existing heartbeat (no new opcode, Option A); the standby's update is **term-guarded and monotonic** and **reconcile-free**; the vote floor reads the fresher value through the unchanged formula. - -## File Structure Plan - -### Modified Files -- `src/include/request.h` + `src/lib/request.c` — add `xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag)`: sends an `OPCode_nop` request with a `heartbeatmsg` body carrying `{term, committed_lsn, committed_config_tag}`. Generic `xce_nop_req` is left unchanged. -- `src/controller/ha.c` — `primary_heartbeat_fn`: replace the `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, ha->term, persist.committed_lsn, ha->committed_config_tag)` (the `XCE_TESTING` `ha_nop_req_hook` seam is preserved). Add `ha_observe_heartbeat_committed`. Migrate the election-path writer of `last_known_committed_config_tag` (`ha.c:925` area) to `atomic_store`. -- `src/controller/cluster.c` — `OP_nop` handler: if `request.body_which == Request_body_heartbeatmsg`, apply the term-guard and call `ha_observe_heartbeat_committed`; otherwise behave exactly as today. Liveness reset + `StateReport` reply unchanged. -- `src/controller/include/ha.h` — change `last_known_committed_config_tag` to `_Atomic uint64_t`; declare `ha_observe_heartbeat_committed`. -- `src/controller/ha_convergence.c` — `cluster_apply_ctrlhello_reply`: write `last_known_committed_config_tag` via `atomic_store` (type migration only; semantics unchanged). -- Tests: `src/controller/testsuite/test_ha_heartbeat_committed.c` (new) for producer/consumer + monotonic + term-guard + advisory-only; registered in `controller_test.c` + `controller/meson.build`. - -### New Files -- `src/controller/testsuite/test_ha_heartbeat_committed.c` — unit tests for the consumer/updater. (No new product files — `HeartbeatMsg` C types are generated; logic slots into existing TUs.) - -## System Flows - -### Heartbeat-borne committed propagation + term-guard -```mermaid -sequenceDiagram - participant P as Primary - participant S as Standby - P->>P: read own committed_lsn and committed_config_tag and term - P->>S: OP_nop with HeartbeatMsg term committed_lsn config_tag - S->>S: reset liveness and reply StateReport as before - alt heartbeat term is at least standby term and committed greater than known - S->>S: monotonic advance last_known_committed_lsn and config_tag - else stale term or not greater or no body - S->>S: liveness only no committed update - end -``` -Gating: the standby updates committed only when the heartbeat's advertised term is `>= ha->term` (term-guard) AND the carried LSN exceeds the current `last_known_committed_lsn` (monotonic). Otherwise the heartbeat is processed for liveness exactly as today. The update never reconciles or truncates the log. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | Primary advertises committed on each heartbeat | `xce_ha_heartbeat_req`, `primary_heartbeat_fn` | `HeartbeatMsg{term, committedLsn, committedConfigTag}` body on `OP_nop` | Propagation | -| 1.2 | Standby consumes it | `cluster.c` `OP_nop` handler, `ha_observe_heartbeat_committed` | `body.heartbeatmsg` read | Propagation | -| 1.3 | Fresh within one heartbeat interval | per-tick send + receive | heartbeat cadence | Propagation | -| 2.1 | Advance on higher carried LSN | `ha_observe_heartbeat_committed` | monotonic-max store | Propagation (alt) | -| 2.2 | Leave unchanged on ≤ (stale/out-of-order/lagging) | `ha_observe_heartbeat_committed` | monotonic guard | Propagation (else) | -| 2.3 | Config tag advances with LSN, never independently regressed | `ha_observe_heartbeat_committed` | paired update under the same advance | Propagation | -| 3.1 | No truncation from heartbeat-borne committed | `ha_observe_heartbeat_committed` (reconcile-free) | no `ha_rejoin_reconcile` call | — | -| 3.2 | No weakening of the vote-freshness floor | vote floor unchanged + term-guard | `max(durable_floor, last_known_committed_lsn)` (unchanged) | Propagation (term-guard) | -| 3.3 | No change to committed production/replication/persist | consume path | read-only on produced committed | — | -| 3.4 | Primary uses its own committed | consume gated to non-primary | inbound heartbeats only on standbys | Propagation | -| 4.1 | Existing liveness behavior unchanged | `OP_nop` handler | `ha_on_heartbeat_received` + `StateReport` untouched | Propagation | -| 4.2 | Absent/zero committed → plain liveness, no-op | `OP_nop` handler, `ha_observe_heartbeat_committed` | body-absent / term-fail / zero → no update | Propagation (else) | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| `xce_ha_heartbeat_req` | libxce | Send `OP_nop` + `HeartbeatMsg` body | 1.1 | request plumbing, `HeartbeatMsg` codegen (P0) | Service | -| `primary_heartbeat_fn` call-site | Controller HA | Stamp own committed/term per tick | 1.1, 3.4 | `persist.committed_lsn`, `ha->committed_config_tag`, `ha->term` (P0) | Service | -| `OP_nop` handler consume branch | Controller | Read body, term-guard, dispatch update | 1.2, 4.1, 4.2 | `ha_observe_heartbeat_committed` (P0) | Service | -| `ha_observe_heartbeat_committed` | Controller HA | Reconcile-free monotonic update of `last_known_committed_*` | 2.x, 3.1 | `ha->last_known_committed_*` atomics (P0) | Service, State | -| `last_known_committed_config_tag` → `_Atomic` | Controller HA | Safe concurrent read/write | 2.3 | existing writers (P1) | State | - -### libxce — HA heartbeat sender (Service) -```c -/* Send the HA heartbeat as OP_nop carrying a HeartbeatMsg body. Clone of the - * nop send path + a heartbeatmsg body. Non-blocking, no reply awaited (the - * standby's StateReport rides the normal nop reply). Generic xce_nop_req is - * left unchanged. */ -int xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, - uint64_t committed_lsn, uint64_t committed_config_tag); -``` -- Preconditions: valid endpoint. -- Postconditions: an `OP_nop` request with `body_which == Request_body_heartbeatmsg` is enqueued; `HeartbeatMsg.term/committedLsn/committedConfigTag` set; `senderPriority` unused (0). -- Invariants: non-blocking; carries no other state. - -### Controller — `ha_observe_heartbeat_committed` (Service / State) -| Field | Detail | -|-------|--------| -| Intent | Monotonically advance a standby's learned committed position from a heartbeat, with no reconciliation | -| Requirements | 2.1, 2.2, 2.3, 3.1 | - -**Responsibilities & Constraints** -- Advance `last_known_committed_lsn` to the carried LSN only if it is strictly greater (monotonic; single receiver thread, so a relaxed compare-then-`atomic_store`/`acquire`-`release` or `atomic_fetch_max` suffices). -- Advance `last_known_committed_config_tag` **together with** the LSN (only when the LSN advances); never regress it independently (2.3). -- **Must NOT** call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state (3.1, 3.3). This is the explicit distinction from `cluster_apply_ctrlhello_reply`. -- No-op when the carried LSN is `<=` current or zero (2.2, 4.2). - -**Dependencies** -- Inbound: `cluster.c` `OP_nop` handler — passes the term-checked payload (P0). -- Outbound: `ha->last_known_committed_lsn` / `last_known_committed_config_tag` atomics (P0). - -### Controller — `OP_nop` handler consume branch (Service) -- After the existing liveness/`StateReport` handling, if `request.body_which == Request_body_heartbeatmsg` and the responder is non-primary: apply the **term-guard** (`hb->term >= atomic_load(ha->term)`) and, if it passes, call `ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag)`. -- If the body is absent, the term-guard fails, or committed is zero → no committed update; liveness handling is unchanged (4.1, 4.2). -- Constraint: this branch must not alter the reply, the liveness reset, or any existing `OP_nop` behavior. - -## Data Models - -No new persistent data and no schema-shape change. `HeartbeatMsg` (existing) is the request-body DTO; its `term`, `committedLsn`, `committedConfigTag` fields are populated by the producer and read by the consumer. The only in-memory state touched is `ha->last_known_committed_lsn` (already `_Atomic`) and `ha->last_known_committed_config_tag` (**migrated to `_Atomic uint64_t`** so the heartbeat-receive writer, the join writer, the election writer, and the `ctrlstatus` reader interact safely). Both are advisory replicas of the primary's committed markers — never authoritative, never persisted by this path. - -## Error Handling - -- **Malformed/absent body:** `body_which != Request_body_heartbeatmsg` → treat as a plain liveness heartbeat; no error (4.2). -- **Stale-term heartbeat:** `hb->term < ha->term` → ignore the committed payload (term-guard); liveness handled normally. This is the safety gate, not an error. -- **Out-of-order / lower committed:** monotonic no-op (2.2); not an error. -- **Send failure:** `xce_ha_heartbeat_req` returns the same negative as the nop path; `primary_heartbeat_fn` logs `WARN` exactly as today — no behavior change (4.1). -- **Observability:** keep heartbeat-committed updates at a low/debug log level (the heartbeat path is hot); do not add per-tick INFO spam. - -## Testing Strategy - -### Unit (`test_ha_heartbeat_committed.c`) -- A standby with `last_known_committed_lsn = 4` receiving a heartbeat (`term >= ha->term`) carrying `committed_lsn = 9` → `last_known_committed_lsn == 9` and `last_known_committed_config_tag` updated (1.2, 2.1, 2.3). -- A heartbeat carrying `committed_lsn = 3` (≤ current 9) → `last_known_committed_lsn` unchanged (2.2). -- A **stale-term** heartbeat (`hb->term < ha->term`) carrying a higher committed → `last_known_committed_*` unchanged (3.2 term-guard); liveness still reset (4.1). -- A heartbeat with no `heartbeatmsg` body / zero committed → no committed update, processed as liveness (4.2). -- `ha_observe_heartbeat_committed` performs no WAL truncation / no reconcile call and mutates nothing beyond `last_known_committed_*` (3.1, 3.3) — assert via a no-truncate / state-snapshot check. - -### Integration -- Producer round-trip: `xce_ha_heartbeat_req` encodes an `OP_nop` whose decoded request has `body_which == heartbeatmsg` with the expected `term/committedLsn/committedConfigTag` (1.1). -- Vote-floor interaction: with a heartbeat-raised `last_known_committed_lsn`, the existing freshness gate rejects a candidate below it and accepts one at/above it — confirming the floor consumes the fresher value through the unchanged formula and is never lowered (3.2). - -### E2E (live `/xce-deploy-ha`) -- On a healthy 3-node cluster, submit jobs so the primary advances committed; `ctrlstatus` on a **standby** shows `committedLsn` tracking the primary within ~a heartbeat interval (no re-join), closing the controller-status Req 10 staleness caveat (1.3, downstream). - -## Security Considerations - -The only safety-relevant change is feeding a heartbeat-fresh `last_known_committed_lsn` into the (unchanged) vote-freshness floor `max(durable_floor, last_known_committed_lsn)`. - -- **Why raising the floor is safe (Leader Completeness):** committed entries are, by definition, durable on a majority and cluster-wide monotonic (State Machine Safety). A heartbeat-borne committed accepted under the **term-guard** (`hb->term >= ha->term`) therefore reflects a genuinely committed position. Requiring vote candidates to be at least that fresh *strengthens* the guarantee that a new leader holds all committed entries; it never weakens it (3.2). The value only ever raises the floor (monotonic), so the floor cannot drop. -- **No liveness wedge:** since a majority holds the committed entries, an eligible candidate (`lastLogLsn >= floor`) always exists; a node behind the floor correctly fails its own freshness check and should not lead. -- **Stale/forged-term defense:** the term-guard rejects committed advertised by a primary whose term is behind the standby's, preventing an old/partitioned primary from inflating the floor. The value is otherwise advisory: it never drives truncation (3.1) and never changes how committed is produced (3.3). -- **No new authorization surface:** the heartbeat is the existing authenticated primary→standby channel; this only adds three integer fields to a message that already flows. diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/requirements.md b/.kiro/specs/ha-heartbeat-committed-propagation/requirements.md deleted file mode 100644 index e5a0d0c9..00000000 --- a/.kiro/specs/ha-heartbeat-committed-propagation/requirements.md +++ /dev/null @@ -1,55 +0,0 @@ -# Requirements Document - -## Introduction - -This feature makes a standby controller's knowledge of the cluster's **committed position** stay current from the primary's **heartbeats**, rather than only being refreshed when the standby (re)joins the primary. Today a standby learns the committed position (committed log position and committed configuration version) only at a join / failover handshake; between joins that knowledge goes stale, and a standby that has applied replicated entries can still report a committed position of zero. The heartbeat already flows continuously between primary and standbys, so it is the natural carrier for a fresh committed marker. - -With this feature, the serving primary includes its current committed position in every heartbeat, and each standby uses it to keep its own known committed position up to date — within roughly one heartbeat interval — without waiting for a re-join. This directly removes the staleness caveat documented by the controller-status feature (whose `ctrlstatus` query reports a standby's known committed position): a standby's reported committed position becomes near-real-time instead of join-stale or zero. - -The propagated committed marker is **advisory** — it exists for observability and standby state freshness only. It must not change how the committed position is produced, advanced, replicated, or persisted, and it must not feed any correctness-critical decision (log truncation, election/vote safety). Existing heartbeat liveness and failover behavior must be unaffected. - -> Full discovery context (problem, approach, scope, boundary candidates, upstream/downstream, constraints) is in `.kiro/specs/ha-heartbeat-committed-propagation/brief.md`. - -## Boundary Context - -- **In scope**: the serving primary advertising its current committed position (committed log position + committed configuration version) on each heartbeat; a standby consuming that value to keep its own **known committed position** current; monotonic (never-regressing) update of that known position; graceful handling of heartbeats that carry no usable committed value; preserving all existing heartbeat liveness/failover behavior. -- **Out of scope**: how the committed position is produced, advanced, replicated, or persisted (owned by the commit engine / durability); the join/failover handshake that already seeds the standby's committed knowledge (unchanged, remains the initial source); the catch-up streaming trigger (remains acknowledgement-driven, not heartbeat-driven); the `ctrlstatus` reporting/rendering of the committed position (owned by controller-status, which already reads the standby's known committed position); any change to how a node truncates its log or evaluates vote freshness. -- **Adjacent expectations**: relies on the existing heartbeat channel and timing (the primary already emits heartbeats and standbys already receive them) and on the existing committed-position semantics, both consumed unchanged. The **downstream** consumer is the controller-status `ctrlstatus` query: once a standby's known committed position is heartbeat-fresh, that query reflects it with no change to controller-status. Vote-freshness evaluation may read the standby's known committed position as part of its floor, but this feature must not let a heartbeat-borne value lower or weaken that floor. - -## Requirements - -### Requirement 1: Heartbeat-borne committed position propagation - -**Objective:** As an operator, I want a standby's known committed position to refresh from the primary's heartbeats, so that it reflects the live cluster committed state without waiting for the standby to re-join. - -#### Acceptance Criteria -1. While a node is the serving primary, when it emits a heartbeat, the Controller shall include in that heartbeat the primary's current committed log position and committed configuration version. -2. When a standby receives a heartbeat that carries a committed position, the Controller shall update the standby's known committed position (committed log position and committed configuration version) from the values in that heartbeat, subject to the monotonicity rule in Requirement 2. -3. While a standby is continuously receiving heartbeats from a serving primary, the standby's known committed position shall reflect the primary's committed position within one heartbeat interval, without requiring a re-join handshake. - -### Requirement 2: Monotonic, non-regressing committed knowledge - -**Objective:** As an operator, I want a standby's known committed position to never move backward, so that observed committed progress stays stable and trustworthy across heartbeats. - -#### Acceptance Criteria -1. When a standby receives a heartbeat whose carried committed log position is greater than the standby's current known committed log position, the Controller shall advance the known committed position to the carried value. -2. If a standby receives a heartbeat whose carried committed log position is less than or equal to the standby's current known committed log position (for example a delayed or out-of-order heartbeat, or a newly started or lagging primary), then the Controller shall leave the standby's known committed position unchanged. -3. The standby's known committed configuration version shall be advanced together with the known committed log position and shall never be regressed independently. - -### Requirement 3: Advisory-only — no effect on correctness-critical paths - -**Objective:** As a maintainer, I want the heartbeat-borne committed value used only for observability and standby state freshness, so that it cannot weaken durability or election-safety guarantees. - -#### Acceptance Criteria -1. The Controller shall not use a heartbeat-borne committed position to truncate, discard, or otherwise modify any write-ahead log entry. -2. The Controller shall not use a heartbeat-borne committed position to lower or weaken the durability-backed freshness floor used when granting votes. -3. The Controller shall not change how the committed position is produced, advanced, replicated, or persisted as a result of propagating it on heartbeats. -4. While a node is the serving primary, the Controller shall continue to use and report its own authoritative committed position; a heartbeat-borne committed value shall be consumed only by non-primary nodes. - -### Requirement 4: No regression to heartbeat liveness behavior - -**Objective:** As an operator, I want existing heartbeat liveness and failover behavior to be unchanged, so that adding committed propagation does not destabilize the cluster. - -#### Acceptance Criteria -1. The Controller shall preserve existing heartbeat liveness detection, missed-heartbeat tolerance, and proactive-heartbeat timing behavior unchanged when committed propagation is added. -2. If a received heartbeat carries no usable committed position (for example a zero or absent value), then the Controller shall process it as a liveness heartbeat exactly as before and shall leave the standby's known committed position unchanged, without error. diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/research.md b/.kiro/specs/ha-heartbeat-committed-propagation/research.md deleted file mode 100644 index f031752f..00000000 --- a/.kiro/specs/ha-heartbeat-committed-propagation/research.md +++ /dev/null @@ -1,64 +0,0 @@ -# Research / Gap Analysis — ha-heartbeat-committed-propagation - -> Generated by `/kiro-validate-gap` (2026-06-20). Brownfield; informs design. Requirements: `requirements.md` (Req 1–4). Brief: `brief.md`. - -## Current State (verified) - -- **The HA primary→standby heartbeat is a bare `OP_nop` with no body.** `primary_heartbeat_fn` sends `xce_nop_req(ep)` (`ha.c:5256`); the standby `OP_nop` handler (`cluster.c:300`) replies a `StateReport` (`ha_nop_reply_state_report`) and resets liveness (`ha_on_heartbeat_received`) — it reads no request body and learns no committed value. -- **`OPCode_heartbeat` is a SEPARATE, client-side keepalive** (`xce_heartbeat(ctx, hb_interval)`, `xceapi.c:48`; `client.c:233`; dispatch-gate `ha.c:2077/2120/2161`) — NOT the HA heartbeat. **Do not repurpose it** (collision). -- **The wire already supports a committed-bearing heartbeat, but nothing uses it:** the Request body union has `heartbeatmsg @25: HeartbeatMsg` (`controller.capnp:653`), and `HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is fully defined. Grep finds **zero** `.c` producers or consumers of `heartbeat_msg_t`/`heartbeatmsg`/its `committedLsn` — dead scaffolding. -- **Standby committed knowledge today:** `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) + `ha->last_known_committed_config_tag` (**plain** `uint64_t`, `ha.h:172`) are written only at join via `cluster_apply_ctrlhello_reply` (`ha_convergence.c:372`) — a **plain `atomic_store` (NOT monotonic)** that ALSO calls `ha_rejoin_reconcile` (truncation) — and at an election safe-commit path (`ha.c:925`). -- **A monotonic-max precedent exists** for committed advance in `commit.c` (`monotonic max() store`, `commit.c:141/151/155`) — mirror it for the heartbeat update. -- **`last_known_committed_lsn` already feeds the vote-freshness floor**: voter floor = `max(durable_floor, last_known_committed_lsn)` (`ha.c:2570–2586`; established by bug-339 / `project_ha_vote_freshness_invariant`). This is the key safety interaction (below). -- `xce_nop_req(ep)` is **generic** (also the client `xce_nop`, `xceapi.c:42`) — must not gain a mandatory committed body. - -## Requirement-to-Asset Map - -| Need (Req) | Existing asset | Gap | -|---|---|---| -| 1.1 primary advertises committed on each heartbeat | `primary_heartbeat_fn` sends bare `xce_nop_req`; `HeartbeatMsg{committedLsn,committedConfigTag}` + `heartbeatmsg` body union member exist | **Missing producer**: a HA-heartbeat sender that attaches a `HeartbeatMsg` body carrying `committed_lsn`/`committed_config_tag` | -| 1.2 standby consumes it | `cluster.c:300` `OP_nop` handler (reads no body); `request.body_which`/`body` union already supports `heartbeatmsg` | **Missing consumer**: read `body.heartbeatmsg` and update `last_known_committed_*` | -| 1.3 fresh within one heartbeat interval | heartbeat tick cadence (`heartbeat_interval`) | property of 1.1+1.2; testable via tick | -| 2.1–2.3 monotonic, never regress (LSN + config tag together) | `commit.c` monotonic-max pattern; `last_known_committed_lsn` is `_Atomic` (single receiver thread) | **Constraint**: existing `cluster_apply_ctrlhello_reply` does a PLAIN store — the heartbeat path needs a **monotonic** update; config tag must move with the LSN | -| 3.1 no truncation from heartbeat-borne committed | `cluster_apply_ctrlhello_reply` calls `ha_rejoin_reconcile` (truncation) | **Critical**: the heartbeat consume path must NOT reconcile/truncate — needs a separate, reconcile-free updater (do NOT reuse `cluster_apply_ctrlhello_reply`) | -| 3.2 must not weaken the vote-freshness floor | voter floor `max(durable_floor, last_known_committed_lsn)` (`ha.c:2570`) | **Research Needed (highest)**: a heartbeat-fresh (higher) `last_known` RAISES the floor — verify that is safe (Leader Completeness) and cannot deadlock liveness (a standby refusing all candidates) | -| 3.3 no change to committed production/replication/persist | consume path is read+store only | none — keep it read-only w.r.t. produced committed | -| 3.4 primary uses its own committed | only standbys process inbound HA heartbeats; primary sends | inherently satisfied; gate the consume to non-primary for clarity | -| 4.1 existing liveness unchanged | `OP_nop` still resets liveness + replies StateReport | additive body-read only; verify no timing change | -| 4.2 absent/zero committed → plain liveness, no-op | monotonic update naturally no-ops on 0 / ≤current | handle `body_which != heartbeatmsg` and zero value as liveness-only | - -## Implementation Approach Options - -**Option A (Extend the `OP_nop` HA heartbeat) — RECOMMENDED.** Add a dedicated HA-heartbeat sender (e.g. `xce_ha_heartbeat_req(ep, committed_lsn, committed_config_tag)`) that sends `OPCode_nop` **with** a `heartbeatmsg` body; the `cluster.c` `OP_nop` handler reads it and monotonically updates `last_known_committed_*` via a new reconcile-free helper. Reuses the already-defined `HeartbeatMsg` + body union member; leaves generic `xce_nop_req` (client nop) untouched. -*Trade-off*: ✅ minimal, reuses dead scaffolding, no new opcode. ❌ touches the hot HA heartbeat path; an `OP_nop` carrying a `heartbeatmsg@25` body is slightly unconventional (a request may carry any body-union member regardless of opcode — design confirms the pairing). - -**Option B (Dedicated HA-heartbeat opcode).** New opcode (distinct from the client `OPCode_heartbeat`) paired with `HeartbeatMsg`, plus dispatch-gate + handler wiring. Cleaner semantic separation. *Trade-off*: ✅ unambiguous. ❌ more churn (opcode, gate, handler, sender swap) for the same effect. - -**Option C (Repurpose `OPCode_heartbeat`) — REJECT.** It is the client keepalive (`xce_heartbeat`/`hb_interval`); reusing it collides with existing client semantics. - -## Effort & Risk - -- **Producer (Req 1.1)**: S — one sender + call-site swap in `primary_heartbeat_fn`. -- **Consumer + monotonic update (Req 1.2, 2)**: S — one handler branch + a reconcile-free monotonic helper. -- **Vote-freshness safety review (Req 3.2)**: M — analysis, not much code, but the gating risk. -- **Combined**: **S–M (3–5 days)**. **Risk Medium** — not the propagation (mechanical) but two interactions: (a) the hot heartbeat path, (b) feeding a fresher `last_known_committed_lsn` into the vote floor. - -## Research Items to Carry to Design - -1. **Vote-freshness safety (do this first).** Confirm against `raft.tla` Leader Completeness that raising `last_known_committed_lsn` from a heartbeat-advertised committed (which the standby has not necessarily applied/durably stored) is safe to feed the voter floor `max(durable_floor, last_known_committed_lsn)`, and that it cannot wedge liveness (standby refusing otherwise-valid candidates). If risky, gate: only let the heartbeat raise `last_known` up to what the standby can corroborate, or keep the floor on durable state and let only reporting consume the heartbeat value. -2. **Reconcile-free consume path.** Do NOT reuse `cluster_apply_ctrlhello_reply` (it truncates). Design a monotonic-only `ha_observe_heartbeat_committed`; decide whether to factor a shared monotonic core used by both join and heartbeat (join adds reconcile, heartbeat does not). -3. **`last_known_committed_config_tag` atomicity.** It is non-atomic (`ha.h:172`); with a new heartbeat-receive writer plus the `ctrlstatus` reader, decide whether to promote to `_Atomic` (and update the 2–3 existing write sites) — or keep best-effort. -4. **Sender scoping.** Add a HA-specific sender; do NOT alter generic `xce_nop_req`. -5. **Opcode/body pairing.** Confirm `OP_nop` + `heartbeatmsg` body is accepted end-to-end by the codec/dispatch, or choose Option B's dedicated opcode. - ---- - -## Synthesis / Design Decisions (2026-06-20) - -Resolutions of the 5 research items, locked into `design.md`: - -1. **Vote-freshness safety — RESOLVED, no vote-path change needed.** The voter floor formula `max(durable_floor, last_known_committed_lsn)` stays UNCHANGED; it already reads `last_known_committed_lsn`. Making that value heartbeat-fresh only RAISES it (monotonic), which *strengthens* the freshness gate (Leader Completeness) rather than weakening it (Req 3.2). Safety rests on three facts: (a) **term-guard** — a standby accepts the heartbeat-borne committed only from a primary whose advertised `term >= ha->term` (`HeartbeatMsg.term@1`), so a stale-term primary cannot inflate the floor; (b) **monotonic** — never regress (Req 2); (c) cluster-wide `committed_lsn` is itself monotonic and majority-durable (State Machine Safety), so any accepted value is genuinely committed. No liveness wedge: committed entries live on a majority, so an eligible candidate (≥ floor) always exists. A node behind the floor correctly fails its own freshness check and should not lead. **Conclusion:** feed the existing floor unchanged; the guard is on the *accept*, not the *use*. -2. **Reconcile-free consume — RESOLVED.** New `ha_observe_heartbeat_committed(c, lsn, tag)`: monotonic-max update only; does NOT call `ha_rejoin_reconcile`/truncate (Req 3.1). Distinct from `cluster_apply_ctrlhello_reply` (which truncates at join). -3. **config_tag atomicity — RESOLVED.** Promote `last_known_committed_config_tag` to `_Atomic uint64_t` (`ha.h:172`); update the 2 existing writers (`ha_convergence.c:377`, `ha.c` election path) to `atomic_store`. New heartbeat writer + the `ctrlstatus` reader justify it. -4. **Sender scoping — RESOLVED.** New `xce_ha_heartbeat_req(ep, term, committed_lsn, committed_config_tag)`; generic `xce_nop_req` untouched (client nop unaffected). -5. **Opcode/body pairing — RESOLVED.** Send `OPCode_nop` WITH a `heartbeatmsg` body (the Request union already has `heartbeatmsg@25`; opcode and body are independent, as `OP_hello` already carries different bodies). Dispatch still routes by `OPCode_nop`; the handler branches on `body_which == Request_body_heartbeatmsg`. No new opcode (Option A over B). diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json b/.kiro/specs/ha-heartbeat-committed-propagation/spec.json deleted file mode 100644 index 104b28de..00000000 --- a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "feature_name": "ha-heartbeat-committed-propagation", - "created_at": "2026-06-20T00:00:00Z", - "updated_at": "2026-06-20T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation": { - "completed": true, - "tasks_done": "1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1", - "notes": "Code + safety verification implemented, reviewed, committed (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562); controller + lib suites green; the vote-floor safety property (Req 3.2) is verified by an executable test against the real ha_vote_self_freshness gate. 4.1 (live E2E, Req 1.2/1.3) VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy: standby committedLsn tracked the primary within one heartbeat interval with no re-join, and continued advancing correctly across a failover. See tasks.md Implementation Notes." - }, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T09:45:00Z", - "note": "RETRY of a validation pass that crashed silently mid-live-soak ~4 hours earlier with no artifacts/logs (stale empty deploy dir /home/claude/tmp/xce-ha-heartbeat-committed-deploy found and removed at start of this pass; no evidence it was a product bug rather than an environment/harness loss). Prior state: all 11 tasks (1.1-4.1) were [x] in tasks.md, 4.1 already live-verified 2026-07-02, but phase was stuck at \"tasks-generated\" with no validate_impl record (same administrative-oversight pattern as ha-wal-replication / ha-entity-snapshots / ha-commit-durability). This pass ran immediately after two sibling specs: raft-drive-async (NO-GO, bug-1447 ASAN SEGV in netio.c SSL handshake) and ha-wal-replication (GO, clean 10/10 soak) -- extra weight was given to reproducing or ruling out bug-1447's signature in this spec's own live soak.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan), binaries confirmed with session-fresh mtimes (controller_unit_test, xce_test, xce_controller). Standalone controller_unit_test run directly with ASAN_OPTIONS=detect_leaks=0 (authoritative over meson's aggregate line per this project's convention): 972 Tests, 0 Failures, 0 Ignored, OK -- zero \":FAIL:\" occurrences, exit 0. All heartbeat-owned tests explicitly confirmed PASS: test_ha_heartbeat_committed.c's 8 tests (test_observe_advances_on_higher, test_observe_noop_on_equal_or_lower, test_observe_noop_on_zero, test_observe_no_reconcile_no_other_mutation, test_consume_term_valid_advances, test_consume_stale_term_ignored, test_consume_primary_does_not_consume, test_consume_null_or_zero_noop) and test_ha_vote_floor_heartbeat.c's 4 tests (test_heartbeat_raises_vote_floor_to_max, test_candidate_below_raised_floor_rejected, test_candidate_at_or_above_raised_floor_accepted, test_lower_heartbeat_never_lowers_floor). TBD/TODO grep on request.c/request.h/ha.c/ha.h/cluster.c/both test files: only pre-existing, already-flagged-unrelated hits (ha.c:1844 ha-term-safety stub; cluster.c group-building and config-generation TODOs that predate and are untouched by this feature). Secrets grep clean.\n\nINTEGRATION (cross-task wiring confirmed by direct code read against design.md's Architecture + File Structure Plan): producer -- primary_heartbeat_fn (ha.c) calls xce_ha_heartbeat_req(ep, ha->term, persist.committed_lsn, ha->committed_config_tag) (request.h:33 decl, request.c:1257 impl) instead of bare xce_nop_req, sending OP_nop with a heartbeatmsg body; consumer -- cluster.c's OP_nop handler (~line 300-375) checks request.body_which == Request_body_heartbeatmsg and calls ha_consume_heartbeat_committed(c, ...) (ha.c:8090-8170), which: (1) gates out CTRL_PRIMARY nodes (Req 3.4), (2) runs the term-guard via the raft library's real UpdateTerm path (zero-entry ha_consume_append_entries call, not a hand-rolled check) and rejects hb->term < myterm (Req 3.2), (3) calls the reconcile-free monotonic updater ha_observe_heartbeat_committed (ha.c:8033, ha.h:1327) which advances last_known_committed_lsn/_config_tag only on strictly-greater LSN with no WAL truncation and no ha_rejoin_reconcile call (Req 2.1/2.2/3.1/3.3/4.2), and (4) calls ha_standby_advance_committed_and_apply -- a later hardening addition (commit 40f510fb, 2026-07-11, bug-1433/1434 fix) that drives this node's own committed_lsn + apply pump from the heartbeat-learned position; this call is additive to the design's Component boundary, not a violation (design.md's Allowed Dependencies already cover read/write of last_known_committed_* and ha->term). Both _Atomic migration writers confirmed: ha.c:8048 (election path) and ha_convergence.c:368 (cluster_apply_ctrlhello_reply join path), both atomic_store_explicit. ha.h declares both new functions (ha_observe_heartbeat_committed:1327, ha_consume_heartbeat_committed:1346) plus the _Atomic field (ha.h:185). Vote-floor formula (ha.c:2570, max(durable_floor, last_known_committed_lsn)) confirmed unchanged -- this spec only makes its input fresher, per design.md's explicit Out-of-Boundary note. All 6 implementation commits (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562) plus the later hardening commit (40f510fb) confirmed present in git history.\n\nCOVERAGE: all 4 requirement sections map to completed tasks with no gaps -- Req 1 (producer) -> tasks 1.2, 3.1; Req 2 (monotonicity) -> task 2.1; Req 3 (advisory-only/term-guard/no-reconcile) -> tasks 2.1, 2.2, 3.2; Req 4 (liveness/no-regression) -> tasks 2.1, 2.2, 4.1. File Structure Plan MINOR drift (non-blocking): design.md's New Files section only names test_ha_heartbeat_committed.c; the vote-floor safety integration test landed as a second file, test_ha_vote_floor_heartbeat.c (both registered in controller_test.c + controller/meson.build) -- functionally equivalent/additive coverage of the same Req 3.2 property design.md calls for, same category of drift already accepted on the ha-wal-replication sibling. design.md's ha.h path is written as src/controller/include/ha.h; actual file is src/controller/ha.h (doc-only path drift, already flagged in tasks.md's own Implementation Notes as a non-blocking learning, not re-litigated here).\n\nRUNTIME LIVENESS (the blocking check, given the bug-1447 sibling finding and this being a retry of a crashed prior attempt): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir (~/tmp/xce-ha-heartbeat-committed-soak, confirmed ext4 via df -T before the run, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, all 3 nodes (ctrl-1, ctrl-2, ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied: ctrl-1 rounds 1-4/6-9, ctrl-2 rounds 1-4/7-10, ctrl-3 rounds 5/6/10). Explicitly grepped the full soak log and all per-node logs for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker): zero matches across all six markers. Also grepped broadly for any other crash/abort signature (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches. This does not prove bug-1447 is fixed or absent globally (it is a rare, timing-dependent race in a different subsystem, raft-drive-async's reconnect-offload path, which this spec's boundary does not touch) -- but it does establish that this spec's own heartbeat-borne committed-propagation path runs clean through a full supported failover lifecycle, satisfying this spec's own regression contract (Req 4).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record); noted here only as a risk actively checked for and ruled out in this run, not adopted as this spec's blocker. No new bugs found or logged during this validation pass; the prior silent crash (~4 hours before this retry) left no diagnosable artifacts and could not be attributed to a specific cause -- treated as an environment/harness loss, not a product regression, since this clean retry reproduced nothing.", - "known_limitations": [ - "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered by raft-drive-async's own reconnect-offload cadence. Did not reproduce in this spec's clean 10/10 run, but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-heartbeat-committed-propagation; tracked entirely under raft-drive-async's remediation.", - "The prior validation attempt (~4 hours before this retry) crashed silently mid-soak with no logs or diagnosable artifacts (only an empty stale deploy-dir marker was found). Could not be root-caused or attributed to a product bug; this retry's clean, fully-instrumented 10/10 soak found nothing reproducing it, so it is treated as a one-off environment/harness loss rather than a regression, but it was not positively explained.", - "File Structure Plan drift (non-blocking, doc-only): design.md's New Files section names only test_ha_heartbeat_committed.c; the vote-floor safety integration test landed as a second registered file, test_ha_vote_floor_heartbeat.c. Coverage is equivalent/additive; a future doc pass could update design.md to match.", - "design.md's File Structure Plan lists the ha.h path as src/controller/include/ha.h; the actual file is src/controller/ha.h. Cosmetic path drift already flagged in tasks.md's own Implementation Notes, not a functional gap." - ] - } -} diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md b/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md deleted file mode 100644 index f9486d11..00000000 --- a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md +++ /dev/null @@ -1,66 +0,0 @@ -# Implementation Plan - -> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor. -> The work runs along a single request path (atomic migration → libxce sender → primary stamp / standby consume → vote-floor verification → E2E), so tasks are sequential by design; no `(P)` markers. The two foundation tasks (1.1 atomic migration, 1.2 sender) touch independent files but are kept ordered for review simplicity. The HA-critical change is the standby consume + term-guard (task 2.2) and its vote-floor interaction (task 3.2). - -- [x] 1. Foundation: atomic state + heartbeat sender -- [x] 1.1 Migrate `last_known_committed_config_tag` to `_Atomic` - - Change `ha->last_known_committed_config_tag` from a plain `uint64_t` to `_Atomic uint64_t` (`src/controller/include/ha.h`) and update its existing writers — `cluster_apply_ctrlhello_reply` (`ha_convergence.c`) and the election safe-commit path (`ha.c`) — to `atomic_store_explicit(..., release)`, and any reader (incl. the controller-status `ctrlstatus_fill`) to `atomic_load_explicit(..., acquire)`. Pure type/access migration — no behavior change. - - Observable: the controller suite builds clean and passes unchanged; every read/write of the field uses atomic accessors (no plain access remains). - - _Requirements: 2.3_ - - _Boundary: ha.h + last_known_committed_config_tag accessors_ -- [x] 1.2 Add the HA heartbeat sender `xce_ha_heartbeat_req` - - Add `xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag)` (`src/include/request.h` + `src/lib/request.c`) that sends an `OPCode_nop` request carrying a `heartbeatmsg` body (`HeartbeatMsg{term, committedLsn, committedConfigTag}`; `senderPriority` unused). Leave the generic `xce_nop_req` untouched. - - TDD: a round-trip test encodes via the new sender and decodes a request with `body_which == Request_body_heartbeatmsg` carrying the three field values; a guard assertion confirms `xce_nop_req` still sends a bodyless nop. - - Observable: the round-trip test decodes the three fields from the new sender, and the bodyless-`xce_nop_req` guard still holds. - - _Requirements: 1.1_ - - _Boundary: libxce sender_ - -- [x] 2. Standby consume (reconcile-free, term-guarded, monotonic) -- [x] 2.1 Add the reconcile-free monotonic updater `ha_observe_heartbeat_committed` - - Add `ha_observe_heartbeat_committed(cluster_t* c, uint64_t committed_lsn, uint64_t committed_config_tag)` (`ha.c`, declared in `ha.h`): advance `last_known_committed_lsn` only when the carried LSN is strictly greater (monotonic; atomic), advancing `last_known_committed_config_tag` together with it; no-op when `<=` current or zero. It must NOT call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state. - - TDD: unit tests assert advance-on-higher (LSN + config tag updated), no-op on equal/lower/zero, config tag never regresses independently, and — critically — that the call performs no WAL truncation / no reconcile and mutates nothing beyond `last_known_committed_*`. - - Observable: the unit tests go red → green; the no-truncation / no-mutation assertions hold. - - _Requirements: 2.1, 2.2, 2.3, 3.1, 3.3_ - - _Boundary: ha_observe_heartbeat_committed_ - - _Depends: 1.1_ -- [x] 2.2 Wire the standby `OP_nop` consume branch + term-guard - - In the `cluster.c` `OP_nop` handler, after the existing liveness reset + `StateReport` reply, if `request.body_which == Request_body_heartbeatmsg` and the responder is non-primary: apply the term-guard (`hb->term >= atomic_load(ha->term)`) and, only if it passes, call `ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag)`. A missing body, a failed term-guard, or a zero committed value updates nothing and leaves all existing `OP_nop` behavior intact. - - TDD: tests confirm a term-valid higher-committed heartbeat advances `last_known_committed_*`; a stale-term heartbeat (`hb->term < ha->term`) carrying a higher committed leaves committed unchanged but STILL resets liveness; a bodyless / zero-committed nop is processed as liveness only; and a primary responder does not consume an inbound committed value. - - Observable: the tests pass for the term-valid, stale-term, bodyless, and primary-responder cases, with liveness reset preserved in every case. - - _Requirements: 1.2, 3.2, 3.4, 4.1, 4.2_ - - _Boundary: cluster.c OP_nop handler_ - - _Depends: 2.1_ - -- [x] 3. Primary produce + safety verification -- [x] 3.1 Stamp committed + term on the primary heartbeat - - In `primary_heartbeat_fn` (`ha.c`), replace the per-standby `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, atomic_load(ha->term), atomic_load(persist.committed_lsn), atomic_load(ha->committed_config_tag))`. Preserve the `XCE_TESTING` `ha_nop_req_hook` seam and the existing heartbeat send-failure `WARN` and `last_push_time` bookkeeping. - - TDD/observable: a test (or the `ha_nop_req_hook` seam) confirms the primary's heartbeat now carries its current committed LSN / committed config tag / term; the send-failure and liveness-eviction behavior of `primary_heartbeat_fn` is unchanged. - - _Requirements: 1.1, 3.4_ - - _Boundary: primary_heartbeat_fn_ - - _Depends: 1.2_ -- [x] 3.2 Verify the vote-freshness floor consumes the fresher value safely - - Add an integration test proving the existing freshness gate, with `last_known_committed_lsn` raised by a heartbeat (via `ha_observe_heartbeat_committed`), rejects a vote candidate whose last-log LSN is below the raised floor and accepts one at/above it — confirming the floor formula `max(durable_floor, last_known_committed_lsn)` is read unchanged and is never lowered by this feature. No production change to the vote path. - - Observable: the test shows the candidate-below-floor rejected and candidate-at/above-floor accepted after a heartbeat raise, and no edit to the floor formula. - - _Requirements: 3.2_ - - _Boundary: vote-freshness interaction test_ - - _Depends: 2.1, 2.2_ - -- [x] 4. Validation (live cluster) <!-- VERIFIED 2026-07-02 --> -- [x] 4.1 End-to-end: standby committed tracks the primary within a heartbeat interval <!-- VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy — see Implementation Notes --> - - On a live 3-node `/xce-deploy-ha` cluster, submit jobs so the primary advances its committed position, then query a STANDBY repeatedly: its `ctrlstatus` `committedLsn` advances to track the primary within roughly one heartbeat interval, without any re-join — closing the controller-status Req 10 staleness caveat. Confirm existing liveness/failover behavior is unaffected. - - Observable: the live run shows a standby's `committedLsn` rising to track the primary between joins (not frozen at the join value), and the cluster's liveness/failover behavior is unchanged; manual environment prerequisite (live cluster, real fs). - - _Requirements: 1.2, 1.3_ - - _Depends: 3.1, 2.2_ - -## Implementation Notes -- 4.1 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy`, queried standby ctrl-2's `ctrlstatus` before/after submitting 5 more jobs (4s wait, inside the 3000ms heartbeat interval): `committedLsn` rose `4 → 9`, exactly matching the 5 new jobs, with NO re-join — confirms committed propagates to standbys via heartbeat, not just at join. Then killed the primary (ctrl-1) and confirmed liveness/failover unaffected: `wait-primary` succeeded, term advanced `3 → 5` (new primary elected), and `committedLsn` continued advancing post-failover (`9 → 10`). PASS on both halves of the Observable. Found and logged (unrelated, low-priority) during this session: bug-1400 (xce_test cmd_jobsub SEGVs on malformed CLI args) and bug-1401 (ctrlstatus reply decode leak in `decode_CtrlStatus_ptr`). -- The HA heartbeat is `OPCode_nop` with a `heartbeatmsg` body — opcode and body are independent in this schema (mirrors `OP_hello` carrying `ctrlhellomsg` vs `replymsg`). Dispatch routes by `OPCode_nop` (unchanged); the consumer branches on `body_which`. -- Do NOT reuse `cluster_apply_ctrlhello_reply` for the heartbeat path — it truncates (`ha_rejoin_reconcile`). The heartbeat updater is reconcile-free (Req 3.1). -- Do NOT touch the client keepalive `OPCode_heartbeat` / `xce_heartbeat` (a different message) or the generic `xce_nop_req` (client nop). -- 1.1 LEARNING: `ha.h` is `src/controller/ha.h` (NOT `src/controller/include/ha.h`). `last_known_committed_config_tag` is now `_Atomic uint64_t` (`ha.h:172`), sibling of `last_known_committed_lsn` (`ha.h:170`); read via `atomic_load_explicit(acquire)`, write via `atomic_store_explicit(release)`. -- TEST-RUN LEARNING (2.x/3.x): the meson test wrapper (`xce-build.sh controller`) is NON-DETERMINISTIC ("OK"/"FAIL"/"Timeout") because of ~149 PRE-EXISTING LeakSanitizer leaks from `ha_start_election_async`/`ha_start_prevote_async` (`test_ha_start_election_async.c`, `test_ha_rdhup_integration.c`) — unrelated to this spec (identical count on a stashed base). AUTHORITATIVE verification = run the binary directly: `cd src && ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test`, then check rc=0 and zero `:FAIL:` markers. -- 3.1 LEARNING: primary stamps its OWN `ha->term` / `c->persist.committed_lsn` / `ha->committed_config_tag` (all `_Atomic`, acquire-load once per tick) — NOT the standby `last_known_*`. Test seam: new `ha_heartbeat_req_hook(ep,term,lsn,tag)` (`XCE_TESTING`, NULL default) with precedence new-hook → legacy `ha_nop_req_hook` → real `xce_ha_heartbeat_req`, so existing `ha_nop_req_hook` liveness tests stay green. -- 1.2 LEARNING: the C-side `heartbeatmsg` body support PRE-EXISTED (no typemap edit needed) — `heartbeat_msg_t` typedef `typemap.h:17`, `xce_request_t.heartbeatmsg` member `typemap.h:287`, `struct heartbeat_msg_s{sender_priority,term,committed_lsn,committed_config_tag}` `typemap.h:445` (order matches schema `@0..@3`); enum `Request_body_heartbeatmsg == 22` (rides `OPCode_nop`). The codec (`encode/decode/free_HeartbeatMsg` + the request arm) was generated; only producer/consumer were missing. Test round-trip pattern: heap-`calloc` packet consumed+freed by `xce_decode` (caller does NOT free pkt), decoded body freed by `xce_free_request` (see `test_wal_wire.c`). Sender uses `xce_request(ep, &req)` (non-blocking). -- 4.1 HAND-OFF (live-cluster manual validation — run on a real-fs deploy, NOT the overlayfs sandbox): on a `/xce-deploy-ha` 3-node cluster, (1) `jobsub` a steady trickle of jobs so the primary's committedLsn keeps advancing; (2) repeatedly `ctrlstatus` a STANDBY (any-node hint view) and watch its `committedLsn` — it should RISE to track the primary within ~one heartbeat interval (no re-join), instead of being frozen at the join value or 0 (the controller-status Req 10 staleness this spec removes); (3) confirm liveness/failover is unaffected (kill the primary, a standby still takes over normally). PASS = standby committedLsn tracks the live primary between joins, with no liveness regression. Rebuild first so the cluster has the producer (task 3.1) + consumer (task 2.2). -- The vote-floor formula is NOT edited — task 3.2 is a verification test only. Safety rests on the term-guard (task 2.2) + monotonicity (task 2.1) + cluster-monotonic committed; see design.md Security Considerations. diff --git a/.kiro/specs/ha-heartbeat-liveness/design.md b/.kiro/specs/ha-heartbeat-liveness/design.md deleted file mode 100644 index e46ea42e..00000000 --- a/.kiro/specs/ha-heartbeat-liveness/design.md +++ /dev/null @@ -1,382 +0,0 @@ -# Design Document: ha-heartbeat-liveness - -## Overview - -This design replaces the dedicated `hb_timeout_thread` + `sem_timedwait` watchdog on standbys with activity-based liveness tracking. The standby records a `time_t last_activity` timestamp on every inbound primary-channel message and a delaymgr periodic task polls it to detect silence. The primary extends its existing `primary_heartbeat_fn` delaymgr task to track per-standby idle time and send a NOP only when no other traffic has been sent recently. - -The change eliminates the race where `committing_timeout_ms` ≈ `heartbeat_timeout_ms` caused standbys that received ctrlhello (but not yet a NOP) to falsely trigger re-election. It also provides explicit missed-heartbeat tolerance: silence must reach `heartbeat_timeout_ms` before failover fires, meaning operators can set `heartbeat_timeout_ms = N × heartbeat_interval_ms` to tolerate N-1 misses. - -### Goals - -- Replace blocking `hb_timeout_thread` with a delaymgr poll task (no new thread) -- Any inbound primary-channel message resets the liveness deadline -- Primary sends NOP only when idle for `heartbeat_interval_ms` toward a standby -- Missed-heartbeat tolerance expressed via existing `heartbeat_timeout_ms` / `heartbeat_interval_ms` ratio -- Startup validation enforces `heartbeat_timeout_ms >= 2 × heartbeat_interval_ms` - -### Non-Goals - -- Raft election, vote RPCs, `ha_start_election_async` — unchanged -- EPOLLRDHUP fast-path failover — unchanged and not modified -- WAL replication protocol, config push protocol, catch-up logic -- Adding a new configurable `inactive_heartbeat_count` parameter - -## Boundary Commitments - -### This Spec Owns - -- `ha_state_t.last_activity` — new `time_t` field replacing the semaphore-based deadline -- `standby_ep_ctx_t.last_push_time` — new `time_t` field tracking per-standby idle time -- `ha_arm_heartbeat_timeout_vtask` / `ha_disarm_heartbeat_timeout_vtask` — replaced with delaymgr-based equivalents -- `hb_vtask_t` struct and `hb_timeout_thread` function — removed -- `primary_heartbeat_fn` — extended to check `last_push_time` per standby before sending NOP -- `ha_on_heartbeat_received` — semantics preserved; updates `last_activity` instead of posting semaphore -- Startup validation: `heartbeat_timeout_ms >= 2 × heartbeat_interval_ms` check in `validate_peer_count` or new `validate_heartbeat_config` - -### Out of Boundary - -- Raft state machine, raft.c, election timeout delaymgr tasks -- EPOLLRDHUP handler `ha_on_primary_ctrl_fd_rdhup` — called by the new poll task but not modified -- `repl.c` replication logic, WAL ACK processing -- `check_workers_timeout` worker liveness (separate delaymgr task, different direction) - -### Allowed Dependencies - -- `delaymgr` (`delay.h`): `delay_task_ms()` for standby poll and primary idle check -- `ha_on_primary_ctrl_fd_rdhup()`: called by the standby poll task when timeout fires -- `xce_nop_req()`: called by primary idle check when standby is idle -- `ha->standby_endpoints` (zarray): iterated by primary heartbeat fn (existing) -- `clock_gettime(CLOCK_MONOTONIC)` for all timestamp reads - -### Revalidation Triggers - -- If `heartbeat_timeout_ms` or `heartbeat_interval_ms` semantics change in `ha_config_t` -- If `standby_ep_ctx_t` ownership or lifetime changes (it is freed in `vote_ep_err_handler`) -- If `ha->standby_endpoints` iteration pattern changes in `primary_heartbeat_fn` -- If a new role state is added between CTRL_STANDBY and CTRL_ELECTING - -## Architecture - -### Existing Architecture Analysis - -**Current standby liveness path:** -1. `ha_arm_heartbeat_timeout_vtask()` spawns `hb_timeout_thread` (pthread) -2. Thread blocks on `sem_timedwait(&t->reset_sem, &deadline)` -3. `ha_on_heartbeat_received()` posts `reset_sem` to reset deadline -4. On timeout: calls `ha_on_primary_ctrl_fd_rdhup()` - -**Current primary heartbeat path:** -1. `ha_start_primary_heartbeat()` schedules `primary_heartbeat_fn` via `delay_task_ms` -2. `primary_heartbeat_fn` iterates `ha->standby_endpoints`, calls `xce_nop_req()` on each -3. Re-arms itself via `delay_task_ms` at `heartbeat_interval_ms` if still PRIMARY - -**Problem**: `ha_arm_heartbeat_timeout_vtask` is only called in `raft_cb_become_follower`. Standbys entering via the voter-only path (no local config, tag=0) go through `myapp_run` directly without `raft_cb_become_follower`, so their watchdog is never armed until ctrlhello triggers `ha_arm_heartbeat_timeout_vtask`. But ctrlhello arrives at the same time as the committing_timeout — the semaphore deadline starts too late and fires immediately. - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph Standby - LastActivity[ha_state_t.last_activity] - PollTask[liveness_poll_fn delaymgr task] - OnMessage[cluster.c any-msg handler] - RdhupFn[ha_on_primary_ctrl_fd_rdhup] - OnMessage -->|clock_gettime write| LastActivity - PollTask -->|reads elapsed| LastActivity - PollTask -->|timeout exceeded| RdhupFn - end - subgraph Primary - LastPush[standby_ep_ctx_t.last_push_time] - HbFn[primary_heartbeat_fn delaymgr task] - AnySend[any xce_request to standby] - NopReq[xce_nop_req] - AnySend -->|clock_gettime write| LastPush - HbFn -->|reads elapsed per standby| LastPush - HbFn -->|idle exceeded| NopReq - NopReq -->|resets| LastActivity - end -``` - -**Key decisions:** -- Standby uses a single `time_t last_activity` in `ha_state_t` (not per-message-type) — simplest, any message counts -- Primary uses per-standby `last_push_time` in `standby_ep_ctx_t` — already iterated by heartbeat fn -- Both tasks use delaymgr; no new pthreads - -## File Structure Plan - -### Modified Files - -- `src/controller/ha.h` — Add `time_t last_activity` to `ha_state_t`; remove `hb_vtask_t` typedef; update `ha_arm_heartbeat_timeout_vtask` / `ha_disarm_heartbeat_timeout_vtask` / `ha_on_heartbeat_received` signatures (no API change — same names, different impl) -- `src/controller/ha.c` — Remove `hb_vtask_t`, `hb_timeout_thread`, `s_hb_vtask` static. Replace `ha_arm_heartbeat_timeout_vtask` with delaymgr version. Add `liveness_poll_fn`. Update `ha_on_heartbeat_received` to write `last_activity`. Extend `primary_heartbeat_fn` to check `standby_ep_ctx_t.last_push_time`. -- `src/controller/cluster.c` — Update `OPCode_nop` handler to call `ha_on_heartbeat_received` unconditionally when role is CTRL_STANDBY (remove `ep->fd == primary_ctrl_fd` guard — `last_activity` write is safe from any primary-connected ep) -- `src/controller/config_repl.c` — Keep `ha_arm_heartbeat_timeout_vtask` call in `handle_ctrlhello` (arms the delaymgr poll). Keep `ha_on_heartbeat_received` call (now writes timestamp). Remove the workaround NOP added in prior session from `on_standby_connected` in ha.c. -- `src/controller/startup_validation.c` — Add `validate_heartbeat_config()` function checking `heartbeat_timeout_ms >= 2 × heartbeat_interval_ms`. -- `src/controller/include/` or `ha.h` — Extend `ha_config_t` with no new fields (uses existing `heartbeat_timeout_ms` and `heartbeat_interval_ms`). - -### Internal struct change (ha.c only, not in .h) - -- `standby_ep_ctx_t` — Add `time_t last_push_time` field (ha.c internal struct) - -## System Flows - -### Standby Liveness Poll - -```mermaid -sequenceDiagram - participant Primary - participant StandbyCluster as cluster.c OPCode handler - participant LastActivity as ha_state_t.last_activity - participant PollTask as liveness_poll_fn - participant Delaymgr - - Primary->>StandbyCluster: any msg ctrlhello / NOP / WAL - StandbyCluster->>LastActivity: clock_gettime write - Delaymgr->>PollTask: fires every heartbeat_interval_ms/2 - PollTask->>LastActivity: read elapsed - alt elapsed < heartbeat_timeout_ms - PollTask->>Delaymgr: re-arm - else elapsed >= heartbeat_timeout_ms - PollTask->>PollTask: ha_on_primary_ctrl_fd_rdhup - end -``` - -### Primary Idle-Triggered NOP - -```mermaid -sequenceDiagram - participant HbFn as primary_heartbeat_fn - participant LastPush as standby_ep_ctx_t.last_push_time - participant StandbyEp as standby endpoint - - HbFn->>LastPush: read elapsed for each standby - alt elapsed >= heartbeat_interval_ms - HbFn->>StandbyEp: xce_nop_req - HbFn->>LastPush: write clock_gettime - else elapsed < heartbeat_interval_ms - HbFn->>HbFn: skip NOP for this standby - end - HbFn->>HbFn: re-arm delay_task_ms -``` - -## Requirements Traceability - -| Requirement | Summary | Component | Key Design Element | -|-------------|---------|-----------|-------------------| -| 1.1 | Record last-activity timestamp | `ha_state_t.last_activity` | `time_t` written by any-msg handler | -| 1.2 | Periodic check elapsed vs timeout | `liveness_poll_fn` | delaymgr task at `heartbeat_interval_ms / 2` | -| 1.3 | Any message resets deadline | cluster.c OPCode dispatch | all opcodes on primary fd write `last_activity` | -| 1.4 | Failover on timeout | `liveness_poll_fn` → `ha_on_primary_ctrl_fd_rdhup` | timeout check in poll fn | -| 1.5 | Stop poll on role change | `ha_disarm_heartbeat_timeout_vtask` | cancel delaymgr task on transition | -| 1.6 | No blocking thread | Replace pthread with delaymgr | `hb_timeout_thread` removed | -| 2.1 | Declare dead after `heartbeat_timeout_ms` | `liveness_poll_fn` threshold | uses existing config field | -| 2.2 | N=3 tolerates 2 missed | configuration convention | `heartbeat_timeout_ms = 3 × interval` | -| 2.3 | N=2 tolerates 1 missed | configuration convention | `heartbeat_timeout_ms = 2 × interval` | -| 2.4 | Startup validation | `validate_heartbeat_config()` | `timeout >= 2 × interval` check | -| 3.1 | Standby tracks idle per ep | `standby_ep_ctx_t.last_push_time` | per-ep field, updated on every send | -| 3.2 | NOP after one idle interval | `primary_heartbeat_fn` | send NOP when `elapsed >= heartbeat_interval_ms` | -| 3.3 | Any send resets idle timer | all send sites in `primary_heartbeat_fn` and `on_standby_connected` | write `last_push_time` after every `xce_request` | -| 3.4 | No redundant NOP | `primary_heartbeat_fn` elapsed check | skip NOP if `elapsed < heartbeat_interval_ms` | -| 3.5 | Stop idle task on demotion | `primary_heartbeat_fn` role guard | existing guard returns 0 on non-PRIMARY | -| 4.1 | ctrlhello counts as activity | cluster.c OPCode_hello handler | calls `ha_on_heartbeat_received` | -| 4.2 | Committing window is safe | ctrlhello resets `last_activity` | standby deadline starts from ctrlhello, not startup | -| 4.3 | Guarantee no premature failover | `liveness_poll_fn` threshold | fires only after full `heartbeat_timeout_ms` | -| 5.1 | Raft unchanged | no changes to raft.c, ha_start_election_async | boundary commitment | -| 5.2 | EPOLLRDHUP immediate | `ha_on_primary_ctrl_fd_rdhup` unchanged | poll fn calls same fn on timeout | -| 5.3 | Single-node no interference | `primary_heartbeat_fn` skips when no standbys | existing check | -| 5.4 | No poll in INIT/ELECTING | `ha_arm_heartbeat_timeout_vtask` called only from STANDBY paths | callsite guard | -| 5.5 | Log on arm/reset/fire | `ha_arm_heartbeat_timeout_vtask`, `ha_on_heartbeat_received`, `liveness_poll_fn` | as-built: INFO on arm, DEBUG3 on reset (avoids log spam per-message), WARN on fire — not uniformly INFO as originally planned | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | -|-----------|-------|--------|--------------|-----------------| -| `ha_state_t.last_activity` | State | Timestamp of last primary msg | 1.1, 1.3, 4.1 | `clock_gettime` | -| `liveness_poll_fn` | ha.c | Delaymgr task replacing hb_timeout_thread | 1.2, 1.4, 1.5, 2.1–2.3, 4.2, 4.3 | delaymgr, `ha_on_primary_ctrl_fd_rdhup` | -| `ha_arm_heartbeat_timeout_vtask` | ha.c | Schedule liveness poll (replaces thread spawn) | 1.5, 1.6 | delaymgr | -| `ha_disarm_heartbeat_timeout_vtask` | ha.c | Cancel liveness poll | 1.5 | delaymgr | -| `ha_on_heartbeat_received` | ha.c | Write `last_activity` timestamp | 1.3, 4.1 | `clock_gettime` | -| `standby_ep_ctx_t.last_push_time` | ha.c | Per-standby idle timer for primary | 3.1, 3.3, 3.4 | `clock_gettime` | -| `primary_heartbeat_fn` (extended) | ha.c | Idle-triggered NOP; checks `last_push_time` | 3.1–3.5 | `xce_nop_req`, delaymgr | -| `validate_heartbeat_config` | startup_validation.c | Enforce `timeout >= 2 × interval` | 2.4 | `ha_config_t` | - -### ha.c — Standby Liveness Poll - -#### `liveness_poll_fn` (new) - -| Field | Detail | -|-------|--------| -| Intent | Delaymgr callback; checks `last_activity` elapsed; triggers failover or re-arms | -| Requirements | 1.2, 1.4, 2.1, 4.2, 4.3 | - -**Responsibilities:** -- Read `ha->last_activity` (volatile `time_t`) -- Compute `elapsed = now - ha->last_activity` -- If `elapsed >= heartbeat_timeout_ms`: call `ha_dispatch_rdhup_failover(ha, c)`; do NOT re-arm (Note: as-built this offloads to a vtask that calls `ha_on_primary_ctrl_fd_rdhup` off the delaymgr thread, rather than calling it inline as originally designed — added by raft-drive-async, see that spec's design.md) -- Else: call `delay_task_ms(&c->delaymgr, poll_interval_ms, 1, liveness_poll_fn, ctx)` to re-arm -- Log INFO on fire; suppress per-poll logging when resetting (only log on state change) - -**Poll interval**: `heartbeat_interval_ms / 2` — polls twice per heartbeat interval, keeping detection latency within one interval of the actual timeout. - -**State struct** (`liveness_poll_ctx_t`, ha.c internal): -```c -typedef struct { - ha_state_t *ha; - cluster_t *c; - int cancelled; /* set to 1 by ha_disarm_heartbeat_timeout_vtask */ -} liveness_poll_ctx_t; -``` - -**Contracts**: State [x] - -#### `ha_arm_heartbeat_timeout_vtask` (replaced) - -**New implementation** (same API, different body): -- Allocate `liveness_poll_ctx_t`, store pointer in `s_liveness_ctx` (static, replaces `s_hb_vtask`) -- Write `clock_gettime(CLOCK_MONOTONIC)` to `ha->last_activity` (arm the deadline from now) -- Schedule `delay_task_ms(&c->delaymgr, poll_interval_ms, 1, liveness_poll_fn, ctx)` -- Log INFO: "HA: standby liveness poll armed (timeout=%ums, poll=%ums)" -- No pthread, no semaphore - -#### `ha_disarm_heartbeat_timeout_vtask` (replaced) - -**As-built implementation** (corrected from the original design below — freeing here would race a concurrently-running delaymgr task): -- Set `s_liveness_ctx->cancelled = 1` (poll fn checks this and skips re-arm + failover) -- Set `s_liveness_ctx` to NULL — does NOT free the ctx here -- The ctx is freed lazily by `liveness_poll_fn` itself, the next time it fires and observes `cancelled == 1` (production builds only; `XCE_TESTING` callers own stack-allocated ctxs) — avoids a use-after-free/double-free race with an in-flight delaymgr thread -- No pthread_join needed — delaymgr tasks run to completion and are not cancellable mid-flight; the `cancelled` flag makes the next invocation a no-op - -#### `ha_on_heartbeat_received` (updated) - -**As-built implementation** (the DEBUG3 log was added, not omitted as originally planned; a PreVote-cancellation block was appended later — see note): -```c -void ha_on_heartbeat_received(ha_state_t *ha) { - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - ha->last_activity = ts.tv_sec; - tlog(TLOG_DEBUG3, "HA: heartbeat received — last_activity updated to %ld", (long)ts.tv_sec); - /* + pre_vote_in_progress cancellation block (Note: added by ha-prevote task 4.2, - * see that spec's design.md — a fresh heartbeat mid-PreVote now abandons the - * round and re-attaches as CTRL_STANDBY) */ -} -``` - -### ha.c — Primary Idle Heartbeat - -#### `primary_heartbeat_fn` (extended) - -**Extension to existing function:** -- For each ep in `ha->standby_endpoints`: cast `ep->data` to `standby_ep_ctx_t` -- Compute `elapsed = now - sctx->last_push_time` -- If `elapsed >= heartbeat_interval_ms`: call `xce_ha_heartbeat_req(ep, term, committed_lsn, committed_config_tag)`, write `sctx->last_push_time = now` (Note: superseded by ha-heartbeat-committed-propagation, see that spec's design.md — the plain `xce_nop_req(ep)` this spec originally specified was replaced with the richer heartbeat that also carries the primary's committed position; both send opcode `OPCode_nop`) -- Else: skip NOP for this standby -- Late-joiner connect path unchanged — `on_standby_connected` writes `last_push_time` on connect - -**`last_push_time` write sites:** -1. `on_standby_connected`: write on TLS handshake complete -2. `primary_heartbeat_fn`: write after sending NOP -3. Replication send path (`repl_broadcast`, WAL entry send): write after any `xce_request` to standby ep - -**Note on Req 3.3**: WAL and config pushes go through `repl_broadcast` in repl.c. Rather than modifying repl.c (out of boundary), the primary_heartbeat_fn can use a simpler heuristic: if the standby's `acked_lsn` advanced since the last poll iteration, treat it as activity (implying a WAL send occurred). This keeps repl.c untouched. Alternatively, `repl_broadcast` can write `last_push_time` — design phase decision deferred to task implementation. - -### startup_validation.c — Timing Validation - -#### `validate_heartbeat_config` (new) - -```c -int validate_heartbeat_config(uint32_t heartbeat_timeout_ms, - uint32_t heartbeat_interval_ms, - char *err_buf, size_t err_len); -``` - -- Returns 0 if `heartbeat_timeout_ms >= 2 * heartbeat_interval_ms` -- Returns -EINVAL with message if violated - -Called from `main.c` after `ha_config_t` is populated, before `ha_init()`. - -### cluster.c — Opcode Dispatch Update - -Remove `ep->fd == c->ha.primary_ctrl_fd` guard from `OPCode_nop` handler; replace with role check: call `ha_on_heartbeat_received` when `role == CTRL_STANDBY` (else `ha_discover_primary` when `role == CTRL_ELECTING`). This avoids tight coupling on fd value while still restricting the call to standby context. (Note: the handler now also gates this on `ha_msg_term_is_stale` — a term-staleness guard added by a later bug-1418-family fix — before checking role; see ha-term-safety's design.md.) - -Also add `ha_on_heartbeat_received` call to `OPCode_hello` (ctrlhello) and `OPCode_walCatchupEntry` handlers — any primary-direction message resets the deadline (Req 1.3, 4.1). (`OPCode_replEntry` is not dispatched through cluster.c's handler switch — that opcode is used internally in ha.c, not at this call site — so the originally-planned call there does not apply.) - -## Data Models - -### State Changes in `ha_state_t` - -| Field | Change | Type | Notes | -|-------|--------|------|-------| -| `last_heartbeat` (struct timespec) | Repurposed / replaced | `time_t last_activity` | Simpler type; CLOCK_MONOTONIC seconds | -| `heartbeat_seen` (sem_t) | Retained | sem_t | Still used for CTRL_INIT heartbeat detection in `ha_wait_for_heartbeat_or_timeout` | - -### State Changes in `standby_ep_ctx_t` (ha.c internal) - -| Field | Change | Type | Notes | -|-------|--------|------|-------| -| `last_push_time` | New | `time_t` | Written on every primary→standby send | - -### Removed State - -| Item | Reason | -|------|--------| -| `hb_vtask_t` struct | Thread-based watchdog removed | -| `s_hb_vtask` static pointer | Replaced by `s_liveness_ctx` | -| `hb_timeout_thread` function | Replaced by `liveness_poll_fn` delaymgr task | - -## Error Handling - -### Liveness Poll Cancellation Race - -**Scenario**: `liveness_poll_fn` fires from delaymgr at the same time `ha_disarm_heartbeat_timeout_vtask` sets `cancelled = 1`. - -**Resolution**: Poll fn checks `cancelled` before calling `ha_on_primary_ctrl_fd_rdhup` and before re-arming. Since `cancelled` is a plain `int` written by the main-context teardown path and read by the delaymgr worker thread, it must be `_Atomic int` to be safe. No mutex needed — it's a one-way flag. - -### EPOLLRDHUP and Poll Task Coexistence (Req 5.2, 5.3, 5.4) - -**Two detection paths, one failover function:** - -| Path | Trigger | Latency | Handler | -|------|---------|---------|---------| -| EPOLLRDHUP fast path | TCP RST / connection close | < 1ms | `ha_on_primary_ctrl_fd_rdhup` directly | -| Liveness poll slow path | Silent dead (no RST) | ≤ `heartbeat_timeout_ms` | `liveness_poll_fn` → `ha_on_primary_ctrl_fd_rdhup` | - -Both paths call the same `ha_on_primary_ctrl_fd_rdhup` function. That function is idempotent on `primary_ctrl_fd == -1` (already correct). The duplicate-fire risk is: -- EPOLLRDHUP fires first → `ha_on_primary_ctrl_fd_rdhup` sets `primary_ctrl_fd = -1` and transitions to CTRL_ELECTING. The liveness poll fn then fires → checks `ha->role == CTRL_STANDBY` before proceeding; if role is already CTRL_ELECTING, the poll fn must skip the failover call. - -**Fix**: `liveness_poll_fn` checks `atomic_load(&ha->role) == CTRL_STANDBY` before calling `ha_dispatch_rdhup_failover`. If role has already transitioned away from CTRL_STANDBY, the poll fn skips the call and does not re-arm. - -**EPOLLRDHUP cancels poll**: `ha_on_primary_ctrl_fd_rdhup` calls `ha_disarm_heartbeat_timeout_vtask`, setting `cancelled = 1` on the poll ctx, before the failover proceeds. This prevents the poll fn from firing a second failover if the delaymgr fires it concurrently. (Note: as-built this call is no longer literally the first step — two guard blocks, an in-progress-election check and an RDHUP debounce window, run first and may return early without disarming; added by raft-ha-core-safety tasks 5.6/5.10b, see that spec's design.md.) - -### `last_activity` Concurrency - -`ha->last_activity` is a `time_t` (typically `int64_t` on LP64). Written by the iomgr vtask thread (via cluster.c dispatch) and read by the delaymgr worker. On LP64 platforms, aligned `time_t` reads/writes are atomic at the hardware level. For strict correctness, declare as `_Atomic time_t` or `volatile time_t` and accept the worst-case one-interval staleness in the poll. - -### `standby_ep_ctx_t` Lifetime - -`standby_ep_ctx_t` is freed in `vote_ep_err_handler` / `standby_ep_err_handler_primary`. `primary_heartbeat_fn` accesses `ep->data` while holding `ha->lock` over the `standby_endpoints` iteration — this is already the existing pattern. No new lifetime risk introduced. - -## Testing Strategy - -### Unit Tests (in `test_ha_failover.c` or new `test_ha_liveness.c`) - -1. `liveness_poll_fn` fires `ha_on_primary_ctrl_fd_rdhup` when elapsed ≥ `heartbeat_timeout_ms` -2. `liveness_poll_fn` re-arms without firing when elapsed < `heartbeat_timeout_ms` -3. `ha_on_heartbeat_received` updates `ha->last_activity` to current time -4. `ha_disarm_heartbeat_timeout_vtask` sets `cancelled=1`; subsequent poll fn invocation is a no-op -5. `validate_heartbeat_config`: `timeout = 2 × interval` passes; `timeout = interval` fails - -### Unit Tests (primary idle NOP, in `test_ha_repl_push.c` or new file) - -6. `primary_heartbeat_fn` sends NOP when `last_push_time` elapsed ≥ `heartbeat_interval_ms` -7. `primary_heartbeat_fn` skips NOP when `last_push_time` elapsed < `heartbeat_interval_ms` - -### Integration Tests (via deploy-ha.sh 3-node cluster) - -8. **Task 11.2 regression**: SIGKILL primary; standby promotes within `heartbeat_timeout_ms + election_timeout_ms + committing_timeout_ms`; no false re-election loops -9. **Race elimination**: 3-node cold start; ctrl-2/3 (tag=0) receive ctrlhello during committing; no RDHUP fires before primary finishes committing (Req 4.2) -10. **Missed-heartbeat tolerance**: Pause primary process for 1 × `heartbeat_interval_ms` (SIGSTOP/SIGCONT); no failover fires; resume; cluster stable - -## Performance & Scalability - -**Poll overhead**: one `clock_gettime` call per standby endpoint per poll period. With 2 standbys and a 1500ms poll interval, this is negligible. - -**Primary NOP suppression**: with active WAL replication, `last_push_time` is continuously fresh — zero NOPs sent during active workloads. NOPs only fire during idle periods, matching the intent. diff --git a/.kiro/specs/ha-heartbeat-liveness/manual-tests.md b/.kiro/specs/ha-heartbeat-liveness/manual-tests.md deleted file mode 100644 index 32c77c6b..00000000 --- a/.kiro/specs/ha-heartbeat-liveness/manual-tests.md +++ /dev/null @@ -1,222 +0,0 @@ -# Manual Integration Tests: ha-heartbeat-liveness — Task 4.5 - -**Requirements:** 2.2, 2.3 -**Depends on:** 4.4 (3-node cluster deployed and primary-crash failover verified) - -Two scenarios test missed-heartbeat tolerance using `SIGSTOP`/`SIGCONT` rather -than `SIGKILL`. `SIGSTOP` freezes a process without closing file descriptors, -so the kernel keeps TCP connections open and no `EPOLLRDHUP` fires on the -standbys. Only the activity-based `liveness_poll_fn` (which polls -`ha->last_activity` every `heartbeat_interval_ms / 2`) can detect the silence. - ---- - -## Environment - -``` -Worktree: /home/rshen/projects/xce/worktrees/persistent_ha -Build dir: src/xce.build -Binary: src/xce.build/controller/xce_controller -Deploy dir: /tmp/xce-deploy (3-node cluster via deploy-ha.sh) -``` - -## Build First - -```bash -# From the worktree -/xce-build controller -``` - ---- - -## Automated Script - -A self-contained bash script runs both scenarios without manual intervention: - -```bash -bash src/tools/test-heartbeat-tolerance.sh -``` - -Expected terminal output: - -``` -=== Task 4.5: Missed-Heartbeat Tolerance (SIGSTOP/SIGCONT) === - ---- Scenario A: SIGSTOP for 1 × heartbeat_interval_ms (3000ms) --- - PASS: Scenario A: no RDHUP fired on ctrl-2 or ctrl-3 during short pause (Req 2.2) - PASS: Scenario A: ctrl-1 still primary after SIGCONT - ---- Scenario B: SIGSTOP for heartbeat_timeout_ms + 500ms (10500ms) --- - PASS: Scenario B: ctrl-2 fired exactly 1 RDHUP during long pause (Req 2.3) - PASS: Scenario B: ctrl-3 fired exactly 1 RDHUP during long pause (Req 2.3) - PASS: Scenario B: ctrl-2 promoted to CTRL_PRIMARY after failover - -========================= -Results: 5 passed, 0 failed -========================= -``` - ---- - -## Manual Step-by-Step (if the script cannot run) - -### One-time setup - -```bash -BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller -DATADIR=/home/rshen/projects/xce/worktrees/persistent_ha/ha/data -PEERS="localhost:9991,localhost:9992,localhost:9993" - -bash src/tools/deploy-ha.sh "$BINARY" "$DATADIR" /tmp/xce-deploy "$PEERS" -``` - -### Start the cluster - -```bash -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 >> /tmp/xce-deploy/logs/ctrl-1.log 2>&1 & -sleep 0.5 -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 >> /tmp/xce-deploy/logs/ctrl-2.log 2>&1 & -sleep 0.5 -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 >> /tmp/xce-deploy/logs/ctrl-3.log 2>&1 & -``` - -Wait for ctrl-1 to reach CTRL_PRIMARY: - -```bash -grep -m1 "CTRL_PRIMARY" /tmp/xce-deploy/logs/ctrl-1.log -``` - ---- - -### Scenario A — Short Pause (no failover expected) - -**Requirement 2.2:** 1 × `heartbeat_interval_ms` (3000ms) silence must not trigger failover. - -```bash -# 1. Get ctrl-1 PID -PID=$(pgrep -f "xce_controller.*ctrl-1" | head -1) -echo "ctrl-1 PID: $PID" - -# 2. Snapshot RDHUP count before -PRE2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) -PRE3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) - -# 3. Stop ctrl-1 for exactly 1 × heartbeat_interval_ms -kill -STOP $PID -sleep 3 # 3000ms = 1 × heartbeat_interval_ms -kill -CONT $PID - -# 4. Wait 3 seconds for any poll reaction -sleep 3 - -# 5. Count new RDHUP events -POST2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) -POST3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) -echo "New RDHUP on ctrl-2: $((POST2 - PRE2)) (expected: 0)" -echo "New RDHUP on ctrl-3: $((POST3 - PRE3)) (expected: 0)" -``` - -**Expected:** -- `New RDHUP on ctrl-2: 0` -- `New RDHUP on ctrl-3: 0` -- ctrl-1 still shows `CTRL_PRIMARY` in its log -- No `CTRL_ELECTING` entries appear in ctrl-2 or ctrl-3 logs during this window - ---- - -### Scenario B — Long Pause (failover expected exactly once) - -**Requirement 2.3:** Silence exceeding `heartbeat_timeout_ms` must trigger failover. - -Restart the cluster cleanly first: - -```bash -kill $(pgrep -f "xce_controller") 2>/dev/null || true -sleep 1 -bash /tmp/xce-deploy/scripts/cleanup.sh -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 >> /tmp/xce-deploy/logs/ctrl-1.log 2>&1 & -sleep 0.5 -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 >> /tmp/xce-deploy/logs/ctrl-2.log 2>&1 & -sleep 0.5 -VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 >> /tmp/xce-deploy/logs/ctrl-3.log 2>&1 & - -# Wait for primary + standbys -sleep 20 -``` - -Then: - -```bash -# 1. Get ctrl-1 PID -PID=$(pgrep -f "xce_controller.*ctrl-1" | head -1) -echo "ctrl-1 PID: $PID" - -# 2. Snapshot before -PRE2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) -PRE3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) - -# 3. Stop ctrl-1 for heartbeat_timeout_ms + 500ms = 10.5s -kill -STOP $PID -sleep 10.5 - -# 4. CONT ctrl-1 (it may lose election and become standby) -kill -CONT $PID - -# 5. Wait for new primary to emerge (allow election + committing time) -sleep 20 - -# 6. Verify exactly 1 RDHUP per standby -POST2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) -POST3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) -echo "New RDHUP on ctrl-2: $((POST2 - PRE2)) (expected: 1)" -echo "New RDHUP on ctrl-3: $((POST3 - PRE3)) (expected: 1)" - -# 7. Verify a new primary emerged -grep "CTRL_PRIMARY" /tmp/xce-deploy/logs/ctrl-{1,2,3}.log | tail -3 -``` - -**Expected:** -- `New RDHUP on ctrl-2: 1` -- `New RDHUP on ctrl-3: 1` -- Exactly one of `ctrl-{1,2,3}.log` shows a final `CTRL_PRIMARY` entry after the failover -- No infinite RDHUP loop: `grep ha_on_primary_ctrl_fd_rdhup` shows total count = 1 per standby - ---- - -## What to Look For in the Logs - -| Log pattern | Scenario A | Scenario B | -|---|---|---| -| `ha_on_primary_ctrl_fd_rdhup` on ctrl-2/ctrl-3 | count unchanged | count +1 each | -| `CTRL_ELECTING` on ctrl-2 or ctrl-3 | absent | present (exactly 1 per node) | -| `CTRL_PRIMARY` on a non-ctrl-1 node | absent | present on one node | -| `hb_timeout_thread` anywhere | absent | absent (thread was removed) | -| Multiple `ha_on_primary_ctrl_fd_rdhup` per standby | n/a | must NOT exceed 1 per standby (Req 5.4) | - ---- - -## Results Template - -Paste filled results back to the developer. - -``` -Task 4.5 SIGSTOP/SIGCONT missed-heartbeat tolerance -Date: _______________ -Build: _______________ - -Scenario A (1 × heartbeat_interval_ms = 3s pause): - ctrl-2 new RDHUP count: ___ (expected: 0) - ctrl-3 new RDHUP count: ___ (expected: 0) - ctrl-1 still primary after SIGCONT: yes / no - Result: PASS / FAIL - -Scenario B (heartbeat_timeout_ms + 500ms = 10.5s pause): - ctrl-2 new RDHUP count: ___ (expected: 1) - ctrl-3 new RDHUP count: ___ (expected: 1) - New primary after failover: ctrl-1 / ctrl-2 / ctrl-3 / none - hb_timeout_thread entries: ___ (expected: 0) - Result: PASS / FAIL - -Overall: PASS / FAIL -Notes: -``` diff --git a/.kiro/specs/ha-heartbeat-liveness/requirements.md b/.kiro/specs/ha-heartbeat-liveness/requirements.md deleted file mode 100644 index f7c85e94..00000000 --- a/.kiro/specs/ha-heartbeat-liveness/requirements.md +++ /dev/null @@ -1,148 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE controller HA cluster uses a dedicated blocking thread (`hb_timeout_thread`) on each -standby to detect primary liveness via a `sem_timedwait` semaphore. This design has two problems: - -1. **Race condition**: when `committing_timeout_ms` equals `heartbeat_timeout_ms` (both default - to 10s), standbys arm their watchdog when the primary's first ctrlhello arrives, but the - primary does not send its first heartbeat NOP until it finishes the committing phase — - exactly at the watchdog deadline. This causes repeated false failovers. - -2. **No missed-heartbeat tolerance**: the timeout fires on the first missed heartbeat, with - no ability to tolerate transient packet loss or scheduling jitter spanning 1–2 intervals. - -This feature replaces the blocking watchdog thread with activity-based liveness tracking: -the standby polls a `last_activity_from_primary` timestamp via delaymgr, declares the primary -dead only after `inactive_heartbeat_count` consecutive missed intervals, and the primary sends -a proactive NOP only when idle for one full interval — so ctrlhello and WAL traffic naturally -suppress redundant heartbeats. - -## Boundary Context - -- **In scope**: Standby liveness detection (replacing `hb_timeout_thread`); missed-heartbeat - tolerance via `inactive_heartbeat_count`; primary idle-triggered NOP policy; timestamp reset - on all inbound primary-channel messages; timing correction so ctrlhello counts as activity; - startup validation of heartbeat timing parameters. -- **Out of scope**: Raft election, vote RPCs, `ha_start_election_async`, raft.c — unchanged. - EPOLLRDHUP fast-path failover (already correct, not modified by this spec). WAL replication - protocol, config push protocol, standby catch-up logic. -- **Adjacent expectations**: The primary must send a ctrlhello to each standby before - `inactive_heartbeat_count × heartbeat_interval_ms` elapses from when the standby enters - CTRL_STANDBY. The delaymgr must be running before liveness polling is armed (guaranteed by - existing startup sequence). - -## Requirements - -### Requirement 1: Activity-Based Standby Liveness Detection - -**Objective:** As a cluster operator, I want the standby to detect primary failure based on -elapsed time since any inbound primary activity, so that transient silence during committing -and single missed heartbeats do not trigger a false re-election. - -#### Acceptance Criteria - -1. While in CTRL_STANDBY state, the controller shall record the wall-clock timestamp of the - most recent inbound message received on the primary control channel. -2. While in CTRL_STANDBY state, the controller shall periodically check whether the elapsed - time since the last recorded primary activity exceeds - `inactive_heartbeat_count × heartbeat_interval_ms`. -3. When any message is received on the primary control channel, the controller shall update - the last-activity timestamp regardless of the message opcode (NOP, ctrlhello, WAL entry, - config push all count as activity). -4. When the elapsed time since the last primary activity exceeds - `inactive_heartbeat_count × heartbeat_interval_ms`, the controller shall transition to - CTRL_ELECTING and initiate the RDHUP failover path. -5. When the controller transitions away from CTRL_STANDBY, the controller shall stop the - liveness polling task. -6. The controller shall not use a dedicated blocking thread for standby liveness detection. - ---- - -### Requirement 2: Missed-Heartbeat Tolerance - -**Objective:** As a cluster operator, I want the cluster to tolerate at least one or two -missed heartbeat intervals before declaring primary failure, so that transient network jitter -or scheduling delay does not cause unnecessary failovers. - -#### Acceptance Criteria - -1. The controller shall declare the primary dead only after a silence period equal to - `heartbeat_timeout_ms`, where `heartbeat_timeout_ms` is already configured to cover the - desired number of missed intervals (e.g. `heartbeat_timeout_ms = 3 × heartbeat_interval_ms` - tolerates 2 missed intervals). -2. When `heartbeat_interval_ms` is 3000 and `heartbeat_timeout_ms` is 9000, the controller - shall tolerate up to 2 consecutive missed heartbeat intervals before triggering failover. -3. When `heartbeat_interval_ms` is 3000 and `heartbeat_timeout_ms` is 6000, the controller - shall tolerate exactly 1 missed heartbeat interval before triggering failover. -4. The controller shall validate at startup that - `heartbeat_timeout_ms >= 2 × heartbeat_interval_ms`; if not, the controller shall reject - startup with a clear error message. - ---- - -### Requirement 3: Primary Proactive Heartbeat (Idle-Triggered) - -**Objective:** As a cluster operator, I want the primary to send a heartbeat NOP when it has -been idle toward a standby for one full heartbeat interval, so that WAL entries and config -pushes suppress redundant NOPs and the standby never misses more than one implied interval. - -#### Acceptance Criteria - -1. While in CTRL_PRIMARY state, the controller shall track the elapsed time since the last - outbound message was sent to each connected standby. -2. When the elapsed time since the last outbound message to a standby reaches - `heartbeat_interval_ms`, the controller shall send an explicit NOP to that standby. -3. When the primary sends any message to a standby (WAL entry, config push, ctrlhello, NOP), - the controller shall reset the idle timer for that standby. -4. The controller shall not send a redundant NOP to a standby if any message was sent to that - standby within the last `heartbeat_interval_ms`. -5. When the controller transitions away from CTRL_PRIMARY, the controller shall stop the - idle-heartbeat task for all standbys. - ---- - -### Requirement 4: Timing Correctness — Race Elimination - -**Objective:** As a cluster operator, I want the standby liveness deadline to be reset by the -primary's ctrlhello during the COMMITTING phase, so that the committing window does not -trigger a false failover regardless of how long committing takes. - -#### Acceptance Criteria - -1. When a standby receives a ctrlhello message from the primary, the controller shall treat - that message as primary activity and reset the liveness deadline. -2. When the primary is in CTRL_PRIMARY_COMMITTING state and sends a ctrlhello to a standby, - the standby shall not trigger failover before - `inactive_heartbeat_count × heartbeat_interval_ms` elapses from that ctrlhello. -3. The controller shall guarantee that a standby which received any primary message within - the liveness window does not declare the primary dead before that window expires. - ---- - -### Requirement 5: No Behavioral Regression - -**Objective:** As a cluster operator, I want all correct existing behaviors preserved after -the heartbeat refactor, so that the cluster operates identically in all paths that were -already working. - -#### Acceptance Criteria - -1. The raft election mechanism, vote RPCs, and async election initiator shall be unchanged by - this feature. -2. When a primary control channel TCP connection drops (EPOLLRDHUP event), the controller - shall immediately call the EPOLLRDHUP failover handler without waiting for the liveness - poll interval. The heartbeat timeout applies only to silent-dead scenarios where the - connection remains open but the primary stops sending traffic. -3. The liveness polling task and the EPOLLRDHUP handler shall both invoke the same failover - path (`ha_on_primary_ctrl_fd_rdhup`), ensuring consistent state transitions regardless of - which detection mechanism fires first. -4. When the EPOLLRDHUP handler fires while a liveness poll task is pending, the controller - shall cancel the poll task so it does not trigger a duplicate failover. -5. When a single-node controller operates with no standbys, the liveness mechanism on the - primary side shall not interfere with normal job dispatch or queue operations. -6. When the controller is in CTRL_INIT or CTRL_ELECTING state, the controller shall not arm - or execute the standby liveness polling task. -7. The controller shall log at INFO level when the liveness deadline is armed, when it is - reset by primary activity, and when it fires the failover transition. diff --git a/.kiro/specs/ha-heartbeat-liveness/research.md b/.kiro/specs/ha-heartbeat-liveness/research.md deleted file mode 100644 index a4dd055f..00000000 --- a/.kiro/specs/ha-heartbeat-liveness/research.md +++ /dev/null @@ -1,400 +0,0 @@ -# Gap Analysis: ha-heartbeat-liveness - -> Run date: 2026-06-02 (post-implementation audit) -> -> Note: this gap analysis was requested after implementation was already -> complete. It therefore serves as a completion audit rather than an -> upfront feasibility study. - ---- - -## Analysis Summary - -- **Scope**: Replace blocking `hb_timeout_thread` watchdog with delaymgr-based - activity-tracking liveness; add primary idle-triggered NOP suppression; - enforce startup timing validation. -- **Implementation status**: All mandatory tasks (1.1, 1.2, 2.1, 2.2, 3.1, - 3.2, 4.1–4.4) are complete. One optional integration test (4.5 — SIGSTOP - tolerance) is marked `[ ]*` (optional). -- **Remaining gap**: `OPCode_replEntry` and `OPCode_walCatchupEntry` are - **not** calling `ha_on_heartbeat_received` in `cluster.c`; only `OPCode_nop` - and `OPCode_hello` do. Task 3.1 intended all opcodes on the primary channel - to reset the deadline, but WAL replication opcodes are dispatched through - `config_repl.c` / `ha.c` directly — not through the `cluster.c` dispatch - table — so the hook was never needed there. -- **Recommendation**: No blocking issues. The design's "any message counts" - invariant is satisfied via ctrlhello + NOP. WAL entries reset the liveness - deadline indirectly because they keep `last_push_time` fresh on the primary, - suppressing NOPs, and on the standby the ctrlhello already armed the deadline - with a fresh timestamp. See §4 below for nuance. - ---- - -## 1. Current State Investigation - -### Existing Assets Used - -| Component | Location | Role in feature | -|-----------|----------|-----------------| -| `liveness_poll_fn` | `ha.c:2796` | New delaymgr task replacing `hb_timeout_thread` | -| `liveness_poll_ctx_t` | `ha.c:2775` | Internal context; `_Atomic int cancelled` | -| `s_liveness_ctx` | `ha.c:2783` | `_Atomic` static replacing `s_hb_vtask` | -| `ha_arm_heartbeat_timeout_vtask` | `ha.c:2860` | Replaced: allocates ctx, calls `delay_task_ms` | -| `ha_disarm_heartbeat_timeout_vtask` | `ha.c:2911` | Sets `cancelled=1`, frees ctx | -| `ha_on_heartbeat_received` | `ha.c:2902` | Writes `clock_gettime` to `ha->last_activity` | -| `liveness_poll_fn_test_entry` | `ha.c:2844` | `XCE_TESTING` seam for unit tests | -| `primary_heartbeat_fn` (extended) | `ha.c:3127` | Per-standby `last_push_time` idle check | -| `standby_ep_ctx_t.last_push_time` | `ha.c:2940` | Per-standby send timestamp | -| `standby_ep_ctx_build_for_test` | `ha.c:2945` | Test helper for ep context | -| `validate_heartbeat_config` | `startup_validation.c:143` | `timeout >= 2×interval` enforcement | -| `ha_state_t.last_activity` | `ha.h / ha.c:668` | Written by `ha_on_heartbeat_received` | -| cluster.c `OPCode_nop` handler | `cluster.c:225–233` | Calls `ha_on_heartbeat_received` (CTRL_STANDBY guard) | -| cluster.c `OPCode_hello` handler | `cluster.c:195–202` | Calls `ha_on_heartbeat_received` (CTRL_STANDBY guard) | -| `test_ha_liveness.c` | `testsuite/` | Unit tests for tasks 2.1, 4.1 | -| `test_ha_rdhup_integration.c` | `testsuite/` | Integration test for 4.3/4.4 | -| Both test files wired into build | `meson.build:120–121` | Confirmed | - -### What Was Removed - -| Removed | Status | -|---------|--------| -| `hb_vtask_t` struct | Only referenced in comments — actual struct gone | -| `hb_timeout_thread` function | Only in comments — function gone | -| `s_hb_vtask` static variable | Replaced by `s_liveness_ctx` | -| Blocking `sem_timedwait` watchdog path | Remaining `sem_timedwait` use is in `ha_wait_for_heartbeat_or_timeout` (CTRL_INIT path — different, intentionally kept) | - ---- - -## 2. Requirements Feasibility — Gap Check - -### Req 1.3 / 4.1: Any message resets liveness deadline - -Task 3.1 listed `OPCode_replEntry` and `OPCode_walCatchupEntry` as needing -`ha_on_heartbeat_received` calls in cluster.c. Investigation shows they are -**not dispatched through cluster.c** on the standby: - -- `OPCode_replEntry` is handled in `ha.c:1124` / `ha.c:1143` (directly in the - HA vtask dispatch table, not cluster.c). -- `OPCode_walCatchupEntry` is handled in `config_repl.c` via the catch-up path. - -**Consequence**: these opcodes do not currently call `ha_on_heartbeat_received`. -In practice this is acceptable because: -1. WAL replication only starts after ctrlhello, which resets `last_activity`. -2. The primary's idle-triggered NOP (`primary_heartbeat_fn`) ensures a NOP is - sent within `heartbeat_interval_ms` of any silence — and `OPCode_nop` does - call `ha_on_heartbeat_received`. -3. Active WAL replication keeps `last_push_time` fresh on the primary, - suppressing redundant NOPs, so there is no silence window where both WAL - replication AND NOP are absent. - -**Risk**: Low. Under very heavy WAL load where NOPs are completely suppressed -and the delaymgr thread is starved, a standby could theoretically accumulate -elapsed time without a liveness reset. This is an edge case unlikely in -practice; the current design matches the approved spec. - -**Recommendation**: If Req 1.3 literal compliance is required (every opcode on -the primary channel resets the deadline), add `ha_on_heartbeat_received` calls -to the `OPCode_replEntry` and `OPCode_walCatchupEntry` dispatch sites in -`ha.c`. This is a 2-line change with no risk. Defer to maintainer judgment. - -### All other requirements: fully satisfied - -| Req | Status | Evidence | -|-----|--------|----------| -| 1.1 `last_activity` field | ✅ Done | `ha.c:668`, `ha.h` | -| 1.2 Periodic poll | ✅ Done | `liveness_poll_fn` delaymgr task | -| 1.3 Any msg resets deadline | ⚠ Partial | NOP + ctrlhello covered; replEntry/walCatchup not wired (low risk) | -| 1.4 Failover on timeout | ✅ Done | `liveness_poll_fn` → `ha_on_primary_ctrl_fd_rdhup` | -| 1.5 Stop poll on role change | ✅ Done | `ha_disarm_heartbeat_timeout_vtask` | -| 1.6 No blocking thread | ✅ Done | `hb_timeout_thread` removed | -| 2.1–2.3 Missed-HB tolerance | ✅ Done | `heartbeat_timeout_ms` threshold in poll fn | -| 2.4 Startup validation | ✅ Done | `validate_heartbeat_config` in `startup_validation.c` | -| 3.1–3.5 Primary idle NOP | ✅ Done | `primary_heartbeat_fn` + `standby_ep_ctx_t.last_push_time` | -| 4.1–4.3 Race elimination | ✅ Done | ctrlhello resets `last_activity`; poll armed from ctrlhello | -| 5.1 Raft unchanged | ✅ Done | raft.c untouched | -| 5.2 EPOLLRDHUP fast path | ✅ Done | `ha_on_primary_ctrl_fd_rdhup` unchanged, called by poll fn too | -| 5.3 Single-node no interference | ✅ Done | existing guard in `primary_heartbeat_fn` | -| 5.4 RDHUP cancels poll | ✅ Done | `ha_on_primary_ctrl_fd_rdhup` calls `ha_disarm_heartbeat_timeout_vtask` at `ha.c:2694` | -| 5.5 INFO logging | ✅ Done | log in arm/fire paths | -| 5.6 No poll in INIT/ELECTING | ✅ Done | `ha_arm_heartbeat_timeout_vtask` callsite guards | - ---- - -## 3. Implementation Approach - -**Option A (Extend existing) was used** — all changes are in existing files -(`ha.c`, `ha.h`, `cluster.c`, `config_repl.c`, `startup_validation.c`, -`main.c`). No new files were created except the two test files -(`test_ha_liveness.c`, `test_ha_rdhup_integration.c`). - -This was the correct choice: the feature is a targeted refactor of the HA -liveness path, not a new subsystem. - ---- - -## 4. Outstanding Items - -| Item | Priority | Effort | -|------|----------|--------| -| Wire `ha_on_heartbeat_received` in `OPCode_replEntry` / `OPCode_walCatchupEntry` dispatch sites in ha.c | Low | S (2 lines each) | -| Task 4.5: SIGSTOP/SIGCONT missed-heartbeat tolerance integration test | Optional | S (manual script) | - ---- - -## 5. Effort and Risk Assessment - -- **Effort**: S — implementation is complete; only minor additions remain. -- **Risk**: Low — all core paths tested; RDHUP concurrency handled with `_Atomic`; no new threads or semaphores. - ---- - -## Next Steps - -- Run `/xce-build controller` to confirm build + test suite green. -- Optionally wire `ha_on_heartbeat_received` into `OPCode_replEntry` / - `OPCode_walCatchupEntry` sites in `ha.c` for full Req 1.3 literal compliance. -- Run task 4.5 integration test (SIGSTOP/SIGCONT) when a 3-node cluster is - available. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **ha-heartbeat-liveness** is **~97% complete**. All core implementation is done. The `hb_timeout_thread` has been replaced with delaymgr-based `liveness_poll_fn`. -- **Remaining gaps**: (1) Req 1.3 partial — `OPCode_replEntry` and `OPCode_walCatchupEntry` dispatch sites in ha.c do not call `ha_on_heartbeat_received`. Low risk: active WAL replication keeps `last_push_time` fresh on primary, suppressing NOPs, and ctrlhello resets the standby's deadline. Adding 2 `ha_on_heartbeat_received` calls in the dispatch sites would achieve full literal compliance. (2) Task 4.5 — SIGSTOP/SIGCONT missed-heartbeat tolerance is an optional manual integration test (script exists at `src/tools/test-heartbeat-tolerance.sh`). -- **Cross-spec relationship**: `ha-repl-hardening` heartbeat-driven convergence tracking relies on this spec's liveness infrastructure. The `primary_heartbeat_fn` idle-triggered NOP suppression is correct and does not interfere with ha-repl-hardening's per-standby state reporting via heartbeat replies. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -> Scope: re-validation of ha-heartbeat-liveness on the persistent_ha working tree -> (uncommitted changes to ha.c/ha.h/myapp.c/raft.c analyzed as on disk), after -> commit e1517bb "heartbeat re-arm broken by delaymgr ms-task ordering". - -## 1. Current State Investigation - -All tasks in tasks.md are checked done (1.1–4.5). On-disk verification: - -| Asset | Location | Verified state | -|-------|----------|----------------| -| `liveness_poll_ctx_t` + `_Atomic cancelled` | `src/controller/ha.c:3308` | Present; `s_liveness_ctx` is `_Atomic` ptr (ha.c:3320) | -| `liveness_poll_fn` | `ha.c:3333` | cancelled check → role guard → elapsed vs `heartbeat_timeout_ms` → `ha_on_primary_ctrl_fd_rdhup` or re-arm at `interval/2` (min 100ms) | -| `ha_arm_heartbeat_timeout_vtask` | `ha.c:3408` | delaymgr-based; disarms stale ctx, seeds `last_activity = now`, INFO log | -| `ha_on_heartbeat_received` | `ha.c:3453` | writes `CLOCK_MONOTONIC` secs to `ha->last_activity`; DEBUG3 log | -| `ha_disarm_heartbeat_timeout_vtask` | `ha.c:3462` | sets cancelled, clears static ptr; poll fn frees ctx | -| RDHUP cancels poll (Req 5.4) | `ha.c:3189` | `ha_disarm_heartbeat_timeout_vtask` is the first action in `ha_on_primary_ctrl_fd_rdhup` | -| `primary_heartbeat_fn` idle-NOP | `ha.c:3692–3740` | per-standby `last_push_time` elapsed check before `xce_nop_req` | -| `ha_start_primary_heartbeat` | `ha.c:3784` | post-e1517bb: `runcount=-1` infinite delaymgr task + static `_Atomic` once-guard; no fragile re-arm path | -| delaymgr ordering fix | `src/lib/delay.c` (e1517bb) | `task_cmp` now orders ms-tasks by `nextrun_ts` (ns precision) — re-arm starvation root cause fixed | -| `validate_heartbeat_config` | `src/controller/startup_validation.c:143` | `timeout >= 2×interval` else `-EINVAL` + message; wired in `cluster.c:885` before `ha_init()` (design said main.c; cluster.c init path is equivalent) | -| Opcode liveness resets | `cluster.c:250` (ctrlhello), `cluster.c:285` (NOP), `config_repl.c:1373` (handle_ctrlhello) | only these three production call sites of `ha_on_heartbeat_received` | -| Arm call sites (Req 5.6) | `ha.c:664` (raft_cb_become_follower), `myapp.c:302` (skip-election standby path), `config_repl.c:1374` (handle_ctrlhello) | all three set/verify CTRL_STANDBY before arming; poll fn re-checks role | -| `hb_timeout_thread` / `hb_vtask_t` / `s_hb_vtask` | — | removed; only historical comments remain | -| Unit tests | `testsuite/test_ha_liveness.c`, `test_ha_startup.c:712ff` | present, registered in `meson.build:120–121` | -| Integration | `testsuite/test_ha_rdhup_integration.c`, `src/tools/test-heartbeat-tolerance.sh` | present (4.5 script exists) | - -Uncommitted working-tree changes to ha.c are election-init refactors (`ha_init_raft` -extraction, term-scoped vote dedup, `primary_ctrl_fd >= 0` vote-gate guard) from the -ha-repl-hardening stream; they do not alter the liveness mechanics of this spec. - -## 2. Requirement-to-Asset Map - -| Req | Summary | Asset(s) | Status | -|-----|---------|----------|--------| -| 1.1 | Record last-activity timestamp | `ha_state_t.last_activity` (ha.h; init ha.c:814; writer ha.c:3453) | Implemented | -| 1.2 | Periodic elapsed check | `liveness_poll_fn` ha.c:3333; poll = `interval/2`, floor 100ms | Implemented | -| 1.3 | ANY opcode resets deadline | NOP + ctrlhello only; `OPCode_walCatchupEntry` (cluster.c:407), `OPCode_replConfigPush`, `OPCode_snapshotTransfer` handlers do NOT call `ha_on_heartbeat_received` | **Partial** — compensated at runtime because NOPs are not actually suppressed during WAL traffic (see 3.3) | -| 1.4 | Failover on timeout | ha.c:3367 → `ha_on_primary_ctrl_fd_rdhup` | Implemented | -| 1.5 | Stop poll on role change | disarm in RDHUP (ha.c:3189) + role guard in poll fn (ha.c:3356) | Implemented (minor: role-guard early-return at ha.c:3357 returns without freeing/cancelling ctx — a one-time ~32-byte leak on the rare promote-without-disarm path) | -| 1.6 | No blocking thread | `hb_timeout_thread` removed | Implemented | -| 2.1 | Dead only after `heartbeat_timeout_ms` | poll threshold ha.c:3365–3367 | Implemented — **Constraint**: whole-second arithmetic (`timeout_s = ms/1000`, `elapsed` in truncated secs) can fire up to ~1s off the ms-precise spec; sub-second intervals unsupported | -| 2.2 | 3000/9000 tolerates 2 misses | config convention + `test-heartbeat-tolerance.sh` | Implemented | -| 2.3 | 3000/6000 tolerates 1 miss | same | Implemented | -| 2.4 | Startup validation | `validate_heartbeat_config` + cluster.c:885 wiring + tests (test_ha_startup.c:727–759) | Implemented | -| 3.1 | Track per-standby outbound idle | `standby_ep_ctx_t.last_push_time` ha.c:3490 | **Partial** — written only at `on_standby_connected` (ha.c:3561) and after NOP (ha.c:3735) | -| 3.2 | NOP after one idle interval | ha.c:3725–3736 | Implemented | -| 3.3 | ANY send resets idle timer | WAL batch (`send_wal_batch`), config push, snapshot send paths never write `last_push_time` (grep confirms only 2 write sites) | **Missing** — design.md:275 deferred the repl.c decision and it never landed | -| 3.4 | No redundant NOP within interval | elapsed check exists, but because 3.3 is missing, NOPs continue at full rate during active WAL/config/snapshot traffic | **Partial** — benign deviation (extra NOPs, no correctness impact; in fact it is what keeps 1.3's gap safe) | -| 3.5 | Stop idle task on demotion | role guard ha.c:3699; `runcount=-1` + `_Atomic` once-guard (ha.c:3796) | Implemented | -| 4.1 | ctrlhello counts as activity | cluster.c:250 + config_repl.c:1373 | Implemented | -| 4.2 | Committing window safe | arm + `last_activity=now` at ctrlhello; threshold = full timeout | Implemented | -| 4.3 | No premature declare within window | poll threshold; subject to ±1s granularity constraint above | Implemented | -| 5.1 | Raft unchanged | raft.c untouched **by this spec** (working-tree raft.c edits belong to ha-repl-hardening) | Implemented | -| 5.2 | RDHUP immediate fast path | handler unchanged; liveness only covers silent-dead | Implemented | -| 5.3 | Both paths → same failover fn | poll fn calls `ha_on_primary_ctrl_fd_rdhup` | Implemented | -| 5.4 | RDHUP cancels pending poll | ha.c:3189 disarm-first | Implemented | -| 5.5 | Single-node non-interference | empty `standby_endpoints` loop is no-op; reconnect loop skips self | Implemented | -| 5.6 | No arm/exec in INIT/ELECTING | all 3 arm sites are standby-only; poll fn role guard | Implemented | -| 5.7 | INFO log arm/reset/fire | arm = INFO (ha.c:3442); fire = WARN (ha.c:3369); reset = DEBUG3 (ha.c:3459) | **Partial** — reset is DEBUG3 by deliberate documented design deviation (log-spam avoidance); fire is WARN not INFO (arguably more correct) | - -**Score: 20 of 25 acceptance criteria fully Implemented; 4 Partial (1.3, 3.1, 3.4, 5.7); 1 Missing (3.3).** -The post-e1517bb runtime behavior satisfies the feature's operational intent: heartbeats -re-arm correctly (delaymgr `task_cmp` ordering fixed + infinite-repeat task), no false -failovers in the committing window, tolerance via timeout ratio works. - -### New cross-cutting defect found (out-of-boundary but caused by this refactor) - -**`ha_heartbeat_is_fresh` (ha.c:1762) reads a dead field.** It computes elapsed from -`ha->last_heartbeat` (struct timespec, ha.h:154), but no production code ever writes -`last_heartbeat` — only tests do (`test_ha_vote_gate.c`). The liveness refactor moved all -activity tracking to `ha->last_activity` (design.md data model even lists `last_heartbeat` -as "Repurposed / replaced") yet the vote-deny gate was never repointed. Consequence: once -process uptime exceeds `heartbeat_timeout_ms` (~10s), `ha_heartbeat_is_fresh` always -returns stale, so the "STANDBY with live primary denies votes" gates at **ha.c:214** -(raft vote callback) and **ha.c:1841** (`handle_request_vote`) are inert. A disturber -candidate can collect votes from standbys whose primary is healthy and usurp leadership. -The uncommitted `primary_ctrl_fd >= 0 &&` guard at ha.c:1841 narrows the deny condition -further (it does not restore it), and the ha.c:214 site has no fd guard at all. -**Fix is a one-liner**: compute freshness from `ha->last_activity` (and either delete -`last_heartbeat` or update the vote-gate tests). Tag: **Missing** (vote-gate liveness -input), Severity: Medium (availability — unnecessary election churn; not data loss, since -term/log freshness gates still apply). - -### Secondary observation — type-punned idle tracking on promoted vote endpoints - -`primary_heartbeat_fn` casts `ep->data` to `standby_ep_ctx_t*` for every entry in -`standby_endpoints`, but promoted vote endpoints carry `vote_ep_ctx_t` (ha.c:45) whose -5th field is `uint64_t my_term` — which aliases `last_push_time` (both at offset 32). -Reading it yields a tiny term number → elapsed always huge → NOP always sent on the first -tick, then `now` is written into `my_term`. Functionally self-correcting, but it silently -clobbers `my_term` and depends on struct-layout coincidence. Tag: **Constraint** (fragile; -unify the ctx structs or skip idle tracking when `sctx->ctx != NULL`). - -## 3. Implementation Approach Options for Remaining Gaps - -### Option A — Targeted point fixes in existing files (recommended) -1. `ha_heartbeat_is_fresh`: read `ha->last_activity`; adjust `test_ha_vote_gate.c`; drop or deprecate `ha_state_t.last_heartbeat`. (~10 lines + test updates) -2. Req 1.3 literal compliance: add `ha_on_heartbeat_received(&c->ha)` (role-guarded) to the `OPCode_walCatchupEntry`, `OPCode_replConfigPush`, `OPCode_snapshotTransfer` dispatch arms in cluster.c. (~6 lines) -3. Req 3.3 (optional): write `last_push_time` in the primary's WAL/config/snapshot send helpers, or explicitly document extra-NOPs-during-replication as the accepted behavior. -- ✅ Smallest diff, stays inside spec boundary files, leverages existing tests -- ❌ Leaves the `vote_ep_ctx_t`/`standby_ep_ctx_t` aliasing in place - -### Option B — Centralized channel-activity layer -Introduce a small `ha_liveness` helper that hooks endpoint send/recv once (lib layer) and owns both `last_activity` and per-ep `last_push_time`; unify the two ep ctx structs. -- ✅ Eliminates per-opcode wiring forever; fixes the aliasing structurally -- ❌ Touches `src/lib` endpoint layer (outside spec boundary), M effort, new revalidation triggers for three other HA specs - -### Option C — Hybrid (A now, struct unification later) -Apply Option A immediately; file the ctx-struct unification and Req 3.3 send-path tracking as a follow-up item under ha-repl-hardening where repl.c is already in boundary. -- ✅ Unblocks completion with low risk; defers structural work to a spec that owns repl.c -- ❌ Two-step bookkeeping; deviation on 3.3/3.4 persists in the interim - -## 4. Effort & Risk - -| Item | Effort | Risk | Justification | -|------|--------|------|---------------| -| Fix `ha_heartbeat_is_fresh` → `last_activity` | S | Low (fixes a Medium-severity defect) | one-line logic change + 2 test helpers; existing vote-gate tests pin behavior | -| Req 1.3 opcode wiring (3 handlers) | S | Low | identical pattern to existing NOP/hello call sites | -| Req 3.3 send-path `last_push_time` | S | Low–Medium | needs ep→sctx access in config_repl.c send helpers; aliasing caveat applies | -| ctx struct unification (aliasing) | M | Medium | touches election ep promotion path; needs 3-node regression | -| Overall remaining work | **S** (≤1 day for Option A) | **Low** | all patterns established; tests and deploy harness exist | - -## 5. Recommendations & Research Needed - -**Recommended approach: Option C** — land Option A's three point fixes now (the -`ha_heartbeat_is_fresh` repoint should go in before the current working tree is -committed, since the uncommitted vote-gate edit at ha.c:1841 builds on the broken -freshness check), and move struct unification + send-path idle tracking into -ha-repl-hardening's backlog. - -**Research Needed:** -- R1: Is `vote_ep_ctx_t.my_term` read anywhere after an endpoint is promoted into `standby_endpoints`? (If yes, the `last_push_time` clobber is an active bug, not just fragility.) -- R2: Are sub-second `heartbeat_interval_ms` configurations a supported operating point? If yes, `liveness_poll_fn`/`primary_heartbeat_fn` must move from whole-second to ms arithmetic (`timespec` diff). -- R3: Confirm with ops/runbook owners that DEBUG3-on-reset (vs Req 5.7's literal INFO) is the accepted trade-off; if so, amend Req 5.7 wording. -- R4: Verify task 4.5 (`test-heartbeat-tolerance.sh`) has been executed against the post-e1517bb binary; the delaymgr ordering fix changed scheduling behavior after the last recorded run. - ---- - -## Gap Re-Audit (2026-06-18) - -Post-implementation audit. Tasks 15/15 `[x]`, `spec.json.phase = implemented`. This pass -re-verifies every EARS acceptance criterion against the current on-disk code (the codebase -evolved after the original 2026-06-10 audit — most notably the `ha-prevote` layer now wraps -the failover trigger). All file:line refs verified via codegraph + grep on this date. - -### Requirement-to-Asset Map - -| AC | Status | Asset (file:line) | Notes | -|----|--------|-------------------|-------| -| 1.1 record last-activity timestamp | CLOSED | `ha_state_t.last_activity` (ha.c:1387 doc); written by `ha_on_heartbeat_received` ha.c:4886-4892 | CLOCK_MONOTONIC seconds | -| 1.2 periodic elapsed check | CLOSED | `liveness_poll_fn` ha.c:4750-4801; re-arms at `heartbeat_interval_ms/2` (min 100ms) ha.c:4795-4798 | delaymgr task, no thread | -| 1.3 any msg resets deadline | CLOSED | cluster.c dispatch: OPCode_nop:313-314, OPCode_hello/ctrlhello:278-279, wal-catchup:178-179, snapshot:210-211; replEntry path config_repl.c:703; config_repl.c:1452 | role-gated `CTRL_STANDBY`; bonus: also steps down from CTRL_ELECTING via `ha_discover_primary` | -| 1.4 failover on timeout | CLOSED (semantics changed) | `liveness_poll_fn` ha.c:4784-4790 → `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 | **No longer transitions directly to CTRL_ELECTING.** Now begins a PreVote round and stays CTRL_STANDBY until PreVote majority (ha.c:4655-4707, `ha_begin_failover`/`ha_start_prevote_async`). Behaviour is a superset of the EARS text — see Residual G1. | -| 1.5 stop poll on role change | CLOSED | `ha_disarm_heartbeat_timeout_vtask` ha.c:4911-4919 (sets `cancelled`, nulls `s_liveness_ctx`); `liveness_poll_fn` self-skips when `role != CTRL_STANDBY` ha.c:4773-4774 | | -| 1.6 no blocking thread | CLOSED | `hb_timeout_thread`/`hb_vtask_t`/`s_hb_vtask` removed (grep: 0 non-comment hits in src). Remaining `sem_timedwait` (ha.c:2380) is the **separate** `heartbeat_seen` semaphore for CTRL_INIT detection, explicitly retained (design.md:305-306) | | -| 2.1 declare dead after `heartbeat_timeout_ms` | CLOSED | `liveness_poll_fn` ha.c:4782-4784 `(uint32_t)elapsed >= timeout_s` | second-granularity; see G2 | -| 2.2 N=3 tolerate 2 misses | CLOSED (convention) | threshold uses existing config ratio | no new param (per Non-Goals) | -| 2.3 N=2 tolerate 1 miss | CLOSED (convention) | same | | -| 2.4 startup validation `timeout >= 2×interval` | CLOSED | `validate_heartbeat_config` startup_validation.c:143-167; wired pre-`ha_init` in cluster.c:1085-1095 (loud ERROR + stderr + `-EINVAL`) | | -| 3.1 primary tracks idle per-standby | CLOSED | `standby_ep_ctx_t.last_push_time` ha.c:4939; read in `primary_heartbeat_fn` ha.c:5242 | | -| 3.2 NOP after one idle interval | CLOSED | `primary_heartbeat_fn` ha.c:5242-5243 `elapsed >= interval_s` → `xce_nop_req` | | -| 3.3 any send resets idle timer | PARTIAL | written on connect (`on_standby_connected` ha.c:5005-5010) and after NOP (ha.c:5252) | WAL/config pushes in repl.c do **not** write `last_push_time` (repl.c out-of-boundary). Effect: a busy primary still emits a redundant NOP up to once/interval. Benign (per spec.json note); see G3 | -| 3.4 no redundant NOP within interval | PARTIAL | suppression honoured for NOP-vs-NOP ha.c:5243 | not suppressed by WAL/config traffic (same root as G3) | -| 3.5 stop idle task on demotion | CLOSED | `primary_heartbeat_fn` role guard ha.c:5179-5182 returns 0 on non-PRIMARY | | -| 4.1 ctrlhello counts as activity | CLOSED | cluster.c:271-282 (handle_ctrlhello → `ha_on_heartbeat_received`); config_repl.c:1452 | | -| 4.2 committing window safe | CLOSED | ctrlhello resets `last_activity` before timeout; deadline armed from ctrlhello (config_repl.c:1450-1453). Additionally `ha_on_heartbeat_received` now ABANDONS an in-progress PreVote round on a fresh heartbeat (ha.c:4901-4908) — stronger race elimination than the original design | Note: `CTRL_PRIMARY_COMMITTING` state was removed project-wide (see memory: ha_consistency_invariants); AC 4.2's literal state name is stale but the guarantee holds | -| 4.3 no premature failover within window | CLOSED | `liveness_poll_fn` fires only after full `heartbeat_timeout_ms` ha.c:4784 | | -| 5.1 raft/vote/async-init unchanged | CLOSED (re-scoped) | raft.c untouched by this spec. NOTE: the failover *initiator* path changed under the separate `ha-prevote` spec (now PreVote-gated), not under this one | boundary held for THIS spec | -| 5.2 EPOLLRDHUP immediate | CLOSED | `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 still invoked directly from RDHUP epoll path; disarms poll first (ha.c:4645) | | -| 5.3 both paths → same failover fn | CLOSED | RDHUP epoll and `liveness_poll_fn` both call `ha_on_primary_ctrl_fd_rdhup` ha.c:4788 | | -| 5.4 RDHUP cancels pending poll | CLOSED | `ha_on_primary_ctrl_fd_rdhup` first step calls `ha_disarm_heartbeat_timeout_vtask` ha.c:4643-4645 | | -| 5.5 single-node no interference | CLOSED | `primary_heartbeat_fn` only iterates `standby_endpoints` (empty for single-node) ha.c:5199 | | -| 5.6 no poll in INIT/ELECTING | CLOSED | arm only from STANDBY callsites (config_repl.c:1447-1454 role-gated; `ha_prevote_rearm_liveness` ha.c:4068-4077); poll self-skips non-STANDBY ha.c:4773 | | -| 5.7 INFO log arm/reset/fire | PARTIAL | arm: INFO ha.c:4859; fire: WARN ha.c:4786 (acceptable severity); reset: DEBUG3 ha.c:4892 (NOT INFO — deliberate anti-spam) | documented trade-off; see G4 | - -Reqs met: **24/27 fully CLOSED**, 3 PARTIAL (3.3, 3.4, 5.7) — all pre-acknowledged benign -in `spec.json.note` and the prior audit's R-list. No regressions found vs the 2026-06-10 pass. - -### Residual / Deferred Gaps - -- **G1 (design drift, not a defect) — RDHUP no longer goes straight to CTRL_ELECTING.** - AC 1.4 / 1.5 / 5.x EARS text says the timeout "transitions to CTRL_ELECTING and initiates the - RDHUP failover path." The implemented `ha_on_primary_ctrl_fd_rdhup` now begins a **PreVote - round and stays CTRL_STANDBY** until a PreVote majority is won (ha.c:4655-4707), a change - introduced by the downstream `ha-prevote` spec to prevent a transient blip from unseating a - healthy primary. The liveness *detection* contract this spec owns is intact; only the - *consequence* fans out through an extra safety gate. The spec docs were never updated to - reflect this. Also: AC 4.2 references `CTRL_PRIMARY_COMMITTING`, a state since removed. - -- **G2 (latent, sub-second configs) — whole-second timer arithmetic.** `liveness_poll_fn` - (ha.c:4782 `timeout_s = ...ms/1000`) and `primary_heartbeat_fn` (ha.c:5195 `interval_s = - ...ms/1000`) truncate to whole seconds. With `heartbeat_interval_ms < 1000` the interval - truncates to 0 (then floored: poll min 100ms, but NOP `interval_s` can be 0 → NOP every - tick). `validate_heartbeat_config` enforces only the ratio, not a sub-second floor. Carried - from prior audit R2 — unresolved. - -- **G3 (Req 3.3/3.4 partial) — WAL/config pushes don't reset `last_push_time`.** `repl.c` - send paths are out-of-boundary, so a busy primary emits one redundant NOP per interval per - standby. Functionally harmless (NOP is cheap; standby treats it as activity). Pre-accepted. - -- **G4 (Req 5.7 partial) — reset logged at DEBUG3 not INFO.** Deliberate to avoid per-heartbeat - log spam. Needs a one-line wording amendment to AC 5.7 (or a runbook sign-off). Prior audit R3. - -- **G5 (interaction w/ never-block-iomgr rule) — VERIFIED SAFE.** `liveness_poll_fn` and - `primary_heartbeat_fn` run on the **delaymgr worker thread**, never in an iomgr vtask handler; - the only blocking-capable call is `xce_nop_req` which is the non-blocking `xce_request` path - (per CLAUDE.md HA invariants). `last_activity` is a plain `time_t` written from vtask context - and read from delaymgr — atomic on LP64 (design.md:347). No iomgr-blocking risk introduced. - -- **G6 (controller-never-self-terminate invariant) — VERIFIED COMPLIANT.** The timeout path - fences/re-elects via PreVote and stays alive; the config-less (tag=0) guard (ha.c:4691-4700) - parks the node as a passive voter in CTRL_STANDBY rather than exiting. No `exit/abort/FATAL` - on the liveness path. Consistent with the repo's process-lifecycle design invariant. - -### Options for the Open Items (G1, G2, G4) - -| Item | Option A (recommended) | Option B | Option C | Effort / Risk | -|------|------------------------|----------|----------|---------------| -| G1 doc drift | Update requirements.md AC 1.4/4.2 + design.md to describe the PreVote-gated consequence and drop `CTRL_PRIMARY_COMMITTING`; add a cross-ref to the `ha-prevote` spec. Doc-only. | Re-open spec and add an EARS AC for the PreVote interaction. | Leave as-is; rely on `ha-prevote` spec to own the consequence. | A: ~30min / none (no code). B: ~2h / low. C: 0 / leaves spec text misleading for future readers. | -| G2 sub-second | Add a sub-second floor check to `validate_heartbeat_config` (`interval_ms >= 1000`) — fail loud at startup. Tiny, matches existing validation pattern. | Convert both timers to `timespec` ms arithmetic to actually support sub-second. | Document "second-granularity only" in steering and close. | A: ~1h + 1 test / very low. B: ~half-day / medium (touches two hot timers; needs new tests). C: 0 / risk if an operator sets sub-second silently. | -| G4 log level | Amend AC 5.7 wording to "DEBUG3 on reset" (one line). | Add an INFO log only on the *first* reset after an arm (state-change, not per-tick). | Keep DEBUG3, runbook sign-off. | A: ~5min / none. B: ~30min / none. C: 0 / minor. | - -**Recommendation:** G1-A and G4-A are zero-risk doc fixes that close the audit cleanly. -G2-A (startup floor) is the cheapest way to neutralize the only latent code hazard; defer -G2-B unless sub-second heartbeats become a real operating requirement. No functional code -change is required for correctness at the documented operating point (interval >= 1s). diff --git a/.kiro/specs/ha-heartbeat-liveness/spec.json b/.kiro/specs/ha-heartbeat-liveness/spec.json deleted file mode 100644 index a819a47a..00000000 --- a/.kiro/specs/ha-heartbeat-liveness/spec.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "feature_name": "ha-heartbeat-liveness", - "created_at": "2026-06-01T09:50:00Z", - "updated_at": "2026-06-01T10:20:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "Req 1.3 fixed 2026-06-10: WAL/snapshot/config handlers update last_activity. Req 3.3/3.4 are benign constraints. Controller suite green." -} diff --git a/.kiro/specs/ha-heartbeat-liveness/tasks.md b/.kiro/specs/ha-heartbeat-liveness/tasks.md deleted file mode 100644 index 8f84a5b1..00000000 --- a/.kiro/specs/ha-heartbeat-liveness/tasks.md +++ /dev/null @@ -1,103 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: state fields and startup validation -- [x] 1.1 Add liveness state fields to ha_state_t and standby_ep_ctx_t - - Add `time_t last_activity` to `ha_state_t` in ha.h; zero-initialize in `ha_init()` - - Add `time_t last_push_time` to `standby_ep_ctx_t` struct (ha.c internal); initialize to 0 in all allocation sites - - Remove `hb_vtask_t` typedef declaration from ha.h (struct moved to ha.c or fully removed) - - Observable: project builds; `ha_init()` test verifies `last_activity == 0` after init - - _Requirements: 1.1, 1.3, 3.1_ - -- [x] 1.2 Add heartbeat timing validation at startup - - Implement `validate_heartbeat_config(heartbeat_timeout_ms, heartbeat_interval_ms, err_buf, err_len)` in `startup_validation.c` - - Returns `-EINVAL` when `heartbeat_timeout_ms < 2 * heartbeat_interval_ms`; writes human-readable error into `err_buf` - - Wire call into `main.c` after `ha_config_t` is populated, before `ha_init()` - - Observable: starting controller with `heartbeat_timeout_ms = heartbeat_interval_ms` prints error and exits non-zero - - _Requirements: 2.4_ - - _Boundary: startup_validation.c, main.c_ - -- [x] 2. Core: standby liveness poll (replaces watchdog thread) -- [x] 2.1 (P) Implement liveness_poll_fn and replace hb_timeout_thread - - Define `liveness_poll_ctx_t` struct in ha.c: `{ ha_state_t *ha; cluster_t *c; _Atomic int cancelled; }` - - Implement `liveness_poll_fn(void *arg)`: check `cancelled` first; check `atomic_load(&ha->role) == CTRL_STANDBY` to guard against duplicate fires with EPOLLRDHUP path; read `ha->last_activity`, compute elapsed; if `elapsed >= heartbeat_timeout_ms` call `ha_on_primary_ctrl_fd_rdhup(ha, c)`; else re-arm via `delay_task_ms` at `heartbeat_interval_ms / 2` - - Replace `ha_arm_heartbeat_timeout_vtask`: allocate `liveness_poll_ctx_t`, write `clock_gettime` to `ha->last_activity`, call `delay_task_ms` to schedule first poll; log INFO "HA: standby liveness poll armed" - - Replace `ha_disarm_heartbeat_timeout_vtask`: set `ctx->cancelled = 1`, free ctx, set static ptr to NULL; no pthread_join needed - - Add `ha_disarm_heartbeat_timeout_vtask` call at the top of `ha_on_primary_ctrl_fd_rdhup` so that EPOLLRDHUP cancels any pending poll task before transitioning to CTRL_ELECTING (Req 5.4) - - Update `ha_on_heartbeat_received`: write `clock_gettime(CLOCK_MONOTONIC)` seconds to `ha->last_activity`; log DEBUG3 on each reset (not INFO to avoid log spam) - - Remove `hb_vtask_t` struct, `hb_timeout_thread` function, `s_hb_vtask` static variable from ha.c - - Observable: `grep -r hb_timeout_thread src/controller/*.c` returns zero matches; build passes with no new warnings; SIGKILL primary → standbys show exactly one `ha_on_primary_ctrl_fd_rdhup` log each (no duplicate fires) - - _Requirements: 1.2, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 4.2, 4.3, 5.2, 5.3, 5.4, 5.6_ - - _Boundary: ha.c (standby liveness path)_ - -- [x] 2.2 (P) Extend primary_heartbeat_fn with per-standby idle tracking - - In `primary_heartbeat_fn`: for each ep in `ha->standby_endpoints`, cast `ep->data` to `standby_ep_ctx_t` and read `sctx->last_push_time` - - Compute `elapsed = now - sctx->last_push_time`; if `elapsed >= heartbeat_interval_ms` call `xce_nop_req(ep)` and write `sctx->last_push_time = now`; else skip NOP for this standby - - In `on_standby_connected`: write `sctx->last_push_time = clock_gettime(...)` immediately after TLS handshake completes (before ctrlhello send); this counts the ctrlhello as the first push - - Remove the workaround `xce_nop_req(ep)` call added at the end of `on_standby_connected` (no longer needed) - - Observable: with 2 standbys active and no WAL traffic, primary sends NOP to each standby at most once per `heartbeat_interval_ms`; verified via DEBUG3 log counting - - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_ - - _Boundary: ha.c (primary heartbeat path, standby_ep_ctx_t)_ - -- [x] 3. Integration: wire liveness reset into opcode dispatch -- [x] 3.1 Update cluster.c opcode handlers to reset standby liveness on any primary message - - `OPCode_nop` handler: replace `if (ep->fd == c->ha.primary_ctrl_fd)` guard with `if (atomic_load(&c->ha.role) == CTRL_STANDBY)` — call `ha_on_heartbeat_received(&c->ha)` - - `OPCode_hello` handler (ctrlhello path): after the existing `handle_ctrlhello` call, additionally call `ha_on_heartbeat_received(&c->ha)` when role is CTRL_STANDBY (ctrlhello counts as primary activity per Req 4.1) - - `OPCode_replEntry` and `OPCode_walCatchupEntry` handlers: add `ha_on_heartbeat_received(&c->ha)` call when role is CTRL_STANDBY - - Observable: in a running 3-node cluster, after primary sends ctrlhello, `ha->last_activity` timestamp is fresh and standby liveness poll does not fire within `heartbeat_timeout_ms` - - _Requirements: 1.3, 4.1_ - - _Depends: 2.1_ - - _Boundary: cluster.c (opcode dispatch)_ - -- [x] 3.2 Verify config_repl.c ctrlhello integration and EPOLLRDHUP cancellation path - - Confirm `ha_arm_heartbeat_timeout_vtask` call in `handle_ctrlhello` (config_repl.c:1380) still correctly arms the delaymgr poll (not a thread) - - Confirm `ha_on_heartbeat_received` call at config_repl.c:1379 writes `last_activity` (new impl) - - Confirm `ha_on_primary_ctrl_fd_rdhup` now calls `ha_disarm_heartbeat_timeout_vtask` as its first step (Req 5.4) - - Observable: build passes; `handle_ctrlhello` unit test (existing) still green; `ha_arm_heartbeat_timeout_vtask` no longer spawns a pthread; SIGKILL primary test shows exactly one RDHUP log per standby (no duplicate from poll) - - _Requirements: 1.3, 4.1, 4.2, 5.2, 5.3, 5.4_ - - _Depends: 2.1, 3.1_ - - _Boundary: config_repl.c, ha.c (ha_on_primary_ctrl_fd_rdhup)_ - -- [x] 4. Validation: unit and integration tests -- [x] 4.1 (P) Unit tests for standby liveness poll - - Test `liveness_poll_fn` fires `ha_on_primary_ctrl_fd_rdhup` when `now - last_activity >= heartbeat_timeout_ms` - - Test `liveness_poll_fn` re-arms (no failover) when `now - last_activity < heartbeat_timeout_ms` - - Test `liveness_poll_fn` with `cancelled = 1` is a no-op (no failover, no re-arm) - - Test `ha_on_heartbeat_received` updates `ha->last_activity` to a non-zero value - - Test `ha_arm_heartbeat_timeout_vtask` does NOT spawn a pthread (thread count unchanged before/after call) - - Test `validate_heartbeat_config`: `timeout = 2 × interval` passes; `timeout = interval - 1` fails - - Observable: `meson test --suite controller` exits 0 with all new test cases green; zero ASAN errors - - _Requirements: 1.2, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4_ - - _Boundary: ha.c unit tests, startup_validation.c unit tests_ - - _Depends: 2.1, 1.2_ - -- [x] 4.2 (P) Unit tests for primary idle NOP suppression - - Test `primary_heartbeat_fn`: sends NOP to standby when `last_push_time` elapsed ≥ `heartbeat_interval_ms` - - Test `primary_heartbeat_fn`: skips NOP when `last_push_time` elapsed < `heartbeat_interval_ms` - - Test `on_standby_connected` writes non-zero `last_push_time` immediately - - Observable: all 3 test cases pass; `meson test --suite controller` exits 0 - - _Requirements: 3.1, 3.2, 3.3, 3.4_ - - _Boundary: ha.c unit tests_ - - _Depends: 2.2_ - -- [x] 4.3 Integration test: 3-node cold start — no false RDHUP loops and race elimination - - Deploy 3-node cluster (ctrl-1 has config, ctrl-2/3 tag=0); verify ctrl-1 elects PRIMARY - - Verify ctrl-2/3 receive ctrlhello, set `last_activity`, and do NOT trigger RDHUP before `heartbeat_timeout_ms` elapses from ctrlhello - - Verify cluster is stable for 60s with no `hb_timeout_thread` log entries (thread removed) and no RDHUP false-positive entries - - Observable: `grep "hb_timeout_thread\|heartbeat timeout" /tmp/xce-deploy/logs/ctrl-{2,3}.log` returns 0 lines during steady state; all 3 processes alive after 60s - - _Requirements: 1.3, 1.4, 4.1, 4.2, 4.3, 5.1, 5.3_ - - _Depends: 3.1, 3.2_ - -- [x] 4.4 Integration test: primary crash triggers failover within timeout window - - Start 3-node cluster; verify ctrl-1 PRIMARY and cluster stable - - SIGKILL ctrl-1; verify ctrl-2 or ctrl-3 detects silence within `heartbeat_timeout_ms`; promotes to PRIMARY via RDHUP path - - Verify promoted primary serves traffic; verify no infinite RDHUP loop - - Observable: promoted standby log shows `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING → CTRL_PRIMARY` exactly once; cluster operational within `heartbeat_timeout_ms + election_timeout_ms + committing_timeout_ms + 5s` - - _Requirements: 1.4, 2.1, 2.2, 2.3, 5.2_ - - _Depends: 4.3_ - -- [x]* 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT) - - Start 3-node cluster; SIGSTOP ctrl-1 for `1 × heartbeat_interval_ms`; SIGCONT; verify no failover fires - - SIGSTOP ctrl-1 for `heartbeat_timeout_ms + 500ms`; verify failover fires exactly once - - Observable: first scenario — no RDHUP log on standbys; second scenario — exactly one `ha_on_primary_ctrl_fd_rdhup` log per standby - - _Requirements: 2.2, 2.3_ - - _Depends: 4.4_ diff --git a/.kiro/specs/ha-peer-connection-reuse/design.md b/.kiro/specs/ha-peer-connection-reuse/design.md deleted file mode 100644 index ba376e01..00000000 --- a/.kiro/specs/ha-peer-connection-reuse/design.md +++ /dev/null @@ -1,427 +0,0 @@ -# Design Document: ha-peer-connection-reuse - -## Overview - -The XCE controller election currently discards outgoing vote connections after the vote exchange, forcing standbys to create new outbound connections to the primary for the persistent control channel. This produces a 10-second heartbeat timeout delay before standbys detect the primary, spurious retry warnings from premature config-fetch attempts, and a complex tier-1/2/3 primary discovery path. - -This spec reuses each candidate's outgoing vote endpoint as the persistent peer control channel: the election winner promotes its connections to replication channels; the loser closes its connections and waits passively for the primary's ctrlhello to arrive on its listener. - -The entire change is confined to `ha.c` and `cluster.c`. iomgr, the capnp wire protocol, and the raft library are not modified. - -### Goals - -- Eliminate the double-connection round-trip after election -- Remove the standby's active primary discovery path (fetch_config_from_primary) -- Arm the standby heartbeat watchdog immediately when CTRL_STANDBY is entered -- Preserve existing HA behaviors: failover, WAL replication, config push, heartbeat detection - -### Non-Goals - -- SSL/TLS certificate management changes -- Vote grant/deny protocol logic changes -- WAL entry format or replication content changes -- Config file format changes -- Cluster membership changes (add/remove peers) - ---- - -## Boundary Commitments - -### This Spec Owns - -- The lifecycle of outgoing vote endpoints from creation through promotion or close -- The `peer_ctrl_msg_handler` state-aware dispatcher (replaces `vote_reply_msg_handler` + explicit reassignment) -- Population of `ha->standby_endpoints` from promoted vote connections on the primary side -- Arming the standby heartbeat watchdog from `raft_cb_become_follower` instead of after `ha_find_and_connect_to_primary` -- Setting `ha->primary_ctrl_fd` from the inbound ctrlhello connection on the standby side -- Removing the `fetch_config_from_primary` delay task from the standby path - -### Out of Boundary - -- SSL/TLS negotiation details — handled by existing `client_ssl_ctx` / `on_vote_connected` path -- iomgr internals — zero changes to `src/lib/netio.c` or `src/include/netio.h` -- Raft library — zero changes to `src/lib/raft/raft.c` -- Heartbeat watchdog thread implementation — only the arming callsite changes -- The existing `ha_find_and_connect_to_primary` function — originally scoped to remain for future use; **drifted**: it was removed entirely during implementation (exceeds this design's original intent — see this spec's own research.md "Gap Re-Audit 2026-06-18", finding R4). The standby path no longer calls it, and the function no longer exists in the tree. - -### Allowed Dependencies - -- `ha_state_t`, `ha_elect_ctx_t`, `ha_vote_entry_t` — extended with new fields -- `vote_ep_ctx_t` — reused as-is for promoted connections -- `endpoint_t`, `iomgr_monitor`, `endpoint_set_msg_handler` — used unchanged -- `ha->standby_endpoints` (zarray) — populated from promoted connections -- `raft_cb_become_leader` / `raft_cb_become_follower` — extended to populate/close endpoints -- `ha_arm_heartbeat_timeout_vtask` — called from `raft_cb_become_follower` - -### Revalidation Triggers - -- Changing `ha_vote_entry_t` struct layout (tasks depending on `votes[].ep`) -- Changing `vote_ep_ctx_t` fields or lifetime contract -- Changing `raft_cb_become_leader` / `raft_cb_become_follower` signatures -- Adding new roles to `ctrl_role_t` (the dispatcher switch must be updated) -- Changing `ha->standby_endpoints` population semantics - ---- - -## Architecture - -### Existing Architecture Analysis - -The current election flow: -1. Candidate creates one `endpoint_t` per peer (outgoing vote), registers with iomgr -2. `on_vote_connected` sends the vote request and reassigns `ep->msg_handler = vote_reply_msg_handler` -3. After the election, the outgoing vote endpoints are closed by `ha_election_cleanup` -4. The winning standby calls `ha_find_and_connect_to_primary` to create a new inbound connection to the primary -5. The heartbeat watchdog is armed after this reconnection succeeds - -The promoted-connection flow (this spec): -1. Same vote endpoint creation -2. `on_vote_connected` sets `ep->msg_handler = peer_ctrl_msg_handler` (state-aware, never reassigned) -3. In `raft_cb_become_leader`: each surviving vote endpoint is added to `ha->standby_endpoints` (primary) -4. In `raft_cb_become_follower`: outgoing vote endpoints are closed; standby waits passively; watchdog armed immediately -5. Primary's ctrlhello arrives on the standby's listener, setting `ha->primary_ctrl_fd` - -### Architecture Pattern & Boundary Map - -```mermaid -graph TD - subgraph Election_Phase - A[raft_cb_send_vote_request] --> B[endpoint_connect + iomgr_monitor] - B --> C[on_vote_connected sets peer_ctrl_msg_handler] - C --> D[send RequestVote via xce_request] - end - - subgraph Election_Result - E[raft_cb_become_leader] --> F[promote: add ep to standby_endpoints] - G[raft_cb_become_follower] --> H[close: endpoint_fini each vote ep] - G --> I[arm heartbeat watchdog] - end - - subgraph Promoted_Connection_PRIMARY - F --> J[peer_ctrl_msg_handler role=PRIMARY] - J --> K[repl_ack_decode_and_dispatch: no-op ack, INFO log] - end - %% Note: CTRL_PRIMARY_COMMITTING was removed (superseded by ha-consistency-invariants / - %% ha-single-leader-role, 2026-06-11); ha_repl_ack_received no longer exists -- promotion - %% is now driven by the heartbeat decision loop + commit engine, not ack-counting on this ep. - - subgraph Standby_Passive - H --> L[wait for ctrlhello on listener] - L --> M[handle_ctrlhello sets primary_ctrl_fd] - I --> N[watchdog arm: fires on silence timeout] - end - - subgraph Dispatcher - D --> O[peer_ctrl_msg_handler] - O -- role=ELECTING --> P[vote_reply_decode_and_dispatch] - O -- role=PRIMARY --> K - O -- role=STANDBY + PreVote round --> R[pre_vote_reply_decode_and_dispatch] - O -- other role --> Q[drop packet] - end - %% pre_vote_reply_decode_and_dispatch branch added later by ha-prevote (2026-07-18) -- - %% shares this dispatcher but was not part of this spec's original scope. -``` - -### Technology Stack - -| Layer | Choice | Role | Notes | -|-------|--------|------|-------| -| HA election | ha.c (C, gnu17) | Endpoint lifecycle, dispatcher | All changes here | -| Cluster runtime | cluster.c (C) | standby_endpoints population, ctrlhello | Minor additions | -| iomgr | netio.c (C) | Unchanged | No modifications | -| Raft library | lib/raft/raft.c | Unchanged | No modifications | - ---- - -## File Structure Plan - -### Modified Files - -- `src/controller/ha.c` — All behavioral changes: `peer_ctrl_msg_handler`, `vote_ep_ctx_t` promoted lifetime, `ha_vote_entry_t.ep` tracking, `raft_cb_become_leader` promotion path, `raft_cb_become_follower` passive path + watchdog arm, remove `fetch_config_from_primary` delay task -- `src/controller/ha.h` — Add `endpoint_t *ep` to `ha_vote_entry_t`; add `peer_ctrl_msg_handler` forward declaration or make it accessible via a new test-seam hook; update comments on `ha_find_and_connect_to_primary` noting standby post-election path no longer calls it -- `src/controller/config_repl.c` — `handle_ctrlhello`: set `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY and ctrlhello arrives for the first time (Note: this design originally cited `cluster.c`; `handle_ctrlhello` actually lives in `config_repl.c` — drift already flagged in this spec's own research.md R4, corrected here) -- `src/controller/testsuite/test_ha_connection_reuse.c` — Tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello (Note: implemented under this filename, not `test_ha_election.c` as originally planned — that file is a pre-existing, unrelated quorum-election test suite) - ---- - -## System Flows - -### Election + Promotion (Winner Path) - -```mermaid -sequenceDiagram - participant C as Candidate ha.c - participant I as iomgr - participant P as Peer Standby - - C->>I: endpoint_connect + iomgr_monitor (per peer) - I->>C: on_vote_connected fires - C->>P: xce_request(OPCode_requestVote) - note over C: ep->msg_handler = peer_ctrl_msg_handler - P->>C: RequestVoteReply - I->>C: peer_ctrl_msg_handler(role=ELECTING) - C->>C: vote_reply_decode_and_dispatch → raft_on_vote_reply - C->>C: raft_cb_become_leader fires - C->>C: votes[i].ep added to ha->standby_endpoints - C->>P: push_config_to_standbys (on promoted ep) - C->>P: ha_send_ctrlhello_to_all_standbys (on promoted ep) - C->>C: repl_add_standby registers promoted ep with WAL replicator (ha-commit-engine addition) - P->>C: OPCode_replCatchUpDone on same ep - I->>C: peer_ctrl_msg_handler(role=PRIMARY) - C->>C: repl_ack_decode_and_dispatch: replCatchUpDone is now a no-op (ack-counting removed) -``` - -> Note: the original `role=COMMITTING` step and `ha_repl_ack_received → ha_promote_to_primary` -> ack-counting tail no longer exist — `CTRL_PRIMARY_COMMITTING` and `ha_repl_ack_received` were -> deleted by `ha-consistency-invariants`/`ha-single-leader-role` (2026-06-11); promotion is now -> driven by the heartbeat decision loop + commit engine (`ha-commit-engine`, `ha-repl-hardening`), -> not by counting `replCatchUpDone` acks on the promoted ep. See those specs' design.md. - -### Election + Passive Wait (Loser Path) - -```mermaid -sequenceDiagram - participant C as Loser ha.c - participant I as iomgr - participant W as Winner Primary - - C->>I: endpoint_connect + iomgr_monitor (per peer) - I->>C: on_vote_connected fires - C->>W: xce_request(OPCode_requestVote) - W->>C: RequestVoteReply (denied) - I->>C: peer_ctrl_msg_handler(role=ELECTING) - C->>C: raft_cb_become_follower fires - C->>C: endpoint_fini(votes[i].ep) — close all vote endpoints - C->>C: ha_arm_heartbeat_timeout_vtask - note over C: CTRL_STANDBY — passive wait on listener - W->>C: ctrlhello arrives on listener (existing iomgr path) - C->>C: handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd - C->>C: ha_on_heartbeat_received resets watchdog deadline -``` - ---- - -## Requirements Traceability - -| Requirement | Summary | Component | Flow | -|-------------|---------|-----------|------| -| 1.1 | Candidate establishes one outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` (unchanged) | Election Phase | -| 1.2 | Candidate reuses each surviving outbound vote connection as persistent peer channel | `raft_cb_become_leader` promotes ep; `peer_ctrl_msg_handler` dispatches by role | Promotion | -| 1.3 | Winner promotes vote connections to standby replication channels | `raft_cb_become_leader`: add `votes[i].ep` to `standby_endpoints` | Winner Path | -| 1.4 | Loser closes all outbound vote connections | `raft_cb_become_follower`: `endpoint_fini(votes[i].ep)` | Loser Path | -| 2.1 | CTRL_STANDBY does not initiate outbound TCP connections | Remove `fetch_config_from_primary` delay task from `raft_cb_become_follower` | Loser Path | -| 2.2 | Standby accepts ctrlhello / config / WAL on inbound connection from winner | Existing vtaskmgr path unchanged; `handle_ctrlhello` sets `primary_ctrl_fd` | Standby Passive | -| 2.3 | No heartbeat from primary within timeout → new election | `ha_arm_heartbeat_timeout_vtask` called in `raft_cb_become_follower` | Loser Path | -| 2.4 | CTRL_STANDBY does not invoke tier-1/2/3 discovery | Remove `fetch_config_from_primary` call; `ha_find_and_connect_to_primary` not called | Loser Path | -| 3.1 | First ctrlhello on promoted connection → designate as primary control channel | `handle_ctrlhello` sets `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY | Standby Passive | -| 3.2 | Primary control channel RDHUP → detect within one epoll cycle → CTRL_ELECTING | Existing `ha_on_primary_ctrl_fd_rdhup` unchanged; armed from ctrlhello ep | Standby Passive | -| 3.3 | Arm heartbeat watchdog with same semantics | `ha_arm_heartbeat_timeout_vtask` called in `raft_cb_become_follower` | Loser Path | -| 4.1 | Loser explicitly closes all outbound vote endpoints on election loss | `raft_cb_become_follower`: iterate `ha->elect_ctx->votes[i].ep`, call `endpoint_fini` | Loser Path | -| 4.2 | Failed outbound vote connection → count peer as denied, no reconnect | Existing `vote_ep_err_handler` unchanged | Error | -| 4.3 | New election → fresh outbound connections | `ha_start_election_async` creates new endpoints each time (unchanged) | Election Phase | -| 5.1 | Primary pushes config via promoted connection | `push_config_to_standbys` uses `standby_endpoints` (populated from promoted eps) | Winner Path | -| 5.2 | WAL replication via promoted connections | Existing catch-up path uses `standby_endpoints`; majority-ack commit tracking now also registers the promoted ep with the replicator via `repl_add_standby` (Note: added by `ha-commit-engine`, 2026-06-12 — see that spec's design.md) | Winner Path | -| 5.3 | Single-node self-elect without peer connections | Existing raft library single-node path unchanged | Election Phase | -| 5.4 | Committing timeout → degraded PRIMARY | `committing_timeout_fn` no longer exists (Note: deleted along with `CTRL_PRIMARY_COMMITTING` by `ha-consistency-invariants`, 2026-06-11; replaced by promotion-fence/own-term-gate machinery — `ha_arm_own_term_gate`, `ha_arm_promotion_fence_timer` — see that spec's design.md) | Winner Path | -| 5.5 | No new ASAN errors | Promoted ep lifetime owned by `vote_ep_err_handler` / `raft_cb_become_follower` cleanup | All | - ---- - -## Components and Interfaces - -| Component | File | Intent | Req Coverage | -|-----------|------|--------|--------------| -| `peer_ctrl_msg_handler` | ha.c | State-aware dispatcher; replaces `vote_reply_msg_handler` | 1.2 | -| `ha_vote_entry_t.ep` field | ha.h | Track per-peer outgoing endpoint for promotion/close | 1.3, 1.4, 4.1 | -| `raft_cb_become_leader` (extended) | ha.c | Promote vote eps → standby_endpoints | 1.3, 5.1, 5.2 | -| `raft_cb_become_follower` (extended) | ha.c | Close vote eps, arm watchdog, remove fetch_config call | 1.4, 2.1, 2.3, 2.4, 3.3 | -| `handle_ctrlhello` (extended) | config_repl.c (design originally cited cluster.c — corrected) | Set primary_ctrl_fd from ctrlhello ep | 3.1, 3.2 | - -### HA Election Layer - -#### peer_ctrl_msg_handler - -| Field | Detail | -|-------|--------| -| Intent | Single iomgr message handler for outgoing vote endpoints; routes by current role | -| Requirements | 1.2 | -| File | `src/controller/ha.c` | - -**Responsibilities & Constraints** - -- Set once in `on_vote_connected`; never reassigned -- Reads `ha->role` atomically at dispatch time -- Routes `CTRL_ELECTING` to vote reply decode (existing `vote_reply_msg_handler` logic, factored out as `vote_reply_decode_and_dispatch`) -- Routes `CTRL_PRIMARY` to repl-ack decode (`repl_ack_decode_and_dispatch`). (Note: `CTRL_PRIMARY_COMMITTING` no longer exists — removed by `ha-consistency-invariants`/`ha-single-leader-role`, 2026-06-11; `repl_ack_decode_and_dispatch`'s `OPCode_replCatchUpDone` branch is now an INFO-logged no-op and `ha_repl_ack_received` was deleted along with the ack-counting promotion path — see those specs' design.md) -- Also routes `CTRL_STANDBY` with an in-progress PreVote round to `pre_vote_reply_decode_and_dispatch` — added later by `ha-prevote` (2026-07-18), not part of this spec's original scope -- Any other role: drop packet (free packet, return 0) -- Runs in iomgr thread — must not block, for the default (dispatch-flag-OFF) path; `raft-drive-async` (2026-07-18) later added an optional vtask-pool dispatch mode (`ha_peerctrl_dispatch_enabled`) that can run this body off the iomgr thread — see that spec's design.md - -**Contracts**: State [ x ] - -##### State Management - -```c -/* Signature: matches msg_handler_t = int (*)(void*) */ -static int peer_ctrl_msg_handler(void *ep_raw); - -/* Internal helpers (static, same file) — current signatures (corrected; both - * gained an io_packet_t *pkt parameter, and vote_reply_decode_and_dispatch - * dropped its endpoint_t *ep parameter, as part of the raft-drive-async - * pkt-ownership refactor, 2026-07-18) */ -static int vote_reply_decode_and_dispatch(vote_ep_ctx_t *vctx, io_packet_t *pkt); -static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx, io_packet_t *pkt); -``` - -- State model: branches on `atomic_load_explicit(&ha->role, memory_order_acquire)` -- ELECTING path: existing vote-reply capnp decode → `vote_reply_dispatch` → `raft_on_vote_reply` -- PRIMARY path: decode `OPCode_replCatchUpDone` — now a no-op INFO log (ack-counting/`ha_repl_ack_received` removed, see note above); the same handler also decodes heartbeat-reply `stateReport` for convergence tracking -- Other: free packet, no-op - -**Implementation Notes** - -- `vote_reply_decode_and_dispatch` is exactly the body of the existing `vote_reply_msg_handler` (factored out); the XCE_TESTING term-fencing and `xce_vote_reply_hook` seams are preserved -- `repl_ack_decode_and_dispatch` decodes an inbound capnp packet; `OPCode_replCatchUpDone` is now a no-op (superseded — see note above), and it separately decodes heartbeat-reply `stateReport` messages for convergence tracking. The original "COMMITTING phase" framing no longer applies since that role was removed. - -#### raft_cb_become_leader (extended) - -| Field | Detail | -|-------|--------| -| Intent | After election win: populate `ha->standby_endpoints` from promoted vote endpoints | -| Requirements | 1.3, 5.1, 5.2 | -| File | `src/controller/ha.c` | - -**Responsibilities & Constraints** - -- Read `ha->elect_ctx` to get `ha_elect_ctx_t*` -- Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)` -- If `votes[i].ep != NULL` (connect succeeded): add to `ha->standby_endpoints` under `ha->lock` -- Existing `push_config_to_standbys` and `ha_send_ctrlhello_to_all_standbys` calls are unchanged; they iterate the now-populated array -- (Note: `ha-commit-engine`, 2026-06-12, added a further registration in this same loop — each promoted ep is also registered with the WAL replicator via `repl_add_standby(&c->persist.repl, ...)`, which is what majority-ack commit tracking now uses; `standby_endpoints` remains the config-push/ctrlhello target only. See that spec's design.md.) - -**Implementation Notes** - -- `votes[0]` is reserved for self in the raft library; peer endpoints start at index 1. Verify index mapping against `raft_cb_send_vote_request` peer_idx. -- If `ha->elect_ctx` is NULL when `raft_cb_become_leader` fires (e.g., single-node), skip promotion loop (no peers to promote). -- Risk: `votes[i].ep` may be NULL if the peer's connect failed; the NULL check guards this. - -#### raft_cb_become_follower (extended) - -| Field | Detail | -|-------|--------| -| Intent | After election loss: close vote endpoints, arm heartbeat watchdog, remove fetch_config call | -| Requirements | 1.4, 2.1, 2.3, 2.4, 3.3 | -| File | `src/controller/ha.c` | - -**Responsibilities & Constraints** - -- Read `ha->elect_ctx`, iterate `votes[i].ep`, call `endpoint_fini(votes[i].ep)` for non-NULL entries, set `votes[i].ep = NULL` -- Do NOT call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary` -- Call `ha_arm_heartbeat_timeout_vtask(ha, c)` after role is set to CTRL_STANDBY -- Existing `cluster_init_phase2` call and `ha->phase2_done` guard are preserved - -**Implementation Notes** - -- `endpoint_fini` closes the fd and releases the endpoint slot; iomgr is notified via epoll RDHUP on the peer side, which is expected -- Watchdog arm must happen after `ha->role = CTRL_STANDBY` is stored (role gate in watchdog logic) -- XCE_TESTING: `ha_arm_heartbeat_timeout_vtask` is a no-op when `c == NULL`; the test seam is preserved - -### Cluster Runtime Layer - -#### handle_ctrlhello (extended) - -| Field | Detail | -|-------|--------| -| Intent | Set `ha->primary_ctrl_fd` when ctrlhello arrives on standby's inbound connection | -| Requirements | 3.1, 3.2, 3.3 | -| File | `src/controller/cluster.c` | - -**Responsibilities & Constraints** - -- Existing ctrlhello handler already processes OP_hello and triggers WAL catch-up -- Add: if `ha->role == CTRL_STANDBY` and `ha->primary_ctrl_fd == -1`, set `ha->primary_ctrl_fd = ep->fd` -- Call `ha_on_heartbeat_received(ha)` to reset the watchdog deadline (already done by heartbeat path; add to ctrlhello path for first message) -- No other changes to the ctrlhello handler - -**Implementation Notes** - -- Guard with `primary_ctrl_fd == -1` for idempotency; a repeat ctrlhello on the same connection does not need re-registration -- `ha_on_primary_ctrl_fd_rdhup` already handles fd close and triggers re-election; no changes needed there - ---- - -## Data Models - -### ha_vote_entry_t (modified) - -```c -typedef struct { - xce_candidate_t *peer; /* Peer being solicited */ - uint64_t my_term; /* Term sent in OP_requestVote */ - uint64_t my_tag; /* config_tag sent in request */ - uint64_t my_lsn; /* WAL LSN sent in request */ - int vote_granted; /* 1 if peer granted vote */ - int replied; /* 1 if peer replied */ - uint64_t peer_tag; /* peer's config_tag from reply */ - uint64_t peer_lsn; /* peer's WAL LSN from reply */ - struct endpoint_t *ep; /* NEW: outgoing vote endpoint; NULL if connect failed */ -} ha_vote_entry_t; -``` - -**Invariants**: -- `ep` is set by `raft_cb_send_vote_request` when `iomgr_monitor` succeeds -- `ep` is NULL if `endpoint_connect` or `iomgr_monitor` failed for this peer -- Winner: `ep` is transferred to `standby_endpoints`; winner does NOT call `endpoint_fini` on promoted eps -- Loser: `ep` is closed via `endpoint_fini` in `raft_cb_become_follower`; set to NULL after close - -### vote_ep_ctx_t - -The `promoted` flag is NOT added — role is read from `ha->role` at dispatch time, which is the authoritative state, as this design intended. (Note: despite the original "(unchanged)" heading, the struct is no longer literally unchanged since this spec closed — `ha-term-safety` (2026-07-17) added `my_last_log_term`/`my_last_log_index` fields, and `ha-elect-ctx-lifecycle` (2026-07-18) changed its lifetime contract: the endpoint's `on_fini` hook now calls `ha_elect_ctx_release` to decrement a refcount on the owning `ha_elect_ctx_t`, rather than the simple election-cleanup-only lifetime this design describes. See those specs' design.md.) - ---- - -## Error Handling - -### Error Strategy - -Treat connection failures and packet decode errors identically to the existing vote path: count the peer as denied (or drop the packet for non-vote phases). - -### Error Categories - -**Connect failure** (vote_ep_err_handler): -- Existing behavior preserved: a connect-time or post-connect failure counts as a denied vote through the production tally (`vote_reply_dispatch`), not the legacy `handle_request_vote_reply` path (hardened later by `raft-adapter-conformance`) -- Promoted connection disconnect: `vote_ep_err_handler` fires when the standby drops. For `PRIMARY` role (Note: the `COMMITTING` role named here was removed — see the peer_ctrl_msg_handler note above), remove `ep` from `ha->standby_endpoints` under `ha->lock`; current code also calls `repl_remove_standby` to deregister from the WAL replicator (`ha-commit-engine` addition) — `ha_repl_ack_received` no longer exists - -**Packet decode error in repl_ack_decode_and_dispatch**: -- Drop packet, return 0 (iomgr continues monitoring endpoint) -- Log WARN with opcode for debugging - -**Unexpected role in peer_ctrl_msg_handler**: -- Drop packet, return 0 (CTRL_INIT: should not happen normally; CTRL_STANDBY on outgoing ep is now an EXPECTED case when a PreVote round is in progress — routed to `pre_vote_reply_decode_and_dispatch` rather than dropped, added by `ha-prevote`, 2026-07-18) -- Log WARN - -### Monitoring - -- Add `INFO` log in `raft_cb_become_leader` for each promoted endpoint: `"HA: promoted vote ep[%d] → standby channel"` -- Add `INFO` log in `raft_cb_become_follower` for endpoint close: `"HA: closing vote ep[%d] (loser path)"` -- Existing heartbeat watchdog log messages unchanged - ---- - -## Testing Strategy - -### Unit Tests (test_ha_election.c) - -1. **Dispatcher routes by role**: set `ep->msg_handler = peer_ctrl_msg_handler`; inject packet with role=ELECTING → verify `xce_vote_reply_hook` fires; inject packet with role=COMMITTING → verify repl-ack hook fires; inject with role=STANDBY → verify packet dropped -2. **Winner promotes endpoints**: call `raft_cb_become_leader` with `ha->elect_ctx` populated; verify `ha->standby_endpoints->count == n_peers_connected` -3. **Loser closes endpoints**: call `raft_cb_become_follower`; verify `votes[i].ep == NULL` for all i; verify `endpoint_fini` called (ASAN validates no leak) -4. **Watchdog armed on CTRL_STANDBY**: mock `ha_arm_heartbeat_timeout_vtask` via test hook; verify it is called exactly once from `raft_cb_become_follower` -5. **fetch_config_from_primary NOT called**: verify no `delay_task_ms` call for `fetch_config_from_primary` in `raft_cb_become_follower` (use test seam to intercept delay_task_ms) -6. **primary_ctrl_fd set on ctrlhello**: call `handle_ctrlhello` with role=CTRL_STANDBY and `primary_ctrl_fd=-1`; verify `primary_ctrl_fd == ep->fd` -7. **Promoted ep err_handler removes from standby_endpoints**: populate `standby_endpoints` with a mock ep; call `vote_ep_err_handler` with role=COMMITTING; verify ep removed from array - -### Integration Tests (test_ha_bootstrap.c / test_p3_integration.c) - -1. **Two-node election → standby receives ctrlhello without outbound connect**: boot two nodes; verify standby never calls `ha_find_and_connect_to_primary`; verify ctrlhello arrives within `election_timeout_ms + 200ms` -2. **Promoted connection carries WAL replication**: verify `OPCode_replCatchUpDone` decoded on promoted ep increments `repl_ack_count` -3. **Primary failover → fresh connections on re-election**: kill primary; verify standby transitions CTRL_ELECTING and creates fresh endpoints (not reusing closed ones) diff --git a/.kiro/specs/ha-peer-connection-reuse/requirements.md b/.kiro/specs/ha-peer-connection-reuse/requirements.md deleted file mode 100644 index 80c192e3..00000000 --- a/.kiro/specs/ha-peer-connection-reuse/requirements.md +++ /dev/null @@ -1,75 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE controller election currently creates outgoing vote connections that are discarded after the vote exchange, forcing standbys to create new outbound connections back to the primary for the persistent control channel. This double-connection pattern causes a 10-second heartbeat timeout delay before standbys detect the primary, retry warnings from premature config fetch attempts, and a complex tier-1/2/3 primary discovery path. This feature reuses the candidate's outgoing vote connections as persistent standby control channels after the election: the winner promotes its connections to replication channels; the loser closes its outbound connections and waits passively for the primary's ctrlhello to arrive on the listener. - -## Boundary Context - -- **In scope**: Lifecycle of outbound vote connections from creation through promotion or close; standby passive connection model; primary control channel setup via promoted connection; heartbeat watchdog against the promoted connection. -- **Out of scope**: SSL/TLS certificate management; vote grant/deny protocol logic; WAL entry format and replication content; config file format; cluster membership changes. -- **Adjacent expectations**: iomgr must support reassigning a `msg_handler` on an existing endpoint without closing and reopening the connection. The existing `endpoint_t` lifetime model is not changed. The existing heartbeat timeout and RDHUP detection semantics are preserved on the promoted connection. - -## Requirements - -### 1. Candidate Initiates All Peer Connections - -**Objective:** As a cluster operator, I want the controller that initiates an election to create and own all peer connections, so that standbys never need to discover or dial out to the primary. - -#### Acceptance Criteria - -1. When a controller enters `CTRL_ELECTING` state and sends vote requests, the controller shall establish one outbound TLS connection per peer for the vote RPC. -2. The controller shall reuse each surviving outbound vote connection as the persistent peer control channel for the duration of that node's role after the election result is known, without closing and re-establishing the connection. -3. When a controller wins the election, the controller shall promote each surviving outbound vote connection to a standby replication channel without creating new outbound connections to standbys. -4. When a controller loses the election, the controller shall close all outbound vote connections it created and remain fully passive on its listener. - ---- - -### 2. Standby Fully Passive Connection Model - -**Objective:** As a cluster operator, I want standby controllers to never initiate outbound connections to the primary, so that connection management is uniform and the standby cannot mis-identify the primary. - -#### Acceptance Criteria - -1. While in `CTRL_STANDBY` state, the controller shall not initiate any outbound TCP connections to peer controllers. -2. When the primary sends a ctrlhello, config push, or WAL replication message on the inbound connection accepted from the winning candidate, the standby controller shall accept and process it on that existing connection. -3. If a controller transitions to `CTRL_STANDBY` and receives no inbound communication from the primary within `heartbeat_timeout_ms`, the controller shall trigger a new election. -4. When a controller becomes a standby, the controller shall not invoke the tier-1/2/3 primary discovery path; the inbound connection from the winning candidate is the only primary control channel for that election round. - ---- - -### 3. Primary Control Channel via Promoted Connection - -**Objective:** As a cluster operator, I want the standby's primary control channel to be the inbound connection established by the winning candidate during the vote, so that heartbeat detection is immediate and no reconnection delay occurs. - -#### Acceptance Criteria - -1. When the primary sends the first ctrlhello on a promoted vote connection, the standby controller shall designate that connection as its primary control channel and arm the heartbeat timeout watchdog against it. -2. When the primary control channel drops (RDHUP), the controller shall detect the failure within one iomgr epoll cycle and transition to `CTRL_ELECTING`. -3. The controller shall arm the heartbeat timeout watchdog against the promoted inbound connection with the same timeout semantics currently applied to `primary_ctrl_fd`. - ---- - -### 4. Explicit Close of Outbound Connections on Election Loss - -**Objective:** As a cluster operator, I want the losing controller to cleanly close its outbound vote connections, so that the primary's promoted connections are the only active inter-controller channels after election. - -#### Acceptance Criteria - -1. When a controller determines the election is lost (via quorum denial or election timeout), the controller shall explicitly close all outbound vote endpoints it created for that election round. -2. If an outbound vote connection fails before a reply is received, the controller shall count that peer as a denied vote and not attempt to reconnect to that peer for replication purposes in that election round. -3. When a controller starts a new election after a failover event, the controller shall create fresh outbound connections to all peers for the new round. - ---- - -### 5. No Regression in Existing Behaviors - -**Objective:** As a cluster operator, I want all current HA behaviors preserved after connection reuse, so that failover, WAL replication, and config push continue to work correctly. - -#### Acceptance Criteria - -1. When the primary wins election, the controller shall push cluster configuration to standbys via the promoted connection with the same correctness guarantees as the current config push path. -2. When the primary enters `CTRL_PRIMARY_COMMITTING`, the controller shall replicate WAL entries to standbys via the promoted connections using the existing catch-up replication protocol. -3. When a single-node cluster starts, the controller shall self-elect to primary without creating any peer connections. -4. If the majority ack threshold is not reached within `committing_timeout_ms`, the controller shall promote to `CTRL_PRIMARY` in degraded mode (Req 8.4 behavior preserved). -5. When all controller tests pass (unit and integration), the controller shall produce no new ASAN errors compared to the baseline before this feature. diff --git a/.kiro/specs/ha-peer-connection-reuse/research.md b/.kiro/specs/ha-peer-connection-reuse/research.md deleted file mode 100644 index bcb0bfde..00000000 --- a/.kiro/specs/ha-peer-connection-reuse/research.md +++ /dev/null @@ -1,404 +0,0 @@ -# Research & Design Decisions: ha-peer-connection-reuse - -## Discovery Scope - -**Type**: Extension — modifies the existing election endpoint lifecycle in `ha.c` and standby connection model in `cluster.c`. - -**Discovery method**: Light discovery — codebase analysis of existing patterns. - -## Key Investigations - -### 1. Existing msg_handler Reassignment Pattern - -**Finding**: `on_vote_connected` (ha.c:1869) already performs `ep->msg_handler = vote_reply_msg_handler` without a lock. This is safe because iomgr is single-threaded and msg_handler is only written from the iomgr thread. - -**Implication**: A state-aware dispatcher (`peer_ctrl_msg_handler`) that reads `ha->role` atomically can replace this explicit reassignment without any iomgr changes. The role is already atomic (`_Atomic int32_t role`). - -### 2. vote_ep_ctx_t Lifetime - -**Finding**: `vctx` (vote_ep_ctx_t) lives in `ep->data`. It is currently freed by `ha_election_cleanup`. The `vote_ep_err_handler` intentionally does NOT free it because `vote_reply_msg_handler` also references `ep->data`. - -**Implication**: With connection reuse, `vctx` outlives the election for promoted connections. Lifetime is extended: freed only when the promoted endpoint closes (via err_handler) or when election cleanup runs for the loser path. - -### 3. ha->standby_endpoints Population - -**Finding**: Currently populated in `cluster.c` when standbys create inbound connections to the primary and register with `HostRole_controller`. With connection reuse, the primary populates it directly from promoted outgoing vote endpoints in `raft_cb_become_leader`. - -**Implication**: The existing `ha_send_ctrlhello_to_all_standbys` and `push_config_to_standbys` iterate `ha->standby_endpoints`. Populating it from promoted connections means no changes are needed to these functions — they work on whatever is in the array. - -### 4. Standby Side: fetch_config_from_primary - -**Finding**: `raft_cb_become_follower` (ha.c:457-462) schedules `fetch_config_from_primary` via `delay_task_ms` with 100ms delay. This is the tier-1/2/3 discovery path the spec wants to eliminate. - -**Implication**: Remove this call. The primary pushes config via promoted connections. The standby receives it passively on the inbound connection (handled by existing vtaskmgr path). - -### 5. Heartbeat Watchdog Arming - -**Finding**: `ha_arm_heartbeat_timeout_vtask` is called after `ha_find_and_connect_to_primary` succeeds (production path). With connection reuse, `ha_find_and_connect_to_primary` is not called on the standby path. - -**Implication**: Arm the watchdog in `raft_cb_become_follower` instead, immediately when transitioning to CTRL_STANDBY. The `ha_on_heartbeat_received` call from ctrlhello/heartbeat handlers resets the deadline. - -### 6. Primary Control Channel (primary_ctrl_fd) on Standby - -**Finding**: `ha->primary_ctrl_fd` is set by `ha_find_and_connect_to_primary`. The RDHUP and heartbeat watchdog use it as the anchor for failover. With connection reuse, the primary's inbound connection (accepted from winner's outgoing vote connection) is the primary control channel. - -**Implication**: Set `ha->primary_ctrl_fd = ep->fd` in `handle_ctrlhello` when ctrlhello arrives and role == CTRL_STANDBY. The existing controller leave handler in cluster.c checks `ep->fd == ha->primary_ctrl_fd` to trigger re-election. - -### 7. Promoted Connection Leave Handling - -**Finding**: The cluster.c controller leave handler removes endpoints from `ha->standby_endpoints`. This fires for inbound connections by checking `priv->role == HostRole_controller`. Promoted outgoing vote endpoints have `ep->data = vctx` (vote_ep_ctx_t), not a role struct, so the existing leave handler cannot identify them. - -**Decision**: Handle promoted endpoint disconnect in `vote_ep_err_handler` instead of the cluster.c leave handler. When role is COMMITTING/PRIMARY, the err_handler removes the endpoint from `ha->standby_endpoints`. - -## Architecture Decisions - -### Decision 1: State-Aware Dispatcher vs. Explicit Reassignment - -**Chosen**: State-aware dispatcher (`peer_ctrl_msg_handler`) that reads `ha->role` atomically. - -**Why not explicit reassignment**: Explicit reassignment in `raft_cb_become_leader` (which runs in the raft callback chain, still in iomgr thread) would be safe, but requires coordinating "which endpoints to reassign" and creates a narrow window where a message could arrive between role update and handler swap. The state-aware dispatcher eliminates this concern entirely. - -**Boundary**: `peer_ctrl_msg_handler` is set once at endpoint creation and never changed. - -### Decision 2: No Changes to iomgr - -**Chosen**: Zero changes to `src/lib/netio.c` or `src/include/netio.h`. - -**Why**: `endpoint_t` already supports `msg_handler` reassignment via `endpoint_set_msg_handler`. The state-aware dispatcher approach doesn't reassign at all — it reads state at dispatch time. No iomgr API changes are needed. - -### Decision 3: vote_ep_ctx_t Reuse for Promoted Connections - -**Chosen**: Keep `vote_ep_ctx_t` as `ep->data` for promoted connections. No separate replication context struct. - -**Why**: `vote_ep_ctx_t` already carries `ha`, `c`, and `peer_idx` — exactly what replication ACK handling needs. Adding a `promoted` flag (or reading role) is sufficient to differentiate phases. No new struct reduces code churn. - -### Decision 4: ha_vote_entry_t.ep for Endpoint Tracking - -**Chosen**: Add `endpoint_t *ep` to `ha_vote_entry_t` to track the outgoing vote endpoint per peer. - -**Why**: `raft_cb_become_leader` and `raft_cb_become_follower` receive only `ha_state_t*`. They read `ha->elect_ctx` to get `ha_elect_ctx_t*` and iterate `votes[]` to find endpoints. This is the cleanest way to pass endpoints from `raft_cb_send_vote_request` to the callbacks without additional global state. - -## Synthesis Outcomes - -**Generalization**: The "peer control channel" concept generalizes both the vote phase and the replication phase. One endpoint, two roles, one dispatcher. The role is the state machine. - -**Build vs. Adopt**: All changes are purely additive/modificative within existing ha.c/cluster.c patterns. No new libraries. - -**Simplification**: Removing `fetch_config_from_primary` from the standby path eliminates the tier-1/2/3 discovery complexity. The standby path becomes purely passive. - ---- - -## Post-Implementation Gap Analysis (2026-05-31) - -### Implementation Status - -Tasks 1–6 (all core implementation) are complete. The following was verified in the live codebase: - -| Component | Status | Evidence | -|-----------|--------|---------| -| `ha_vote_entry_t.ep` field in ha.h | ✅ Done | ha.h:82 | -| `peer_ctrl_msg_handler` (state-aware dispatcher) | ✅ Done | ha.c:2082 | -| `on_vote_connected` uses `peer_ctrl_msg_handler` | ✅ Done | ha.c:1905 | -| `raft_cb_become_leader` promotes vote eps to `standby_endpoints` | ✅ Done | ha.c:353–430 | -| `raft_cb_become_follower` closes vote eps + arms watchdog | ✅ Done | ha.c:440–496 | -| `handle_ctrlhello` sets `primary_ctrl_fd` (config_repl.c) | ✅ Done | config_repl.c:1596–1603 | -| `vote_ep_err_handler` removes ep from `standby_endpoints` | ✅ Done | ha.c:2437–2442 | -| Test file `test_ha_connection_reuse.c` (8 unit tests) | ✅ Done | testsuite/ | -| Test file wired into `meson.build` | ✅ Done | meson.build:120 | - ---- - -### Gap 1 (CRITICAL): `fetch_config_from_primary` Still Present in `ha_elect_evaluate_and_signal` - -**Location**: `src/controller/ha.c`, lines 1737–1743 - -```c -/* ha_elect_evaluate_and_signal — STANDBY result path (old election path) */ -#ifndef XCE_TESTING -if (c != NULL) { - extern int fetch_config_from_primary(cluster_t *c); - delay_task_ms(&c->delaymgr, 100, 1, - (delay_task_entry_fn_t)fetch_config_from_primary, c); -} -#endif -``` - -**Analysis**: There are **two code paths** that can transition a node to `CTRL_STANDBY`: - -1. `raft_cb_become_follower` (raft library callback, ha.c:440–496) — the new path introduced by the election-iomgr-refactor spec. This path **correctly does NOT call** `fetch_config_from_primary`. ✅ -2. `ha_elect_evaluate_and_signal` (ha.c:~1702–1744) — the older custom election path (invoked directly by `handle_request_vote_reply` / vote-thread path). This path **still calls** `fetch_config_from_primary`. ❌ - -**Requirement violated**: Req 2.1 ("CTRL_STANDBY does not initiate outbound TCP connections") and Req 2.4 ("CTRL_STANDBY does not invoke tier-1/2/3 discovery"). - -**Risk**: Whether this matters depends on which path is actually active at runtime. If the system now exclusively uses the raft library callback path (and `ha_elect_evaluate_and_signal` is never called with a STANDBY result), the bug is dormant. But the old path still exists in the code and can be triggered by direct vote-reply handling, so it is a latent correctness issue and contradicts the spec intent. - -**Fix**: Remove the `#ifndef XCE_TESTING` block at ha.c:1737–1743 (the `fetch_config_from_primary` call) from `ha_elect_evaluate_and_signal`. Matches task 4.1 intent, which was scoped only to `raft_cb_become_follower`. - ---- - -### Gap 2 (LOW): `ha_arm_heartbeat_timeout_vtask` Not Called from `ha_elect_evaluate_and_signal` - -**Analysis**: Similarly to Gap 1, `ha_elect_evaluate_and_signal` does not call `ha_arm_heartbeat_timeout_vtask` in its STANDBY result path, while `raft_cb_become_follower` does (ha.c:495). If the old path is still reachable, the heartbeat watchdog is not armed from that path. - -**Requirement potentially violated**: Req 2.3 and Req 3.3. - -**Fix**: Mirror the watchdog arm from `raft_cb_become_follower` into `ha_elect_evaluate_and_signal`'s STANDBY path, or confirm the old path is dead code and remove it. - ---- - -### Gap 3 (MEDIUM): Unit Tests Written But Not Verified Passing - -**Location**: `src/controller/testsuite/test_ha_connection_reuse.c` - -**Analysis**: The test file exists with 8 unit tests (task 7.1) and is wired into `meson.build`. However, task 7.1 is marked open in `tasks.md`. The tests reference several extern symbols that must be defined under `XCE_TESTING` in ha.c: - -- `peer_ctrl_msg_handler_for_test` — present (ha.c:2325) -- `vote_ep_err_handler_for_test` — present (ha.c:2316) -- `vote_reply_build_pkt_for_test` — present (ha.c:2123) -- `repl_catchup_done_pkt_for_test` — present (ha.c:2282) -- `vote_ep_ctx_build_for_test` — present (ha.c:2210) -- `raft_cb_become_leader_for_test` — present (ha.c:2336) -- `raft_cb_become_follower_for_test` — present (ha.c:2343) -- `ha_push_config_hook`, `ha_send_ctrlhello_hook` — present (ha.c:1523, 1530) - -All seams appear to be in place. The tests should pass, but a build run (`/xce-build controller`) is needed to confirm. - ---- - -### Gap 4 (LOW): Integration Tests Not Written (Tasks 7.2, 7.3) - -**Tasks open**: -- 7.2: Two-node integration test verifying standby never calls `ha_find_and_connect_to_primary` -- 7.3: Integration test confirming WAL replication ACK on promoted ep increments `repl_ack_count` - -**Risk**: Without integration tests, the full end-to-end behaviour (standby passive model, WAL replication via promoted connection) is unverified at the system level. The unit tests cover the individual components but not their interaction with the running cluster state machine. - ---- - -### Requirements Feasibility Assessment - -| Req | Summary | Status | Gap | -|-----|---------|--------|-----| -| 1.1 | Candidate establishes outbound TLS per peer | ✅ existing path unchanged | None | -| 1.2 | Vote connection reused as peer channel | ✅ `peer_ctrl_msg_handler` implemented | None | -| 1.3 | Winner promotes to `standby_endpoints` | ✅ `raft_cb_become_leader` extended | None | -| 1.4 | Loser closes vote endpoints | ✅ `raft_cb_become_follower` closes | None | -| 2.1 | CTRL_STANDBY no outbound TCP | ⚠️ partially | Old path still calls `fetch_config_from_primary` | -| 2.2 | Standby accepts ctrlhello/config/WAL passively | ✅ existing path unchanged | None | -| 2.3 | No heartbeat → new election | ✅ watchdog armed in new path | Old path unarmed | -| 2.4 | No tier-1/2/3 discovery | ⚠️ partially | Old path still schedules it | -| 3.1 | ctrlhello → designate primary control channel | ✅ `handle_ctrlhello` updated | None | -| 3.2 | RDHUP detected in one epoll cycle | ✅ existing mechanism | None | -| 3.3 | Watchdog arm semantics preserved | ✅ new path | Old path unarmed | -| 4.1 | Loser closes all outbound vote eps | ✅ `raft_cb_become_follower` | None | -| 4.2 | Failed connect = denied vote, no reconnect | ✅ `vote_ep_err_handler` unchanged | None | -| 4.3 | New election = fresh connections | ✅ `ha_start_election_async` unchanged | None | -| 5.1–5.4 | No regression in HA behaviors | ✅ promoted path uses existing iterators | Integration tests needed | -| 5.5 | No new ASAN errors | Unknown | Needs build + test run | - ---- - -### Implementation Approach Summary - -**Chosen approach**: Option A (extend existing components) — completed for tasks 1–6. - -**Remaining work** for full compliance: - -1. **(Critical — Req 2.1, 2.4)** Remove `fetch_config_from_primary` call from `ha_elect_evaluate_and_signal` STANDBY path (ha.c:1737–1743), or confirm the function is never reached on the STANDBY branch in production. Mirror `ha_arm_heartbeat_timeout_vtask` call if the path is still live. -2. **(Verification — task 7.1)** Run `/xce-build controller` to confirm all unit tests in `test_ha_connection_reuse.c` pass and ASAN is clean. -3. **(Integration — task 7.2)** Add two-node test verifying standby does not call `ha_find_and_connect_to_primary`. -4. **(Integration — task 7.3)** Add promoted-connection WAL ACK test. - ---- - -### Effort and Risk - -| Item | Effort | Risk | -|------|--------|------| -| Remove `fetch_config_from_primary` from old path | S (1 hour) | Low — isolated removal; old path analysis needed first | -| Verify unit tests pass | S (build + inspect) | Low — seams all present | -| Task 7.2 integration test | M (1–2 days) | Medium — requires two-node test harness plumbing | -| Task 7.3 integration test | M (1–2 days) | Medium — requires promoted ep + WAL harness | - -**Overall**: The core feature is implemented. The remaining work is test validation + one latent correctness issue in the old election path. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **ha-peer-connection-reuse** is **~98% complete**. All 7 task groups (1–7) are `[x]` in tasks.md. -- **Prior CRITICAL gap RESOLVED**: The `fetch_config_from_primary` call in `ha_elect_evaluate_and_signal`'s STANDBY path was removed (confirmed: only comments remain in ha.c). The standby is fully passive — no outbound TCP connections from `CTRL_STANDBY`. -- **Integration tests (7.2, 7.3)**: Both marked `[x]`. The two-node integration test and WAL ACK test are done. -- **Remaining**: Build verification (`/xce-build controller`) to confirm all unit tests in `test_ha_connection_reuse.c` pass under ASAN. -- **Cross-spec relationships**: `ha-repl-hardening` commit `0762577` (task 5.1: "reduce handle_ctrlhello to connection setup only") directly depends on this spec's `peer_ctrl_msg_handler` + promoted connection model. The two specs are consistent — ha-repl-hardening moves all replication decisions to the heartbeat decision loop, which works over the promoted connections established by this spec. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -> Scope: requirements.md (Reqs 1–5) vs working tree on `persistent_ha` (including uncommitted changes to `ha.c`, `ha.h`, `myapp.c`, `raft.c`). All tasks in tasks.md are checked `[x]`. This pass re-verifies each acceptance criterion against on-disk code and a live run of the controller unit-test binary. - -## 1. Current State Summary - -- All seven task groups are implemented in `src/controller/ha.c`, `src/controller/ha.h`, `src/controller/config_repl.c`, with tests in `src/controller/testsuite/test_ha_connection_reuse.c` (registered in `src/controller/meson.build:119`). -- The dispatcher (`peer_ctrl_msg_handler`, ha.c:2570) is set once in `on_vote_connected` (ha.c:2321, 3586) and routes by `ha->role`: ELECTING → `vote_reply_decode_and_dispatch` (ha.c:2370), COMMITTING/PRIMARY → `repl_ack_decode_and_dispatch` (ha.c:2456), other → WARN + drop (ha.c:2589). -- Winner promotion: `raft_cb_become_leader` (ha.c:513–529) appends every non-NULL `votes[i].ep` to `ha->standby_endpoints` under `ha->lock`, then pushes config (ha.c:544) and ctrlhello (ha.c:559) over those endpoints. -- Loser path: `raft_cb_become_follower` (ha.c:640–664) calls `endpoint_fini` on each vote ep, sets `CTRL_STANDBY`, then arms `ha_arm_heartbeat_timeout_vtask` — no outbound activity afterward. -- Standby side: `handle_ctrlhello` (config_repl.c:1357–1374) sets `primary_ctrl_fd = ep->fd` (idempotent `== -1` guard) and resets the liveness watchdog via `ha_on_heartbeat_received`. -- The tier-1/2/3 discovery path is fully removed: `ha_find_and_connect_to_primary` and `fetch_config_from_primary` no longer exist as functions (only stale comment references remain — ha.h:741, ha.c:3136–3137, cluster.c:1006, main.c:168). myapp.c:289 documents the new passive-standby contract. -- Implementation went beyond the spec in one compliant way: late joiners are reconnected by the primary heartbeat task (`primary_heartbeat_fn` / `on_standby_connected`, ha.c:85–86) via fresh outbound connections — this respects Req 4.2's "in that election round" restriction. -- Live test run (controller_unit_test built 2026-06-09 13:07, newer than all sources): 769 PASS, 0 FAIL; all 10 connection-reuse tests pass. However ASAN reports **37,360 bytes leaked in 397 allocations** at process exit (see Req 5.5 below). - -## 2. Requirement-to-Asset Map - -| Req | Summary | Code Assets | Status | -|-----|---------|-------------|--------| -| 1.1 | One outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` ha.c:~147–190; SSL via `ha->client_ssl_ctx` | **Implemented** (Constraint: TLS only when `client_ssl_ctx` is configured; plaintext fallback otherwise) | -| 1.2 | Reuse surviving vote connection as persistent peer channel | `peer_ctrl_msg_handler` ha.c:2570; set once at ha.c:2321/3586, never reassigned | **Implemented** — test `test_dispatcher_*` x3 PASS | -| 1.3 | Winner promotes vote connections to standby channels | `raft_cb_become_leader` promotion loop ha.c:513–529 | **Implemented** — `test_become_leader_promotes_vote_endpoints`, `..._skips_null_endpoints` PASS | -| 1.4 | Loser closes all outbound vote connections | `raft_cb_become_follower` close loop ha.c:640–655 | **Implemented** — `test_become_follower_closes_vote_endpoints` PASS | -| 2.1 | Standby initiates no outbound TCP to peers | `fetch_config_from_primary` deleted; passive wait myapp.c:288–300 | **Implemented** — `test_follower_passive_no_outbound_connect` PASS | -| 2.2 | Standby processes ctrlhello/config/WAL on inbound connection | cluster.c:246 dispatch → `handle_ctrlhello`; existing config/WAL handlers unchanged | **Implemented** | -| 2.3 | No inbound comm within timeout → new election | `ha_arm_heartbeat_timeout_vtask` ha.c:3408 armed at ha.c:664 and myapp.c skip-election path; fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING | **Implemented** (Note: comment at ha.c:662–663 says "election_timeout_ms"; actual watchdog uses `heartbeat_timeout_ms` — stale comment) | -| 2.4 | No tier-1/2/3 discovery on standby | `ha_find_and_connect_to_primary` removed entirely (exceeds design, which said it would "remain for future use") | **Implemented** | -| 3.1 | First ctrlhello designates primary control channel + arms watchdog | config_repl.c:1357–1374, idempotency guard `primary_ctrl_fd == -1` | **Implemented** — `test_handle_ctrlhello_sets_primary_ctrl_fd` PASS | -| 3.2 | RDHUP detected within one epoll cycle → CTRL_ELECTING | `ha_on_primary_ctrl_fd_rdhup` ha.c:3183–3199; iomgr err-handler path unchanged | **Implemented** | -| 3.3 | Watchdog same timeout semantics as `primary_ctrl_fd` | Same `ha_arm_heartbeat_timeout_vtask` vtask, `heartbeat_timeout_ms` check ha.c:3302+ | **Implemented** | -| 4.1 | Explicit close of vote endpoints on loss | `endpoint_fini` + NULL-out, ha.c:648–652 | **Implemented** | -| 4.2 | Failed vote connection → denied vote, no reconnect that round | `vote_ep_err_handler` ha.c:2942: ELECTING → `handle_request_vote_reply(grant=0)`; COMMITTING/PRIMARY → remove ep from `standby_endpoints` under lock | **Implemented** — `test_err_handler_removes_promoted_ep_from_standby` PASS | -| 4.3 | New election → fresh outbound connections | `ha_start_election_async` allocates fresh `ha_elect_ctx_t` per round; eps created in `raft_cb_send_vote_request` | **Implemented** | -| 5.1 | Config push via promoted connections | ha.c:531–553 `push_config_to_standbys` over populated `standby_endpoints` | **Implemented** | -| 5.2 | WAL replication via promoted connections | ctrlhello → catch-up over promoted eps; `repl_ack_decode_and_dispatch` → `ha_repl_ack_received` | **Implemented** — `test_promoted_ep_repl_ack_promotes_to_primary` PASS | -| 5.3 | Single-node self-elect, no peer connections | raft library shortcircuit (unchanged); `ectx`/NULL guards in both callbacks | **Implemented** | -| 5.4 | Committing timeout → degraded PRIMARY | `committing_timeout_fn` armed ha.c:567–591; cancel-on-stepdown ha.c:611–621 | **Implemented** | -| 5.5 | No new ASAN errors vs baseline | Live run: 769 PASS / 0 FAIL, but ASAN leak report at exit: 37,360 B / 397 allocs. Traces include this spec's test file (fixture callocs at test_ha_connection_reuse.c:334–335, 470–471 — test-only) AND one production allocation: `ha_arm_heartbeat_timeout_vtask` ctx (ha.c:3424) leaked when invoked via `handle_ctrlhello` (config_repl.c:1374) in the test harness (no live delaymgr to consume/free the ctx) | **Partial** — leaks exist; whether they are "new vs baseline" is unverified (Unknown) | - -**Score: 18 of 19 acceptance criteria Implemented; 1 Partial (5.5).** - -### Documentation/metadata drift (not functional gaps) - -- ha.c:3136–3137 comment still says "myapp_run schedules fetch_config_from_primary" — the function no longer exists. -- ha.h:741 comment references removed `ha_find_and_connect_to_primary`. -- ha.c:662 watchdog comment says `election_timeout_ms`; code uses `heartbeat_timeout_ms`. -- design.md places `handle_ctrlhello` in `cluster.c`; the actual handler lives in `config_repl.c:1357` (tasks.md is correct). -- design.md says `ha_find_and_connect_to_primary` "remains for future use"; it was removed entirely (the stronger guarantee — acceptable, but design should be noted as superseded). -- spec.json `phase` is still `"tasks-generated"` although all tasks are `[x]` and tests pass. -- Working tree carries uncommitted changes (`ha_init_raft` extraction, vote-gate now requires `primary_ctrl_fd >= 0` for the fresh-heartbeat deny, raft.c changes). These interact with this spec's assets and pass tests, but are not committed — provenance vs adjacent specs (ha-repl-hardening / ha-config-bootstrap) is unclear. - -## 3. Implementation Approach Options (for the remaining gap) - -### Option A: Extend Existing (close Req 5.5 + comment hygiene in place) — RECOMMENDED - -- Free test fixtures in `test_ha_connection_reuse.c` (callocs at lines 334–335, 470–471, and ctrlhello test path) at test teardown. -- For the `ha_arm_heartbeat_timeout_vtask` ctx leak: either free the armed ctx in the test teardown via the existing cancel flag, or gate the arm behind the existing `c == NULL` test seam in `handle_ctrlhello`'s call. -- Fix the four stale comments; update spec.json phase. -- ✅ Smallest diff, no interface change, all in files this spec already owns. ❌ Leaves "baseline comparison" implicit — pair with the Research item below. - -### Option B: New Components (ASAN baseline harness) - -- Add a `tools/asan-baseline.sh` that runs the controller suite on a pinned baseline commit and diffs leak reports, making Req 5.5 mechanically checkable for every spec. -- ✅ Permanent, reusable verification for all HA specs' "no new ASAN errors" criteria. ❌ New tooling outside this spec's boundary; more effort than the gap warrants on its own. - -### Option C: Hybrid - -- Do Option A now (fix the known leaks regardless of baseline status), and file Option B as a follow-up task under a tooling/steering spec. -- ✅ Closes 5.5 definitively (zero leaks ⇒ trivially "no new errors") while seeding durable verification. ❌ Slightly more coordination. - -## 4. Effort & Risk - -| Item | Effort | Risk | Justification | -|------|--------|------|---------------| -| Test-fixture leak frees (test file only) | S | Low | Mechanical `free()`/teardown additions in one test file | -| Watchdog-ctx leak under test harness | S | Low | Existing cancel-flag/test-seam patterns apply; production path (delaymgr consumes ctx) unaffected | -| Stale comment + design.md/spec.json corrections | S | Low | Documentation-only | -| Commit hygiene for uncommitted ha.c/ha.h/myapp.c/raft.c changes | S | Medium | Changes span two specs' boundaries; mis-attribution complicates future validation passes | -| Option B baseline-diff tooling (optional) | M | Low | New script + CI wiring, no production code | - -**Overall remaining effort: S. Overall risk: Low.** - -## 5. Recommendations for Design/Closure - -1. **Adopt Option A (or C)**: eliminate the reported leaks rather than litigating the baseline — a leak-free run satisfies Req 5.5 unconditionally. -2. Verify the production-side lifetime of the `ha_arm_heartbeat_timeout_vtask` ctx when `handle_ctrlhello` re-arms repeatedly (cancel-flag of the previous ctx must lead to a free in the delaymgr callback; confirm no slow accumulation on long-lived standbys). -3. Update spec.json to an implementation-complete phase and annotate design.md with the two superseded statements (handle_ctrlhello location; full removal of `ha_find_and_connect_to_primary`). -4. Commit or re-home the uncommitted working-tree changes with spec attribution before the next validation pass. - -### Research Needed - -- **ASAN baseline diff**: run the controller suite at the pre-feature commit to confirm which of the 397 leaked allocations predate this spec (only needed if Option A is rejected). -- **Watchdog ctx lifetime in production**: trace `heartbeat_timeout_ctx` ownership through cancel/re-arm cycles on a long-lived standby (ha.c:3337–3449). -- **Uncommitted raft.c changes**: determine which spec owns the `ha_init_raft` extraction and the `primary_ctrl_fd >= 0` vote-gate tightening (interacts with Req 3.1's designation semantics), and validate against the TLA+ checklist in `src/controller/CLAUDE.md`. - ---- - -## Gap Re-Audit (2026-06-18) - -> POST-IMPLEMENTATION GAP AUDIT. Scope: requirements.md (Reqs 1–5, 19 acceptance criteria) vs on-disk code on `persistent_ha`. tasks.md = 17/17 `[x]`; spec.json `phase: "implemented"`. Verified via codegraph_explore over the live source. This pass re-confirms the 2026-06-09 findings against code that has since absorbed adjacent specs (ha-prevote, ha-repl-hardening, ha-commit-durability, ha-convergence-recovery). - -### 1. What changed since the last audit (2026-06-09) - -- **Req 5.5 ASAN leak — CLOSED.** spec.json note (2026-06-10): test-fixture leaks fixed via `free(ep0)/free(ep1)` in the two affected tests; controller suite green under ASAN. The prior "Partial" verdict is upgraded to **Implemented**. -- **spec.json phase drift — FIXED.** Now `"implemented"` (was `"tasks-generated"`). -- **Dispatcher evolved (cross-spec, still compliant).** `peer_ctrl_msg_handler` (ha.c:3600) now routes `CTRL_PRIMARY` → `repl_ack_decode_and_dispatch` (ha.c:3618) and added a `CTRL_STANDBY && ha->raft.pre_vote_in_progress` → `pre_vote_reply_decode_and_dispatch` branch (ha.c:3610). The original design's `CTRL_PRIMARY_COMMITTING` case is gone because that role was removed by ha-consistency-invariants. Still "set once, never reassigned" — Req 1.2 intact. -- **`handle_ctrlhello` strengthened.** Now lives in config_repl.c:1436 (design.md said cluster.c — superseded, tasks.md was right). On first ctrlhello in `CTRL_STANDBY` with `primary_ctrl_fd == -1` it sets `primary_ctrl_fd = ep->fd`, calls `ha_on_heartbeat_received`, AND arms `ha_arm_heartbeat_timeout_vtask` (ha.c:1447–1455). Exceeds original Req 3.1 (which only required designation) — also covers 3.3 from the receive path. -- **Compliant superset on the primary side.** Late/failed peers are reconnected by `primary_heartbeat_fn` → `ha_connect_to_peer` (ha.c:5103) via fresh outbound connects with `standby_ep_err_handler_primary`. This is the primary acting as primary — does NOT violate Req 2.1 (standby passivity) and respects Req 4.2's "in that election round" scope. - -### 2. Requirement-to-Asset Map - -| Req | Summary | Code Asset (file:line) | Tag | -|-----|---------|------------------------|-----| -| 1.1 | One outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` ha.c (+ PreVote `raft_cb_send_pre_vote_request` ha.c:223); SSL via `ha->client_ssl_ctx` | CLOSED | -| 1.2 | Reuse surviving vote conn as persistent peer channel | `peer_ctrl_msg_handler` ha.c:3600 (set once `on_vote_connected`, never reassigned) | CLOSED | -| 1.3 | Winner promotes vote conns → standby channels | `raft_cb_become_leader` promotion loop ha.c:~1056 (`zarray_store(standby_endpoints, votes[i].ep)`) | CLOSED | -| 1.4 | Loser closes all outbound vote conns | `raft_cb_become_follower` close loop ha.c:~1179 (`endpoint_fini(votes[i].ep)`; NULL-out) | CLOSED | -| 2.1 | Standby initiates no outbound TCP to peers | `fetch_config_from_primary` + `ha_find_and_connect_to_primary` deleted; passive contract myapp.c:312 | CLOSED | -| 2.2 | Standby processes ctrlhello/config/WAL on inbound conn | cluster.c `myreq_handler` → `handle_ctrlhello` config_repl.c:1436; config/WAL handlers unchanged | CLOSED | -| 2.3 | No inbound comm within timeout → new election | `ha_arm_heartbeat_timeout_vtask` armed in `raft_cb_become_follower` (~ha.c:1238) + `handle_ctrlhello` (config_repl.c:1453) | CLOSED | -| 2.4 | No tier-1/2/3 discovery on standby | `ha_find_and_connect_to_primary` removed entirely (exceeds design "remain for future use") | CLOSED | -| 3.1 | First ctrlhello → designate primary control channel | `handle_ctrlhello` config_repl.c:1447–1455, idempotency guard `primary_ctrl_fd == -1` | CLOSED | -| 3.2 | RDHUP detected in one epoll cycle → CTRL_ELECTING | iomgr `io_handler`/`handle_error` netio.c:444–525 → err-handler → re-election | CLOSED | -| 3.3 | Watchdog same timeout semantics as `primary_ctrl_fd` | Same `ha_arm_heartbeat_timeout_vtask` (`heartbeat_timeout_ms`) ha.c:4825 | CLOSED | -| 4.1 | Explicit close of vote endpoints on loss | `endpoint_fini` + NULL-out in `raft_cb_become_follower` (~ha.c:1181) | CLOSED | -| 4.2 | Failed vote conn → denied vote, no reconnect that round | `vote_ep_err_handler` ha.c:3937: ELECTING→`handle_request_vote_reply(grant=0)`; PRIMARY→remove from standby_endpoints + `repl_remove_standby` | CLOSED | -| 4.3 | New election → fresh outbound connections | fresh `ha_elect_ctx_t`/eps per round in send-vote callback | CLOSED | -| 5.1 | Config push via promoted connections | `push_config_transport_only` config_repl.c:879 over populated `standby_endpoints` | CLOSED | -| 5.2 | WAL replication via promoted connections | `repl_ack_decode_and_dispatch` → `ha_repl_ack_received`; promoted ep registered via `repl_add`/`repl_remove_standby` | CLOSED | -| 5.3 | Single-node self-elect, no peer connections | raft single-node shortcircuit (unchanged); `ectx`/NULL guards in both callbacks | CLOSED | -| 5.4 | Committing timeout → degraded PRIMARY | promotion-fence/own-term-gate machinery (`ha_arm_own_term_gate`, fence timer ha.c:692) — superset of original `committing_timeout_fn` | CLOSED | -| 5.5 | No new ASAN errors vs baseline | Test-fixture leaks fixed 2026-06-10 (spec.json note); suite green under ASAN | CLOSED | - -**Score: 19 of 19 acceptance criteria CLOSED.** (Up from 18/19 + 1 Partial on 2026-06-09.) - -### 3. Residual gaps - -No functional or correctness gaps remain. Only documentation/metadata drift (cosmetic, non-blocking): - -- **R1 (LOW, doc):** Stale comment ha.c:~4498 "Transition to CTRL_STANDBY so myapp_run schedules fetch_config_from_primary" — the function no longer exists; the standby path is passive. Comment is now misleading. -- **R2 (LOW, doc):** `raft_cb_become_follower` watchdog comment still says "election_timeout_ms" while the armed vtask uses `heartbeat_timeout_ms` (`ha_arm_heartbeat_timeout_vtask`). Same stale wording flagged on 2026-06-09; still present. -- **R3 (LOW, doc):** Comment refs to `fetch_config_from_primary()` / `ha_find_and_connect_to_primary` persist in main.c:218, cluster.c:1209, ha.h:1091 as historical breadcrumbs. Harmless but reference removed symbols. -- **R4 (LOW, design drift):** design.md still (a) places `handle_ctrlhello` in cluster.c (actual: config_repl.c) and (b) says `ha_find_and_connect_to_primary` "remains for future use" (actual: removed). design.md should be annotated "superseded" — tasks.md/spec.json are correct. - -### 4. Integration risks (forward-looking, not gaps) - -- **Shared-asset coupling:** `peer_ctrl_msg_handler`, `standby_endpoints`, and the promoted-connection model are now load-bearing for ha-prevote (PreVote reply routing), ha-repl-hardening (heartbeat decision loop over promoted eps), and ha-commit-durability (promotion fence + own-term gate run on the primary that owns these eps). Any future change to the dispatcher's role switch or to `standby_endpoints` population must re-run those specs' suites. The design.md "Revalidation Triggers" list already names these — keep it authoritative. -- **vctx lifetime:** `vote_ep_err_handler` deliberately does NOT free `vctx` (ha.c:3978) — it is freed in `ha_election_cleanup`. With promoted connections outliving the election, confirm `ha_election_cleanup` (or endpoint teardown) still reaches every promoted ep's vctx on a long-lived primary. Currently consistent, but it is the one lifetime invariant most exposed to drift. - -### 5. Options for the open (cosmetic-only) work - -**Option A — Comment/design hygiene only (RECOMMENDED).** Fix R1–R4 in place (4 comment edits + 2 design.md annotations). Effort: S (~30 min). Risk: Low (no code-behavior change). Closes the spec cleanly with zero outstanding items. - -**Option B — Leave as-is, note in steering.** Record the stale-comment debt in `.wolf/memory.md` / a docs backlog and move on, since all 19 criteria are functionally CLOSED. Effort: XS. Risk: Low, but the misleading ha.c:4498 comment can mislead a future reader on the standby path. - -**Option C — Bundle into a cross-spec doc-sweep.** Defer R1–R4 to a single documentation pass across the HA spec family (several specs carry similar removed-symbol breadcrumbs). Effort: M (coordinated). Risk: Low; best ROI if other specs share the debt. - -| Item | Effort | Risk | -|------|--------|------| -| R1–R3 stale comment fixes | S | Low | -| R4 design.md "superseded" annotations | S | Low | -| Optional: vctx-lifetime trace on long-lived primary (integration risk #2) | S | Low | - -**Verdict: feature COMPLETE — 19/19 acceptance criteria CLOSED. Remaining work is documentation hygiene only (Option A recommended).** diff --git a/.kiro/specs/ha-peer-connection-reuse/spec.json b/.kiro/specs/ha-peer-connection-reuse/spec.json deleted file mode 100644 index cc594f37..00000000 --- a/.kiro/specs/ha-peer-connection-reuse/spec.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "feature_name": "ha-peer-connection-reuse", - "created_at": "2026-05-31T11:30:00Z", - "updated_at": "2026-05-31T12:30:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "Test-fixture ASAN leaks fixed 2026-06-10 (free(ep0)/free(ep1) in two tests). Controller suite green under ASAN. Req 5.5 closed." -} diff --git a/.kiro/specs/ha-peer-connection-reuse/tasks.md b/.kiro/specs/ha-peer-connection-reuse/tasks.md deleted file mode 100644 index 2489c0e4..00000000 --- a/.kiro/specs/ha-peer-connection-reuse/tasks.md +++ /dev/null @@ -1,92 +0,0 @@ -# Implementation Plan: ha-peer-connection-reuse - -- [x] 1. Foundation: track outgoing vote endpoint per peer - -- [x] 1.1 Add `ep` field to `ha_vote_entry_t` and populate it during vote endpoint creation - - Add `struct endpoint_t *ep` field to `ha_vote_entry_t` in `ha.h` (forward-declare `endpoint_t` if not already present in that header) - - In `raft_cb_send_vote_request` (ha.c): after `iomgr_monitor(ep)` succeeds, store `ep` in `ectx->votes[votes_idx].ep`; leave `votes[votes_idx].ep = NULL` on connect or monitor failure - - In the test helper `vote_ep_ctx_build_for_test`: also accept an `endpoint_t *ep` parameter and store it in `votes[peer_idx].ep` so tests can verify the field - - Observable: ASAN-clean build; `ectx->votes[i].ep` is non-NULL for successfully connected peers and NULL for failed peers in existing vote-send tests - - _Requirements: 1.1, 1.2, 1.3, 1.4, 4.1_ - -- [x] 2. Core: implement state-aware message dispatcher - -- [x] 2.1 (P) Implement `peer_ctrl_msg_handler` as a role-dispatching iomgr handler - - Add `static int peer_ctrl_msg_handler(void *ep_raw)` in ha.c - - On `CTRL_ELECTING`: call `vote_reply_decode_and_dispatch(ep, vctx)` — factor the body of the existing `vote_reply_msg_handler` into this static helper (preserving XCE_TESTING term-fencing and `xce_vote_reply_hook` seam) - - On `CTRL_PRIMARY_COMMITTING` or `CTRL_PRIMARY`: call `repl_ack_decode_and_dispatch(ep, vctx)` — new static helper that decodes an inbound capnp packet, checks opcode == `OPCode_replCatchUpDone`, then calls `ha_repl_ack_received(ha, c)`; logs WARN and drops on unexpected opcode - - On any other role (`CTRL_STANDBY`, `CTRL_INIT`): free packet, return 0, log WARN - - Observable: unit test verifies that injecting a vote-reply packet with role=ELECTING calls `xce_vote_reply_hook`; injecting a replCatchUpDone packet with role=COMMITTING calls the repl-ack hook; injecting with role=STANDBY results in packet dropped and no hook called - - _Requirements: 1.2_ - - _Boundary: peer_ctrl_msg_handler_ - -- [x] 3. Core: promote winner's vote endpoints to standby control channels - -- [x] 3.1 (P) Extend `raft_cb_become_leader` to populate `ha->standby_endpoints` from surviving vote endpoints - - Read `ha->elect_ctx`; if NULL (single-node path), skip the loop - - Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)` (note: votes[0] is self in the raft library; peer eps start at index 1 — verify against `raft_cb_send_vote_request` peer_idx mapping) - - For each non-NULL `votes[i].ep`: acquire `ha->lock`, append to `ha->standby_endpoints`, release lock - - Log `INFO "HA: promoted vote ep[%d] → standby channel"` for each promoted endpoint - - Observable: after `raft_cb_become_leader` fires in a unit test with 2 populated `votes[i].ep`, `ha->standby_endpoints->count == 2`; subsequent call to `ha_send_ctrlhello_to_all_standbys` iterates the promoted endpoints - - _Requirements: 1.3, 5.1, 5.2_ - - _Boundary: raft_cb_become_leader_ - -- [x] 4. Core: passive standby path — close vote endpoints and arm watchdog - -- [x] 4.1 (P) Extend `raft_cb_become_follower` to close vote endpoints and arm heartbeat watchdog; remove `fetch_config_from_primary` call - - Read `ha->elect_ctx`; if NULL, skip close loop - - Iterate `ctx->votes[i].ep` for all i: if non-NULL, call `endpoint_fini(votes[i].ep)`, set `votes[i].ep = NULL` - - Log `INFO "HA: closing vote ep[%d] (loser path)"` for each closed endpoint - - Remove the `#ifndef XCE_TESTING` block that calls `fetch_config_from_primary` via `delay_task_ms` — delete it entirely - - After `ha->role = CTRL_STANDBY` is stored, call `ha_arm_heartbeat_timeout_vtask(ha, c)` (no-op when `c == NULL` in tests) - - Observable: unit test with mocked `ha_arm_heartbeat_timeout_vtask` hook verifies it is called exactly once; `votes[i].ep == NULL` for all i after the call; no `delay_task_ms` for `fetch_config_from_primary` is dispatched - - _Requirements: 1.4, 2.1, 2.3, 2.4, 3.3, 4.1_ - - _Boundary: raft_cb_become_follower_ - -- [x] 5. Core: designate primary control channel from ctrlhello - -- [x] 5.1 (P) Extend `handle_ctrlhello` to set `primary_ctrl_fd` on first ctrlhello when role is CTRL_STANDBY - - In the ctrlhello handler (config_repl.c): if `ha->role == CTRL_STANDBY` and `ha->primary_ctrl_fd == -1`, set `ha->primary_ctrl_fd = ep->fd` - - Call `ha_on_heartbeat_received(ha)` to reset the watchdog deadline (marks the first live message from primary) - - Guard with `primary_ctrl_fd == -1` for idempotency — repeated ctrlhellos on the same connection do not re-register - - Observable: unit test calls `handle_ctrlhello` with `ha->role = CTRL_STANDBY`, `ha->primary_ctrl_fd = -1`; afterward `ha->primary_ctrl_fd == ep->fd`; calling again with the same ep leaves `primary_ctrl_fd` unchanged - - _Requirements: 3.1, 3.2, 3.3_ - - _Boundary: handle_ctrlhello_ - -- [x] 6. Integration: wire dispatcher and promoted-connection lifecycle - -- [x] 6.1 Replace `vote_reply_msg_handler` assignment in `on_vote_connected` with `peer_ctrl_msg_handler` - - In `on_vote_connected` (ha.c): change `ep->msg_handler = vote_reply_msg_handler` to `ep->msg_handler = peer_ctrl_msg_handler` - - `vote_reply_msg_handler` retained for `vote_reply_msg_handler_for_test` under XCE_TESTING; body factored into `vote_reply_decode_and_dispatch` - - Observable: existing vote-send tests still pass with no behavior change; the `xce_vote_reply_hook` seam path is reached via `peer_ctrl_msg_handler` → `vote_reply_decode_and_dispatch` - - _Depends: 2.1_ - - _Requirements: 1.2_ - -- [x] 6.2 Extend `vote_ep_err_handler` to remove promoted endpoint from `standby_endpoints` on disconnect - - When `vote_ep_err_handler` fires and `ha->role` is `CTRL_PRIMARY_COMMITTING` or `CTRL_PRIMARY`: acquire `ha->lock`, scan `ha->standby_endpoints` for `ep`, remove it, release lock - - Existing deny-vote path (fired during CTRL_ELECTING) is preserved unchanged — role gate ensures no double-action - - Observable: unit test populates `standby_endpoints` with a mock endpoint, sets role=COMMITTING, calls `vote_ep_err_handler`; verifies endpoint removed from array and `standby_endpoints->count` decremented - - _Requirements: 4.2, 5.5_ - -- [x] 7. Validation: tests and regression coverage - -- [x] 7.1 Unit tests for dispatcher, promotion, and passive path - - `peer_ctrl_msg_handler` role-routing: three scenarios (ELECTING → vote hook; COMMITTING → repl-ack hook; STANDBY → drop), each confirmed by mock hooks - - `raft_cb_become_leader` promotion: `standby_endpoints` populated from non-NULL `votes[i].ep` entries; NULL entries skipped - - `raft_cb_become_follower` close path: all `votes[i].ep` set to NULL; `ha_arm_heartbeat_timeout_vtask` called; no `fetch_config_from_primary` dispatched - - `handle_ctrlhello` idempotency: first call sets `primary_ctrl_fd`; second call is a no-op - - `vote_ep_err_handler` standby removal: endpoint removed from `standby_endpoints` when role=COMMITTING - - Observable: `make test` for controller testsuite passes with no new failures; ASAN reports no leaks or UAF - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.1, 4.2, 5.5_ - -- [x] 7.2 Integration test: standby receives ctrlhello without active outbound primary connect - - Using existing two-node test harness in `test_ha_bootstrap.c` or `test_p3_integration.c`: boot two nodes, verify the standby never invokes `ha_find_and_connect_to_primary` after election - - Verify ctrlhello arrives on the standby within `election_timeout_ms + 200ms` via the promoted inbound connection - - Verify `ha->primary_ctrl_fd` is set on the standby after ctrlhello - - Observable: test passes end-to-end; ASAN reports no new errors compared to baseline - - _Requirements: 2.2, 2.4, 3.1, 5.3, 5.5_ - -- [x] 7.3 Integration test: promoted connection carries WAL replication ACK - - Verify `OPCode_replCatchUpDone` decoded on the promoted ep increments `ha->repl_ack_count` and triggers `ha_promote_to_primary` when majority threshold reached - - Observable: `repl_ack_count` reaches quorum - 1 and role transitions to CTRL_PRIMARY; ASAN clean - - _Requirements: 5.2, 5.4_ diff --git a/.kiro/specs/ha-prevote/design.md b/.kiro/specs/ha-prevote/design.md deleted file mode 100644 index 41b788eb..00000000 --- a/.kiro/specs/ha-prevote/design.md +++ /dev/null @@ -1,336 +0,0 @@ -# Design Document - -## Overview - -**Purpose.** This feature adds a **PreVote phase** to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its persistent term or unseat a healthy primary. Before a node increments its term and campaigns, it must first win a hypothetical PreVote majority — a "would you vote for me in term+1?" round that mutates no node's persistent term or recorded vote. - -**Users.** Operators of 3-node controller HA clusters, and the controllers themselves, benefit: a frozen-then-resumed standby rejoins as a no-op instead of triggering the ~30s demote-and-restart re-election storm observed live on 2026-06-15. - -**Impact.** The election trigger path changes from `trigger → real election (term++)` to `trigger → PreVote round → (only on majority) → real election (term++)`. The real-vote state machine, replication, and commit engine are unchanged. The change is additive: two new wire opcodes reusing the existing vote payload structs, pure PreVote decision **and tally** functions in the raft library (mirroring the production real-vote path `raft_on_vote_reply`), and PreVote orchestration in the controller that reuses the existing eligibility gate. - -### Goals -- A node never increments its persistent term unless a PreVote majority is first obtained. -- A voter denies a PreVote whenever it still believes a leader is alive (leader stickiness), reusing the existing heartbeat-freshness predicate. -- A frozen-then-resumed standby in a healthy cluster gets no PreVote majority and re-attaches as a standby with the cluster term unchanged. -- Genuine leadership changes (true primary death, partition heal) still elect within existing timeout bounds. -- All Raft safety invariants and the bug-339/bug-340 fixes are preserved. - -### Non-Goals -- Changes to replication, the commit/barrier engine, snapshot/WAL catch-up transfer, or config bootstrap. -- Redesign of the freshness / log-completeness comparison (reused as-is). -- Backward/rolling-upgrade compatibility (product is pre-release; all controllers run the same build). -- Leader-side CheckQuorum self-demotion and the defense-in-depth real-vote stickiness guard (see Architecture → Deferred decisions). -- **The startup / first-boot election** (the `ha_elect_ctx_t.done` semaphore path). PreVote gates only the **failover** trigger; on first boot there is no leader to protect and every node's heartbeat is stale, so a PreVote round would trivially pass. The startup election path is untouched. - -## Boundary Commitments - -### This Spec Owns -- The **PreVote round** on the election-trigger path: a side-effect-free majority probe that runs before any persistent term increment. -- The **voter-side PreVote decision**: term check + reuse of the existing eligibility gate + reuse of the dispatcher freshness gate, with **zero persistent mutation**. -- The two new wire opcodes (`preVoteRequest`, `preVoteReply`) and their dispatch. -- The trigger-path restructuring so that `truncate-WAL + CTRL_ELECTING` happens only after PreVote succeeds. -- PreVote observability (denial reasons, abandoned-campaign logging). - -### Out of Boundary -- The real-vote state machine (`raft_on_vote_request`, `raft_on_vote_reply`, `raft_start_election`) — invoked unchanged after PreVote success. -- The freshness floor computation (`ha_vote_self_freshness`) and the eligibility gate (`raft_cb_vote_eligible`) — reused, not modified (except that the eligibility gate is now also called from the PreVote receiver path). -- Replication, commit barrier, snapshot transfer, demotion-convergence mechanics (truncate-above-committed + self-restart) — unchanged. -- Leader self-demotion / CheckQuorum. -- The startup-election `ha_elect_ctx_t.done` semaphore path — not gated by PreVote. - -### Allowed Dependencies -- raft library callback seams (`raft_callbacks_t`), the raft-library vote-tally pattern (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`), `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, the capnp election transport (`xce_request` / vote endpoints / `peer_ctrl_msg_handler`), and the `xce_vote_request_hook` test seam. -- Dependency direction (must not be violated): **capnp/types → raft library (pure) → controller adapter (ha.c) → dispatch/transport (cluster.c)**. The raft library must not depend on controller state. - -### Revalidation Triggers -- Any change to the `request_vote_req_t` / `request_vote_reply_t` payload shape (shared by real-vote and PreVote). -- Any change to `raft_cb_vote_eligible` semantics (now shared by both vote paths). -- Any change to the leader-stickiness predicate (`ha_heartbeat_is_fresh` / `heartbeat_timeout_ms`). -- Any change to the election-trigger entry points (`ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn`). -- Introduction of mixed-version clusters (would reintroduce the deferred wire-compat concern). - -## Architecture - -### Existing Architecture Analysis - -The election path is split across two layers (preserved by this design): - -- **raft library** (`src/lib/raft/raft.c`, `raft.h`) — the canonical, TLA+-checkable state machine. Owns term, `voted_for`, role, and the pure vote-decision logic. Communicates outward only through `raft_callbacks_t` (no controller dependency). Today: `raft_start_election` increments+persists the term before any RPC; `raft_on_vote_request` adopts a higher term and steps down a leader in **Step 2 — before** the `on_vote_eligible` gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side. -- **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election. (Note: `handle_request_vote_reply` is now the test-mode-only reply path; production reply collection — including endpoint/connect failures — routes through `vote_reply_dispatch`/`pre_vote_reply_dispatch` as the sole tally driver, per bug-1423/raft-adapter-conformance. `handle_request_vote_reply` is retained only for `XCE_TESTING` callers.) - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph trigger [Election Trigger] - Rdhup[ha_on_primary_ctrl_fd_rdhup] - Liveness[liveness_poll_fn] - end - subgraph controller [Controller Adapter ha.c cluster.c] - BeginFailover[ha_begin_failover] - PrevoteAsync[ha_start_prevote_async] - ElectAsync[ha_start_election_async] - Collect[raft_on_pre_vote_reply tally] - DispReq[cluster.c preVoteRequest case] - DispRep[cluster.c preVoteReply case] - Eligible[raft_cb_vote_eligible reused] - Fresh[ha_vote_self_freshness reused] - end - subgraph raftlib [Raft Library pure] - Begin[raft_begin_pre_vote] - OnReq[raft_on_pre_vote_request] - OnRep[raft_on_pre_vote_reply] - StartElection[raft_start_election unchanged] - end - subgraph wire [Transport] - Capnp[preVoteRequest preVoteReply opcodes] - end - - Rdhup --> BeginFailover - Liveness --> BeginFailover - BeginFailover --> PrevoteAsync - PrevoteAsync --> Begin - Begin --> Capnp - Capnp --> DispReq - DispReq --> Fresh - DispReq --> Eligible - DispReq --> OnReq - OnReq --> Capnp - Capnp --> DispRep - DispRep --> OnRep - OnRep --> Collect - Collect -->|majority| ElectAsync - ElectAsync --> StartElection - Collect -->|lost or timeout| Rearm[re-arm liveness stay STANDBY] -``` - -**Architecture Integration** -- **Selected pattern**: side-effect-free probe interposed before an existing state transition (Raft dissertation §9.6 PreVote). Adopted, not invented. -- **Boundaries**: raft library owns the pure PreVote *decision and tally*; the controller owns the *orchestration/transport*. The voter eligibility gate is **shared** with the real-vote path via reuse, not duplication; the PreVote tally mirrors the production real-vote tally (`raft_on_vote_reply`) rather than the test-only `ha_elect_ctx_t` path. -- **Existing patterns preserved**: callback-seam isolation of the raft library; raft-library-owns-tally (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`); dispatcher-applies-freshness-before-library; `xce_vote_request_hook` test seam. -- **New components rationale**: a new opcode pair keeps the real-vote handler byte-for-byte unchanged (zero regression risk to bug-339/340); pure PreVote functions in the library keep the state machine TLA+-checkable. -- **Steering compliance**: async, non-blocking vtask handlers; long-lived connections; inbound connections vote-only; immutable-term discipline (PreVote mutates nothing). - -### Deferred decisions (documented, not implemented) -- **Defense-in-depth real-vote stickiness guard** — rejecting a real higher-term RequestVote while a leader is believed alive. PreVote already guarantees a candidate only sends a real RequestVote after a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the safety-critical real-vote term-adoption path and risk regressing bug-339/340 for no in-scope benefit. Deferred; revisit only if mixed-version clusters or leader CheckQuorum are introduced. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Messaging / Events | Cap'n Proto (capnpc-c 1.0.1) | New `preVoteRequest`/`preVoteReply` opcodes reusing existing vote payload structs | No new structs; follows existing opcode-enum pattern | -| Backend / Services | C (raft library + controller) | Pure PreVote decision + tally functions; controller orchestration | No new libraries; reuses iomgr/delaymgr and the raft tally pattern | -| Infrastructure / Runtime | iomgr async transport, delaymgr | PreVote send/collect + round timeout | Non-blocking, mirrors election path | - -## File Structure Plan - -### Modified Files -- `src/schema/controller.capnp` — add two `OPCode` enum entries: `preVoteRequest`, `preVoteReply`. Reuse existing `RequestVoteRequest` / `RequestVoteReply` structs and their `Request.body` union arms (`requestvotemsg` / `requestvotereplymsg`); no new payload structs. -- `src/lib/raft/raft.h` — add pre-vote return codes (e.g. `RAFT_PRE_VOTE_WON`, `RAFT_PRE_VOTE_LOST`, `RAFT_PRE_VOTE_PENDING`), an optional `on_send_pre_vote_request` callback in `raft_callbacks_t`, pre-vote tally fields in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`), and declarations for the three new functions. -- `src/lib/raft/raft.c` — add `raft_begin_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply` (all pure: no `on_persist`, no `term`/`voted_for`/`role` mutation). `raft_on_pre_vote_request` reuses `on_vote_eligible`. **`raft_on_pre_vote_reply` owns the PreVote tally in `raft_state_t` (mirroring `raft_on_vote_reply`) and returns `WON/LOST/PENDING` — it never fires `on_become_leader`/`on_become_follower`.** -- `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here). (Note: as built, this handle is a module-static `prevote_round_ctx_t * _Atomic s_prevote_round_ctx` in `ha.c`, not a field added to `ha_state_t`/`ha.h` — `ha.h` was not modified.) -- `src/controller/ha.c` — add `ha_begin_failover` (common entry), `ha_start_prevote_async` (calls `raft_begin_pre_vote`, arms round timeout), and `pre_vote_reply_dispatch` (calls `raft_on_pre_vote_reply`, mirroring `vote_reply_dispatch → raft_on_vote_reply`); on `RAFT_PRE_VOTE_WON` perform truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST`/round-timeout stay `CTRL_STANDBY`, re-arm liveness, log abandonment. Modify `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn` to call `ha_begin_failover` (PreVote first; **no** truncate/`CTRL_ELECTING` until PreVote wins); modify `ha_on_heartbeat_received` to cancel an in-progress PreVote and re-attach. -- `src/controller/cluster.c` — add `OPCode_preVoteRequest` and `OPCode_preVoteReply` dispatch cases mirroring the real-vote cases, calling `raft_on_pre_vote_request` (no term adoption) and routing replies to `pre_vote_reply_dispatch`. - -### New Files -- `src/controller/testsuite/test_ha_prevote.c` — PreVote gating + frozen-then-resumed no-op scenario + genuine-failover scenario. - -### Test files extended -- `src/lib/testsuite/test_raft.c` — pure PreVote decision/tally unit tests + invariant assertions (no term mutation). -- `src/controller/testsuite/test_ha_vote_gate.c` — PreVote eligibility reuse of the role/heartbeat/catch-up gates. - -## System Flows - -### PreVote round (candidate side) - -```mermaid -sequenceDiagram - participant Trig as Trigger - participant Cand as Candidate ha.c - participant Peer as Peer voter - Trig->>Cand: ha_begin_failover (stay CTRL_STANDBY) - Cand->>Cand: ha_start_prevote_async (prospective term = current+1, no persist) - Cand->>Peer: preVoteRequest term+1 tag lsn - Note over Peer: stickiness gate then freshness gate then term check - Peer-->>Cand: preVoteReply granted or denied (peer term unchanged) - alt PreVote majority - Cand->>Cand: now truncate WAL, CTRL_ELECTING - Cand->>Cand: ha_start_election_async then raft_start_election (term++) - else lost or round timeout - Cand->>Cand: stay CTRL_STANDBY, re-arm liveness watchdog - Note over Cand: next primary heartbeat re-attaches as standby - end -``` - -Key decisions: the candidate remains `CTRL_STANDBY` and mutates nothing during the PreVote round; the destructive `truncate + CTRL_ELECTING` step is deferred to PreVote success. An incoming primary heartbeat during the round cancels it and re-attaches. The round is bounded by `election_timeout_ms`; on loss the watchdog is re-armed so a later genuine failure still triggers, and repeated PreVote attempts succeed once a majority's stickiness lapses. - -### PreVote decision (voter side) - -```mermaid -flowchart TD - A[preVoteRequest received] --> B{role == CTRL_PRIMARY} - B -- yes --> D[deny] - B -- no --> C{role == CTRL_STANDBY and heartbeat fresh} - C -- yes --> D - C -- no --> E{in catch-up} - E -- yes --> D - E -- no --> F{req.term lt current_term} - F -- yes --> D - F -- no --> G{freshness floor ok} - G -- no --> D - G -- yes --> H[grant, no term change, no persist, no voted_for] -``` - -Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Step G is exactly the existing dispatcher freshness floor (`ha_vote_self_freshness`). The only PreVote-specific rule is the **absence of any state mutation** on either grant or deny. - -(Note: superseded by `raft-commit-safety`, see that spec's design.md — `raft_on_pre_vote_request` now also runs an internal `raft_log_ok(lastLogTerm, lastLogIndex)` check between step F and step B/C/E, denying stale-log candidates before the eligibility gate is reached; not shown in the flowchart above.) - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces / Flows | -|-------------|---------|------------|--------------------| -| 1.1 | PreVote before term increment on trigger | `ha_begin_failover`, `ha_start_prevote_async` | PreVote round flow | -| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote`, `ha_start_prevote_async` | candidate stays STANDBY | -| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` | preVoteRequest payload | -| 1.4 | No majority → no real election | `raft_on_pre_vote_reply` (LOST), `pre_vote_reply_dispatch` | round flow "lost" path | -| 1.5 | Majority → start real election | `ha_start_election_async`, `raft_start_election` (unchanged) | round flow "won" path | -| 2.1 | Grant only if real-vote freshness gate passes | `cluster.c preVoteRequest`, `ha_vote_self_freshness` | voter flow step G | -| 2.2 | Fail freshness → deny | same | voter flow step G | -| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` | voter flow leaf H | -| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` (no `voted_for` write) | — | -| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` (`ha_heartbeat_is_fresh`) | voter flow step C | -| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` | voter flow C→E→F→G | -| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` (PRIMARY-deny) | voter flow step B | -| 4.1 | Frozen-resumed gets no majority | full path | frozen-rejoin scenario test | -| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` | round flow "lost"/cancel | -| 4.3 | Healthy primary not demoted/restarted | PreVote denies → no real RequestVote | — | -| 4.4 | Term unchanged, no re-election cycle | candidate stays STANDBY | — | -| 5.1 | True death → PreVotes granted → elect | `raft_cb_vote_eligible` (stale heartbeat → eligible) | genuine-failover test | -| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` (unchanged) | real-vote path | -| 5.3 | No regression of bug-339/bug-340 | real-vote path untouched | — | -| 5.4 | Complete within election-timeout bounds | `election_timeout_ms` round timeout | round flow | -| 6.1 | Election Safety preserved | real-vote path untouched; PreVote grants no leadership | RN-1 (research.md) | -| 6.2 | Term monotonicity; PreVote never changes term | `raft_on_pre_vote_*` (pure) | — | -| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | — | -| 6.4 | Consistent with raft.tla + checklist | `test_raft.c` invariant tests | RN-1 | -| 7.1 | PreVote over existing transport, no disruption to real votes | new opcodes, reused arms | — | -| 7.2 | Log PreVote denials with candidate + reason | `cluster.c preVoteRequest`, `on_trace`/`tlog` | — | -| 7.3 | Log abandoned campaign | `pre_vote_reply_dispatch` (LOST/timeout) | round flow | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|-------|--------|--------------|--------------------------|-----------| -| raft pre-vote functions | raft library | Pure PreVote decision + tally | 1.2, 1.3, 2.3, 2.4, 6.2 | `on_vote_eligible` (P0), `on_send_pre_vote_request` (P1) | Service, State | -| PreVote dispatch | cluster.c | Receive/answer PreVote, no mutation | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | `ha_vote_self_freshness` (P0), `raft_on_pre_vote_request` (P0) | Service, Event | -| PreVote orchestration | ha.c | Run round, gate trigger, branch to election | 1.1, 1.4, 1.5, 4.x, 5.4, 7.3 | `raft_on_pre_vote_reply` (P0), `ha_start_election_async` (P0), `liveness` (P1) | Service, State | -| Trigger restructuring | ha.c | Defer truncate/CTRL_ELECTING until PreVote wins | 1.1, 4.2, 4.3, 4.4 | `ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn` (P0) | State | - -### Raft library — pre-vote functions - -| Field | Detail | -|-------|--------| -| Intent | Pure PreVote decision and tally; canonical, TLA+-checkable; mutates no persistent state | -| Requirements | 1.2, 1.3, 2.3, 2.4, 6.2 | - -**Responsibilities & Constraints** -- `raft_begin_pre_vote`: compute prospective term = `term + 1` **without** assigning it to `s->term` or persisting; set `pre_vote_in_progress`, `pre_votes_granted = 1` (self); emit `on_send_pre_vote_request(ctx, peer_idx, prospective_term)` per peer. -- `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote). (Note: superseded by `raft-commit-safety`, see that spec's design.md — that spec (closed 2026-06-25, i.e. before this spec's own close) added a `raft_log_ok(lastLogTerm, lastLogIndex)` gate as an internal Step 2 inside `raft_on_pre_vote_request` itself, run **after** the stale-term check and **before** `on_vote_eligible`. The function signature grew two params — `cand_last_log_term`, `cand_last_log_index` — see the corrected Service Interface below. The dispatcher's `ha_vote_self_freshness` floor check in `cluster.c` still also runs, so freshness is now enforced at both layers.) -- `raft_on_pre_vote_reply`: **owns the PreVote tally** in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`), mirroring `raft_on_vote_reply`; return `RAFT_PRE_VOTE_WON` at quorum, `RAFT_PRE_VOTE_LOST` when all replied without quorum, else `RAFT_PRE_VOTE_PENDING`. Unlike `raft_on_vote_reply`, it **never** fires `on_become_leader`/`on_become_follower` and makes no role/term change — the controller acts on the returned status. -- **Invariant**: across all three functions, `s->term`, `s->voted_for`, `s->voted_for_term`, and `s->role` are unchanged, and `on_persist` is never invoked. - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -/* All pure: no persist, no term/role/voted_for mutation. */ -int raft_begin_pre_vote(raft_state_t *s); /* sends pre-votes, self-grant in raft_state_t */ -int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term, - uint64_t cand_last_log_term, - uint64_t cand_last_log_index); /* RAFT_GRANT=1 / RAFT_DENY=0 */ -int raft_on_pre_vote_reply(raft_state_t *s, - int peer_idx, int granted); /* tally in raft_state_t -> WON / LOST / PENDING */ -``` -(Note: superseded by `raft-commit-safety` — `raft_on_pre_vote_request` gained the `cand_last_log_term`/`cand_last_log_index` params and an internal `raft_log_ok` freshness check; see the Responsibilities & Constraints note above.) -- Preconditions: `raft_begin_pre_vote` requires `role == RAFT_FOLLOWER`. -- Postconditions: on `RAFT_PRE_VOTE_WON` the caller invokes the unchanged `raft_start_election`. -- Invariants: persistent state untouched (asserted in `test_raft.c`). - -**Implementation Notes** -- Integration: `on_vote_eligible` is the existing `raft_cb_vote_eligible`; reusing it gives PRIMARY-deny + heartbeat-stickiness + catch-up-deny for free (Req 3.x). -- Validation: unit tests assert `term`/`voted_for` unchanged after a granted PreVote. -- Risks: accidental term mutation — mitigated by keeping all term writes in `raft_start_election` only. - -### Controller — PreVote dispatch (cluster.c) - -| Field | Detail | -|-------|--------| -| Intent | Answer `preVoteRequest` with freshness + stickiness gates, no mutation; route `preVoteReply` to the tally | -| Requirements | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | - -**Responsibilities & Constraints** -- `OPCode_preVoteRequest`: parse `request_vote_req_t`; compute freshness floor via `ha_vote_self_freshness`; deny on freshness failure; otherwise call `raft_on_pre_vote_request`; reply `request_vote_reply_t` with **unchanged** `term`, the grant bit, and `peer_config_tag`/`peer_last_log_lsn`. Log denials with candidate id + reason (Req 7.2). -- `OPCode_preVoteReply`: route to `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply` (raft-library tally), paralleling the production `vote_reply_dispatch` → `raft_on_vote_reply`. -- Inbound connections remain vote-only; the reply carries no state push. - -**Contracts**: Service [x] / Event [x] - -##### Event Contract -- Published: `preVoteRequest` (term+1, candidate_id, config_tag, last_log_lsn). -- Subscribed: `preVoteReply` (term, vote_granted, peer_config_tag, peer_last_log_lsn). -- Delivery: best-effort over the existing election transport; missing/denied replies count as non-grants. No ordering requirement beyond per-peer idempotency: `raft_on_pre_vote_reply` receives `peer_idx` and tracks a per-peer replied flag in `raft_state_t` to avoid double-counting duplicate replies. - -### Controller — PreVote orchestration & trigger restructuring (ha.c) - -| Field | Detail | -|-------|--------| -| Intent | Run the side-effect-free round, gate the failover trigger, collect replies, branch to real election or clean abandon | -| Requirements | 1.1, 1.4, 1.5, 4.1–4.4, 5.4, 7.3 | - -**Responsibilities & Constraints** -- `ha_begin_failover`: single entry from both `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn`; starts `ha_start_prevote_async`; **does not** truncate WAL or transition to `CTRL_ELECTING` (deferred to PreVote success). -- `ha_start_prevote_async`: call `raft_begin_pre_vote` (self-grant + send `preVoteRequest` per peer via `on_send_pre_vote_request`); arm an `election_timeout_ms` round timeout. The PreVote tally lives in `raft_state_t`, not a controller struct. -- `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply`: on `RAFT_PRE_VOTE_WON` → truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST` or round timeout → stay `CTRL_STANDBY`, re-arm liveness, log abandonment (Req 7.3). -- `ha_on_heartbeat_received`: if a PreVote round is in progress, cancel it (clear the round handle) and re-attach as standby (Req 4.2) — no term change. -- **Invariant**: while a PreVote round is in progress, `ha->role` stays `CTRL_STANDBY` and no WAL/term mutation occurs. - -**Contracts**: Service [x] / State [x] - -##### State Management -- State model: the PreVote tally (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`) lives in `raft_state_t`, mirroring the real-vote tally; quorum formula `(n_peers / 2) + 1` reused from `raft_on_vote_reply`. The controller holds only a round-timeout handle + in-progress flag for cancellation. -- Concurrency: PreVote reply handling runs on the same single-threaded dispatch path as `vote_reply_dispatch`; no new lock introduced. - -**Implementation Notes** -- Integration: PreVote success calls the **unchanged** `ha_start_election_async`/`raft_start_election`, so the real election (and bug-339/340 behavior) is unaffected. -- Validation: `test_ha_prevote.c` asserts the frozen-rejoin no-op (term constant, primary not demoted) and genuine-failover success. -- Risks: a failed PreVote leaving the watchdog un-armed (would freeze failover) — covered by an explicit re-arm test. - -## Error Handling - -### Error Strategy -- **No PreVote majority** (expected, frozen-rejoin or split): not an error — abandon cleanly, stay `CTRL_STANDBY`, re-arm liveness, log at debug/info with reason. The cluster term is unchanged. -- **Round timeout**: treated identically to "lost"; the next liveness cycle retries PreVote, succeeding once a majority's stickiness lapses (bounds legitimate failover by heartbeat-timeout, not unbounded — Req 5.4). -- **Reply from a higher-term peer**: the PreVote reply path does **not** adopt the peer's term (PreVote is side-effect-free); a genuinely higher term is observed and adopted only through the normal real-vote / heartbeat paths, unchanged. -- **Unknown candidate id / malformed payload**: deny PreVote (same as real-vote dispatch), log. - -### Monitoring -- Log every PreVote denial with candidate id + reason (`primary`, `fresh_heartbeat`, `catch_up`, `stale_term`, `freshness`) — Req 7.2. -- Log campaign abandonment distinctly from genuine failover so operators can tell a suppressed disruptive election from a real one — Req 7.3. - -## Testing Strategy - -### Unit Tests (raft library — `test_raft.c`) -- `raft_on_pre_vote_request` grants for a fresh, higher-term candidate when `on_vote_eligible` returns 1, and asserts `term`/`voted_for`/`role` unchanged (2.3, 6.2). -- `raft_on_pre_vote_request` denies on `cand_term < term` and when `on_vote_eligible` returns 0 (2.2, 3.x). -- `raft_on_pre_vote_reply` returns WON at quorum, LOST when all replied without quorum, with no role/term change (1.4, 1.5, 6.2). -- A granted PreVote does not set `voted_for`, so a later real vote in the same term is still grantable (2.4). - -### Integration Tests (controller — `test_ha_prevote.c`, `test_ha_vote_gate.c`) -- **Frozen-then-resumed no-op**: 3-node, primary alive; a resumed standby starts PreVote; both peers deny (PRIMARY + fresh heartbeat); assert no real RequestVote sent, cluster term unchanged, primary still `CTRL_PRIMARY` (4.1, 4.3, 4.4). -- **Genuine failover**: primary dead, peers' heartbeats stale; PreVote granted → real election → new primary within bounds (5.1, 5.4). -- **Re-attach on heartbeat mid-round**: PreVote in progress, primary heartbeat arrives → round cancelled, node re-attaches as standby, no term change (4.2). -- **PreVote eligibility reuse**: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny exercised through the PreVote path (3.1–3.3). -- **Liveness re-arm after loss**: failed PreVote re-arms the watchdog; a subsequent genuine failure still triggers (1.4 robustness). - -### Verification (Req 6) -- `test_raft.c` invariant assertions and a `raft/raft.tla` consistency argument (RN-1, tracked in `research.md`) confirm Election Safety and term monotonicity hold with the PreVote path; the real-vote path is unchanged so bug-339/340 coverage is preserved (5.3). diff --git a/.kiro/specs/ha-prevote/requirements.md b/.kiro/specs/ha-prevote/requirements.md deleted file mode 100644 index b20a45a1..00000000 --- a/.kiro/specs/ha-prevote/requirements.md +++ /dev/null @@ -1,211 +0,0 @@ -# Requirements Document - -## Project Description (Input) - -**Who has the problem.** Operators of XCE 3-node controller HA clusters, and the -controllers themselves, when a controller is temporarily unreachable to its -peers but not crashed — e.g. a process freeze (SIGSTOP, long GC/allocator -stall), a one-way or transient network partition, or a slow host — and then -becomes reachable again ("frozen-then-resumed" / "rejoining" node). - -**Current situation.** The XCE election layer (Raft library `src/lib/raft/raft.c` -plus the controller adapter in `src/controller/cluster.c`, `ha.c`, `myapp.c`) -has no PreVote phase. A node that misses heartbeats hits its liveness timeout -(`liveness_poll_fn` / RDHUP → `ha_on_primary_ctrl_fd_rdhup`) and immediately -starts a real election, which **increments its persistent term** before any RPC -(`raft_start_election`, `raft.c:68`). When such a node rejoins with an inflated -term and sends `RequestVote`, the live primary adopts the higher term and steps -down: `raft_on_vote_request` (`raft.c:125-134`, Step 2) updates `term`, sets -role to follower, and fires `on_become_follower` **before** any freshness or -role gate. The controller dispatcher (`cluster.c` `OPCode_requestVote`) only -guards this with the `freshness_ok` check, so a rejoining node that was -caught-up before the stall is fresh enough to pass and topples the leader. -Because `raft_cb_become_follower` treats a was-PRIMARY demotion as -demotion-convergence (truncate-above-committed-LSN + **self-restart**, -`ha.c`), a single disruptive `RequestVote` makes the healthy primary truncate -and restart, triggering a multi-term re-election storm. - -Observed live (2026-06-15, 3-node ASAN deploy): a SIGSTOP'd-then-resumed standby -forced the primary to demote and self-restart; terms climbed 3 → 6 over ~30s -before the cluster reconverged. No committed data was lost (`committed_lsn` and -all submitted jobs survived), and the cluster self-healed — but availability -blipped for ~30s on an event that should be a no-op. A normal crash+restart does -NOT trigger this: a restarted node restores its term from disk and observes -`leader already known → skipping own election` (`myapp.c`). The problem is -specific to nodes that inflate their term without a genuine leadership change. - -This is the classic Raft "disruptive server" problem (Raft dissertation §9.6). -The repository's TLA+ reference (`raft/raft.tla`) and the controller CLAUDE.md -verification checklist require term monotonicity and Election Safety to be -preserved by any change to this path. - -**What should change.** Introduce a PreVote phase so a node cannot inflate its -term or disrupt a healthy leader merely by becoming temporarily unreachable. -Before incrementing its term and campaigning, a candidate must first win a -PreVote majority — a hypothetical "would you vote for me in term+1?" round that -does NOT change any node's persistent term or recorded vote. Peers grant a -PreVote only if they (a) would grant the corresponding real vote under the -existing freshness/log-completeness gate AND (b) have not heard from a current -leader within the minimum election timeout (leader stickiness). A node that has -a live primary connection (fresh heartbeat / valid `primary_ctrl_fd`) denies -PreVotes, so a rejoining node in a cluster with a healthy leader gets no PreVote -majority, never increments its term, and simply re-attaches as a standby when -the primary's next heartbeat/ctrlhello arrives. - -The change must: -- Preserve all existing Raft safety invariants (Election Safety ≤1 leader/term, - Leader Completeness, term monotonicity, monotonic commit index) and remain - consistent with `raft/raft.tla` and the controller verification checklist. -- Not regress the already-fixed failover (bug-339) and rejoin (bug-340) paths, - nor the legitimate cases where a real leadership change SHOULD occur (true - primary death, network healing into a higher-term majority). -- Integrate with the existing two-tier transport (capnp `RequestVote` / - `RequestVoteReply` over iomgr, the `ha_elect_ctx_t` vote collection, and the - raft library callback seams) without breaking the unit-test seams. - -Scope: the raft library election state machine and its PreVote RPC, the -controller vote dispatch/freshness/heartbeat-stickiness gates, the election -trigger path (liveness timeout / RDHUP → PreVote → real election), and the -wire message (a PreVote flag on the existing vote RPC or a new PreVote message). -Out of scope: changes to replication, the commit engine, snapshot transfer, or -config bootstrap beyond what the PreVote gating requires. - -## Introduction - -This feature adds a **PreVote phase** to the XCE controller HA election path so -that a controller which is merely temporarily unreachable cannot inflate its -term or unseat a healthy primary. Before a node increments its persistent term -and starts a real election, it must first win a hypothetical PreVote majority. -Peers grant a PreVote only when the candidate could legitimately become leader -(the existing freshness / log-completeness gate) **and** the peer no longer -believes a current leader is alive (leader stickiness). The net effect: a -frozen-then-resumed standby in a cluster with a healthy primary gets no PreVote -majority, never increments its term, and re-attaches as a standby — while -genuine leadership changes (true primary death, healing into a higher-term -majority) still proceed. The change must preserve every existing Raft safety -invariant and must not regress the previously fixed failover (bug-339) and -rejoin (bug-340) behavior. - -Throughout this document, "the Election Module" refers to the controller's HA -election state machine (raft library plus its controller adapter) as observed -through cluster behavior; "the voting controller" refers to a peer evaluating an -incoming PreVote request; and "the HA cluster" refers to the externally observed -behavior of the 3-node controller cluster as a whole. - -## Boundary Context - -- **In scope**: - - A PreVote round that precedes any persistent term increment and real - election on the election-trigger path (liveness timeout / primary-connection - loss). - - Voter-side PreVote evaluation combining the existing freshness / - log-completeness gate with a new leader-stickiness gate. - - The user-/operator-observable outcomes for frozen-then-resumed rejoin, - genuine failover, and partition healing. - - PreVote requests and replies traveling over the existing election transport. -- **Out of scope**: - - Replication, the commit/barrier engine, snapshot/WAL catch-up transfer, and - config bootstrap — except where PreVote gating must read state those paths - already expose. - - Redesign of the existing freshness / log-completeness comparison itself; this - feature reuses it as the real-vote eligibility criterion. -- **Adjacent expectations** (relied on, not owned by this feature): - - The existing freshness / log-completeness gate (config-tag and last-log-LSN - floor) remains the authority on whether a candidate could win a real vote. - - The existing heartbeat-liveness / primary-connection tracking remains the - signal for whether a peer still believes a leader is alive. - - The existing two-tier election transport and vote-collection mechanism - continue to deliver and tally PreVote and real-vote messages. - - Term restore-from-disk on crash+restart continues to suppress unnecessary - elections; this feature does not change that path. - -## Requirements - -### Requirement 1: PreVote precedes any persistent term increment - -**Objective:** As an XCE controller node, I want to win a hypothetical PreVote -majority before starting a real election, so that I never inflate my persistent -term unless a genuine leadership change is actually possible. - -#### Acceptance Criteria - -1. When a node's election trigger fires (liveness/heartbeat timeout or primary-connection RDHUP), the Election Module shall initiate a PreVote round before incrementing its persistent term. -2. While a PreVote round is in progress, the Election Module shall not modify the node's persistent term or recorded vote. -3. When a node requests PreVotes, the Election Module shall request them for the prospective next term (current term + 1) without persisting that prospective term. -4. If a node does not obtain a PreVote majority, then the Election Module shall not start a real election and shall not increment its persistent term. -5. When a node obtains a PreVote majority, the Election Module shall start a real election by incrementing its persistent term and casting its self-vote, following the existing real-election path. - -### Requirement 2: Voter-side PreVote eligibility reuses the real-vote gate - -**Objective:** As a voting controller, I want to grant a PreVote only to a -candidate I would actually elect, so that PreVote screens out candidates that -could never legitimately become leader. - -#### Acceptance Criteria - -1. When a voting controller receives a PreVote request, the voting controller shall grant it only if the candidate would also pass the existing real-vote freshness and log-completeness gate (config-tag and last-log-LSN floor). -2. If a PreVote candidate fails the freshness or log-completeness gate, then the voting controller shall deny the PreVote. -3. When a voting controller grants or denies a PreVote, the voting controller shall not record a vote, shall not change its voted-for state, and shall not change its persistent term. -4. When a voting controller has granted a PreVote to one candidate, the voting controller shall remain able to grant a real vote later in the same term to any eligible candidate, regardless of which candidate received the PreVote. - -### Requirement 3: Leader stickiness gate - -**Objective:** As a voting controller that still believes its leader is alive, I -want to deny PreVotes, so that a temporarily-unreachable peer cannot disrupt a -healthy primary. - -#### Acceptance Criteria - -1. While a voting controller still considers a current leader alive — it has received leader traffic recently enough that it would not itself start an election — the voting controller shall deny all PreVote requests, even when the candidate passes the freshness gate. -2. If a voting controller has gone long enough without leader traffic that it would itself be eligible to start an election, then the voting controller shall evaluate PreVote requests under the freshness / log-completeness gate alone. -3. While a controller is itself the current primary holding quorum, the controller shall deny all PreVote requests. - -### Requirement 4: Frozen-then-resumed rejoin is non-disruptive - -**Objective:** As an operator, I want a frozen-then-resumed standby to rejoin -without disrupting the cluster, so that transient unreachability is effectively a -no-op for availability. - -#### Acceptance Criteria - -1. When a previously-frozen standby resumes and initiates a PreVote round while a healthy primary still holds the cluster, the HA cluster shall deny it a PreVote majority so that it neither increments its term nor sends a real RequestVote. -2. When a non-disruptive rejoining standby next receives a primary heartbeat or ctrlhello, the standby shall re-attach as a standby under the existing primary without any intervening persistent-term change. -3. While a healthy primary holds quorum, the HA cluster shall not demote, truncate-and-converge, or self-restart the primary in response to a rejoining standby's election attempt. -4. When a frozen-then-resumed event occurs in an otherwise healthy 3-node cluster, the HA cluster shall keep the cluster's persistent term unchanged across the event and shall not enter a multi-term re-election cycle. - -### Requirement 5: Legitimate leadership changes still proceed - -**Objective:** As an XCE HA cluster, I want genuine leadership changes to -complete, so that PreVote suppresses only disruptive elections and never blocks -real failover. - -#### Acceptance Criteria - -1. When the primary genuinely fails and no peer has heard from a leader within its election-eligibility window, the HA cluster shall grant PreVotes to an eligible candidate so that a real election proceeds and a new primary is elected. -2. When a partition heals and a node observes a higher-term majority, the Election Module shall still adopt the higher term and step down according to the existing term-monotonicity rules. -3. The Election Module shall preserve the previously fixed failover behavior (bug-339) and rejoin behavior (bug-340) with PreVote enabled. -4. Where a real leadership change is warranted, the HA cluster shall complete the new election within the existing election-timeout bounds, introducing no additional unbounded delay from the PreVote round. - -### Requirement 6: Raft safety invariants preserved - -**Objective:** As an XCE HA cluster, I want every existing Raft safety invariant -preserved, so that adding PreVote introduces no correctness regression. - -#### Acceptance Criteria - -1. The Election Module shall preserve Election Safety (at most one leader per term) across all PreVote and real-vote paths. -2. The Election Module shall preserve term monotonicity, such that no PreVote causes any node's persistent term to decrease and no node's persistent term increases without a completed real election. -3. The HA cluster shall preserve Leader Completeness and a monotonic commit index across PreVote-gated elections. -4. The PreVote behavior shall remain consistent with the repository's TLA+ reference (`raft/raft.tla`) and the controller verification checklist for term monotonicity and Election Safety. - -### Requirement 7: PreVote transport compatibility and operator observability - -**Objective:** As an operator, I want PreVote activity to be diagnosable and to -not break existing election exchanges, so that I can distinguish a suppressed -disruptive election from a real failover. - -#### Acceptance Criteria - -1. The HA cluster shall carry PreVote requests and replies over the existing election transport without disrupting real RequestVote / RequestVoteReply exchanges. -2. When a controller denies a PreVote due to leader stickiness or the freshness gate, the HA cluster shall record the denial with the candidate identity and the denial reason at a diagnosable log level. -3. When a node abandons a campaign after failing to obtain a PreVote majority, the HA cluster shall record that outcome so operators can distinguish a suppressed disruptive election from a genuine failover. diff --git a/.kiro/specs/ha-prevote/research.md b/.kiro/specs/ha-prevote/research.md deleted file mode 100644 index ec450e6e..00000000 --- a/.kiro/specs/ha-prevote/research.md +++ /dev/null @@ -1,366 +0,0 @@ -# Gap Analysis: ha-prevote - -_Generated by `/kiro-validate-gap`. Informational — presents options and research items, not final design decisions._ - -## Analysis Summary - -- **The fix must be sender-side.** The disruptive step-down occurs in `raft_on_vote_request` Step 2 (raft.c:125-134) — term adoption + leader demotion — which runs **before** the `on_vote_eligible` gate (Step 5). No receiver gate today can save a healthy leader from a *fresh, higher-term* `RequestVote`, because the term is already adopted before any stickiness check. PreVote prevents the term inflation at the source. -- **Most receiver-side gating already exists.** `raft_cb_vote_eligible` (ha.c:217-245) already implements PRIMARY-deny, fresh-heartbeat-on-standby deny (leader stickiness), and catch-up deny; the dispatcher already enforces the freshness floor via `ha_vote_self_freshness`. PreVote evaluation can largely **reuse these gates** but must run with **zero state mutation**. -- **The new machinery is the candidate-side PreVote round** plus a wire message and a way to tally PreVote replies without touching `term`/`voted_for`. -- **Recommended shape: Hybrid (Option C).** New PreVote state in the raft library + new wire message/opcode, reusing the existing `ha_elect_ctx_t` collection, dispatcher freshness gate, and `raft_cb_vote_eligible` semantics. -- **Effort: M (3–7 days). Risk: Medium** — touches the safety-critical election path and the TLA+-referenced invariants, but integrates with well-factored, test-covered seams. - ---- - -## 1. Current State Investigation - -### Election path (raft library — `src/lib/raft/raft.c`, `raft.h`) - -- **State** (`raft_state_t`, raft.h:23-38): `term`, `role` (`RAFT_FOLLOWER|CANDIDATE|LEADER`), `voted_for`, `voted_for_term`, `votes_granted`, `replies_received`, `election_decided`, `election_timeout_ms`, `my_priority`. **No pre-vote fields.** -- **Callbacks** (`raft_callbacks_t`, raft.h:66-96): `on_send_vote_request(ctx, peer_idx, term)`, `on_vote_eligible(ctx, from_idx, term)`, `on_become_leader`, `on_become_follower`, `on_persist(ctx, term, voted_for)`, `on_schedule_timeout`, `on_cancel_timeout`, `on_start_election_after_delay`, `on_trace`. -- **`raft_start_election`** (raft.c:61-108): increments `term`, sets self-vote, **persists**, then sends `on_send_vote_request` to each peer, schedules timeout. Single-node short-circuit promotes immediately. -- **`raft_on_vote_request`** (raft.c:110-191): 6-step flow — (1) stale-term deny; (2) **term upgrade → adopt term, clear vote, step down, fire `on_become_follower` if was LEADER** (the disruptive path); (3) LEADER role deny; (4) one-vote-per-term; (5) `on_vote_eligible` policy gate; (6) persist-then-grant. -- **`raft_on_vote_reply`** (raft.c:193-252): term-upgrade demotion; quorum = `(n_peers/2)+1`; all-replied-no-quorum fallback. -- **Triggers**: `raft_on_leader_lost` → `on_start_election_after_delay` (priority jitter); `raft_check_and_start_election` (raft.c:282-294) skips if already voted for another, else `raft_start_election`. - -### Controller adapter (`src/controller/cluster.c`, `ha.c`, `ha.h`) - -- **Dispatch** (`cluster.c` `OPCode_requestVote`, lines 359-408): parses `request_vote_req_t`, computes freshness floor via `ha_vote_self_freshness` (max of local and learned-committed tag/LSN), applies `freshness_ok`, maps candidate name → index, then calls `raft_on_vote_request`. -- **Candidate send** (`ha.c` `on_vote_connected`, lines 2579-2610): builds `request_vote_req_t` (term, candidate_id, config_tag, last_log_lsn), sends via `xce_request` (or `xce_vote_request_hook` under `XCE_TESTING`). -- **Collection** (`ha_elect_ctx_t`, ha.h:75-116): flexible-array `votes[]`, `votes_granted` (init 1 = self), `n_replied`, `max_resp_tag/lsn` (yield gate), `done` semaphore. `handle_request_vote_reply` (ha.c:2473-2528) tallies and calls `ha_elect_evaluate_and_signal`. Quorum = `(n_peers+1+1)/2`. -- **Voter gate** (`raft_cb_vote_eligible`, ha.c:217-245): PRIMARY → deny; STANDBY + `ha_heartbeat_is_fresh` → deny; in catch-up → deny; else eligible (freshness already enforced by dispatcher). -- **Leader stickiness primitive** (`ha_heartbeat_is_fresh`, ha.c:2013-2022): `(now - last_activity) < heartbeat_timeout_ms`. `ha_on_heartbeat_received` updates `last_activity`. -- **Election initiation** (`ha_start_election_async`, ha.c:3341-3410): allocates `ha_elect_ctx_t`, sets self-vote, stores `ha->elect_ctx`, delegates to `raft_start_election`. -- **Election triggers**: `ha_on_primary_ctrl_fd_rdhup` (ha.c:3486) on connection loss; `liveness_poll_fn` (ha.c:3684-3722) on heartbeat timeout. - -### Wire schema (`src/schema/controller.capnp`) - -```capnp -struct RequestVoteRequest { # lines 796-801 - term @0; candidateId @1; configTag @2; lastLogLsn @3; -} -struct RequestVoteReply { # lines 803-808 - term @0; voteGranted @1; peerConfigTag @2; peerLastLogLsn @3; -} -``` -`OPCode` is an enum (lines 10-49); `requestVote @32` (line 43). **No PreVote opcode.** - -### Test seams - -- `xce_vote_request_hook` (ha.c:2560) intercepts the candidate send without live iomgr/SSL. -- `src/lib/testsuite/test_raft.c` — raft library unit tests with mock callbacks (TLA+ invariants). -- `src/controller/testsuite/`: `test_ha_election.c`, `test_ha_vote_gate.c`, `test_ha_start_election_async.c`, `test_ha_vote_send.c`, `test_ha_liveness.c`, `test_ha_role_lifecycle.c`, `test_ha_failover.c`. - -### Relevant constants (`ha.h:121-129`) - -`HEARTBEAT_INTERVAL_MS=3000`, `HEARTBEAT_TIMEOUT_MS=10000`, `ELECTION_TIMEOUT_MS=5000`, `ELECTION_JITTER_MAX_MS=200`, `ELECTION_PRIORITY_STEP_MS=500`. - ---- - -## 2. Requirement-to-Asset Map - -| Req | Capability needed | Existing asset | Gap | -|-----|-------------------|----------------|-----| -| **1** PreVote precedes term increment | A pre-election round that runs before `raft_start_election`; no persist/term change | `raft_start_election` (raft.c:61), `ha_start_election_async` (ha.c:3341) as the call site to gate | **Missing**: PreVote state + entry point; must interpose between trigger and `raft_start_election` | -| **1.2/1.3** No persistent mutation during PreVote | — | `on_persist` is only called by `raft_start_election`/`raft_on_vote_request` | **Constraint**: PreVote path must never call `on_persist` or mutate `term`/`voted_for` | -| **2** Voter eligibility reuses real-vote gate | Freshness + log-completeness check | Dispatcher freshness (`ha_vote_self_freshness`, cluster.c:389) + `raft_cb_vote_eligible` (ha.c:217) | **Constraint/Reuse**: run the same gate; must NOT record a vote | -| **2.3/2.4** No vote recorded; can still grant real vote later | — | one-vote-per-term lives in `raft_on_vote_request` Step 4 | **Constraint**: PreVote evaluation must bypass Steps 2/4/6 side effects | -| **3** Leader stickiness | Deny PreVote while leader believed alive; PRIMARY denies | `ha_heartbeat_is_fresh` (ha.c:2013), PRIMARY-deny in `raft_cb_vote_eligible` | **Reuse**: already present; just invoke from the PreVote receiver path | -| **3.1** Stickiness window semantics | "would not itself start an election" | Today: `heartbeat_timeout_ms` (10s) for freshness vs. `election_timeout_ms` (5s) probe | **Unknown (Research)**: which window defines stickiness — see RN-2 | -| **4** Frozen-then-resumed is non-disruptive | Candidate gets no PreVote majority → no term bump → re-attach | `ha_on_heartbeat_received` re-attach path | **Missing**: end-to-end behavior emerges from Req 1+3; needs integration test | -| **5** Legitimate changes still proceed | True death / partition heal still elect | `raft_on_vote_reply` term-upgrade demotion; existing failover (bug-339), rejoin (bug-340) | **Constraint**: do not regress; real-vote path unchanged after PreVote success | -| **6** Safety invariants | Election Safety, term monotonicity, Leader Completeness | `raft/raft.tla`, `test_raft.c`, controller verification checklist | **Constraint**: PreVote must be modeled/argued against TLA+; add invariant tests | -| **7.1** Transport compatibility | PreVote over existing election transport | capnp `OPCode`, `xce_request`, `peer_ctrl_msg_handler` | **Missing**: new opcode/flag + reply + dispatch case | -| **7.2/7.3** Observability | Log PreVote denials / abandoned campaigns | `on_trace` callback, `tlog` | **Missing**: trace/log points on PreVote deny + campaign abandonment | - -Gap tags: **Missing** = no asset; **Constraint** = existing behavior bounds the design; **Reuse** = asset directly applicable; **Unknown** = research item. - ---- - -## 3. Implementation Approach Options - -### Option A — Extend the existing RequestVote path with an `isPreVote` flag -Add `isPreVote @4 :Bool = false` to `RequestVoteRequest` (and a matching flag/struct for the reply). Reuse the same dispatch case, same `on_vote_connected` send, and branch inside `raft_on_vote_request` / `raft_cb_vote_eligible` on the flag to skip all state mutation. - -- ✅ Smallest wire change; backward-compatible capnp default; one RPC path. -- ✅ Reuses send/collect/gate machinery almost verbatim. -- ❌ Risks branching the safety-critical `raft_on_vote_request` with "if pre_vote skip steps 2/4/6" — easy to get a side-effect leak wrong. -- ❌ Overloads one function with two semantics (real vs. hypothetical). - -### Option B — New PreVote opcode + dedicated PreVote functions -Add `preVoteRequest`/`preVoteReply` opcodes and `PreVoteRequest`/`PreVoteReply` structs; add `raft_start_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply`, and `raft_cb_pre_vote_eligible`; a parallel (or shared) `ha_elect_ctx_t` pre-vote tally. - -- ✅ Clean separation; the pure (no-side-effect) PreVote functions are trivially auditable against the safety invariants. -- ✅ Real-vote path stays byte-for-byte unchanged (zero regression risk to bug-339/340). -- ❌ More new files/symbols and a second dispatch/collection path. -- ❌ Some duplication of freshness/eligibility logic unless carefully shared. - -### Option C — Hybrid (recommended) -New PreVote **wire message + raft state-machine functions** (Option B's clarity for the safety-critical core), but **reuse** the controller-side assets: the dispatcher freshness computation, `raft_cb_vote_eligible` semantics (called read-only), the `ha_elect_ctx_t` collection (add a `pre_votes_granted` counter or a `phase` field rather than a whole second struct), and `xce_vote_request_hook` for tests. - -- ✅ Keeps the pure PreVote decision isolated and auditable while avoiding duplicate collection/gate code. -- ✅ Real-vote path untouched; PreVote success simply calls the existing `raft_start_election`. -- ✅ Maps onto existing test seams (extend `test_ha_vote_gate.c`, add PreVote cases to `test_raft.c`). -- ❌ Requires a deliberate decision on shared-vs-separate eligibility helper to avoid drift. - ---- - -## 4. Effort & Risk - -| Dimension | Rating | Justification | -|-----------|--------|---------------| -| **Effort** | **M (3–7 days)** | New wire message + raft PreVote functions + candidate round + tests; but heavy reuse of existing gates, collection, and transport. | -| **Risk** | **Medium** | Touches the election safety core and TLA+-referenced invariants; mitigated by well-factored callback seams, existing gate logic to reuse, and strong test coverage. Highest-risk subtlety: ensuring the PreVote path has **zero** persistent side effects. | - ---- - -## 5. Recommendations for Design Phase - -**Preferred approach:** Option C (Hybrid). - -**Key decisions to finalize in design:** -1. **Wire representation** — separate `preVoteRequest`/`preVoteReply` opcodes (recommended for auditability) vs. `isPreVote` flag on the existing struct. Either is capnp-compatible; the opcode keeps the real-vote path provably untouched. -2. **Eligibility sharing** — factor the freshness + `raft_cb_vote_eligible` checks into a read-only helper callable from both the real-vote and PreVote receiver paths, so the two cannot drift. -3. **Collection reuse** — extend `ha_elect_ctx_t` with a PreVote phase/counter rather than a second struct. -4. **PreVote success → real election** — on PreVote majority, call the **unchanged** `raft_start_election`; confirm the candidate re-checks leader-stickiness was still absent at the moment of starting the real round. -5. **Defense-in-depth question** — decide whether to *also* add leader-stickiness gating to the real `raft_on_vote_request` term-upgrade path (Raft "reject RequestVote within min election timeout"), since PreVote alone leaves Step 2 able to step down a leader if a stray higher-term real RequestVote ever arrives. - -**Research Needed (carry to design):** -- **RN-1**: Model/argue the PreVote phase against `raft/raft.tla` and the controller verification checklist; confirm Election Safety + term monotonicity hold with the new path. (Req 6) -- **RN-2**: ✅ **RESOLVED — reuse `heartbeat_timeout_ms` via `ha_heartbeat_is_fresh()`.** See "Resolved Decisions → RN-2" below for the full comparison. (Req 3.1) -- **RN-3**: Confirm whether the PRIMARY should deny PreVotes via the existing `raft_cb_vote_eligible` PRIMARY-deny (Req 3.3) and that this composes with CheckQuorum-style behavior already present. -- **RN-4**: Decide PreVote round timeout/abandonment behavior and its interaction with the existing election jitter/priority and `election_timeout_ms` bounds (Req 5.4) — ensure no added unbounded delay. - -**Explicitly out of scope:** Backward/rolling-upgrade compatibility with peers that predate PreVote. The product is **pre-release**, so all controllers in a cluster deploy from the same build — there is no mixed-version (old-binary ↔ new-binary) cluster to support. The wire change (new opcode or struct field) may be made without a negotiation/fallback path. Req 7.1's "without disrupting real RequestVote/RequestVoteReply exchanges" refers to PreVote coexisting with the *concurrent same-version* real-vote path, not to inter-version compatibility. - ---- - -## 6. Resolved Decisions - -### RN-2 — Leader-stickiness window → reuse `heartbeat_timeout_ms` (Opt 1) - -**Decision:** The voter's leader-stickiness predicate is the existing `ha_heartbeat_is_fresh()` (`now − last_activity < heartbeat_timeout_ms`, currently 10s), gated on `role == CTRL_STANDBY`. PreVote introduces **no new timing concept**. - -**The governing invariant.** Stickiness is a single voter-side predicate: *"do I still believe a leader is alive?"* It must be **pinned to the failure-detection window** — the period during which the voter would not itself start an election. If the two diverge: -- **stickiness < failure-detection** → a gap opens where the voter grants a PreVote while still believing its leader is alive and not yet failing over → **reopens the disruption hole**. -- **stickiness > failure-detection** → a voter that has already decided the leader is dead (and is campaigning) still denies others' PreVotes → mutual denial → **PreVote majorities can't form → liveness stall**. - -So the window must equal whichever constant is XCE's *failure detector*. - -**Naming trap.** Textbook Raft overloads one "election timeout" for two roles; XCE separates them, so the dissertation's "reject votes within the minimum election timeout" maps to `HEARTBEAT_TIMEOUT_MS`, **not** `ELECTION_TIMEOUT_MS`: - -| Raft concept | XCE constant | Value | Role in XCE | -|---|---|---|---| -| election timeout *as leader-failure detector* | `HEARTBEAT_TIMEOUT_MS` | 10000 | Follower decides primary is dead → triggers failover (`liveness_poll_fn` / RDHUP) | -| election timeout *as vote-collection deadline* | `ELECTION_TIMEOUT_MS` | 5000 | Candidate waits this long for vote replies before giving up | -| heartbeat interval | `HEARTBEAT_INTERVAL_MS` | 3000 | Primary re-sends heartbeat every 3s | - -**Option comparison.** - -| | Opt 1: `heartbeat_timeout_ms` (10s) | Opt 2: `election_timeout_ms` (5s) | Opt 3: new `prevote_stickiness_ms` | Opt 4: `heartbeat_interval_ms` (3s) | -|---|---|---|---|---| -| Matches XCE failure-detector | ✅ exactly | ❌ wrong constant (vote-collection deadline) | ⚠️ only if set == 10s | ❌ far shorter | -| Consistent w/ existing real-vote gate (`raft_cb_vote_eligible`) | ✅ same predicate reused | ❌ real-vote 10s vs PreVote 5s drift | ⚠️ drift unless tuned | ❌ drift | -| Disruption gap (grant while leader alive) | ✅ none | ❌ [5s,10s] gap on flaky/asymmetric link | ⚠️ gap if < 10s | ❌ [3s,10s] gap | -| Tolerance to transient heartbeat loss | ✅ ~3 missed heartbeats | ⚠️ ~1 missed heartbeat | ⚠️ depends | ❌ 0 missed | -| Liveness-stall risk | ✅ none | ✅ none | ⚠️ stall if > 10s | ✅ none | -| Speeds up legitimate failover | n/a (matches today) | ❌ no real benefit¹ | ⚠️ only by also lowering failover timeout | ❌ no | -| New config surface | none | none | +1 tunable | none | - -¹ Under symmetric silent death nobody campaigns until the 10s failure-detection fires, so a shorter stickiness window has no candidate to grant to in [5s,10s] — zero speedup. A real crash closes sockets → RDHUP → standbys leave `CTRL_STANDBY` and campaign immediately, independent of the window. The shorter window's only observable effect is the disruption gap on a flaky link (a healthy primary missing ~2 of 3 heartbeats to one voter would get a PreVote granted against it at ~6s). - -**Why Opt 1.** It is the single value where the stickiness window, the failure-detection trigger (`liveness_poll_fn`), and the existing real-vote stickiness gate (`raft_cb_vote_eligible`) all key off **one predicate** (`ha_heartbeat_is_fresh`). Gating on `role == CTRL_STANDBY` also handles connection loss for free: RDHUP moves the node out of `CTRL_STANDBY`, so a genuinely disconnected voter stops being sticky immediately instead of waiting the full 10s. This is exactly what Req 3.1's observable phrasing ("received leader traffic recently enough that it would not itself start an election") already describes. If failover latency is later reduced by lowering `HEARTBEAT_TIMEOUT_MS`, stickiness follows automatically — which is itself the argument against Opt 3's separable constant. - -**Requirement impact:** Req 3.1 can be tightened in design (Boundary Commitment) to name `ha_heartbeat_is_fresh` / `heartbeat_timeout_ms` as the binding, while keeping the requirement text observable. - ---- - -## 7. Design Synthesis Outcomes (design phase) - -Applied the generalization / build-vs-adopt / simplification lenses before writing `design.md`. - -### Generalization — share interfaces, not implementations -PreVote and real-vote are two variations of "solicit a vote decision." Two seams are shared rather than duplicated: -- **Voter eligibility** — `raft_cb_vote_eligible` already encodes exactly the PreVote gates (PRIMARY-deny + fresh-heartbeat stickiness + catch-up-deny). The PreVote receiver reuses it verbatim via the existing `on_vote_eligible` callback; the only PreVote-specific rule is "decide without adopting the term." -- **Candidate tally** — `ha_elect_ctx_t` is reused with a `phase` discriminator (`HA_ELECT_PREVOTE` / `HA_ELECT_REAL`) instead of a second collection struct. Same quorum formula `(n_peers + 2) / 2`, same `votes[]` idempotency. - -The raft state-machine *functions* stay distinct (`raft_on_pre_vote_*` vs `raft_on_vote_*`) so the pure, no-side-effect PreVote path is independently auditable against the TLA+ model. - -### Build vs. Adopt — adopt -- **Algorithm**: Raft dissertation §9.6 PreVote (standard, well-specified). Not invented. -- **Infrastructure**: capnp election transport, iomgr/delaymgr, `ha_elect_ctx_t`, `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, `xce_vote_request_hook`. No new libraries. - -### Simplification decisions -- **Wire**: new `preVoteRequest`/`preVoteReply` **opcodes that reuse the existing `RequestVoteRequest`/`RequestVoteReply` structs and union arms** — no new capnp structs, and the real-vote handler stays byte-for-byte unchanged (zero regression risk to bug-339/340). Chosen over an `isPreVote` flag on the existing struct (which would branch the safety-critical real-vote handler). -- **Defer defense-in-depth** (the RN-4-adjacent / open "real-vote stickiness guard"): PreVote guarantees a candidate only sends a real RequestVote *after* a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the real-vote term-adoption path for no in-scope benefit. **Deferred**, documented in `design.md` → Architecture → Deferred decisions. (Resolves the open defense-in-depth question; RN-4 reduces to "bound the PreVote round with `election_timeout_ms` and re-arm liveness on loss," which the design specifies.) -- **No new role state**: the candidate stays `CTRL_STANDBY` during the PreVote round (tracked by `ctx.phase`), so a lost PreVote needs no rollback. The destructive `truncate-WAL + CTRL_ELECTING` step is **moved** from the failover trigger to PreVote success — the key change that makes a transient freeze a true no-op (Req 4). - -### Remaining open research item -- **RN-1**: TLA+ / verification-checklist argument that Election Safety + term monotonicity hold with the PreVote path (the real-vote path is unchanged, so this is largely an argument that the pure PreVote functions mutate nothing). Carried into implementation as `test_raft.c` invariant assertions. RN-3 (PRIMARY-deny composition) is resolved by reusing `raft_cb_vote_eligible`. - ---- - -## RN-1 resolved (task 6.1) - -**Decision/outcome:** The PreVote phase is consistent with the repository's -`raft/raft.tla` reference and the controller verification checklist — -Election Safety (≤1 leader per term), Leader Completeness, term monotonicity, -and a monotonic commit index all continue to hold with PreVote enabled. - -**Argument (purity ⇒ no new transitions).** - -1. **The PreVote functions are pure w.r.t. persistent Raft state.** Across an - arbitrary sequence of `raft_begin_pre_vote` / `raft_on_pre_vote_request` / - `raft_on_pre_vote_reply` calls — mixed grant+deny, duplicate replies, - out-of-range peers, and both WON and LOST terminal outcomes — none of them - ever writes `s->term`, `s->voted_for`, `s->voted_for_term`, or `s->role`, - and none invokes `on_persist`. Mechanical evidence: - `test_invariant_prevote_sequence_pure_no_persist` and - `test_invariant_prevote_lost_sequence_pure_no_persist` in - `src/lib/testsuite/test_raft.c`. - -2. **Term monotonicity is preserved.** Because the PreVote path writes no term, - the only writers of `s->term` remain the UNCHANGED `raft_start_election` - (the real, gated term bump) and `raft_on_vote_request` Step 2 (adopt a - strictly higher term). PreVote can only *gate* whether a real election - later runs; it never advances a term itself, and it never decreases one. - -3. **Election Safety is preserved.** Because the PreVote path creates no leader - (no `on_become_leader`, no `role := RAFT_LEADER`), it adds no new way to - satisfy the antecedent of Election Safety. A PreVote round reaching - `RAFT_PRE_VOTE_WON` returns a *status* only; the node stays `RAFT_FOLLOWER` - until the separate, unchanged `raft_start_election` / `raft_on_vote_reply` - path runs. Evidence: `test_invariant_prevote_won_grants_no_leadership`. - -4. **The bug-339/bug-340 failover path is unchanged.** The legitimate - higher-term-adopt + leader-step-down behavior lives entirely in - `raft_on_vote_request` / `raft_on_vote_reply`, which this feature does not - touch. PreVote sits strictly before that path and (by 1) leaves its inputs - untouched. Evidence: `test_invariant_real_vote_higher_term_adopt_unchanged` - demonstrates the real-vote path still adopts a higher term, fires - `on_become_follower` to step a leader down, and elects at quorum via - `on_become_leader` — even after a full PreVote round has run on the node. - -**Conclusion.** PreVote introduces no transition that writes a term or creates -a leader; every term-writing / leadership-creating transition in the model is -byte-for-byte the pre-PreVote transition. The set of reachable -`(term, leader)` configurations is therefore unchanged, so the `raft/raft.tla` -invariants (Election Safety, Leader Completeness, term monotonicity, monotonic -commit index) and the controller verification checklist continue to hold. The -four `test_invariant_*` tests are the mechanical evidence for premises (1)–(4), -and the existing failover/rejoin regression suites remain green with PreVote -enabled (controller suite 563 pass / 0 fail after task 4.2). Requirements -covered: 5.2, 5.3, 6.1, 6.2, 6.3, 6.4. - ---- - -## Gap Re-Audit (2026-06-18) - -**Method.** Verified every EARS acceptance criterion in `requirements.md` against -the current on-disk source via codegraph + grep. tasks.md shows phase -`tasks-generated` with 6 unchecked `[ ]` and 12 checked `[x]`; this audit -classifies each `[ ]` and tags each requirement MET/Missing/Unknown/Constraint. - -### STATUS: CLOSED — requirements met 7/7, all 12 leaf tasks implemented + tested - -Every production seam the design names exists, is wired into the real failover -path, and is exercised by tests. The 6 unchecked boxes are **group-header -artifacts**, not work. - -### Open-task classification (the 6 unchecked `[ ]`) - -All 6 unchecked boxes are the **phase/group-header lines** (task numbers `1`–`6`, -the section titles). Every one of their leaf sub-tasks is checked `[x]`: - -| `[ ]` header | Children (all `[x]`) | Classification | -|---|---|---| -| 1. Foundation: wire schema + raft scaffolding | 1.1, 1.2 | group-header-artifact | -| 2. Core: raft-library pure decision + tally | 2.1, 2.2, 2.3 | group-header-artifact | -| 3. Core: controller voter-side dispatch | 3.1 | group-header-artifact | -| 4. Core: candidate orchestration + trigger restructure | 4.1, 4.2 | group-header-artifact | -| 5. Integration: transport wiring + E2E | 5.1, 5.2, 5.3 | group-header-artifact | -| 6. Validation: safety invariants + regression guard | 6.1 | group-header-artifact | - -**Genuine-gap: 0. Deferred: 0. Artifact: 6.** A group header is complete by -definition once all its children are done; these were never intended as -separately-checkable work items. No effort/risk applies (nothing to build). - -### Requirement-to-Asset map - -| Req | Criterion (summary) | Asset (file:line) | Tag | -|---|---|---|---| -| 1.1 | PreVote before term increment on trigger | `ha_begin_failover` ha.c:4606; `ha_start_prevote_async` ha.c:4193; triggers `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 / `liveness_poll_fn` (calls rdhup) ha.c:~4789 | MET | -| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote` raft.c:232 (writes only `pre_vote_*` tally; no term/voted_for/role/persist) | MET | -| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` raft.c:243 (`prospective_term = s->term + 1`, local only) | MET | -| 1.4 | No majority → no real election | `pre_vote_reply_dispatch` ha.c:4296 LOST branch (stay STANDBY, re-arm) | MET | -| 1.5 | Majority → start real election | `pre_vote_reply_dispatch` WON → `ha_enter_real_election` ha.c:4103 → unchanged `ha_start_election_async`/`raft_start_election` | MET | -| 2.1 | Grant only if real-vote freshness gate passes | cluster.c:480 `ha_vote_self_freshness` floor + cluster.c:512 `raft_on_pre_vote_request` | MET | -| 2.2 | Fail freshness → deny | cluster.c:482-506 `freshness_ok` → `deny_reason="freshness"` | MET | -| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` raft.c:193 (pure); reply carries unchanged term cluster.c:500/556 | MET | -| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` writes no `voted_for`; asserted `test_genuine_failover...` test_ha_prevote.c:709 | MET | -| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` ha.c:299 (STANDBY + fresh heartbeat + primary_ctrl_fd>=0 → deny), reused by PreVote | MET | -| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` falls through to grant once heartbeat stale; `test_genuine_failover_stale_peer_grants_and_elects` | MET | -| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` ha.c:307 (CTRL_PRIMARY → 0) | MET | -| 4.1 | Frozen-resumed gets no majority | scenario test `test_ha_prevote.c` (Angle B deny) + cluster.c deny path | MET | -| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` ha.c:4886-4909 (cancel round mid-flight, stay STANDBY); `test_heartbeat_mid_round_cancels_prevote` test_ha_failover_prevote.c:198 | MET | -| 4.3 | Healthy primary not demoted/restarted | trigger defers truncate/CTRL_ELECTING to WON only (ha.c:4655-4662) | MET | -| 4.4 | Term unchanged, no re-election cycle | candidate stays CTRL_STANDBY through round; round state in `raft_state_t` only | MET | -| 5.1 | True death → PreVotes granted → elect | `test_genuine_failover_stale_peer_grants_and_elects` test_ha_prevote.c:680 | MET | -| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` raft.c:346 (unchanged real-vote path) | Constraint (unchanged, preserved) | -| 5.3 | No regression of bug-339/bug-340 | real-vote path byte-for-byte unchanged; `test_invariant_real_vote_higher_term_adopt_unchanged` (RN-1) | MET | -| 5.4 | Complete within election-timeout bounds | round timeout `election_timeout_ms` ha.c:4258; rdhup disarms liveness first | MET | -| 6.1 | Election Safety preserved | PreVote fires no `on_become_leader`; RN-1 + `test_invariant_prevote_won_grants_no_leadership` | MET | -| 6.2 | Term monotonicity; PreVote never changes term | RN-1 + `test_invariant_prevote_sequence_pure_no_persist` | MET | -| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | Constraint (unchanged, preserved) | -| 6.4 | Consistent with raft.tla + checklist | RN-1 argument (research.md §RN-1) + invariant tests | MET | -| 7.1 | PreVote over existing transport, no real-vote disruption | new opcodes `preVoteRequest @38`/`preVoteReply @39` controller.capnp:49-50, reuse vote structs; reply routed via `peer_ctrl_msg_handler` ha.c:3610 (pre_vote_in_progress gate) → `pre_vote_reply_decode_and_dispatch` ha.c:3340 | MET | -| 7.2 | Log denials w/ candidate + reason | cluster.c:546-552 `INFO("HA: PreVote DENIED candidate=... reason=...")` (gate-order-faithful reason) | MET | -| 7.3 | Log abandoned campaign distinctly | `pre_vote_reply_dispatch` LOST branch ha.c:4313 INFO; round-timeout fn | MET | - -**Tally: MET 25, Constraint (preserved-unchanged) 2 (5.2, 6.3), Missing 0, Unknown 0.** - -### Notable implementation facts (beyond the design baseline) - -- **Reply routing** is NOT a dedicated `OPCode_preVoteReply` switch case in - cluster.c (as task 5.1 literally phrased it). Instead `peer_ctrl_msg_handler` - (ha.c:3600) routes by role+state: `CTRL_STANDBY && pre_vote_in_progress` → - `pre_vote_reply_decode_and_dispatch` → `pre_vote_reply_dispatch`. The request - side does emit/reply over the dedicated `OPCode_preVoteReply` opcode - (cluster.c:564). Functionally equivalent to the design intent; the dispatch - discriminator is round-state rather than a third reply opcode case. -- **Two extra robustness mechanisms** were added beyond the spec, both benign - and in-scope-adjacent: (a) a **per-node stagger** (`ha_election_stagger_ms` - → `ha_failover_stagger_fn`, ha.c:4586) before starting PreVote to avoid - lock-step split votes (bug-606); (b) a retained **config-less-node guard** - (tag==0 && n_peers>1 → voter-only, ha.c:4693) that short-circuits before the - round (ha-config-bootstrap Req 2.8). -- **Test coverage** spans three files: `src/lib/testsuite/test_raft.c` (pure - decision/tally + 4 invariant tests), `src/controller/testsuite/test_ha_prevote.c` - (voter decide + frozen-rejoin + genuine-failover scenarios), - `src/controller/testsuite/test_ha_failover_prevote.c` (trigger rewire: - rdhup/liveness start PreVote not election, heartbeat-mid-round cancel, - WON→real-election, config-less-node no-PreVote). Per Implementation Notes - (task 4.2): controller suite 563 PASS / 0 FAIL with PreVote live. - -### Carry-forward (NOT a ha-prevote gap) - -The deliberate per-round `elect_ctx` ASAN leak (`ha_start_prevote_async` -allocates a metadata `ha_elect_ctx_t` that is intentionally not freed because -live vote endpoints may hold a bare pointer — same UAF caution as the rdhup -path) is pre-existing, bounded (~one per round), and explicitly carried into a -separate follow-up spec `.kiro/specs/ha-elect-ctx-lifecycle/`. It is out of -scope for ha-prevote and does not affect any ha-prevote requirement. - -### Options for the (zero) genuine remaining work - -- **Option A (recommended):** Mark ha-prevote CLOSED. Optionally check the 6 - group-header boxes for bookkeeping cleanliness (cosmetic; all children done). - **Effort: S. Risk: none.** -- **Option B:** Run `/kiro-verify-completion ha-prevote` to mechanically confirm - every acceptance criterion against the suite before flipping spec.json phase - to a completed state. **Effort: S. Risk: low** (known `test_commit_barrier_reply` - 300s TIMEOUT in the full controller suite — use the targeted run and check for - zero `:FAIL`, per Implementation Note 3.1). -- **Option C:** No action; treat the unchecked headers as a tasks.md rendering - convention. **Effort: none. Risk: none** (the only downside is `[ ]` headers - reading as "incomplete" to a casual `/kiro-spec-status` glance). diff --git a/.kiro/specs/ha-prevote/spec.json b/.kiro/specs/ha-prevote/spec.json deleted file mode 100644 index c0d8957c..00000000 --- a/.kiro/specs/ha-prevote/spec.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "feature_name": "ha-prevote", - "created_at": "2026-06-15T23:45:43Z", - "updated_at": "2026-07-18T09:55:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation": { - "completed": true, - "tasks_done": "1.1, 1.2, 2.1, 2.2, 2.3, 3.1, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1 (all 12 leaf tasks; the 6 unchecked tasks.md group-header lines are bookkeeping artifacts, not work -- see research.md's 2026-06-18 Gap Re-Audit)", - "notes": "PreVote phase (Raft dissertation Sec 9.6) added ahead of the failover trigger: raft_begin_pre_vote/raft_on_pre_vote_request/raft_on_pre_vote_reply (raft.c, pure -- no term/voted_for/role mutation, no persist) plus ha_begin_failover/ha_start_prevote_async/pre_vote_reply_dispatch/ha_enter_real_election (ha.c) orchestrating the round and deferring truncate+CTRL_ELECTING to PreVote success. Original implementation landed 2026-06-16 (commits 0d66026..e6b234a, controller suite 563/0 clean); research.md's own RN-1 TLA+/verification-checklist argument and 2026-06-18 gap re-audit closed all 25 MET + 2 preserved-unchanged criteria with zero gaps. Substantial hardening landed on top since: bug-605 (dispatch-gate wiring), bug-606 (per-node failover stagger to avoid lock-step split vote), bug-660 (dead-primary fresh-heartbeat convergence), bug-1421/1424 (PreVote-reply registration on connect/monitor failure), bug-1430 (UAF CAS fix in ha_prevote_round_timeout_fn), bug-1435 (raft_lock serialization), task 5.7 jittered [T,2T] election-timeout backoff, and ha-elect-ctx-lifecycle's reference-counted elect_ctx carrier (commit 6529dcf, replacing the deliberate per-round leak this spec's tasks.md Implementation Notes had carried forward as a known, deferred limitation -- now resolved by that sibling spec, itself pending its own validate-impl pass)." - }, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T09:55:00Z", - "note": "Cross-task/feature-level validation of a spec whose 18/18 tasks.md checkboxes (12 leaf + 6 group-header) had been complete since 2026-06-16, but whose spec.json phase was never advanced past tasks-generated (same administrative-oversight pattern as several sibling specs validated earlier today: ha-wal-replication, ha-heartbeat-committed-propagation). Ran immediately after raft-drive-async (NO-GO, bug-1447 ASAN SEGV in src/lib/netio.c SSL handshake, round 9/10 of its own sequential soak) and after ha-wal-replication / ha-heartbeat-committed-propagation (both GO, clean 10/10 soaks) -- extra weight was given to reproducing or ruling out bug-1447's signature in this spec's own live soak, especially since round 9 is the exact round bug-1447 hit.\n\nCROSS-TASK CONSISTENCY / DESIGN ALIGNMENT (codegraph-verified against design.md's Architecture, Boundary Commitments, and Requirements Traceability table, plus research.md's own 2026-06-18 gap re-audit which independently mapped all 27 acceptance criteria to file:line evidence): confirmed live on-disk source for raft_begin_pre_vote/raft_on_pre_vote_request/raft_on_pre_vote_reply (src/lib/raft/raft.c) and ha_begin_failover/ha_start_prevote_async/pre_vote_reply_dispatch/ha_enter_real_election (src/controller/ha.c) all still present with the designed pure/side-effect-free shape (zero term/voted_for/role mutation, zero on_persist calls in the three raft-library functions -- verified by direct source read, not just test titles). ha_dispatch_gate (ha.c:3794) admits OPCode_preVoteRequest/preVoteReply in CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY exactly as bug-605's fix requires. Confirmed the dependency direction (capnp/types -> raft library (pure) -> controller adapter -> dispatch/transport) still holds with no upward imports. One legitimate, expected architectural evolution beyond the original design.md snapshot: raft_on_pre_vote_request's signature grew cand_last_log_term/cand_last_log_index parameters and now runs the (lastLogTerm,lastLogIndex) raft_log_ok freshness gate INSIDE the raft library itself, rather than the dispatcher applying freshness before the call as design.md originally specified -- this is the G6 vote-freshness closure from the later raft-commit-safety spec (own boundary, own validation), layered onto this spec's foundation; functionally strictly stronger, not a regression, and does not change any of this spec's own requirements or invariants. research.md documents one further non-blocking, functionally-equivalent routing drift (PreVote replies route via peer_ctrl_msg_handler's role+pre_vote_in_progress state check rather than a dedicated cluster.c OPCode_preVoteReply switch case as task 5.1 literally phrased it) -- already flagged in that file, not re-litigated here.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) -- zero compiler/linker errors, binaries confirmed with session-fresh mtimes. Both authoritative standalone binaries run directly with ASAN_OPTIONS=detect_leaks=0 (per this project's convention, authoritative over meson's aggregate line): xcelib_test (raft library) 293 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- includes 24 dedicated pre_vote-tagged RUN_TEST entries in test_raft.c (decision/tally purity, quorum WON/LOST/PENDING, duplicate-reply idempotency, out-of-range-peer guard, 5-node quorum, and 4 explicit invariant assertions incl. test_invariant_prevote_sequence_pure_no_persist and test_invariant_prevote_won_grants_no_leadership). controller_unit_test 972 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- includes 28 dedicated PreVote-tagged tests across test_ha_prevote.c (12: start/no-mutation, majority-grant, no-majority-rearm, round-timeout-as-loss, reusable-after-loss, pending-noop, reply-over-real-transport, frozen-rejoin no-op x2, genuine-failover, heartbeat-mid-round-cancel, lost-then-later-elects), test_ha_failover_prevote.c (6: trigger rewire rdhup/liveness->PreVote not election, heartbeat-cancel, WON->real-election, config-less-node guard), test_ha_vote_gate.c (8: primary-role-denies, standby-fresh-heartbeat-denies, catch-up-grants-when-primary-unreachable, catch-up-denies-while-primary-live, stale-freshness-denies, stale-term-denies-no-adoption, granted-when-stickiness-lapsed, dead-primary-grants-despite-fresh-activity -- this last one is bug-660's own regression test), test_ha_role.c (1: dispatch-gate admits PreVote in voter roles), and test_ha_elect_ctx_lifecycle.c (1: 20 PreVote rounds x 2 votes with no carrier leak -- the sibling ha-elect-ctx-lifecycle spec's own regression test, confirming its refcounted-carrier fix holds under repeated PreVote churn). Zero :FAIL anywhere in either log, zero ASAN/crash signatures (AddressSanitizer, heap-buffer-overflow, use-after-free, double-free, SEGV) in either log. TBD/TODO/FIXME/HACK/XXX grep on the PreVote-boundary files (raft.c, raft.h, ha.c, ha.h, cluster.c, controller.capnp, test_ha_prevote.c, test_ha_failover_prevote.c, test_raft.c, test_ha_vote_gate.c): zero PreVote-related matches. Secrets grep: clean.\n\nRUNTIME LIVENESS (the blocking check, given the bug-1447 sibling finding and this spec's own live-election subject matter -- PreVote specifically governs election behavior under partition/unreachability): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir (~/tmp/xce-ha-prevote-soak, confirmed ext4 via df -T before the run, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, zero walcompare divergence at any round boundary, all 3 nodes (ctrl-1, ctrl-2, ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied by round 4). Term progressed 2->13 over 10 genuine kills -- climbing by exactly +1 per round in 9 of 10 rounds (clean single-jump elections, no runaway multi-round election-storm churn, exactly the disruption-suppression behavior this spec exists to produce) and +2 in exactly one round (round 9, term 10->12), coinciding with the log's single live PreVote DENIED event (candidate=ctrl-2 term=8 reason=freshness) -- a legitimate freshness-gate denial of a behind candidate, not a fault. Live PreVote activity confirmed directly from the per-node logs (not just inferred from job-loss absence): 13 PreVote round WON events across the three nodes' logs (6+5+2), 1 explicit PreVote DENIED (reason=freshness) log line carrying candidate id + reason exactly as Req 7.2 requires, and 2 'primary heartbeat arrived mid-PreVote -- cancelling the round' events (Req 4.2's mid-round-cancel path) -- meaning this run incidentally live-exercised the frozen-then-resumed re-attach path in addition to the genuine-failover path the sequential kill schedule directly drives, without needing a dedicated SIGSTOP-based scenario (design.md's Testing Strategy already covers the frozen-then-resumed case at the controller integration-test level via test_ha_prevote.c/test_ha_failover_prevote.c, run above; it does not call for a live SIGSTOP soak scenario, and none exists in run-ha-soak.sh's mode set -- confirmed by inspection). Explicitly grepped the full soak driver log AND all three per-node controller logs (11268/12012/11515 lines) for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker): zero matches anywhere, including at round 9 -- the exact round bug-1447 hit in raft-drive-async's own soak. Also grepped broadly for any other crash/abort signature (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches. This does not prove bug-1447 is fixed or globally absent (it is a rare, timing-dependent race in raft-drive-async's own reconnect-offload cadence, a different subsystem this spec's boundary does not touch) -- but it does establish that this spec's own PreVote-gated election path runs clean through a full 10-round supported failover lifecycle with live PreVote activity on every round, satisfying this spec's own regression contract (Req 5.4, 4.x).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record, bug-1447 in buglog.json); noted here only as a risk actively checked for and ruled out in this run, not adopted as this spec's blocker. No new bugs found or fixed during this validation pass -- the tree's PreVote-adjacent bug history (605, 606, 660, 1421, 1424, 1430, 1435) was reviewed and every fix confirmed present and correct in the current source (e.g. bug-1430's CAS-based UAF fix in ha_prevote_round_timeout_fn read directly from ha.c:7216-7240).", - "known_limitations": [ - "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered by raft-drive-async's own reconnect-offload cadence. Did not reproduce in this spec's clean 10/10 run (including at round 9, the exact round it hit for raft-drive-async), but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-prevote; tracked entirely under raft-drive-async's remediation.", - "The deliberate per-round ha_elect_ctx_t carrier leak that this spec's own tasks.md Implementation Notes (task 4.2) had explicitly deferred to a follow-up spec is now resolved in the current tree by ha-elect-ctx-lifecycle (commit 6529dcf, reference-counted carrier via atomic_exchange + release-on-last-ref in ha_start_prevote_async) -- confirmed present in the current ha.c source and covered by a live-passing regression test (test_many_prevote_rounds_no_carrier_leak, 20 PreVote rounds x 2 votes, no leak). That sibling spec's own spec.json is still phase=tasks-generated pending its own validate-impl pass (scheduled to run after this one in the same worktree); this note records that ha-prevote's own known-limitation is already satisfied in-tree regardless of that sibling's administrative validation status.", - "design.md's original Service Interface signature for raft_on_pre_vote_request (2 params: from_idx, cand_term) is now stale documentation -- the shipped function additionally takes cand_last_log_term/cand_last_log_index and runs the (lastLogTerm,lastLogIndex) freshness gate inside the raft library itself (the later raft-commit-safety spec's G6 vote-freshness closure, layered onto this spec's foundation). Functionally strictly stronger than the original design, not a regression; a future doc pass could update design.md's Service Interface block to match the shipped signature.", - "research.md's own file already documents one non-blocking, functionally-equivalent routing drift: PreVote replies are routed via peer_ctrl_msg_handler's role+pre_vote_in_progress state check rather than a dedicated cluster.c OPCode_preVoteReply switch case as task 5.1's text literally phrased it. Not re-verified independently in this pass beyond confirming the request-side opcode (preVoteReply @39) is still emitted and the reply still reaches the tally live (confirmed via the soak's 13 live PreVote round WON events)." - ] - } -} diff --git a/.kiro/specs/ha-prevote/tasks.md b/.kiro/specs/ha-prevote/tasks.md deleted file mode 100644 index fa4f978b..00000000 --- a/.kiro/specs/ha-prevote/tasks.md +++ /dev/null @@ -1,100 +0,0 @@ -# Implementation Plan - -> TDD is mandatory (project rule): for each implementation sub-task, add the failing test in the relevant `testsuite/` first (RED), then implement to green. Dependency direction: capnp/types → raft library → controller adapter → dispatch/transport. The real-vote state machine is reused unchanged after PreVote success. - -- [x] 1. Foundation: wire schema and raft-library PreVote scaffolding -- [x] 1.1 Add PreVote opcodes to the election wire protocol - - Add two opcodes for the PreVote request and PreVote reply, reusing the existing vote request/reply payload structures and their message union arms (no new payload structs). - - Regenerate the generated bindings and confirm the build produces the new opcode constants while existing vote opcode ordinals are unchanged. - - _Requirements: 7.1_ - - _Boundary: PreVote dispatch (wire schema)_ - -- [x] 1.2 (P) Add PreVote types, state, and callback seam to the raft library - - Add PreVote outcome codes (won / lost / pending), a self-contained PreVote tally on the raft state (granted count, per-peer replied tracking, in-progress flag), and an optional send-pre-vote callback in the callback table. - - Header compiles with the new declarations and existing election state/layout is unchanged; no behavior added yet. - - _Requirements: 1.2, 6.2_ - - _Boundary: raft pre-vote functions_ - -- [x] 2. Core: raft-library pure PreVote decision and tally (test-first, no side effects) -- [x] 2.1 Implement the voter-side PreVote decision - - RED first: add raft unit tests asserting grant for a fresh, higher-or-equal-term candidate when the eligibility callback allows it, and deny on stale term or when the eligibility callback rejects. - - Reuse the existing vote-eligibility callback (primary-deny, fresh-heartbeat stickiness, catch-up deny) and let the dispatcher apply freshness beforehand, exactly as the real-vote path does. - - Observable: after a granted PreVote the test confirms term, voted-for, and role are all unchanged and no persist callback fired. - - _Requirements: 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 6.2_ - - _Boundary: raft pre-vote functions_ - -- [x] 2.2 Implement the candidate-side PreVote start - - RED first: add a raft unit test asserting the prospective term used is current+1, that it is never written to the persistent term and never persisted, and that the send-pre-vote callback fires once per peer with a self-grant recorded. - - Observable: test confirms the persistent term/voted-for are byte-for-byte unchanged after starting the round. - - _Requirements: 1.1, 1.2, 1.3, 6.2_ - - _Boundary: raft pre-vote functions_ - - _Depends: 1.2_ - -- [x] 2.3 Implement the PreVote reply tally - - RED first: add raft unit tests asserting the tally lives on the raft state, returns won at quorum, lost when all peers reply without quorum, pending otherwise, and ignores duplicate replies from the same peer. - - Mirror the production real-vote tally pattern, but never fire become-leader/become-follower — return a status the caller acts on. - - Observable: test confirms role and term unchanged across won/lost/pending outcomes. - - _Requirements: 1.4, 1.5, 6.2_ - - _Boundary: raft pre-vote functions_ - - _Depends: 1.2_ - -- [x] 3. Core: controller voter-side PreVote dispatch (test-first) -- [x] 3.1 Answer incoming PreVote requests with stickiness + freshness, no mutation - - RED first: extend the vote-gate tests so a PreVote from a fresh candidate is denied while a leader is believed alive (primary role, or standby with fresh heartbeat, or catch-up) and granted once stickiness lapses and freshness passes. - - Compute the freshness floor and run the eligibility gate before the raft decision, reusing the real-vote dispatch shape; reply with the voter's unchanged term plus the grant bit and freshness fields; log every denial with candidate id and reason. - - Observable: a denied PreVote reply carries the voter's original term (no term adoption) and a diagnosable denial-reason log line is emitted. - - _Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 7.1, 7.2_ - - _Boundary: PreVote dispatch_ - - _Depends: 1.1, 2.1_ - -- [x] 4. Core: controller candidate-side orchestration and trigger restructuring (test-first) -- [x] 4.1 (P) Run the PreVote round and branch on its outcome - - RED first: add controller tests driving the reply handler through the existing vote-reply hook (no real transport): majority grant → real election started; no majority → node stays standby with term unchanged and the liveness watchdog re-armed; round timeout treated as a loss. - - Owns the candidate orchestration in the controller: start the round (no persist, no role change), arm a round timeout bounded by the existing election-probe deadline, and the PreVote-reply handler that feeds the raft tally and branches — on a win perform the existing truncate + electing-transition + start-real-election sequence (unchanged real-election path); on loss log campaign abandonment. The send path emits the PreVote request opcode from task 1.1; transport-side reply routing is wired separately in task 5.1. - - Observable: on no-majority the cluster term is unchanged, role stays standby, and a subsequent genuine failure still triggers a new round. - - _Requirements: 1.1, 1.4, 1.5, 5.4, 7.3_ - - _Boundary: PreVote orchestration_ - - _Depends: 1.1, 2.2, 2.3_ (parallel-safe with 3.1: orchestration lives in a different file than the voter-side dispatch) - -- [x] 4.2 Route failover triggers through PreVote and defer the destructive step - - RED first: add tests asserting the failover trigger no longer truncates the WAL or increments the term until PreVote wins, and that an incoming primary heartbeat mid-round cancels the round and re-attaches as standby. - - Make both failover entry points (connection RDHUP and heartbeat-timeout poll) begin the PreVote round instead of starting an election directly; move truncate + electing-transition to PreVote success; cancel an in-progress round on heartbeat receipt. - - Observable: a frozen-then-resumed standby that loses PreVote leaves the WAL and term untouched and re-attaches when the next heartbeat arrives. - - _Requirements: 1.1, 4.2, 4.3, 4.4_ - - _Boundary: PreVote orchestration (trigger path)_ - - _Depends: 4.1_ - -- [x] 5. Integration: transport wiring and end-to-end scenarios -- [x] 5.1 Wire the PreVote reply opcode case to the candidate handler - - Add the transport dispatch case for the incoming PreVote-reply opcode so it routes to the reply handler built in 4.1 (replacing the test hook in production), paralleling the real-vote reply dispatch case. - - Observable: a PreVote reply arriving over the real transport reaches the candidate's tally and can drive the round to a decision end-to-end (not just via the test hook). - - _Requirements: 7.1_ - - _Boundary: PreVote dispatch_ - - _Depends: 1.1, 3.1, 4.1_ (shares the dispatch file with 3.1; owns only the opcode-to-handler routing, while 4.1 owns the handler) - -- [x] 5.2 Frozen-then-resumed no-op scenario test - - Add a 3-node scenario test: primary alive, a resumed standby starts a PreVote round; both peers deny (primary role + fresh heartbeat). - - Observable: no real vote request is sent, the cluster term is unchanged across the event, and the primary stays primary (no demotion/truncate/self-restart). - - _Requirements: 4.1, 4.3, 4.4_ - - _Depends: 3.1, 4.1, 4.2, 5.1_ - -- [x] 5.3 Genuine-failover and re-attach scenario tests - - Add scenario tests: dead primary with stale peer heartbeats → PreVotes granted → a new primary is elected within the existing election-timeout bounds; heartbeat mid-round → clean re-attach; failed PreVote → liveness watchdog re-armed so a later real failure still elects. - - Observable: a genuine primary death elects a new primary, and the suppressed cases never do. - - _Requirements: 5.1, 5.4, 4.2_ - - _Depends: 4.1, 4.2, 5.1_ - -- [x] 6. Validation: safety invariants and regression guard -- [x] 6.1 Assert Raft safety invariants hold with PreVote - - Add raft-library invariant assertions that the PreVote functions mutate no persistent state (Election Safety and term monotonicity preserved), and record the TLA+/verification-checklist consistency argument (the real-vote path is unchanged, so the higher-term-adopt failover and the bug-339/bug-340 fixes are preserved). - - Observable: invariant tests pass and the existing failover/rejoin regression tests remain green with PreVote enabled. - - _Requirements: 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_ - - _Depends: 2.1, 2.2, 2.3_ - -## Implementation Notes -- (1.1) Wire opcodes `preVoteRequest @38` / `preVoteReply @39` reuse existing `RequestVoteRequest`/`RequestVoteReply` structs and union arms (`requestvotemsg @27` / `requestvotereplymsg @14`). Generated bindings live under `src/xce.build/lib/controller.capnp.h` (build artifacts, not source-committed). -- (1.2) Raft PreVote scaffolding in `raft.h`: outcome codes `RAFT_PRE_VOTE_WON/LOST/PENDING` = 7/8/9; tally fields `pre_vote_in_progress`, `pre_votes_granted`, `pre_vote_replies[RAFT_MAX_PEERS]` appended to `raft_state_t` (after `my_priority`); optional `on_send_pre_vote_request` callback (NULL-checkable). New `#define RAFT_MAX_PEERS 8` bounds the per-peer dedup array — safe: production cap is `MAX_HA_CANDIDATES=4` (src/include/config.h:78), and it matches `RAFT_SIM_MAX_NODES=8`. Tasks 2.1/2.2/2.3 supply the function bodies and MUST keep them pure (no `term`/`voted_for`/`role` mutation, no `on_persist`). Build/test the raft layer with `cd src && nix develop --command bash tools/xce-build.sh lib`. -- (3.1) IMPORTANT for controller tasks: `cluster.c` is NOT linked into the unit-test binary (`controller_unit_test_src` in `src/controller/testsuite/meson.build`) — it's daemon-only (`controller_src`). So dispatch cases in cluster.c can't be unit-tested directly; test their composed primitives + contract asserts, and defer end-to-end to scenario tasks 5.2/5.3. `ha.c` IS unit-test-linked (the `test_ha_*` suites exercise it). Controller build/test: `cd src && nix develop --command bash tools/xce-build.sh controller`; the registered meson test name is `"controller test"` (single Unity binary). Known pre-existing `test_commit_barrier_reply` 300s TIMEOUT makes the full suite exit non-zero — validate via the targeted run `nix develop --command meson test -C xce.build --no-rebuild --print-errorlogs "controller test"` and check for zero `:FAIL` (Timeout:1 is the known one). The new prevote opcodes' generated constants are `OPCode_preVoteRequest=38` / `OPCode_preVoteReply=39`. -- (3.1, LOW nit) cluster.c PreVote denial LOG reason uses elimination order (stale_term→primary→catch_up→fresh_heartbeat) which differs from `raft_cb_vote_eligible`'s gate order (primary→fresh_heartbeat→catch_up); in the rare STANDBY catch-up+fresh-heartbeat overlap the log string may say `catch_up` while the gate denied on `fresh_heartbeat`. Decision is identical (deny); log-string only. RESOLVED (commit 47f7338): exposed pure `ha_heartbeat_is_fresh` and rewrote the elimination to mirror the gate's exact order (stale_term → primary → fresh_heartbeat → catch_up). -- (4.1) ha.c now has `ha_start_prevote_async`, `pre_vote_reply_dispatch`, `ha_enter_real_election` (WON branch = the factored truncate+CTRL_ELECTING+ha_start_election_async helper), a `raft_cb_send_pre_vote_request` callback (registered in `s_real_cbs`), and a round-timeout task. KEY for task 5.1: `pre_vote_reply_dispatch(ha, peer_idx, granted)` takes the ABSOLUTE cluster peer index (matches `raft_begin_pre_vote`'s send loop), NOT the real-vote local `votes[]` index. `pre_vote_reply_dispatch` is `__attribute__((used))` with NO production caller yet — task 5.1's cluster.c `OPCode_preVoteReply` case must call it (replacing the `xce_*` test seam in production). Triggers (rdhup/liveness/heartbeat) are still UNMODIFIED — task 4.2 rewires them to call `ha_start_prevote_async`. -- (4.2) Triggers now rewired: `ha_on_primary_ctrl_fd_rdhup` + `liveness_poll_fn` → `ha_begin_failover` → `ha_start_prevote_async` (PreVote first; truncate+CTRL_ELECTING deferred to `ha_enter_real_election` on WON). `ha_on_heartbeat_received` cancels an in-progress round (only when `pre_vote_in_progress`) + re-attaches CTRL_STANDBY, no term change. LOAD-BEARING: `ha_start_prevote_async` checks `raft_begin_pre_vote`'s return and on immediate `RAFT_PRE_VOTE_WON` (single-node, n_peers==1) synchronously calls `ha_enter_real_election` — without this a single-node/immediate-majority node never elects. Full controller suite 563 PASS/0 FAIL with PreVote live. (NOTE: pre-existing deliberate `elect_ctx` ASAN leak ~one-per-round carried into PreVote; out of scope. Tracked as follow-up spec `.kiro/specs/ha-elect-ctx-lifecycle/` — pre-existing + bounded + UAF-sensitive carrier-lifetime fix, deferred by decision.) diff --git a/.kiro/specs/ha-repl-hardening/design.md b/.kiro/specs/ha-repl-hardening/design.md deleted file mode 100644 index df490e7a..00000000 --- a/.kiro/specs/ha-repl-hardening/design.md +++ /dev/null @@ -1,468 +0,0 @@ -# Design: HA Replication Hardening - -## Overview - -**Purpose**: This feature hardens the XCE controller replication path so that the primary maintains majority-consistent state as a continuous invariant throughout its lifecycle — from post-election bootstrap through every state-changing operation until failover. - -**Users**: Cluster operators who require CP semantics (consistency over availability) — no acknowledged state change is ever lost on primary failover. - -**Impact**: Replaces the current timeout-only COMMITTING path with a unified heartbeat-driven replication protocol. The standby is fully passive — it only reports its current `(config_tag, applied_lsn)` in heartbeat replies. The primary makes all replication decisions (config push, snapshot, WAL streaming) from that single data point. The same code path handles fresh standby bootstrap, catch-up after disconnection, and continuous state verification. - -### Goals - -- Unify replication into a single heartbeat-driven decision loop on the primary -- Standby remains fully passive — replies with state, receives what the primary sends -- Primary establishes majority consistency before serving first client request -- Primary continuously verifies majority consistency for every state-changing operation -- Lagging standbys receive snapshot + batch WAL catch-up to converge -- `handle_ctrlhello` reduces to connection setup only - -### Non-Goals - -- Config push comparison (already implemented — pushed only when tags differ) -- WAL segment retention policy changes -- Dynamic cluster membership changes -- Client-visible API changes -- Election protocol changes - -## Majority Consistency Invariant - -> **Any state change acknowledged to a client has been durably committed on a majority of controllers.** - -Enforced at three stages: - -| Stage | Mechanism | When | -|-------|-----------|------| -| **Bootstrap** | Heartbeat reply convergence check *(Note: superseded by `ha-single-leader-role`, closed 2026-06-11 — `CTRL_PRIMARY_COMMITTING` and the convergence-gated bootstrap were deleted; the elected node now enters `CTRL_PRIMARY` directly and write-confirmation is gated by a `WAL_NOOP` own-term commit through the commit engine, not by standby convergence tracking. See that spec's design.md.)* | After winning election, before serving client traffic | -| **Ongoing WAL** | `repl_wait_majority_ack` per entry (existing) | On every job submit, state transition, queue change | -| **Ongoing Config** | `push_config_to_standbys` + majority ack (existing) | On every config upgrade | - -## Boundary Commitments - -### This Spec Owns - -- Heartbeat reply protocol: standby reports `(applied_lsn, config_tag)` to primary -- Unified replication decision loop: primary reads heartbeat state → decides config/snapshot/WAL -- Per-standby convergence tracking in `ha_state_t` -- WAL catch-up entry application on standby side -- Batch WAL entry encoding and transfer -- Snapshot creation and transfer -- COMMITTING→serving transition gated on majority convergence *(Note: superseded by `ha-single-leader-role` — the COMMITTING role and this gate were removed entirely; see note under Majority Consistency Invariant above)* - -### Out of Boundary - -- Election protocol (`election-iomgr-refactor`) -- WAL write path and `repl_wait_majority_ack` (existing in `ha-config-bootstrap`) -- Config push path and config tag comparison (existing — reused by decision loop) -- Heartbeat send scheduling (existing `ha_start_primary_heartbeat`) -- Connection lifecycle (existing iomgr; `handle_ctrlhello` reduces to setup only) - -### Allowed Dependencies - -- `primary_heartbeat_fn` — existing heartbeat send loop; extended with reply processing -- `apply_wal_entry()` — existing WAL entry replay -- `persist_checkpoint()` — existing snapshot creation -- `repl_wait_majority_ack()` — existing quorum ack for ongoing operations -- `push_config_to_standbys()` — existing config push (called from decision loop when tag differs) - -### Revalidation Triggers - -- Changes to `RequestReply` union schema in `controller.capnp` -- Changes to heartbeat interval or send mechanism -- Changes to WAL segment format or `apply_wal_entry` signature -- Changes to `standby_conn_t` or replicator tracking - -## Architecture - -### Unified Heartbeat-Driven Protocol - -The standby is fully passive. The primary drives everything: - -``` -PRIMARY (heartbeat loop, fires every heartbeat_interval_ms) - -For each standby in standby_endpoints: - - 1. Send OPCode_nop (or skip if recent traffic was sent) - 2. Receive reply → (standby_tag, standby_lsn) - 3. Decision: - - if standby_tag < primary_committed_tag: - → push_config_to_standby(standby) - return // re-evaluate next heartbeat - - if standby_lsn < oldest_wal_lsn: - → persist_checkpoint() → send OPCode_snapshotTransfer - return - - if standby_lsn < primary_committed_lsn: - → stream_wal_entries(from=standby_lsn+1, to=committed_lsn) - return // re-evaluate next heartbeat after catch-up - - if standby_tag >= primary_tag AND standby_lsn >= committed_lsn: - → mark converged ✓ - - 4. After loop: if majority converged and role == COMMITTING: - → transition to serving state -``` - -*(Note: step 4 and the COMMITTING role it references were removed by `ha-single-leader-role` (closed 2026-06-11) — see note under Majority Consistency Invariant above. Steps 1-3 (per-standby config/snapshot/WAL decision) remain accurate and are unaffected.)* - -**Key insight**: The same decision tree handles all standby states — fresh bootstrap (tag=0, lsn=0), catch-up after disconnect (tag=current, lsn=stale), and converged (tag=current, lsn=current). No separate code paths needed. - -### handle_ctrlhello — Reduced to Connection Setup - -Before this change, `handle_ctrlhello` (then at config_repl.c:1359, now config_repl.c:2128 post-implementation) orchestrated replication: -- Registered standby in replication set -- Checked snapshot need -- Streamed WAL entries -- Pushed config - -After this change, it only does connection bookkeeping: -1. Set `primary_ctrl_fd = ep->fd` (first ctrlhello from winner) -2. Call `ha_on_heartbeat_received` to reset liveness watchdog -3. Arm `ha_arm_heartbeat_timeout_vtask` - -All replication decisions move to the heartbeat reply handler. The next heartbeat cycle will pick up the new standby and drive replication. - -### Architecture Diagram - -```mermaid -sequenceDiagram - participant P as Primary - participant S as Standby - - Note over P,S: --- Connection Setup --- - S->>P: OPCode_hello (applied_lsn=0, config_tag=0) - P-->>S: CtrlHelloReply (committed_lsn=10000, config_tag=T1) - P->>P: handle_ctrlhello: set primary_ctrl_fd, arm liveness - - Note over P,S: --- First Heartbeat Cycle --- - P->>S: OPCode_nop (heartbeat) - S-->>P: reply {applied_lsn=0, config_tag=0} - P->>P: tag=0 < T1 → push config - P->>S: OPCode_replConfigPush {config blob, tag=T1} - - Note over P,S: --- Second Heartbeat Cycle --- - P->>S: OPCode_nop - S-->>P: reply {applied_lsn=0, config_tag=T1} - P->>P: lsn=0 < 10000 → stream WAL [1..10000] - P->>S: OPCode_walCatchupEntry (batch: 1..256) - P->>S: OPCode_walCatchupEntry (batch: 257..512) - P->>S: ... (more batches) - - Note over P,S: --- Third Heartbeat Cycle --- - P->>S: OPCode_nop - S-->>P: reply {applied_lsn=10000, config_tag=T1} - P->>P: tag=T1 ✓ AND lsn=10000 ✓ → converged! - P->>P: majority reached → CTRL_PRIMARY -``` - -### Key Decisions - -| Decision | Rationale | -|----------|-----------| -| Primary drives all replication from heartbeat reply | Single decision point; no separate orchestrator code paths | -| Standby only reports state, never requests replication | Eliminates standby→primary request messages; standby is purely passive | -| Config/snapshot/WAL decided from `(tag, lsn)` alone | Two scalars encode all replication state needed | -| handle_ctrlhello reduces to connection setup | Clean separation: connection lifecycle vs replication logic | -| One action per heartbeat cycle per standby | Avoids flooding a standby with config + WAL simultaneously; natural pacing. `ha_stream_wal_to_standby()` (named `stream_wal_entries()` in this design; actual as-built name differs) enqueues all batch messages via `xce_request()` (non-blocking) in a single invocation — "one action" means no further decisions for this standby until the next heartbeat reply confirms the new LSN | - -## File Structure Plan - -### Modified Files - -| File | Change | Responsibility | -|------|--------|---------------| -| `src/schema/controller.capnp` | Add `stateReport :group {appliedLsn, configTag}` to `RequestReply.data` | Wire protocol | -| `src/controller/ha.h` | Add `standby_reported_tag[N]`, `standby_reported_lsn[N]`, `standby_converged[N]` | Per-standby tracking | -| `src/controller/ha.c` | Add `ha_process_standby_state()` — the unified decision loop. Wire `OPCode_walCatchupEntry` handler. Process heartbeat replies. | Core replication logic | -| `src/controller/cluster.c` | Extend `OPCode_nop` handler: populate `stateReport` with `applied_lsn`/`config_tag` | Standby state reporting | -| `src/controller/config_repl.c` | Strip `handle_ctrlhello` to connection setup only. Add batch WAL streaming. Implement snapshot transfer. | Replication actions | -| `src/controller/repl.c` | Minor: add helper to read convergence state from heartbeat tracking | Quorum helper | - -### No New Files - -## Components and Interfaces - -### Component: Unified Replication Decision Loop - -| Field | Detail | -|-------|--------| -| Intent | On each heartbeat reply, read standby state, decide next replication action | -| Requirements | 1.1–1.4, 2.1–2.5, 6.1–6.5 | -| Location | `ha.c` — called from heartbeat reply processing | - -**Responsibilities & Constraints**: -- Read `(standby_tag, standby_lsn)` from heartbeat reply -- Compare against `primary_committed_tag` and `primary_committed_lsn` -- Decide: push config, send snapshot, stream WAL, or mark converged -- One action per heartbeat cycle per standby — config first, then WAL, then convergence - -**Contract**: -``` -Decision(standby_tag, standby_lsn) → Action - -Action ∈ { - PUSH_CONFIG, // standby_tag < committed_tag - SEND_SNAPSHOT, // standby_lsn < oldest_wal_lsn - STREAM_WAL, // standby_lsn < committed_lsn - CONVERGED, // tag >= committed AND lsn >= committed - WAIT // action in progress, re-evaluate next cycle -} -``` - -**Invariants**: -- Config is pushed before WAL (tag must be current before LSN comparison is meaningful) -- Snapshot takes priority over WAL streaming (full state transfer needed) -- Convergence is terminal for this cycle — standby won't be re-evaluated until next heartbeat - -### Component: Heartbeat State Reporter (Standby Side) - -| Field | Detail | -|-------|--------| -| Intent | Populate `stateReport` in every `OPCode_nop` reply | -| Requirements | 1.1, 1.3, 1.4 | -| Location | `cluster.c` OPCode_nop handler | - -**Cap'n Proto Schema Change** (as built — a named struct referenced from the union, not an inline group as originally sketched here): -```capnp -# In RequestReply.data union: -stateReportMsg @15: StateReport; - -struct StateReport { - appliedLsn @0 :UInt64; # standby's last applied WAL LSN - configTag @1 :UInt64; # standby's current config version tag - term @2 :UInt64; # standby's current raft term (added post-close by bug-1421, - # AE-reject term-discovery — not part of this design) -} -``` - -**Implementation**: In the `OPCode_nop` handler, `data_which` is set to `RequestReply_data_stateReportMsg`, populated from `ha->applied_lsn`, `myconf->config_version_tag`, and (post-close addition) `ha->term`. Function: `ha_nop_reply_state_report()` in `ha.c`. - -### Component: Per-Standby State Tracking - -| Field | Detail | -|-------|--------| -| Intent | Store each standby's last reported state for convergence evaluation | -| Requirements | 2.1–2.5 | -| Location | `ha.h` ha_state_t | - -**Fields**: -```c -/* MAX_HA_CANDIDATES is 4 (not 8), defined in src/include/config.h — not ha.h */ -uint64_t standby_reported_tag[MAX_HA_CANDIDATES]; -uint64_t standby_reported_lsn[MAX_HA_CANDIDATES]; -bool standby_converged[MAX_HA_CANDIDATES]; -``` - -- Indexed by peer index (0..N-1), matching `--peers` order -- Zero-initialized by `ha_init` -- Updated atomically from IO worker context on each heartbeat reply -- Read by convergence evaluator and committing timeout fn - -### Component: WAL Catch-Up Handler (Standby Side) - -| Field | Detail | -|-------|--------| -| Intent | Apply WAL catch-up entries from primary | -| Requirements | 3.1–3.5 | -| Location | `ha.c` or `cluster.c` dispatch switch | - -Wires the current stub (`OPCode_walCatchupEntry` returns 0) to `apply_wal_entry()`. For batch entries, applies each in order, advances `applied_lsn` after the full batch is durably written. Error entries are logged and skipped. - -*(Note: `ha-term-safety` (closed 2026-07-17) materially extended the `WalCatchupEntry` wire message beyond this description — it now carries a full Raft AppendEntries-conformant envelope (`term`, `prevLogIndex`, `prevLogTerm`, `leaderCommit`, `leaderCurrentTerm`, and a real `entries: List(WalCatchupEntry)` for batching) so the standby intake runs through `raft_on_append_entries_request`'s logOk/conflict-truncation gate rather than blind sequential apply. See that spec's design.md and bug-1067/bug-1421.)* - -**Durability**: `applied_lsn` is not persisted to disk. On controller restart the standby always reports `lsn=0`; the primary drives full re-bootstrap (config push → WAL streaming → converged) via the unified decision loop. This is correct and expected — the decision loop handles `lsn=0` identically to a fresh standby. - -### Component: Snapshot Transfer - -| Field | Detail | -|-------|--------| -| Intent | Transfer full state snapshot when standby cannot catch up via WAL | -| Requirements | 4.1–4.5 | -| Location | `config_repl.c`, `persist.c` | - -#### When Snapshot Is Needed - -The unified decision loop checks two conditions on each heartbeat reply: - -1. **WAL gap** (existing `repl_needs_snapshot`): `standby_lsn < oldest_wal_lsn` — the standby's position predates the oldest retained WAL segment. WAL entries needed for catch-up no longer exist. - -2. **Excessive WAL backlog** (heuristic): `committed_lsn - standby_lsn > wal_streaming_threshold` (configurable, default 10M entries). Even within the WAL window, streaming millions of entries would take too long — a snapshot is faster. - - *(Note: unverified against current code — `wal_streaming_threshold` and this backlog-size heuristic do not exist anywhere in the source tree. `ha_evaluate_repl_action()` (ha.c) implements only condition 1 above (`standby_lsn + 1 < oldest_wal_lsn`); condition 2 was never built. Not attributable to any later spec — appears to have simply not been implemented.)* - -**Design principle**: Prefer WAL retention over snapshot. WAL entries are incremental (O(changes per operation)); a snapshot is O(total state size). Operators should configure `wal_retention_hours` large enough that snapshots are rare. Snapshot transfer is the fallback for long-disconnected standbys. - -#### Transfer Strategy: Chunked + Resumable - -A full snapshot may be hundreds of MB in a large cluster. Single-message transfer would block the control channel and be unrecoverable on failure. - -``` -Primary Standby - │ │ - │ persist_checkpoint() → snap.bin │ - │ OPCode_snapshotTransfer │ - │ {chunk=0, offset=0, │ - │ total=524288000, data=[64KB]} │ - │ ─────────────────────────────────→ │ write to snap.part - │ │ - │ OPCode_snapshotTransfer │ - │ {chunk=1, offset=65536, ...} │ - │ ─────────────────────────────────→ │ append to snap.part - │ ... (8000 chunks for 500MB) ... │ - │ │ - │ OPCode_snapshotTransfer │ - │ {chunk=N, offset=..., │ - │ data=[final], isLast=true} │ - │ ─────────────────────────────────→ │ verify size, atomic rename - │ │ → load snapshot - │ │ → applied_lsn = checkpoint_lsn - │ ←── heartbeat reply ────────────── │ reports new lsn -``` - -**Chunk format** (per `OPCode_snapshotTransfer` message, as built — `struct SnapshotTransfer` in `controller.capnp`): -``` -snapshot_lsn :UInt64 — LSN this snapshot corresponds to (design.md named it checkpoint_lsn) -payload :Data — chunk payload (64KB max) (design.md named it data) -chunk_index :UInt32 — 0-based chunk number -chunk_offset :UInt64 — byte offset in the full snapshot -total_size :UInt64 — total snapshot size in bytes -is_last :Bool — true for the final chunk -checkpoint_term :UInt64 — (Note: field added by `raft-ha-core-safety`, closed 2026-07-09, - not part of this design — the leader's term for snapshot_lsn, - reconciled into the receiver's per-LSN term index after install - so the snapshot boundary establishes a verifiable (term, lsn) pair) -``` - -**Failure handling**: Mid-transfer reconnect always restarts snapshot from chunk 0 — the standby discards its partial `snap.part` file on reconnect and the primary re-creates the checkpoint from scratch. Resumption is intentionally not implemented: snapshots are rare (WAL retention is the primary path), and the extra protocol state (`checkpoint_lsn`, `last_chunk_index`) in `stateReport` would complicate the common heartbeat path for an edge case. The existing reconnect model (standby re-sends ctrlhello, primary drives next heartbeat cycle) naturally handles this without schema changes. Req 4.4's "retry by reconnecting" is satisfied by this restart-on-reconnect approach. - -**Completion**: When all chunks are received, the standby verifies `sum(chunk_sizes) == total_size`, atomically renames `snap.part` → `snap-{checkpoint_lsn}.bin`, and loads it. The next heartbeat reply reports `applied_lsn = checkpoint_lsn`. `applied_lsn` is in-memory only — it is not persisted to disk. On a subsequent restart the standby reports `lsn=0` and the primary re-runs the full bootstrap sequence. - -#### Future: Block-Based Differential (Deferred) - -For very large clusters where even chunked full transfer is slow, a block-based differential approach can be added later: - -1. Primary divides snapshot into fixed-size blocks (e.g., 4KB), computes SHA256 of each -2. Standby computes SHA256 of its current state using same block alignment -3. Primary sends only blocks whose SHA256 differs from the standby's -4. Standby merges received blocks into its state - -This is deferred — the chunked approach handles the common case, and WAL retention minimizes how often snapshots are needed. - -### Component: Batch WAL Sender - -| Field | Detail | -|-------|--------| -| Intent | Transfer WAL entries in batches during catch-up | -| Requirements | 5.1–5.5 | -| Location | `config_repl.c` | - -Accumulates entries up to batch limit (64KB/256 entries), sends as single message. Standby applies all entries in batch, advances `applied_lsn` after batch complete. - -`ha_stream_wal_to_standby()` (this design's `stream_wal_entries()`) enqueues all batch `OPCode_walCatchupEntry` messages via `xce_request()` (non-blocking send-buffer enqueue) in a single handler invocation — it never blocks the vtaskmgr thread. "One action per heartbeat cycle" means the primary makes no further replication decisions for this standby until the next heartbeat reply arrives confirming the standby's updated `applied_lsn`. - -*(Note: as built, each `WalCatchupEntry` message also carries the Raft AppendEntries envelope (`term`, `prevLogIndex`, `prevLogTerm`, `leaderCommit`, `leaderCurrentTerm`) added by `ha-term-safety` (closed 2026-07-17) — see the WAL Catch-Up Handler component above.)* - -## System Flows - -### Flow 0: Post-Election — Determine Safe Commit Point (Critical Safety) - -**Optimization**: The winning candidate already has each voter's `peerLastLogLsn` and `peerConfigTag` from the `RequestVoteReply` messages (stored in `ha_elect_ctx_t.votes[]`). This eliminates the need to wait for the first heartbeat round — the safe commit point is computed immediately upon winning. - -``` -raft_cb_become_leader(): - 1. Collect (peerLastLogLsn, peerConfigTag) from votes[i] for all voters - 2. safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority) - 3. safe_config_tag = max(tag : count of voters with peerConfigTag >= tag) + 1 >= majority) - 4. persist_truncate_wal_above(safe_committed_lsn) - 5. committed_lsn = safe_committed_lsn -``` - -*(Note: step 4 is stale — bug-1060 (fixed 2026-06-24, closely tied to `raft-commit-safety`'s Leader Completeness work, closed 2026-06-25) proved a bare `safe_committed_lsn` truncation can cut a new leader's own durable committed entries when a slower peer sits in the granting majority. As built, `raft_cb_become_leader` (ha.c:1615) truncates to `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor) = max(safe_committed_lsn, own durable tip)` via `ha_compute_safe_committed_lsn()` (ha.c:1144) — a new leader never truncates its own durable log. See `src/controller/CLAUDE.md` Leader Completeness section.)* - -**Raft TLA+ alignment**: Raft's `BecomeLeader` resets `matchIndex[j] = 0` for all followers and determines `commitIndex` via subsequent `AppendEntries` round-trips. Our optimization uses the vote replies (which carry the Raft-equivalent of each voter's lastLogIndex) to compute the safe commit point with zero additional round-trips. This is safe because any committed entry was on a majority and at least one voter in the winning quorum has it. - -### Flow 1: Full Standby Bootstrap (config_tag=0, applied_lsn=0) - -**Heartbeat Cycle 1**: standby reports `(tag=0, lsn=0)` → primary pushes config -**Heartbeat Cycle 2**: standby reports `(tag=T, lsn=0)` → primary streams WAL [1..N] -**Heartbeat Cycle 3+**: standby reports `(tag=T, lsn=N)` → converged - -Each cycle is one heartbeat interval (3s). For a fresh cluster with empty WAL, bootstrap completes in 2 cycles (~6s). For a large cluster, WAL streaming spans multiple cycles. - -### Flow 2: Standby Rejoin After Brief Disconnect - -Standby reports `(tag=T, lsn=5000)`, primary has `committed_lsn=10000`. Standby is within WAL window → primary streams WAL [5001..10000]. Config already current → no config push. One or more heartbeat cycles depending on WAL volume. - -### Flow 3: Standby Rejoin After Long Disconnect (Snapshot) - -Standby reports `(tag=T, lsn=100)`, primary's `oldest_wal_lsn=5000`. Standby predates WAL window → primary creates checkpoint at lsn=10000, sends snapshot. Standby loads snapshot, reports `lsn=10000` next heartbeat → converged. - -## Requirements Traceability - -| Req | Design Element | -|-----|---------------| -| 1.1–1.4 | `cluster.c` OPCode_nop handler: populate `stateReport`; capnp schema change | -| 2.1–2.5 | `ha.c`: unified decision loop + per-standby tracking arrays | -| 3.1–3.5 | `ha.c`/`cluster.c`: wire `OPCode_walCatchupEntry` → `apply_wal_entry` | -| 4.1–4.5 | `config_repl.c`: snapshot creation + `OPCode_snapshotTransfer` | -| 5.1–5.5 | `config_repl.c`: batch accumulation in WAL streaming loop | -| 6.1–6.5 | `ha.c`: heartbeat-driven majority check; committing timeout as 60s fallback *(Note: COMMITTING and its 60s timeout were removed by `ha-single-leader-role` — see note under Majority Consistency Invariant)* | -| 7.1–7.5 | INFO/WARN log statements throughout new code paths | - -## Data Models - -### Cap'n Proto Schema Change - -```capnp -# In RequestReply.data union, add: -stateReport :group { - appliedLsn @15 :UInt64; - configTag @16 :UInt64; -} -``` - -### In-Memory State - -``` -ha_state_t -├── standby_reported_tag[N] uint64_t per-standby last reported config tag -├── standby_reported_lsn[N] uint64_t per-standby last reported applied LSN -└── standby_converged[N] bool derived: tag>=committed && lsn>=committed -``` - -## Error Handling - -| Error | Handling | -|-------|----------| -| WAL entry apply fails | Log ERROR, skip entry, continue; standby reports partial progress next heartbeat | -| Snapshot transfer fails | Standby logs ERROR, reconnects; primary retries next time standby appears in heartbeat loop | -| Heartbeat reply lost | Primary re-sends next cycle (3s); no single-reply dependency | -| Majority lost (standbys disconnect) | Primary continues serving; logs WARN; `repl_wait_majority_ack` returns timeout for new writes | -| Committing timeout fires (60s) | Primary enters degraded mode; logs WARN with per-standby convergence state *(Note: removed by `ha-single-leader-role` along with the COMMITTING role — see note under Majority Consistency Invariant)* | -| Config push fails mid-cycle | Standby reports old tag next heartbeat → primary retries config push | - -## Testing Strategy - -### Unit Tests -- Decision loop: `(tag=0, lsn=0)` → PUSH_CONFIG; `(tag=T, lsn=0)` → STREAM_WAL; `(tag<T, lsn=N)` → PUSH_CONFIG; `(tag=T, lsn=N)` → CONVERGED -- Heartbeat reply populates `stateReport` correctly -- Majority threshold: `n_peers / 2` (C integer division — primary + converged standbys form a cluster majority): - - n=3: 3/2 = 1 converged standby needed → primary + 1 = 2 of 3 = majority - - n=5: 5/2 = 2 converged standbys needed → primary + 2 = 3 of 5 = majority - - n=1: 1/2 = 0 → immediate promotion *(reconciled 2026-06-09)* -- WAL catch-up: apply single entry, batch, error entry skipped -- `handle_ctrlhello` no longer triggers replication — only connection setup - -### Integration Tests -- 3-node cluster: primary enters serving via majority convergence (not timeout) *(Note: superseded by `ha-single-leader-role` — primary now enters CTRL_PRIMARY directly and gates confirmation on a WAL_NOOP own-term commit; see note under Majority Consistency Invariant)* -- Fresh standby (tag=0, lsn=0): bootstrap through config → WAL → converged in heartbeat cycles -- Standby rejoin within WAL window: WAL streaming only, no config push, no snapshot -- Standby rejoin outside WAL window: snapshot transfer → WAL streaming → converged -- Config upgrade: majority ack before committed; standby reports updated tag next heartbeat - -### E2E Tests -- Full cluster startup via `/xce-deploy-ha`: verify single PRIMARY via convergence, no degraded mode -- Failover: kill primary, standby promotes, serves from committed state diff --git a/.kiro/specs/ha-repl-hardening/raft-tla-audit.md b/.kiro/specs/ha-repl-hardening/raft-tla-audit.md deleted file mode 100644 index c10fcab7..00000000 --- a/.kiro/specs/ha-repl-hardening/raft-tla-audit.md +++ /dev/null @@ -1,91 +0,0 @@ -# Raft TLA+ Specification Audit - -Comparing XCE HA implementation against the formal Raft TLA+ specification (ongardie/raft.tla). - -## State Variable Mapping - -| TLA+ Variable | XCE Implementation | Location | Status | -|--------------|-------------------|----------|--------| -| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | ha.h:139 | ✅ | -| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | ha.h:161 | ✅ | -| `log[i]` | WAL segments in `persist->wal_path` | persist.c | ✅ | -| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | persist.h | ✅ | -| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | ha.h:141 | ✅ | -| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,COMMITTING,PRIMARY,STANDBY} | ha.h:139 | ✅ | -| `leaderId[i]` | `ha->primary_host` / `primary_port` | ha.h:142-143 | ✅ | -| `nextIndex[i, j]` | `standby.join_applied_lsn` (where to start sending) | repl.h | ⚠️ Implicit | -| `matchIndex[i, j]` | `standby.acked_lsn` (highest replicated) | repl.h | ⚠️ Implicit | - -**Gap**: `nextIndex`/`matchIndex` are implicit in the replicator, not exposed as first-class per-standby state. The heartbeat-driven design (`standby_reported_lsn`) makes `matchIndex` explicit and visible to the convergence check. - -## Action Mapping - -| TLA+ Action | XCE Implementation | Location | Status | -|------------|-------------------|----------|--------| -| `RequestVote(i, j)` | `raft_on_vote_request()` | raft.c:106 | ✅ | -| `RequestVoteReply(i, j)` | `raft_on_vote_reply()` | raft.c:169 | ✅ | -| `AppendEntries(i, j)` | `OPCode_nop` heartbeat + WAL streaming | ha.c, config_repl.c | ⚠️ No follower state in reply | -| `AppendEntriesReply(i, j)` | Heartbeat reply (currently empty `nodata`) | cluster.c:225 | ⚠️ Missing `success` + `matchIndex` | -| `BecomeLeader(i)` | `raft_cb_become_leader()` | ha.c:347 | ✅ | -| `AdvanceCommitIndex(i)` | `committed_lsn` advancement after `repl_wait_majority_ack` | repl.c:408 | ✅ | -| `ClientRequest(i, v)` | Job submit → WAL append → `repl_wait_majority_ack` | persist.c, repl.c | ✅ | -| `Timeout(i)` | `raft_cb_schedule_timeout` via delaymgr | ha.c | ✅ | - -**Key gap**: `AppendEntries` in Raft is bidirectional — leader sends `(term, prevLogIndex, prevLogTerm, entries[], leaderCommit)`, follower replies `(term, success, matchIndex)`. Our heartbeat (`OPCode_nop`) is unidirectional. This spec fixes exactly this gap by adding `stateReport` to the heartbeat reply. - -## Safety Property Audit - -### Election Safety (at most one leader per term) - -**TLA+**: `∀ i,j ∈ servers : state[i] = Leader ∧ state[j] = Leader ⇒ currentTerm[i] ≠ currentTerm[j]` - -**XCE**: ✅ Enforced through `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). To become leader, a candidate must receive votes from a majority, and each server votes at most once per term. Therefore at most one leader per term. - -### Log Matching (logs identical up to matching index+term) - -**TLA+**: `∀ i,j, index : log[i][index].term = log[j][index].term ⇒ ∀ k ≤ index : log[i][k] = log[j][k]` - -**XCE**: ✅ Enforced through election freshness gate. A candidate must have `config_tag ≥ voter's config_tag` and `applied_lsn ≥ voter's applied_lsn` to receive a vote. This guarantees the winner has the most complete log. `persist_truncate_wal_above(committed_lsn)` on promotion removes any divergent uncommitted entries. - -### Leader Completeness (committed entries survive leader changes) - -**TLA+**: `∀ i, index, term : committed(log[i][index], term) ⇒ ∀ j, futureTerm : leader(j, futureTerm) ⇒ log[j][index].term = term` - -**XCE**: ✅ A newly elected leader truncates WAL entries above `last_known_committed_lsn` (from CtrlHelloReply). The freshness gate ensures the winner has at least the committed entries of any voter. Since the winner needs a majority, and any committed entry was on a majority, the winner must have all committed entries. - -### State Machine Safety (same index → same entry) - -**TLA+**: `∀ i, j, index : applied(log[i][index]) ∧ applied(log[j][index]) ⇒ log[i][index] = log[j][index]` - -**XCE**: ✅ `committed_lsn` advances monotonically and WAL entries are applied in order. The truncation on leader change removes divergent entries above committed_lsn, and the freshness gate ensures the new leader has all entries at or below committed_lsn. - -### Commit Rule (current-term entry required) - -**TLA+**: A leader can commit `log[i][index]` when: -- `index > commitIndex[i]` -- `log[i][index].term = currentTerm[i]` -- A majority of servers have `matchIndex[j] ≥ index` - -**XCE**: ✅ Equivalent achieved through per-entry majority ack. Each WAL entry is committed individually via `repl_wait_majority_ack()` — the commit only succeeds after a majority have durably written the entry AND replied with an ACK. Since entries are written by the current leader (single-writer), every committed entry is implicitly from the current leader's term. The "entry from current term" requirement is satisfied because `persist_truncate_wal_above` removes any uncommitted entries from previous terms before the new leader writes its own. - -## Findings Summary - -### Correctly Implemented -1. Term management and monotonicity ✅ -2. One-vote-per-term enforcement ✅ -3. Election freshness gate (equivalent to Raft's log completeness check) ✅ -4. Committed entry durability through majority ack ✅ -5. WAL truncation on leader change ✅ -6. Leader completeness through election gate ✅ - -### Gaps Being Addressed by This Spec -7. **AppendEntries bidirectional state**: Missing follower state in heartbeat reply → fixed by `stateReport` in this spec -8. **`matchIndex` visibility**: Implicit in replicator → made explicit via `standby_reported_lsn[N]` in this spec - -### Not Applicable (Intentional Differences) -9. **`nextIndex` decrement on inconsistency**: Not needed — we truncate uncommitted entries on leader change, so there are no divergent entries for the consistency check to catch -10. **No-op commit of current-term entry**: Not needed — we use per-entry synchronous majority ack instead of batched commit - -## Recommendation - -No safety gaps found. The two items this spec addresses (heartbeat state reporting, matchIndex visibility) close the remaining protocol gaps vs the formal Raft specification. The per-entry majority ack model is functionally equivalent to Raft's batched commit rule for our append-only single-writer WAL. diff --git a/.kiro/specs/ha-repl-hardening/requirements.md b/.kiro/specs/ha-repl-hardening/requirements.md deleted file mode 100644 index 7a020b58..00000000 --- a/.kiro/specs/ha-repl-hardening/requirements.md +++ /dev/null @@ -1,103 +0,0 @@ -# Requirements Document - -## Introduction - -XCE controller operators running HA clusters face three replication gaps that prevent the system from reaching a safe majority-consistent state after election. The majority-ack fast path (`ha_repl_ack_received`) exists in code but is unreachable because no standby reports its catch-up status back to the primary. The committing timeout (60s degraded-mode fallback, task 5.3) is the safety net when majority convergence fails. Snapshot transfer for standbys that fall behind the WAL retention window is wired via the payload data plane (task 7.2). WAL catch-up streams entries one `xce_request` per entry, blocking a vtaskmgr thread for the entire duration. - -This feature implements a heartbeat-driven state synchronization protocol: the primary pushes config and WAL entries to standbys, each standby acknowledges with its current `(config_tag, applied_lsn)` in heartbeat replies, and the primary transitions to `CTRL_PRIMARY` when a majority of standbys report state converged with the primary. Separate catch-up-done and per-entry ack opcodes are eliminated in favor of piggybacked state reporting. - -## Boundary Context - -- **In scope**: standby-side handling of `OPCode_walCatchupEntry` with local WAL application, heartbeat reply carrying standby state `(config_tag, applied_lsn)`, primary tracking per-standby state and determining majority convergence, snapshot creation and transfer when a standby predates the WAL retention window, batch transfer of multiple WAL entries per message, and gating the `CTRL_PRIMARY` transition on majority state convergence rather than a timeout -- **Out of scope**: config push optimization (already implemented — config is pushed only when tags differ), WAL segment retention policy changes, dynamic cluster membership, mTLS between controllers, the election protocol itself, new wire protocol opcodes (the feature reuses existing heartbeat and ctrlhello messages extended with state fields) -- **Adjacent expectations**: the primary already sends heartbeats to standbys via `ha_start_primary_heartbeat` during `CTRL_PRIMARY_COMMITTING`; `CtrlHelloReply` already carries `committed_lsn` and `config_tag` fields in the capnp schema; `apply_wal_entry` already exists and can replay individual WAL entries; `persist_checkpoint` already exists and can create a consistent snapshot - -## Requirements - -### Requirement 1: Heartbeat Reply Carries Standby State - -**Objective:** As a cluster operator, I want each standby to report its current replication state in every heartbeat reply, so that the primary can determine when the cluster has reached majority consistency without a separate catch-up-complete signal. - -#### Acceptance Criteria -1. When a standby receives a heartbeat message from the primary, the system shall reply with a heartbeat acknowledgement containing the standby's current config version tag and applied WAL log sequence number. -2. When the primary receives a heartbeat reply from a standby, the system shall record that standby's `(config_tag, applied_lsn)` and evaluate whether the standby has converged with the primary's committed state. -3. A standby is considered converged when its reported config tag is greater than or equal to the primary's committed config tag and its reported applied log sequence number is greater than or equal to the primary's committed log sequence number. -4. If a standby has not yet received the current configuration, the system shall report its config tag as zero in heartbeat replies until the config push completes. - ---- - -### Requirement 2: Primary Tracks Per-Standby Convergence State - -**Objective:** As a cluster operator, I want the primary to continuously track each standby's reported state, so that it can determine when a majority of standbys have converged and the cluster is safe to enter `CTRL_PRIMARY`. - -#### Acceptance Criteria -1. When the primary enters `CTRL_PRIMARY_COMMITTING` state, the system shall initialize per-standby tracking with each standby marked as not-converged. -2. When a heartbeat reply arrives from a standby and the standby's reported state satisfies the convergence criteria, the system shall mark that standby as converged and re-evaluate the majority threshold. -3. When the number of converged standbys reaches the majority threshold `n_peers / 2` (C integer division — primary + converged standbys form a cluster majority; n=3→1 standby, n=5→2 standbys; raft rejects even n_peers > 1), the system shall transition the primary to `CTRL_PRIMARY` immediately. *(formula reconciled 2026-06-09)* -4. When the primary transitions to `CTRL_PRIMARY` via majority convergence, the system shall log at INFO level the number of converged standbys and the quorum size. -5. If a converged standby later reports a config tag or applied log sequence number that no longer satisfies the convergence criteria (e.g., after a restart), the system shall mark that standby as not-converged and re-evaluate the majority threshold. - ---- - -### Requirement 3: Standby Applies WAL Catch-Up Entries to Local State - -**Objective:** As a cluster operator, I want a standby that receives WAL catch-up entries from the primary to apply them to its local durable state, so that the standby's state converges with the primary's and the standby is eligible to become primary on failover. - -#### Acceptance Criteria -1. When a standby controller receives an `OPCode_walCatchupEntry` message, the system shall decode the WAL entry and apply it to local persistent storage. -2. The system shall update the standby's applied log sequence number after each successfully applied WAL entry, so that the next heartbeat reply reflects the latest state. -3. If a WAL entry fails to apply (corrupt payload, disk error), the system shall log an ERROR and continue processing subsequent entries without crashing. -4. While the standby is processing WAL catch-up entries, the system shall remain in `CTRL_STANDBY` state and shall not accept job dispatch or queue management requests. -5. The system shall process WAL catch-up entries in log sequence number order, preserving the same ordering the primary committed them in. - ---- - -### Requirement 4: Snapshot Transfer for Standbys Outside WAL Retention Window - -**Objective:** As a cluster operator, I want a standby whose applied log sequence number predates the oldest retained WAL segment to receive a full state snapshot, so that the standby can catch up and rejoin the replication set rather than being permanently excluded. - -#### Acceptance Criteria -1. When the primary determines that a standby's applied log sequence number predates the oldest retained WAL segment, the system shall create a consistent checkpoint of its current in-memory state and transfer the resulting snapshot data to the standby. -2. When a standby receives a snapshot transfer, the system shall load the snapshot to replace its local state, set its applied log sequence number to the snapshot's log sequence number, and then report the updated log sequence number in subsequent heartbeat replies. -3. When a standby receives a snapshot transfer, the system shall not accept job dispatch or queue management requests until the snapshot is fully loaded and all subsequent WAL catch-up entries are applied. -4. If snapshot transfer fails (connection drops, disk full), the standby shall log an ERROR and retry by reconnecting to the primary and initiating a new ctrlhello handshake. -5. When the cluster is a single node, the system shall not initiate snapshot transfer (no standbys exist). - ---- - -### Requirement 5: Batch WAL Catch-Up Entry Transfer - -**Objective:** As a cluster operator, I want the primary to transfer multiple WAL entries in a single message during standby catch-up, so that a fresh standby joining a long-running cluster can catch up in reasonable time rather than being blocked by per-entry overhead. - -#### Acceptance Criteria -1. When the primary streams WAL entries to a catching-up standby, the system shall pack multiple entries into each `OPCode_walCatchupEntry` message rather than sending one message per entry. -2. The system shall accumulate entries up to a configurable maximum batch size (in bytes or entry count) before sending. -3. When a standby receives a batch of WAL catch-up entries, the system shall apply each entry in order and update the applied log sequence number after the entire batch is durably written, so that the next heartbeat reply reflects the post-batch state. -4. If the primary has fewer entries remaining than the batch size, the system shall send a partial batch without waiting for more entries. -5. The batch transfer mechanism shall not change the ordering or durability guarantees of WAL entries — entries within a batch are applied in log sequence number order, and all entries in the batch are durably written before the applied log sequence number is advanced. - ---- - -### Requirement 6: COMMITTING Phase Gated on Majority Convergence - -**Objective:** As a cluster operator, I want the elected primary to remain in `CTRL_PRIMARY_COMMITTING` state until a majority of standbys report converged state via heartbeat replies, so that the cluster is guaranteed to be in a majority-consistent state before accepting client traffic. - -#### Acceptance Criteria -1. While in `CTRL_PRIMARY_COMMITTING` state, the system shall continue sending heartbeats to all standbys and processing heartbeat replies to track convergence. -2. While in `CTRL_PRIMARY_COMMITTING` state, the system shall accept only replication-related opcodes (WAL ack, config push ack, ctrlhello) and shall reply with a not-ready error to job dispatch, queue management, and resource management opcodes. -3. When a majority of standbys have reported converged state via heartbeat replies, the system shall transition to `CTRL_PRIMARY` immediately without waiting for any timeout. -4. If the committing timeout expires before majority convergence is reached, the system shall log a WARN and may transition to `CTRL_PRIMARY` in degraded mode, allowing the cluster to make progress when standbys are genuinely unavailable. -5. When the cluster is a single node, the system shall consider the convergence criterion satisfied immediately (no standbys needed). - ---- - -### Requirement 7: Observability for Replication Progress - -**Objective:** As a cluster operator, I want structured log events for replication state transitions, so that catch-up progress, snapshot transfers, and majority-convergence status can be monitored without external tooling. - -#### Acceptance Criteria -1. When a standby begins WAL catch-up, the system shall log at INFO level the starting and target log sequence numbers. -2. When the primary detects that a standby has converged via heartbeat reply, the system shall log at INFO level the standby's identity and its reported config tag and applied log sequence number. -3. When a snapshot transfer begins and completes, the system shall log at INFO level the snapshot log sequence number and the transfer size. -4. When the primary reaches majority convergence and transitions to `CTRL_PRIMARY`, the system shall log at INFO level the number of standbys that have converged and the quorum size. -5. When the committing timeout fires before majority convergence is reached, the system shall log at WARN level indicating the cluster is entering degraded mode, including the number of standbys that have converged and the number still pending. diff --git a/.kiro/specs/ha-repl-hardening/research.md b/.kiro/specs/ha-repl-hardening/research.md deleted file mode 100644 index 3830ebc4..00000000 --- a/.kiro/specs/ha-repl-hardening/research.md +++ /dev/null @@ -1,588 +0,0 @@ -# Gap Analysis: HA Replication Hardening - -> Analysis date: 2026-06-04. Phase: `requirements-generated`. -> Purpose: identify what exists, what's missing, and viable implementation approaches. - -## Analysis Summary - -- **Scope**: 7 requirements — heartbeat-driven state convergence, WAL catch-up wiring, per-standby tracking, snapshot transfer, batch WAL transfer, COMMITTING gate, observability -- **Key finding**: The primary already sends heartbeats (`OPCode_nop`) and receives replies; the `CtrlHelloReply` already carries `committed_lsn` and `config_tag`; `persist_checkpoint` already exists. The gaps are in wiring, not new infrastructure. -- **Largest gaps**: (1) heartbeat reply doesn't carry standby's `applied_lsn`/`config_tag`; (2) WAL catch-up handler is a stub; (3) snapshot transfer path is a stub; (4) no per-standby convergence tracking in the primary. -- **Recommendation**: Extend existing heartbeat reply with state fields; wire the WAL catch-up stub to `apply_wal_entry()`; implement snapshot transfer using existing `persist_checkpoint()`; add per-standby convergence tracking to `ha_state_t` or extend `standby_conn_t`. - -## 1. Current State Investigation - -### 1.1 Heartbeat Protocol - -**Current state**: Asymmetric, unidirectional state push. -- Primary sends `OPCode_nop` via `primary_heartbeat_fn` (ha.c:2880) every `heartbeat_interval_ms` (3s default) -- Standby receives nop in `myreq_handler` (cluster.c:225), sends empty `RequestReply`, calls `ha_on_heartbeat_received` (ha.c:2655) to reset liveness watchdog -- `CtrlHelloRequest`/`CtrlHelloReply` (controller.capnp:835-846) carry `appliedLsn`, `configTag`, `committedLsn` — but only exchanged once on connect, not periodically - -**Gap**: Heartbeat reply is empty. Standby's current `(applied_lsn, config_tag)` is not reported back to the primary on each heartbeat cycle. - -### 1.2 Per-Standby State Tracking - -**Current state**: `standby_conn_t` (repl.h:22-33) tracks per-standby `acked_lsn` and `acked_config_lsn` (for WAL and config quorum), plus `in_catch_up` and `pending_ack_count`. `ha_state_t` has `standby_endpoints` (zarray of endpoints) and `n_peers`, but no per-standby convergence tracking. - -**Gap**: No field for the standby's self-reported `applied_lsn` or `config_tag` from heartbeat replies. The primary has no way to know whether a standby has converged without receiving explicit WAL/config ACKs. - -### 1.3 WAL Catch-Up Handler - -**Current state**: Stub at ha.c dispatch gate (OPCode_walCatchupEntry returns 0). The primary-side streaming loop in config_repl.c:1489-1520 reads WAL segments and sends `OPCode_walCatchupEntry` messages, but the standby does nothing with them. - -**Gap**: Standby never applies WAL catch-up entries to local state. `apply_wal_entry()` (persist.c:685) exists and is used during startup WAL replay — can be reused. - -### 1.4 Snapshot Transfer - -**Current state**: `handle_ctrlhello` (config_repl.c:1359) checks `repl_needs_snapshot()` (repl.c:491) and if true, logs WARN at line 1418 ("snapshot transfer not yet implemented") and skips. `persist_checkpoint()` (persist.c:610) exists and creates a consistent checkpoint. - -**Gap**: The snapshot creation and transfer path is entirely unimplemented. Without it, any standby whose LSN predates the WAL retention window is permanently dead. - -### 1.5 WAL Entry Batching - -**Current state**: The WAL streaming loop in config_repl.c:1481-1521 sends one `xce_request` per WAL entry. `xce_request` enqueues to the endpoint's send buffer (non-blocking), but the loop processes entries sequentially with `pread()` per entry + `malloc`/`free` per payload. - -**Gap**: No batching. For a standby catching up on millions of entries, this is CPU-intensive and slow. The wire protocol needs a batch message format. - -### 1.6 COMMITTING Gate - -**Current state**: `ha_repl_ack_received()` (ha.c:2197) exists with correct quorum math, but is only called from `repl_ack_decode_and_dispatch()` (ha.c:1859) which processes `OPCode_replCatchUpDone`. No standby sends this opcode. The committing timeout (10s) is the only path to PRIMARY. - -**Gap**: The majority-convergence check is unreachable. The committing timeout degrades the cluster on every startup. - -## 2. Requirement-to-Asset Map - -| Req | Key Gap | Key Reusable Asset | Gap Tag | -|-----|---------|-------------------|---------| -| 1 — Heartbeat carries standby state | Reply body is empty; no state fields in nop reply | `OPCode_nop` send/receive loop, `CtrlHelloReply` fields in capnp schema, `xce_reply` pattern in cluster.c:225-228 | **Missing** | -| 2 — Per-standby convergence tracking | No `applied_lsn`/`config_tag` reported per standby | `standby_conn_t` with `acked_lsn`, `ha->standby_endpoints`, `repl_wait_majority_ack` quorum math | **Missing** | -| 3 — WAL catch-up handler | Stub returns 0 (ha.c:882) | `apply_wal_entry()` (persist.c:685), primary WAL streaming loop (config_repl.c:1489) | **Stub** | -| 4 — Snapshot transfer | Stub logs WARN, skips (config_repl.c:1417-1425) | `persist_checkpoint()` (persist.c:610), `repl_needs_snapshot()` (repl.c:491), `CtrlHelloReply.needsSnapshot` field | **Missing** | -| 5 — Batch WAL transfer | One entry per `xce_request` | Existing WAL streaming loop, capnp message format for walCatchupEntry | **Missing** | -| 6 — COMMITTING gate | `ha_repl_ack_received` unreachable; timeout-only path | `ha_repl_ack_received()` (ha.c:2197) with correct quorum math, `ha_promote_to_primary()` | **Missing** | -| 7 — Observability | Logs exist but need new convergence/snapshot events | `INFO`/`WARN` macros in xcelog.h | **Missing (alongside)** | - -## 3. Implementation Approach Options - -### Option A: Extend heartbeat reply with state fields (Recommended) - -Extend the existing `OPCode_nop` reply or create a lightweight heartbeat-ack to carry `(applied_lsn, config_tag)`: -- **Standby side**: In the nop handler (cluster.c:225), populate reply with current `applied_lsn` and `config_tag` before `xce_reply` -- **Primary side**: In the heartbeat send path or a reply handler, read the standby's reported state and update per-standby tracking -- **Why**: Minimal new code — reuses existing heartbeat loop; no new opcodes needed if we extend the reply body; follows Raft AppendEntries pattern - -**Trade-offs**: -- ✅ Reuses existing infrastructure; minimal new opcodes -- ✅ Heartbeat already fires every 3s — convergence detected promptly -- ❌ Need capnp schema change if RequestReply needs new fields -- ❌ Primary must process heartbeat replies (currently fire-and-forget) - -### Option B: Use CtrlHelloReply periodically - -Send `CtrlHelloRequest` periodically (not just on connect) and use the existing `CtrlHelloReply` fields: -- Already carries `committedLsn`, `configTag` in reply -- Already carries `appliedLsn`, `configTag` in request -- **Why**: No capnp schema changes; ctrlhello already carries all needed fields - -**Trade-offs**: -- ✅ No schema changes -- ❌ CtrlHelloRequest triggers WAL streaming (heavy) — not suitable for periodic use -- ❌ Semantic mismatch: ctrlhello is a connection-setup message, not a state-reporting message - -### Option C: New lightweight opcode - -Add `OPCode_stateReport` / `OPCode_stateAck` to capnp schema: -- Dedicated message type with `(applied_lsn, config_tag, in_catch_up)` fields -- Sent by standby in response to heartbeat, or periodically -- **Why**: Clean separation of concerns - -**Trade-offs**: -- ✅ Clean protocol design -- ❌ New opcode, new capnp struct, new handler on both sides - -## 4. Effort and Risk Summary - -| Req | Effort | Risk | Justification | -|-----|--------|------|---------------| -| 1 — Heartbeat state report | S (1-2d) | Low | Extend existing reply; capnp schema change if needed | -| 2 — Per-standby tracking | S (1d) | Low | New fields in ha_state_t or extend standby_conn_t | -| 3 — WAL catch-up wiring | S (1d) | Low | Wire stub to existing apply_wal_entry() | -| 4 — Snapshot transfer | M (3-5d) | Medium | persist_checkpoint exists; transfer protocol is new | -| 5 — Batch WAL transfer | S (1-2d) | Low | Accumulate entries in loop; capnp schema change | -| 6 — COMMITTING gate | S (1d) | Low | Replace ha_repl_ack_received with heartbeat-driven check | -| 7 — Observability | S (alongside) | Low | INFO/WARN log statements | - -**Total estimated effort**: 1-2 weeks. Risk: Low-Medium (snapshot transfer is the main risk). - -## 5. Key Reusable Assets - -| Asset | Location | Reuse in | -|-------|----------|----------| -| `OPCode_nop` send/reply loop | cluster.c:225, ha.c:2880 | Req 1 (heartbeat state report) | -| `CtrlHelloReply` fields | controller.capnp:835-846 | Req 1 (capnp schema template) | -| `standby_conn_t` + `repl_wait_majority_ack` | repl.h:22, repl.c:338 | Req 2 (convergence tracking) | -| `apply_wal_entry()` | persist.c:685 | Req 3 (WAL catch-up) | -| `persist_checkpoint()` | persist.c:610 | Req 4 (snapshot creation) | -| WAL streaming loop | config_repl.c:1481 | Req 5 (batch send) | -| `ha_repl_ack_received()` | ha.c:2197 | Req 6 (quorum math reuse) | -| `INFO`/`WARN` macros | xcelog.h | Req 7 (observability) | - -## 6. Research Needed (Design Phase) - -- **Capnp schema design**: Whether to extend `RequestReply` with state fields, add a new `stateReport` struct to existing heartbeat messages, or create a new opcode. Evaluate against existing capnp codegen patterns. -- **Heartbeat reply processing**: Currently fire-and-forget; need to add a reply handler on the primary side. Determine best insertion point in the dispatch path. -- **Snapshot wire format**: How to serialize `persist_checkpoint` output for transfer. COW zarray vs raw file transfer. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **ha-repl-hardening** is **~98% complete**. All 6 task groups (1–6), 27 subtasks (1.1–6.7) are `[x]` in tasks.md. Recent commits (aab41fa through e1517bb) confirm active implementation. The git log shows all tasks through 6.7 implemented with bug fixes applied. -- **Remaining gaps**: Task groups 1–4 and 5–6 all show `[ ]` as section headers, but all subtasks within them are `[x]`. The section checkboxes are cosmetic. -- **Deferred items**: None. The original research's "Research Needed" items (capnp schema design, heartbeat reply processing, snapshot wire format) were resolved during the design phase. -- **Cross-spec relationships**: This spec extends `ha-config-bootstrap` Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence. It depends on `ha-heartbeat-liveness` for the idle-triggered NOP policy on the primary and activity-based liveness detection on the standby. It depends on `ha-peer-connection-reuse` for the promoted connection model — `handle_ctrlhello` was reduced to connection setup only (task 5.1), with all replication decisions moved to the heartbeat decision loop. -- **Single-node mode**: Verified working — `n_peers=1`, immediate convergence, `CTRL_PRIMARY` with no heartbeat wait. -- **Recommendation**: Run `/xce-build controller` to confirm ASAN-clean build, then mark the spec `implemented`. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -> Validation-phase gap analysis: requirements.md vs code **as on disk** (including uncommitted -> changes to ha.c/ha.h/myapp.c/raft.c). All 27 subtasks in tasks.md are `[x]`; this pass verifies -> what the code actually does against each acceptance criterion. - -## Headline Finding - -**The control plane is complete; the data plane is not.** Every protocol mechanism in this spec -(heartbeat stateReport, per-standby tracking, decision loop, batch framing, snapshot chunking, -convergence-gated promotion, safe commit point) is implemented and unit/integration tested. But -**payload bytes are never transmitted or applied**: - -- `ha_send_snapshot_to_standby` sends every chunk with `chunk.payload = NULL` — - `config_repl.c:1638` (`/* encoding deferred */`, explained at config_repl.c:1613-1618: the - capnp `Data` ↔ `xce_data_t` tagged-union encode path is not wired). -- The standby snapshot handler passes `payload_bytes=NULL, payload_len=0` — - `cluster.c:191-197`. Consequently `snap.part` is never written, the `is_last` size check - (`ha.c:1094-1106`) fails (or `stat` fails on a missing file), and `persist_load_snapshot` is - never reached in production. Snapshot transfer works only in unit tests via - `xce_snapshot_chunk_hook`. -- WAL catch-up applies entries **header-only**: `ha_apply_single_entry` (ha.c:974-984) builds - `payload_len=0` and calls `apply_wal_entry(p, &hdr, NULL)`. `apply_wal_entry` - (persist.c:691) then only advances the LSN chain — payload-bearing entry types - (`WAL_JOB_SUBMIT` → capnp job, jobstate entries → `wal_jobstate_t`) restore **nothing**. - -**Safety implication**: a standby's `applied_lsn` advances without the corresponding state, so it -reports "converged" via stateReport, the primary promotes out of COMMITTING, and a later failover -to that standby would serve from a state missing all payload-bearing entries. The Majority -Consistency Invariant (design §Overview) is currently tracked but not actually established. - -**Cross-spec constraint**: the live replication path has the same hole — `repl_broadcast` -(repl.c:193-234) discards `payload` and sends nothing ("TCP send wiring deferred to task 7.6", -owned by ha-config-bootstrap). Today no WAL payload bytes move between controllers on any path. - -## Requirement-to-Asset Map - -| Req / AC | Asset (file:line) | Status | -|---|---|---| -| **1.1** heartbeat reply carries (tag, lsn) | `cluster.c:273-287` OPCode_nop handler; `ha_nop_reply_state_report` ha.c:1140; `StateReport` controller.capnp:788 | Implemented | -| **1.2** primary records + evaluates | `repl_ack_decode_and_dispatch` ha.c:2456-2547 → `ha_process_standby_state` ha.c:1198 | Implemented | -| **1.3** convergence criteria (tag≥ AND lsn≥) | ha.c:1211-1212 | Implemented | -| **1.4** tag=0 before config push | reads `myconf->config_version_tag` (0 pre-push) | Implemented | -| **2.1** init tracking on COMMITTING entry | Arrays zeroed only in `ha_init` (ha.c:816-819 comment claims the invariant); `raft_cb_become_leader` (ha.c:445-529) resets `repl_ack_count` but **not** `standby_converged[]`/`standby_reported_*[]` → stale convergence flags survive a demote→re-elect cycle | **Partial** | -| **2.2** update + re-evaluate on reply | ha.c:1198-1223, 2525-2533 | Implemented | -| **2.3** majority threshold ceil(n/2)−1 | `ha_check_majority_convergence` ha.c:1244-1278 requires **all** standbys (`n_converged >= n_peers−1`), not majority. Three conflicting formulas: requirements.md 2.3 = ceil(n/2)−1 (1 standby for n=3); tasks.md 5.2 = ceil(n/2) (2 for n=3); code = n−1 (2 for n=3, 4 for n=5) | **Constraint / deviation** (stricter than spec — safe but availability hit) | -| **2.4** INFO on promotion | ha.c:1266-1267 | Implemented | -| **2.5** regression → not-converged | ha.c:1210-1213 | Implemented | -| **3.1** decode + apply WAL entry | Handler wired (`cluster.c:161-173` → `ha_apply_wal_catchup_entry` ha.c:986) but applies **header-only, payload dropped** (ha.c:974-984) | **Partial (CRITICAL)** | -| **3.2** advance applied_lsn | `ha_advance_applied_lsn` ha.c:960 (atomic, monotonic) | Implemented | -| **3.3** ERROR + skip on failure | ha.c:1003-1016, 1032-1037 | Implemented | -| **3.4** stays CTRL_STANDBY, no job ops | `ha_dispatch_gate` CTRL_STANDBY case ha.c:1456-1483 | Implemented | -| **3.5** LSN order | sequential batch iteration | Implemented | -| **4.1** create checkpoint + transfer | `ha_send_snapshot_to_standby` config_repl.c:1578 chunks an **existing** `snap-{checkpoint_lsn}.capnp` — never calls `persist_checkpoint()` to create a fresh one; payload=NULL | **Partial** | -| **4.2** load snapshot, set applied_lsn | `ha_handle_snapshot_chunk` ha.c:1058-1131 complete (verify/rename/load/applied_lsn) but unreachable in production (NULL payload, cluster.c:196) | **Partial** | -| **4.3** no job ops until loaded | dispatch gate (standby role unchanged) | Implemented | -| **4.4** retry via reconnect | Restart-on-reconnect documented (ha.c:1054, cluster.c:178-180) but **no production code unlinks `snap.part`** on reconnect — stale partial file breaks the next transfer's size check | **Partial** | -| **4.5** single-node: no snapshot | no standbys → decision loop never fires | Implemented | -| **5.1** multiple entries per message | `entries :List(WalCatchupEntry)` controller.capnp:850; `send_wal_batch` config_repl.c:1409 | Implemented | -| **5.2** configurable max batch size | `WAL_STREAM_BATCH_ENTRIES`=256 / `WAL_STREAM_BATCH_BYTES`=64KB are compile-time #defines, not runtime config | Implemented (minor constraint) | -| **5.3** apply in order, advance after batch | ha.c:992-1027 (`last_good_lsn` post-batch advance) — but payload gap applies | **Partial** | -| **5.4** partial batch sent immediately | config_repl.c:1556-1563 | Implemented | -| **5.5** ordering + durability | Ordering yes; "durably written" weak — entities restored in-memory, no fsync on standby (applied_lsn in-memory-only is a documented design choice), and payload gap applies | **Partial** | -| **6.1** heartbeats continue in COMMITTING | self-re-arming heartbeat task (fix e1517bb) | Implemented | -| **6.2** NOT_READY for non-repl opcodes | `ha_dispatch_gate` CTRL_PRIMARY_COMMITTING allowlist ha.c:1436-1454 | Implemented | -| **6.3** immediate promotion on majority | ha.c:2529-2533 (before repl dispatch) — subject to 2.3 threshold deviation | Implemented (constraint) | -| **6.4** timeout → WARN + degraded mode | `committing_timeout_fn` ha.c:2885-2914; default 60000 (`HA_DEFAULT_COMMITTING_TIMEOUT_MS` ha.h:131) | Implemented | -| **6.5** single-node immediate | n_peers=1 → threshold 0 (ha.c:1261-1264; fixed in e10cf9a) | Implemented | -| **7.1** catch-up start INFO | config_repl.c:1445-1447 | Implemented | -| **7.2** converged INFO | ha.c:1216-1222 | Implemented | -| **7.3** snapshot start/complete INFO | config_repl.c:1595, 1663 | Implemented | -| **7.4** majority INFO | ha.c:1266 | Implemented | -| **7.5** timeout WARN w/ counts | ha.c:2905-2908 | Implemented | -| **Flow 0** safe commit point (tasks 4.1/4.2) | `ha_compute_safe_committed_lsn` ha.c:373-432; truncation + committed baseline in `raft_cb_become_leader` ha.c:456-485 | Implemented | - -**Score**: 26 of 33 acceptance criteria fully implemented. Requirements 1, 6, 7 (and Flow 0 -safety) Implemented; Requirement 2 Partial (init-reset + threshold semantics); Requirements 3, 4, -5 Partial — all blocked on the same payload data-plane gap. - -**Uncommitted working-tree changes** (ha.c/ha.h/myapp.c/raft.c/deploy-ha.sh) are election-protocol -hardening adjacent to this spec: `ha_init_raft` one-time init (prevents raft state reset on -re-election), term-scoped candidate-agnostic vote denial in `raft_on_vote_request` with idempotent -re-grant, startup self-vote/stale-vote handling, and watchdog arming on the skip-election path. -These strengthen Election Safety per raft-tla-audit.md and should be committed, but they do not -close any of this spec's acceptance criteria. - -## Implementation Approach Options (remaining gaps) - -### Gap 1 — Payload encoding for `WalCatchupEntry.payload` and `SnapshotTransfer.payload` (the blocker) - -**Option A — Extend the typemap codegen to map capnp `Data` ↔ raw buffer for these structs.** -The schema already declares `payload :Data` on both structs; only the C-side encode/decode of the -`xce_data_t` tagged union is missing. -- ✅ Fixes both messages at once; follows the established schema/typemap convention; benefits any - future Data-carrying message -- ❌ Touches generated-code path — regression risk across all opcodes; needs codegen expertise - -**Option B — Hand-rolled encode/decode for just these two opcodes.** -Bypass the tagged union: serialize `lsn/type/len/bytes` with direct capn calls (or a length-prefixed -raw layout) in `send_wal_batch` / `ha_send_snapshot_to_standby` and their decode counterparts. -- ✅ Localized, no codegen risk, fastest path to a working data plane -- ❌ Two bespoke wire paths diverging from every other message; long-term maintenance debt - -**Option C — Phased hybrid (recommended).** -Phase 1: hand-wire the snapshot payload (raw file bytes — trivially a byte stream, no per-entry -structure) to make Req 4 production-real. Phase 2: solve `Data` encoding properly in the typemap -for `WalCatchupEntry` (structured, batched, used on the hot path) and migrate the snapshot path to -it. Until both land, keep the spec phase below `implemented` or gate promotion on a data-plane -flag. -- ✅ De-risks incrementally; snapshot (rare, big) and WAL (frequent, small) have different shapes -- ❌ Two-step migration; temporary inconsistency between the two paths - -### Gap 2 — Convergence threshold semantics (Req 2.3 vs code) - -- **Option A**: change code to the requirement's majority formula (`ceil(n_peers/2) − 1` standbys). - Restores availability (one dead standby in a 3-node cluster no longer forces the 60s degraded - timeout) and matches Raft quorum; safe because safe_committed_lsn already derives from a majority. -- **Option B**: keep all-standbys convergence and amend requirements.md + tasks.md with the - rationale (strongest consistency at startup; degraded timeout covers minority loss). -- Either way, **reconcile the three formulas** (requirements 2.3, task 5.2, code) — they currently - disagree for any n ≥ 5. - -### Gap 3 — Small correctness fixes (extend existing components) - -1. Zero `standby_converged[]` / `standby_reported_*[]` in `raft_cb_become_leader` on COMMITTING - entry (Req 2.1 — re-election staleness). -2. `unlink(snap.part)` on standby reconnect/ctrlhello (Req 4.4 — stale partial file). -3. Create a fresh checkpoint (via delaymgr — `persist_checkpoint` is blocking I/O, must not run in - the vtask/heartbeat-reply context) when SEND_SNAPSHOT fires and the on-disk snapshot is older - than `oldest_wal_lsn` coverage requires (Req 4.1). -4. Consider caching `ha_compute_oldest_wal_lsn` — currently an opendir/readdir scan per heartbeat - reply per standby in vtask context (ha.c:2538-2543). -5. Optionally promote batch limits to `ha_config_t` (Req 5.2 "configurable"). - -## Effort and Risk - -| Gap | Effort | Risk | Justification | -|---|---|---|---| -| 1 — Payload data plane (Option C) | **M–L (5–10d)** | **Medium** | Snapshot raw-bytes path is S; typemap `Data` codegen + batched WAL payload apply + ASAN-clean lifetime mgmt (current `(xce_data_t *)pbuf` cast at config_repl.c:1529 is already suspect) + E2E re-verification is the bulk | -| 2 — Threshold reconciliation | **S (0.5–1d)** | **Low** | One comparison + spec text alignment; existing tests pin the 3-node case | -| 3.1 — Tracking reset on COMMITTING | **S (<0.5d)** | **Low** | memset 3 arrays at ha.c:510; add re-election unit test | -| 3.2 — snap.part discard | **S (<0.5d)** | **Low** | unlink in handle_ctrlhello standby path | -| 3.3 — Fresh checkpoint on SEND_SNAPSHOT | **M (2–3d)** | **Medium** | Blocking checkpoint must hop to delaymgr; interaction with concurrent WAL writes | -| 3.4 — oldest_wal_lsn caching | **S (0.5d)** | **Low** | Cache + invalidate on segment GC | -| 3.5 — Configurable batch limits | **S (0.5d)** | **Low** | Plumb two fields through ha_config_t | - -## Recommendations - -1. **Do not mark this spec `implemented`.** All tasks are `[x]` but Requirements 3, 4, 5 are only - control-plane complete; the system currently reports convergence it has not achieved. Reopen - targeted tasks (suggest: 7.x series) for the payload data plane rather than re-flagging 2.x/3.5. -2. Land Gap 3.1 and 3.2 immediately (S, Low) — they are latent correctness bugs independent of the - payload work. -3. Decide Gap 2 (threshold) explicitly and record it in the Decision Log; today three documents - disagree. -4. Sequence the payload work after coordinating with **ha-config-bootstrap task 7.6** - (`repl_broadcast` TCP wiring) — the live path and catch-up path should share one entry - serialization, or the two specs will produce divergent wire formats for the same WAL entry. -5. Add an E2E assertion that verifies **entity state** (e.g., a submitted job visible on the - promoted standby), not just `applied_lsn` equality — the current tests 6.5–6.7 pass despite the - data-plane gap because they only observe LSN/log convergence. - -## Research Needed - -- How the typemap codegen represents capnp `Data` (`xce_data_t` tagged union table dispatch) and - what it takes to encode an arbitrary byte buffer — prerequisite for Gap 1 Option A/C phase 2. -- Whether convergence should additionally verify content (e.g., last-entry CRC chained via - `wal_entry_hdr_t.crc/prev_crc` in stateReport) so header-only application can never report - false convergence again. -- Ownership boundary with ha-config-bootstrap task 7.6: which spec owns live WAL entry - serialization, and can `send_wal_batch` be reused for it. -- Req 5.5 "durably written" on the standby: is in-memory entity restore + in-memory applied_lsn - acceptable (current documented design), or should batch apply fsync a standby-side WAL copy. - ---- - -# Gap Re-Audit (2026-06-18) - -> Post-implementation gap audit. spec.json `phase=implemented`; all 41 sub-tasks `[x]` -> (1.1–7.6 incl. the reopened payload data-plane series). This pass re-verifies every EARS -> criterion against the code **as on disk** — specifically whether the two CRITICAL data-plane -> gaps and the small correctness gaps from the 2026-06-09 pass are now closed, and whether the -> COMMITTING-gate requirements still describe the live code after `ha-single-leader-role` landed. - -## Headline Finding - -**Both 2026-06-09 CRITICAL blockers are CLOSED. The data plane is real.** - -- WAL catch-up now applies payload-bearing entries. `ha_apply_single_entry` (ha.c:1558) does - persist-before-apply: `persist_wal_write_replicated(p, e->lsn, type, e->raw_payload, len)` - (fsync, no re-broadcast, idempotent) THEN `apply_wal_entry(p, &hdr, e->raw_payload)`. - `apply_wal_entry` (persist.c:1305) restores entities by type — `WAL_JOB_SUBMIT`→`restore_job`, - job-state→`restore_job_state`, `WAL_WORKER_CTRL`→`worker_from_capnp`, - `WAL_QUEUE_CTRL`→`restore_queue_state` (persist.c:1342-1389). Header-only application is gone. -- Snapshot payload bytes are transmitted. `ha_send_snapshot_to_standby` (config_repl.c:1857) - stats the real file, chunks 64KB, and ships each chunk via `xce_snapshot_chunk_req_raw(ep, - &chunk, buf, chunk_len)` (config_repl.c:1928) — raw `Data` bytes, not NULL. Standby - `ha_handle_snapshot_chunk` (ha.c:1683→ via `handle_snapshot_transfer` cluster.c:196, dispatched - at cluster.c:589) writes chunks to `snap.part`, verifies total size on `is_last`, atomic-renames - to `snap-{lsn}.bin`, loads via `persist_load_snapshot`, sets `applied_lsn`. - -**The Majority Consistency Invariant is now actually established, not merely tracked.** A standby -that reports converged holds the entity state behind its `applied_lsn`. - -## Architectural supersession (read before re-opening any 6.x task) - -`CTRL_PRIMARY_COMMITTING` **no longer exists** — `grep -rn CTRL_PRIMARY_COMMITTING src/controller` -returns zero hits. `ha-single-leader-role` removed the transitional COMMITTING role: -`raft_cb_become_leader` now stores `CTRL_PRIMARY` directly (ha.c:1034) with the comment "there is -no transitional COMMITTING state." New-leader safety is provided by a different, stronger -mechanism than this spec's heartbeat-convergence gate: - -1. **Own-term WAL_NOOP + writable_lsn gate** (`ha_arm_own_term_gate` ha.c:2259) — every WAIT - completion is withheld until `committed_lsn >= own-term noop LSN`, i.e. until this term has - committed an entry on a majority. -2. **Per-operation write quorum** (`ha_dispatch_gate` ha.c:2061 → `ha_has_write_quorum` ha.c:2031) - — state-changing opcodes return `-XCE_ERR_NOT_READY` unless `repl_active_standby_count >= - n_peers/2`; replication and read-only/fact-bearing opcodes are always admitted (allowlist at - ha.c:2078-2106). -3. **Vote-derived safe commit point** (`ha_compute_safe_committed_lsn` ha.c:480, called from - raft_cb_become_leader ha.c:908) — `safe_committed_lsn` = majority-threshold element of granted - voters' LSNs; `persist_truncate_wal_above(safe_lsn)` + `committed_lsn=safe_lsn` (ha.c:914). - -Net effect: the *intent* of Req 6 (don't serve un-replicated state; gate on majority) is met -**more strongly and continuously** than the original timeout-bounded COMMITTING design — it is now -enforced per-operation for the life of the term, not just at startup. The original Req 2.3 / 6.1–6.4 -wording (COMMITTING role, 60s `committing_timeout_ms` degraded fallback, immediate -COMMITTING→PRIMARY on majority) describes a design that was deliberately replaced. This is a -**documentation drift**, not a functional gap. - -## Requirement-to-Asset Map (re-verified) - -| Req / AC | Asset (file:line) | Status | -|---|---|---| -| **1.1** heartbeat reply carries (tag,lsn) | `OPCode_nop` handler cluster.c:300-301; populates from `myconf->config_version_tag` + `applied_lsn` | CLOSED | -| **1.2** primary records + evaluates | `repl_ack_decode_and_dispatch` → `ha_process_standby_state` ha.c:1851 | CLOSED | -| **1.3** convergence criteria tag≥ AND lsn≥ | ha.c:1864-1865 | CLOSED | -| **1.4** tag=0 before config push | reads live `config_version_tag` (0 pre-push) | CLOSED | -| **2.1** init tracking on term entry | **FIXED** — `raft_cb_become_leader` memsets `standby_reported_tag/_lsn/_converged` (ha.c:1043-1045) on CTRL_PRIMARY entry. Closes 2026-06-09 Gap 3.1. | CLOSED | -| **2.2** update + re-evaluate on reply | `ha_process_standby_state` ha.c:1851-1876 | CLOSED | -| **2.3** majority threshold | **RECONCILED** to `n_peers/2` (C int div) per task 7.5 + Decision Log 2026-06-09. `ha_has_write_quorum` uses `connected >= n_peers/2` (ha.c:2058); `safe_committed_lsn` uses `n_granted/2` (ha.c:527). Three-doc disagreement resolved. | CLOSED | -| **2.4** INFO on promotion/converged | ha.c:1869-1875 (per-standby converged INFO) | CLOSED | -| **2.5** regression → not-converged | ha.c:1863-1866 (recomputed every reply) | CLOSED | -| **3.1** decode + apply WAL entry **with payload** | **FIXED** — `ha_apply_single_entry` ha.c:1558-1589 passes `raw_payload` to `apply_wal_entry`; entities restored persist.c:1342-1389. Closes 2026-06-09 CRITICAL. | CLOSED | -| **3.2** advance applied_lsn | `ha_advance_applied_lsn` ha.c:1541 (atomic monotonic); batch advances after full batch (ha.c:1656-1659) | CLOSED | -| **3.3** ERROR + skip on failure | ha.c:1633-1647, 1666-1672 (NULL/failed sub-entries skipped, no crash) | CLOSED | -| **3.4** stays STANDBY, no job ops | `ha_dispatch_gate` STANDBY case | CLOSED | -| **3.5** LSN order | sequential batch iteration + persist-before-apply | CLOSED | -| **4.1** fresh checkpoint + transfer | **FIXED** — `ha_dispatch_repl_action_for_standby` calls `persist_checkpoint(&c->persist)` before send (ha.c:1961), -EBUSY falls back to existing. Closes 2026-06-09 Gap 3.3. | CLOSED | -| **4.2** load snapshot, set applied_lsn | `ha_handle_snapshot_chunk` ha.c:1683 (verify→rename→`persist_load_snapshot`→applied_lsn); reachable in production (real payload) | CLOSED | -| **4.3** no job ops until loaded | dispatch gate (standby role) | CLOSED | -| **4.4** retry via reconnect / discard stale | **FIXED (cleaner than proposed)** — `chunk_index==0` unlinks any stale `snap.part` (ha.c:1693, 1708-1713) instead of relying on a reconnect-time unlink. Verified by `test_snapshot_recv_reconnect_discards_snappart` (test_config_repl.c:2045). Closes 2026-06-09 Gap 3.2. | CLOSED | -| **4.5** single-node: no snapshot | no standbys → decision loop never fires | CLOSED | -| **5.1** multiple entries/message | `entries :List(WalCatchupEntry)` + `send_wal_batch` (config_repl.c) | CLOSED | -| **5.2** configurable max batch | `WAL_STREAM_BATCH_ENTRIES`=256 / `_BYTES`=64KB compile-time #defines | CLOSED (minor: limits are compile-time, not runtime ha_config_t — Req says "configurable maximum"; defensible) | -| **5.3** apply in order, advance after batch | `ha_apply_wal_catchup_entry` ha.c:1615-1681 (`last_good_lsn` post-batch advance) — now with real payload | CLOSED | -| **5.4** partial batch sent immediately | config_repl.c batch flush on range end | CLOSED | -| **5.5** ordering + durability | Ordering preserved; durability now STRONGER than original design — `ha_apply_single_entry` fsyncs via `persist_wal_write_replicated` BEFORE apply/ACK (ha.c:1570). Supersedes the "in-memory only" caveat for catch-up. | CLOSED (exceeds spec) | -| **6.1** heartbeats continue / gate on majority | superseded: continuous per-op `ha_has_write_quorum` (ha.c:2031) replaces COMMITTING-only check | CLOSED (via supersession) | -| **6.2** NOT_READY for non-repl opcodes | `ha_dispatch_gate` ha.c:2061-2110 (allowlist + `-XCE_ERR_NOT_READY` default) — now enforced for the whole term, not just COMMITTING | CLOSED (stronger) | -| **6.3** immediate promotion on majority | superseded by own-term gate + safe_committed_lsn (no COMMITTING dwell to exit) | CLOSED (via supersession) | -| **6.4** timeout → WARN + degraded | **OBSOLETE** — no COMMITTING state ⇒ no `committing_timeout_ms`. Degraded operation handled by write-quorum gate (rejects writes, keeps serving reads) rather than a one-shot 60s promotion fallback. | N/A (superseded) | -| **6.5** single-node immediate | `ha_has_write_quorum` returns true when `n_peers<=1` (ha.c:2037); safe_committed_lsn single-node path ha.c:486 | CLOSED | -| **7.1** catch-up start INFO | config_repl.c (send_wal_batch start log) | CLOSED | -| **7.2** converged INFO | ha.c:1869-1875 | CLOSED | -| **7.3** snapshot start/complete INFO | config_repl.c:1874-1876, 1939-1941 | CLOSED | -| **7.4** majority INFO | per-standby converged INFO (ha.c:1870); promotion is direct-to-PRIMARY INFO (ha.c:1036) | CLOSED | -| **7.5** timeout WARN | OBSOLETE with COMMITTING removal; quorum-loss edge logged once via `ha_log_quorum_transition` (ha.c:2076) instead | N/A (superseded by quorum-transition logging) | -| **Flow 0** safe commit point | `ha_compute_safe_committed_lsn` ha.c:480-540; truncate+baseline raft_cb_become_leader ha.c:898-923 | CLOSED | - -**Score: 31/33 original acceptance criteria CLOSED; 2 (6.4, 7.5) OBSOLETE by deliberate -architectural supersession (CTRL_PRIMARY_COMMITTING removed by ha-single-leader-role).** -Counting the superseded pair as satisfied-by-replacement (their intent — gate on majority, observe -degraded mode — is met more strongly), effective coverage is 33/33. No CRITICAL or HIGH functional -gap remains. - -## Residual / Deferred Items (all LOW, none blocking) - -1. **Documentation drift (HIGH-doc, LOW-code).** requirements.md Req 6 + design.md still describe - `CTRL_PRIMARY_COMMITTING`, `committing_timeout_ms`, and the 60s degraded-mode timeout, none of - which exist in code. The Introduction (requirements.md:5-7) and Adjacent-expectations also - reference COMMITTING. A reader of the spec will mis-model the live FSM. Code is correct; the - prose is stale. -2. **Req 5.2 "configurable" batch size** is compile-time (`#define`), not runtime config. Minor; - no operator has needed to tune it. Defensible as "configurable at build." -3. **`ha_compute_oldest_wal_lsn`** is an `opendir`/`readdir` scan per heartbeat reply per standby - in vtask context (config_repl.c:1946). Performance-only; fine at current cluster sizes. The - 2026-06-09 caching suggestion (Gap 3.4) was never actioned and remains optional. -4. **No covering tests** flagged by codegraph for `ha_process_standby_state`, - `ha_evaluate_repl_action`, `ha_dispatch_repl_action_for_standby`, `ha_send_snapshot_to_standby` - at the unit level (decision-loop logic is exercised via hooks in test_ha_role.c / - test_config_repl.c, and convergence via test_convergence_verification.c). Coverage is - integration-weighted, not absent — but the pure decision functions lack direct unit pins. -5. **E2E entity-state assertion** (2026-06-09 Recommendation 5): tests 6.5–6.7 historically - asserted LSN convergence only. With the data plane now real this is the right place to add a - "submitted job visible on promoted standby" assertion to guard against future regressions to - header-only application. Not done; LOW (data plane verified by unit round-trips per spec.json - note). - -## Effort / Risk for Open Work - -| Item | Option | Effort | Risk | -|---|---|---|---| -| 1 — Spec/design doc reconciliation (remove COMMITTING; document own-term gate + write-quorum + safe_committed_lsn as the actual Req 6 mechanism) | **A (recommended):** update requirements.md Req 6 + Introduction + design.md to describe the superseding mechanism and cross-link `ha-single-leader-role` / `ha-commit-durability`. **B:** leave a one-line "SUPERSEDED" banner pointing to those specs. | A: S (0.5–1d) · B: XS (<0.5d) | Low (docs only) | -| 2 — Promote batch limits to `ha_config_t` | A: plumb two fields. B: leave as build-time. | S (0.5d) | Low | -| 3 — Cache `oldest_wal_lsn` | A: cache + invalidate on segment GC. B: defer (current scale fine). | S (0.5d) | Low | -| 4 — Direct unit tests for decision functions | A: add `ha_evaluate_repl_action` truth-table + `ha_process_standby_state` convergence-flag tests. B: rely on hook/integration coverage. | S (1d) | Low | -| 5 — E2E entity-state assertion | A: extend test 6.7 to assert a submitted job is present post-failover. B: rely on unit payload round-trips. | S (0.5–1d) | Low | - -**Recommendation:** the spec is functionally complete and may remain `implemented`. The only item -worth scheduling is **Item 1 (doc reconciliation)** — the requirements/design now misdescribe the -live FSM after `ha-single-leader-role`, which will mislead future readers and any -`/kiro-verify-completion` run that checks prose against code. Items 2–5 are optional hardening. - ---- - -# Gap Analysis (re-validation) — 2026-06-24 - -> Re-validation pass. spec.json `phase=implemented` (all 41 sub-tasks `[x]`). This pass re-verifies -> every EARS criterion against the code **as on disk** after the heavy `ha-term-safety` rework -> (commits 004a40b…b77d412, tasks 1.1–4.3) plus bug-828/991/1028/1060/1067 fixes landed since the -> 2026-06-18 audit. Question answered: does this spec still hold, and how much of it is now owned by -> sibling specs? - -## Headline Finding - -**Still functionally complete; now LARGELY SUPERSEDED in mechanism, but its data-plane code is the -live load-bearing path.** Two distinct things are true at once: - -1. **The control-plane *design* of this spec (Req 2.3 / Req 6: `CTRL_PRIMARY_COMMITTING` role + - 60 s `committing_timeout_ms` degraded fallback + COMMITTING→PRIMARY-on-majority) is GONE.** - `grep -rn CTRL_PRIMARY_COMMITTING src/controller/*.{c,h}` → 0 hits; `committing_timeout` → 0 hits. - Removed by `ha-single-leader-role` and replaced by a stronger, continuous mechanism - (own-term WAL_NOOP + `writable_lsn` gate `ha_arm_own_term_gate` ha.c:107/1206; per-op write - quorum `ha_has_write_quorum` ha.c:2781 → `-XCE_ERR_NOT_READY`; vote-derived - `ha_compute_safe_committed_lsn` ha.c:510). This is **documentation drift**, not a functional gap — - identical to the 2026-06-18 finding and **still unactioned**. - -2. **The data-plane *code* of this spec (Req 1,3,4,5,7) is intact, hardened, and is the path every - sibling spec builds on.** It was NOT rewritten away — `ha-term-safety` routes through it: - - WAL catch-up payload apply: `ha_apply_single_entry` ha.c:2044 → `persist_wal_write_replicated` - (fsync, idempotent) ha.c:2056 THEN `apply_wal_entry` ha.c:2081 (entity restore). Config markers - also flow here (ha.c:2168). Header-only application remains gone. - - Batched streaming: `send_wal_batch` config_repl.c:2005, `WAL_STREAM_BATCH_ENTRIES`=256 / - `_BYTES`=64 KB (config_repl.c:1981-1982); batch flush loop config_repl.c:2071-2173. - - Snapshot raw payload: `ha_send_snapshot_to_standby` config_repl.c:2425 → - `xce_snapshot_chunk_req_raw` config_repl.c:2496 (real bytes). - - Per-standby convergence: `ha_process_standby_state` ha.c:2592 (sets - `standby_reported_tag/_lsn/_converged`, ha.c:2600-2607); reset on term entry ha.c:1115-1117; - fed from `OPCode_reply` stateReport at ha.c:4620. - - `safe_committed_lsn` (this spec's Flow 0, tasks 4.1/4.2) is *explicitly cited by code* as - ha-repl-hardening: ha.c:940 "Task 4.1 (ha-repl-hardening)". - - bug-1028 (FIXED 2026-06-23) was inside this very streaming path (`ha_stream_wal_to_standby` gained - a `const persist_t *p` param) — confirming the path is live and maintained, not dead. - -## Which sibling spec now owns what - -| Concern | Originating in ha-repl-hardening | Now owned / superseded by | -|---|---|---| -| Heartbeat stateReport (Req 1) | yes | retained as-is; consumed by `ha-heartbeat-committed-propagation` (adds committed+term stamp) | -| Per-standby convergence tracking (Req 2.1/2.2/2.5) | yes | retained as-is | -| Majority threshold `n_peers/2` (Req 2.3) | yes | reconciled + **absorbed** into `ha_has_write_quorum` / `safe_committed_lsn` (raft-commit-safety, ha-term-safety) | -| WAL catch-up payload apply (Req 3) | yes | retained; now also drives `ha-term-safety` config-marker apply + bug-991 lsn-advance fix | -| Snapshot transfer (Req 4) | yes | retained as-is | -| Batch WAL transfer (Req 5) | yes | retained; bug-1028 hardened the streamer signature | -| COMMITTING gate (Req 6) | yes | **SUPERSEDED & DELETED** by `ha-single-leader-role` (own-term gate) + `ha-commit-durability` (durable-commit fence) | -| Flow 0 safe commit point (tasks 4.1/4.2) | yes | retained; extended by `raft-commit-safety` (Figure-8) + `ha-term-safety` (per-entry term, (term,lsn) reconcile) — the LSN-only safe point is now term-aware upstream | - -## Requirement-to-Asset Map (delta vs 2026-06-18 — only changes noted) - -All 31 CLOSED criteria from 2026-06-18 remain CLOSED (assets re-verified, line numbers shifted +~860 -in ha.c / +~560 in config_repl.c due to ha-term-safety growth; spot-checked above). No regression. - -| Req / AC | Change since 2026-06-18 | Status | -|---|---|---| -| **3.x** payload apply | Path unchanged in shape; bug-991 fix made the three live-apply setters advance applied_lsn to the LAST-held LSN (correctness hardening). | CLOSED (hardened) | -| **5.x** batch streaming | bug-1028 added `const persist_t *p` to `ha_stream_wal_to_standby` (off-by-one / hdr-source fix). | CLOSED (hardened) | -| **Flow 0** safe commit | Now also gated by ha-term-safety per-entry term + bug-1060 promotion-truncate floor `max(safe_lsn,durable_tip)` (prevents new leader truncating its own committed log) + bug-1067 AE header-term separation. | CLOSED (strengthened upstream) | -| **6.4 / 7.5** COMMITTING timeout/WARN | Still OBSOLETE (no COMMITTING). | N/A (superseded) | - -**Score: 31/33 CLOSED, 2 OBSOLETE-by-supersession (6.4, 7.5). Effective 33/33. No CRITICAL/HIGH -functional gap. Unchanged from 2026-06-18, now re-confirmed post ha-term-safety.** - -## Disposition: RESCOPE (close as data-plane-complete + supersede the control-plane prose) - -This spec should **not** be reopened for new feature work and should **not** be deleted. Recommended -disposition: - -- **Keep `phase=implemented`.** Every acceptance criterion is satisfied or satisfied-by-replacement. -- **CLOSE-as-superseded the control plane (Req 2.3 + Req 6 entirely).** Their mechanism was - deliberately replaced. Carry a one-line "SUPERSEDED by ha-single-leader-role / ha-commit-durability" - banner on Req 6 and on the Introduction's COMMITTING references. -- **RETAIN the data plane (Req 1,3,4,5,7 + Flow 0) as the canonical owner.** No sibling spec - re-implements WAL-catch-up payload apply, batching, snapshot transfer, or stateReport — they all - consume this code. If anything, this spec is now the *foundation*, not the obsolete layer. - -The 2026-06-09 project-memory note ("data plane MISSING, critical, bugs 335-338") is **stale** — it -was closed by the reopened 7.x payload series (audited 2026-06-18) and is unaffected by later work. - -## Residual / Deferred (all LOW, none blocking — unchanged + re-confirmed) - -1. **Documentation drift (HIGH-doc, LOW-code) — STILL OPEN.** requirements.md Req 6 + Introduction + - design.md describe `CTRL_PRIMARY_COMMITTING` / `committing_timeout_ms` / 60 s degraded fallback, - none of which exist. This is the *only* item worth scheduling. A `/kiro-verify-completion` run that - checks prose-vs-code will flag Req 6 falsely. -2. **Req 5.2 "configurable" batch size** is compile-time `#define`, not `ha_config_t`. Defensible. -3. **`ha_compute_oldest_wal_lsn`** opendir/readdir per heartbeat reply per standby. Perf-only. -4. **Direct unit tests** for `ha_process_standby_state` / `ha_evaluate_repl_action` / - `ha_send_snapshot_to_standby` — coverage is integration-weighted (test_config_repl.c, - test_convergence_verification.c), not absent. -5. **E2E entity-state assertion** (submitted job visible on promoted standby post-failover) — the - live HA soak (`/xce-test-ha`) now exercises this end-to-end, partially discharging the gap; a - pinned unit/E2E assertion is still nice-to-have. - -## Effort / Risk for the one schedulable item - -| Item | Option | Effort | Risk | -|---|---|---|---| -| Doc reconciliation: remove COMMITTING from requirements.md Req 6 + Introduction + design.md; document own-term gate + write-quorum + safe_committed_lsn as the real Req 6 mechanism; cross-link ha-single-leader-role / ha-commit-durability / ha-term-safety | **A (recommended):** rewrite Req 6 + Introduction (S, 0.5–1d). **B:** one-line SUPERSEDED banner (XS, <0.5d). | A: S · B: XS | Low (docs only) | - -## Recommendation - -1. **No open implementation work.** Spec is functionally complete and is the live owner of the HA - replication data plane. Mark project-memory note (2026-06-09 "data plane MISSING") as resolved. -2. **Schedule only the doc reconciliation (Residual #1).** It has been flagged in two prior audits - (2026-06-09 doc-drift, 2026-06-18 Item 1) and remains unactioned. Either rewrite Req 6 (Option A) - or banner it (Option B). Everything else is optional hardening. -3. **Treat this spec as RESCOPED:** control plane CLOSED-as-superseded; data plane RETAINED-as-owner. - -## Research Needed - -- None new. Prior research items (typemap `Data` encoding, snapshot wire format, ownership boundary - with ha-config-bootstrap task 7.6 / live `repl_broadcast`) were all resolved by the payload series - and ha-term-safety task 3.4 (`repl_broadcast` now carries leader term, term-0 live send retired). diff --git a/.kiro/specs/ha-repl-hardening/spec.json b/.kiro/specs/ha-repl-hardening/spec.json deleted file mode 100644 index f7d8115c..00000000 --- a/.kiro/specs/ha-repl-hardening/spec.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "feature_name": "ha-repl-hardening", - "created_at": "2026-06-04T03:30:00Z", - "updated_at": "2026-06-09T17:45:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "Payload data plane tasks 7.1-7.7 complete 2026-06-10. Formula reconciled to n_peers/2. All unit tests green (lib wire round-trips + controller convergence). Snapshot E2E assertion gated on WAL accumulation infrastructure. Spec closed." -} diff --git a/.kiro/specs/ha-repl-hardening/tasks.md b/.kiro/specs/ha-repl-hardening/tasks.md deleted file mode 100644 index 826e1f9c..00000000 --- a/.kiro/specs/ha-repl-hardening/tasks.md +++ /dev/null @@ -1,303 +0,0 @@ -# Implementation Plan: HA Replication Hardening - ---- - -## Task Overview - -7 requirements, 6 design components. Progression: schema first → standby-side handlers → primary-side decision loop → post-election safety → integration cleanup → validation. - ---- - -- [x] 1. Foundation: Schema changes and state tracking fields -- [x] 1.1 Add `stateReport` group to `RequestReply.data` union in controller.capnp - - Add `appliedLsn :UInt64 @15` and `configTag :UInt64 @16` fields inside a `stateReport :group` variant - - Regenerate capnp bindings; verify compilation succeeds - - Existing decode paths and opcode dispatch unaffected - - Compilation succeeds and existing controller tests remain green - - _Requirements: 1.1, 1.3_ - - _Boundary: Wire Protocol (controller.capnp)_ - -- [x] 1.2 Extend `WalCatchupEntry` for batch support in controller.capnp - - Add `entries :List(WalCatchupEntry)` field for multiple entries per message - - Regenerate capnp bindings; verify compilation - - Compilation succeeds; single-entry decode path still works - - _Requirements: 5.1_ - - _Boundary: Wire Protocol (controller.capnp)_ - -- [x] 1.3 Add snapshot chunk fields to capnp schema - - Add `checkpointLsn`, `chunkIndex`, `chunkOffset`, `totalSize`, `isLast` fields to the snapshot transfer message - - No resumption fields — restart-on-reconnect policy means no snapshot progress state in `stateReport` - - Regenerate capnp bindings; verify compilation - - Compilation succeeds - - _Requirements: 4.1, 4.2_ - - _Boundary: Wire Protocol (controller.capnp)_ - -- [x] 1.4 Add per-standby convergence tracking fields to `ha_state_t` - - Add `standby_reported_tag[MAX_HA_CANDIDATES]`, `standby_reported_lsn[MAX_HA_CANDIDATES]`, `standby_converged[MAX_HA_CANDIDATES]` arrays - - Zero-initialize in `ha_init()`; these fields are read by the convergence evaluator in task 3.1 - - Unit test: verify fields are zero after `ha_init()` and writable from test context - - _Requirements: 2.1_ - - _Boundary: Controller FSM (ha.h)_ - ---- - -- [x] 2. Standby-side: Wire WAL catch-up and heartbeat state reporting -- [x] 2.1 (P) Wire `OPCode_walCatchupEntry` single-entry handler on standby - - Replace the stub in the dispatch switch (returns 0) with a call to `apply_wal_entry()` for single entries - - On success, advance `applied_lsn` atomically; on error, log ERROR and skip — do not crash - - Standby remains in `CTRL_STANDBY` throughout; does not accept job dispatch during catch-up - - `applied_lsn` is in-memory only; on restart standby reports `lsn=0` and re-bootstraps via heartbeat loop - - Unit test: apply a single WAL entry, verify `applied_lsn` advances; error entry is skipped, not crashed - - _Requirements: 3.1, 3.3, 3.4, 3.5_ - - _Boundary: Standby Dispatch (cluster.c / ha.c)_ - -- [x] 2.2 Handle batch WAL catch-up entries on standby - - Extend the `OPCode_walCatchupEntry` handler from task 2.1 to decode the `entries` list for batch messages - - Apply each entry in order via `apply_wal_entry()`; advance `applied_lsn` only after the full batch is durably written - - Standby remains in `CTRL_STANDBY` during batch apply; `applied_lsn` not persisted to disk - - Unit test: apply a batch of 3 entries, verify `applied_lsn` advances by 3; partial batch (< max) applied correctly - - _Requirements: 5.3, 5.5_ - - _Boundary: Standby Dispatch (cluster.c / ha.c)_ - - _Depends: 1.2, 2.1_ - -- [x] 2.3 (P) Populate `stateReport` in heartbeat reply on standby - - In the `OPCode_nop` handler (`cluster.c`), replace `data_which = nodata` with `data_which = stateReport` - - Populate `appliedLsn` from `ha->applied_lsn` and `configTag` from `myconf->config_version_tag` - - Unit test: verify reply carries correct `applied_lsn` and `config_tag` after WAL catch-up - - _Requirements: 1.1, 1.3, 1.4_ - - _Boundary: Standby Heartbeat (cluster.c)_ - - _Depends: 1.1_ - ---- - -- [x] 3. Primary-side: Unified heartbeat-driven replication decision loop -- [x] 3.1 Implement per-standby convergence tracking update - - On each heartbeat reply from a standby, extract `appliedLsn` and `configTag` from `stateReport` - - Store in `standby_reported_tag[idx]` and `standby_reported_lsn[idx]` - - Evaluate convergence: `reported_tag >= committed_tag AND reported_lsn >= committed_lsn` - - Set `standby_converged[idx] = true/false`; log INFO when a standby first transitions to converged - - Unit test: send heartbeat reply with `lsn=N`, verify `standby_reported_lsn[idx]=N` and convergence flag updated correctly - - _Requirements: 1.2, 2.2, 2.5_ - - _Boundary: Primary Heartbeat Handler (ha.c)_ - - _Depends: 1.1, 1.4_ - -- [x] 3.2 Implement unified decision loop in heartbeat reply processing - - After updating per-standby tracking, evaluate: tag stale → push config; `lsn < oldest_wal_lsn` → send snapshot; `lsn < committed_lsn` → stream WAL; both current → converged - - One action per heartbeat cycle per standby; `stream_wal_entries()` enqueues all batches non-blocking via `xce_request()` in a single invocation — never blocks vtaskmgr thread - - Log decision at INFO: which standby, which action triggered - - Unit test: decision function returns `PUSH_CONFIG` / `SEND_SNAPSHOT` / `STREAM_WAL` / `CONVERGED` for each representative `(tag, lsn)` input - - _Requirements: 2.2, 2.3, 2.4_ - - _Boundary: Primary Heartbeat Handler (ha.c)_ - - _Depends: 3.1_ - -- [x] 3.3 Trigger per-standby config push from decision loop - - When `standby_tag < committed_tag`, call `push_config_to_standbys()` targeting the specific standby - - Reuse existing config push path; trigger is now heartbeat decision loop instead of `handle_ctrlhello` - - After config push, primary skips further decisions for this standby until the next heartbeat reply confirms the updated tag - - Unit test: decision loop with stale tag triggers config push; subsequent heartbeat with updated tag does not re-push - - _Requirements: 1.4_ - - _Boundary: Config Replication (config_repl.c)_ - - _Depends: 3.2_ - -- [x] 3.4 Implement batch WAL streaming from decision loop - - When `standby_lsn < committed_lsn AND standby_lsn >= oldest_wal_lsn`, stream entries `[standby_lsn+1, committed_lsn]` - - Accumulate entries up to 64KB/256 entries per batch; enqueue all batches via `xce_request()` in a single non-blocking invocation — do not block vtaskmgr thread - - Send each batch as `OPCode_walCatchupEntry` with the `entries` list; partial final batch sent immediately (no wait for fill) - - Unit test: stream 500 entries, verify 2 batches sent; all entries applied in LSN order on standby - - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_ - - _Boundary: WAL Streaming (config_repl.c)_ - - _Depends: 1.2, 3.2_ - -- [x] 3.5a Primary-side snapshot chunk sender (ha_send_snapshot_to_standby) - - Read snapshot file, split into 64KB chunks, enqueue each via xce_request() with correct metadata - - Chunk fields: snapshotLsn, chunkIndex, chunkOffset, totalSize, isLast; payload=raw_chunk_bytes (wired via xce_snapshot_chunk_req_raw, task 7.2) - - Unit test: 128KB file → 2 chunks with correct offsets; small file → 1 chunk - - _Requirements: 4.1, 4.2_ - - _Boundary: Snapshot Transfer (config_repl.c)_ - - _Depends: 1.3_ - -- [x] 3.5b Fix SEND_SNAPSHOT production wiring and oldest_wal_lsn - - Use persist.checkpoint_lsn (not committed_lsn) for snapshot file lookup - - Stat the snapshot file to get actual total_size (not hardcoded 131072) - - Wire persist.oldest_wal_lsn into ha_evaluate_repl_action call site (remove the hardcoded 0) - - Unit test: verify decision loop triggers SEND_SNAPSHOT when standby_lsn < oldest_wal_lsn - - _Requirements: 4.1, 4.3_ - - _Boundary: Controller FSM (ha.c)_ - - _Depends: 3.5a, 3.2_ - -- [x] 3.5c Standby-side snapshot receive handler (OPCode_snapshotTransfer) - - Add case OPCode_snapshotTransfer in myreq_handler dispatch (cluster.c) - - Write each chunk payload to snap.part temp file; on isLast=true, verify total size, atomic rename to snap-{lsn}.bin, load via load_snapshot() - - Set applied_lsn = checkpoint_lsn (in-memory) after successful load - - On mid-transfer reconnect: discard snap.part (restart from chunk 0 on next reconnect) - - Unit test: receive 2 chunks, verify file assembled and renamed correctly - - _Requirements: 4.2, 4.3, 4.4_ - - _Boundary: Standby Dispatch (cluster.c, ha.c)_ - - _Depends: 1.3, 3.5a_ - -- [x] 3.5d Snapshot transfer integration and dispatch-path tests - - Test: SEND_SNAPSHOT dispatch path triggers hook (like PUSH_CONFIG/STREAM_WAL tests in test_ha_role.c) - - Test: standby receive → snap.part assembled → isLast rename → applied_lsn set - - Test: reconnected standby discards snap.part (restart from chunk 0) - - All tests pass under ASAN - - _Requirements: 4.3, 4.4, 4.5_ - - _Boundary: Integration (ha.c, cluster.c, test_ha_role.c, test_config_repl.c)_ - - _Depends: 3.5b, 3.5c_ - ---- - -- [x] 4. Post-election safety: Safe commit point from vote replies -- [x] 4.1 Determine `safe_committed_lsn` from vote replies in `raft_cb_become_leader` - - `ha_elect_ctx_t.votes[]` is already populated by the vote reply handler (`peerLastLogLsn`, `peerConfigTag` per voter); this task reads those fields after winning election - - Compute `safe_committed_lsn` = max LSN held by a majority of voters (iterate all voters with `granted=true`) - - Compute `safe_config_tag` from `peerConfigTag` using the same majority logic - - Unit test: voters at LSNs `[99, 100]`, self at `100` → `safe_committed_lsn=100`; voters at `[50, 51]` → `safe_committed_lsn=51` - - _Requirements: 2.3_ - - _Boundary: Controller FSM (ha.c)_ - -- [x] 4.2 Replace stale-ctrlhello truncation with vote-derived safe commit point - - Remove `persist_truncate_wal_above(last_known_committed_lsn)` from the become-leader path - - Replace with `persist_truncate_wal_above(safe_committed_lsn)` using the value from task 4.1 - - Set `committed_lsn = safe_committed_lsn` after truncation - - Unit test: verify entry at `safe_committed_lsn` is NOT truncated; entry above it IS - - _Requirements: 2.3_ - - _Boundary: Controller FSM (ha.c)_ - - _Depends: 4.1_ - ---- - -- [x] 5. Integration: Simplify ctrlhello and harden COMMITTING gate -- [x] 5.1 Reduce `handle_ctrlhello` to connection setup only - - Remove config push, snapshot check, and WAL streaming from `handle_ctrlhello` - - Keep: set `primary_ctrl_fd`, call `ha_on_heartbeat_received`, arm liveness watchdog (`ha_arm_heartbeat_timeout_vtask`) - - All replication decisions handled by the heartbeat decision loop (tasks 3.2–3.5) - - Unit test: call `handle_ctrlhello`, verify no `OPCode_replConfigPush` or `OPCode_walCatchupEntry` messages enqueued - - _Requirements: 1.1, 1.2_ - - _Boundary: Connection Setup (config_repl.c)_ - -- [x] 5.2 Wire majority-convergence → PRIMARY transition - - In the heartbeat reply handler, after updating convergence tracking, count `standby_converged` and compare to majority threshold - - Threshold: `count >= n_peers / 2` (C integer division — for odd n the primary + - converged standbys form a cluster majority; n=3→1, n=5→2, n=1→0). - Raft rejects even n_peers > 1. *(reconciled 2026-06-09, task 7.5)* - promote to `CTRL_PRIMARY` *(formula reconciled 2026-06-09, task 7.5)* - - Single-node: `n_peers <= 1`, threshold = 0 — satisfied immediately - - Log INFO: "majority converged (N/M standbys, threshold T) — CTRL_PRIMARY" - - Unit test: 3-node — 0/2 → COMMITTING; 1/2 → PRIMARY (majority); 5-node — 1/4 → COMMITTING; 2/4 → PRIMARY; single-node → immediate - - _Requirements: 2.3, 2.4, 6.1, 6.3, 6.5_ - - _Boundary: Controller FSM (ha.c)_ - - _Depends: 3.1_ - -- [x] 5.3 Extend committing timeout to degraded-mode safety net - - Change `committing_timeout_ms` default from `10000` to `60000` - - Timeout handler logs WARN with per-standby convergence state, promotes in degraded mode - - Majority-convergence path (task 5.2) is the PRIMARY promotion mechanism; timeout is the fallback - - WARN log includes: number of standbys converged and number still pending - - _Requirements: 6.4_ - - _Boundary: Controller FSM (ha.c)_ - -- [x] 5.4 Add observability log statements - - WAL catch-up start: INFO with start/target LSN — covered by unit test in task 6.2 - - Standby converged: INFO with identity, config_tag, applied_lsn — covered by unit test in task 6.1 - - Snapshot transfer start/complete: INFO with checkpoint_lsn, size — covered by unit test in task 6.4 - - Majority reached → CTRL_PRIMARY: INFO with converged count and quorum size — covered by unit test in task 5.2 - - Committing timeout: WARN with per-standby state — covered by unit test in task 5.3 - - All 5 log statements visible in test output; `meson test --suite controller` green - - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5_ - - _Boundary: Logging (ha.c, config_repl.c)_ - ---- - -- [x] 6. Validation: Tests and integration verification -- [x] 6.1 Unit tests for decision loop and convergence tracking - - Test: `(tag=0, lsn=0)` → PUSH_CONFIG; `(tag=T, lsn=0)` → STREAM_WAL; `(tag=T, lsn=N)` → CONVERGED - - Test: majority detection (0/2 → not majority; 1/2 → majority for 3-node; per n_peers/2 formula) - - Test: heartbeat reply populates `stateReport` with correct values - - All tests pass under ASAN - - _Requirements: 1.1, 1.2, 2.2, 2.3, 2.4_ - -- [x] 6.2 Unit tests for WAL catch-up and batch handling - - Test: apply single entry, apply batch of 256, error entry skipped, `applied_lsn` advances correctly - - Test: batch size limit enforced; `applied_lsn` only advances after full batch written - - All tests pass under ASAN - - _Requirements: 3.1, 3.3, 3.4, 3.5, 5.3, 5.5_ - -- [x] 6.3 Unit tests for safe commit point from vote replies - - Test: voters at LSNs `[99, 100]`, self at `100` → `safe_committed_lsn=100` - - Test: voters at LSNs `[50, 51]`, self at `50` → `safe_committed_lsn=51` (majority has 51) - - Test: single node → `safe_committed_lsn = self.applied_lsn` - - All tests pass under ASAN - - _Requirements: 2.3_ - -- [x] 6.4 Unit tests for snapshot chunk transfer - - Test: 128KB snapshot in 2 chunks → standby receives, verifies total size, atomically renames, loads; `applied_lsn = checkpoint_lsn` - - Test: mid-transfer reconnect → standby discards `snap.part`, primary restarts from chunk 0 on reconnect - - Test: snapshot transfer only triggered when `standby_lsn < oldest_wal_lsn` - - All tests pass under ASAN - - _Requirements: 4.1, 4.2, 4.3, 4.4_ - -- [x] 6.5 Integration test: 3-node startup via majority convergence - - Requires task 5.3 (`committing_timeout_ms=60000`) deployed so timeout does not fire during test - - Deploy via `/xce-deploy-ha`; all 3 nodes start within 15s - - Verify primary enters `CTRL_PRIMARY` via majority-convergence INFO log (not committing timeout WARN) - - Verify standby with `config_tag=0` receives config push, then WAL, then reports converged - - Verify no degraded-mode WARN in logs - - _Requirements: 6.1, 6.3, 6.5, 7.4_ - -- [x] 6.6 Integration test: snapshot transfer for lagging standby - - Start cluster, let WAL accumulate, kill one standby - - Trim WAL on primary so `oldest_wal_lsn > standby's last applied_lsn` - - Restart standby; verify snapshot transfer triggered (INFO log: checkpoint_lsn, size) and completes - - Verify standby converges and enters `CTRL_STANDBY` - - _Requirements: 4.1, 4.2, 4.5_ - -- [x] 6.7 E2E test: full cluster lifecycle - - Deploy 3 nodes via `/xce-deploy-ha`; verify single PRIMARY, zero degraded mode - - Kill primary; verify standby promotes via RDHUP → election → majority convergence - - Verify promoted primary serves from committed state (no data loss) - - Full ASAN build: `meson test --suite controller` green - - _Requirements: 6.2, 6.3, 6.4_ - ---- - -## Reopened 2026-06-09 — Gap Analysis Validation Pass - -The 2026-06-09 gap analysis (research.md) found the replication **data plane missing**: -snapshot chunks ship with `payload = NULL` (config_repl.c "encoding deferred"; cluster.c) -and WAL catch-up applies entries header-only (`payload_len=0`), so `apply_wal_entry` -advances LSN without restoring entities. Standbys report "converged" without holding the -state. Tests 6.5–6.7 pass because they assert LSN convergence only. The spec MUST NOT be -marked `implemented` until this series lands. - -- [x] 7. Replication payload data plane -- [x] 7.1 Define the payload wire format: capnp `Data` field + typemap codegen for WAL - entries and snapshot chunks. Coordinate with ha-config-bootstrap task 7.6 — live - `repl_broadcast` is also a no-send stub and must share the same format. - - _Requirements: 3.1, 4.2, 5.3_ - - _Boundary: Wire format (controller.capnp, config_repl.c)_ -- [x] 7.2 Snapshot chunk payload transmission (primary) + chunk write to `snap.part` - (standby); verify total size before atomic rename (phased option: hand-wire raw bytes - first if typemap codegen is blocked) - - _Requirements: 4.1, 4.2, 4.3_ - - _Boundary: Snapshot transfer (config_repl.c, cluster.c)_ -- [x] 7.3 WAL catch-up entry payload transmission + entity restoration in - `apply_wal_entry` (e.g. `WAL_JOB_SUBMIT` recreates the job); `applied_lsn` advances - only after the entity is applied - - _Requirements: 3.1, 3.4, 5.3_ - - _Boundary: WAL catch-up (ha.c, config_repl.c, persist.c)_ -- [x] 7.4 Re-zero convergence tracking arrays (`standby_reported_*`, - `standby_converged`) on entry to CTRL_PRIMARY_COMMITTING — stale convergence from a - prior term must not satisfy the gate after re-election - - _Requirements: 2.1_ - - _Boundary: Convergence tracking (ha.c)_ -- [x] 7.5 Reconcile the convergence threshold: code requires ALL standbys (`n_peers−1`), - requirements say `ceil(n/2)−1`, tasks say `ceil(n/2)`, code was `n_peers−1` — reconciled to `n_peers / 2` (C integer division), recorded in - Decision Log 2026-06-09 - - _Requirements: 2.4, 6.1_ - - _Boundary: Quorum policy (ha.c, design.md)_ -- [x] 7.6 Production `unlink(snap.part)` on mid-transfer reconnect (Req 4.4) and fresh - checkpoint creation at snapshot start instead of reusing the old checkpoint (Req 4.1) - - _Requirements: 4.1, 4.4_ - - _Boundary: Snapshot lifecycle (config_repl.c)_ - - _Requirements: 6.2, 6.4_ - - _Boundary: Integration test (deploy-ha)_ diff --git a/.kiro/specs/ha-single-leader-role/brief.md b/.kiro/specs/ha-single-leader-role/brief.md deleted file mode 100644 index b22e862c..00000000 --- a/.kiro/specs/ha-single-leader-role/brief.md +++ /dev/null @@ -1,25 +0,0 @@ -# Brief: ha-single-leader-role - -> Sub-spec 3 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> Authoritative requirements/design/research: `.kiro/specs/ha-consistency-invariants/` (design.md §HA Role Lifecycle + role lifecycle diagram). This brief scopes the slice; do not re-derive decisions. -> **HARD ORDERING**: must land AFTER ha-commit-barrier-rollout. Removing the transitional state before the universal barrier opens a window where a fresh primary accepts unreplicated writes. - -## Problem -`CTRL_PRIMARY_COMMITTING` is a transitional leader state Raft does not have: it causes a redirect loop (`ha_should_redirect` points clients at a primary that doesn't exist yet), `committing_timeout_fn` force-promotes after 60s even with zero converged standbys (violating the commit rule outright), and `ha_is_primary` lies during the window. - -## Desired Outcome -Raft's three-state model: election winner becomes `CTRL_PRIMARY` directly; post-election write safety comes from committing a `WAL_NOOP` entry of the current term (own-term commit rule §5.4.2) via the commit engine's `writable_lsn` gate; no wall-clock path ever changes leadership or confirms operations. - -## Scope -- **In**: `WAL_NOOP` entry type (empty payload, replay no-op); `raft_cb_become_leader` rewrite (direct PRIMARY, hook registration moved here, append+broadcast no-op, `commit_set_writable_lsn`); deletions — `CTRL_PRIMARY_COMMITTING` enum, dispatch-gate COMMITTING branch, `committing_timeout_fn` + watchdog, `ha_check_majority_convergence`, `repl_ack_count`/`ha_repl_ack_received`; `ha_is_primary`/`ha_should_redirect`/vote-deny simplification; legacy `ha_elect_evaluate_and_signal` (XCE_TESTING path) updated symmetrically; test-fixture sweep (broad — every test referencing COMMITTING); `testsuite/test_ha_role_lifecycle.c`. -- **Out**: election/vote logic itself (raft-library); demotion behavior (ha-convergence-recovery); barrier mechanics (specs 1–2). - -## Requirements (umbrella IDs) -1.1, 1.2, 1.3, 1.4, 1.5, 3.1, 3.2, 3.3. - -## Upstream / Downstream -- **Upstream**: ha-commit-engine (writable_lsn gate), ha-commit-barrier-rollout (universal barrier in force). -- **Downstream**: ha-convergence-recovery E2E relies on the simplified lifecycle; ha-async-io.md steering sync (roadmap item) after this lands. - -## Constraints -The COMMITTING reference sweep spans ha.c, cluster.c, steering docs, and many test fixtures — mechanical but must be complete; grep-verified zero references at completion. diff --git a/.kiro/specs/ha-single-leader-role/design.md b/.kiro/specs/ha-single-leader-role/design.md deleted file mode 100644 index 0aeb93a5..00000000 --- a/.kiro/specs/ha-single-leader-role/design.md +++ /dev/null @@ -1,338 +0,0 @@ -# Design: ha-single-leader-role - -> Sub-spec 3 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **HA Role Lifecycle** component and role-lifecycle state diagram (`.kiro/specs/ha-consistency-invariants/design.md`), implementing umbrella Migration Strategy phase 3 ("Promotion no-op + own-term gate; then delete COMMITTING (+ watchdog, convergence checker); redirect fix"). Discovery is complete via the umbrella's `research.md` plus this spec's slice-verification pass (`research.md` here — verified anchors and the COMMITTING census). The commit engine interface (`include/commit.h`) is consumed exactly as shipped by ha-commit-engine — this spec is the designated first caller of `commit_set_writable_lsn` and never modifies engine internals. The classification table and dispatch-gate fact allowlist shipped by ha-commit-barrier-rollout are preserved untouched except for the additions/deletions they themselves pre-declared for this spec. Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping). -> **HARD ORDERING**: implementation must not begin until ha-commit-barrier-rollout is implemented and green — removing the transitional state before the universal barrier is in force opens a window where a fresh primary accepts unreplicated writes. - -## Overview - -**Purpose**: This feature gives cluster operators Raft's three-state leadership model: an election winner becomes `CTRL_PRIMARY` directly, post-election write safety comes from committing a `WAL_NOOP` entry of the current term through the commit engine's `writable_lsn` gate (Raft §5.4.2), and no wall-clock path can ever change leadership or confirm operations. - -**Users**: Cluster operators stop seeing redirect loops to a primary that does not exist yet, dishonest "not primary" status from the node that won the election, and 60-second force-promotions that violate the commit rule with zero converged standbys. Downstream, ha-convergence-recovery gains the simplified lifecycle its E2E validation assumes. - -**Impact**: Rewrites `raft_cb_become_leader` (direct PRIMARY, hook registration moved here, no-op append+broadcast, `commit_set_writable_lsn`); deletes `CTRL_PRIMARY_COMMITTING` and everything that existed only to serve it — the dispatch-gate COMMITTING branch, `committing_timeout_fn` + watchdog, `ha_check_majority_convergence`, `repl_ack_count`/`ha_repl_ack_received`; collapses every `PRIMARY || COMMITTING` compound check; updates the legacy XCE_TESTING election path symmetrically; sweeps every COMMITTING reference across ha.c, ha.h, cluster.c, raft.c comments, ten test fixtures, and four documentation files to grep-verified zero. - -### Goals -- Election winner assumes `CTRL_PRIMARY` directly; the role enum has exactly one leader value (1.1, 1.2) -- Promotion appends a `WAL_NOOP` of the current term and arms the engine's own-term gate; confirmation of any operation waits behind the no-op's commit (2.1, 2.2) -- Zero wall-clock paths that change leadership or confirm operations — watchdog and ack/convergence counters deleted without replacement (1.5, 2.3) -- A live primary never redirects and always self-reports as primary, even before its own-term entry commits (1.3, 1.4) -- Grep-verified zero references to the transitional state across code, tests, and docs (1.2) - -### Non-Goals -- Election/vote correctness, term management, freshness gates (raft-library — untouched except one stale comment) -- Commit-engine internals: gate enforcement mechanics, per-LSN evaluation, admission cap (ha-commit-engine — consumed) -- Operation classification, reply ordering, fact allowlist contents (ha-commit-barrier-rollout — preserved) -- Demotion truncate+restart, rejoin reconciliation, egress filter, ACK-driven catch-up, quorum logging (ha-convergence-recovery) -- Catch-up decision loop changes — `ha_process_standby_state` and `ha_dispatch_repl_action_for_standby` are unchanged - -## Boundary Commitments - -### This Spec Owns -- The promotion sequence on election win (both the raft callback and the legacy XCE_TESTING evaluation path) and the single shared promotion helper realizing it -- The `WAL_NOOP` entry type: enum value, append/broadcast at promotion, replay no-op, classification row -- The lifetime of `CTRL_PRIMARY_COMMITTING` and all machinery existing solely for it (watchdog, convergence checker, ack counters, gate branch, config knob) — i.e., their complete removal -- The role-predicate truth contract: `ha_is_primary`/`ha_should_redirect`/vote-deny/heartbeat re-arm semantics under the three-state model -- `testsuite/test_ha_role_lifecycle.c` and the disposition of every test fixture referencing the deleted state -- Synchronization of `ha-async-io.md` (dispatch-gate table, replication-consistency section, heartbeat re-arm, Raft mapping) and the other three documentation references - -### Out of Boundary -- `commit.c` / `include/commit.h` — consumed, never modified (the `writable_lsn` gate semantics are engine-owned) -- `wal_classify` dispatch mechanics and the fact allowlist in the gate's PRIMARY branch (ha-commit-barrier-rollout) — this spec adds the pre-declared WAL_NOOP row and deletes the pre-declared COMMITTING branch, nothing else in either -- `raft_cb_become_follower` demotion semantics beyond deleting its committing-timeout cancel block (truncate+self-restart is ha-convergence-recovery) -- Standby-side apply, ACK wire format, catch-up streaming (ha-wal-replication) -- `ha_process_standby_state` convergence bookkeeping and the per-standby replication decision loop (kept as-is; only its promotion consumer dies) - -### Allowed Dependencies -- `commit_submit(ce, lsn, COMMIT_MODE_RETAIN, 0)` and `commit_set_writable_lsn(ce, noop_lsn)` on `cluster_t.commit` (ha-commit-engine, P0) -- `persist_wal_async` append+broadcast primitive (persist.c:295; broadcasts via `repl_broadcast` at persist.c:358) and `apply_wal_entry`/`replay_wal` (P0) -- `wal_classify` classification seam + totality guard (ha-commit-barrier-rollout, P0) -- Existing promotion-adjacent machinery reused unchanged: safe-LSN truncation at become-leader, `cluster_init_phase2`, vote-endpoint promotion, config push, ctrlhello, `ha_start_primary_heartbeat` -- Threading rules per `ha-async-io.md`: `raft_cb_become_leader` runs where the raft FSM fires today (no new blocking; `persist_wal_async` and `commit_submit(RETAIN)` are non-blocking) -- Dependency direction: ha.c → persist.c append → commit engine; the engine calls back into nothing - -### Revalidation Triggers -- Any change to `commit_set_writable_lsn` semantics or the engine's NOT_READY contract (upstream) → re-verify promotion ordering and 2.2 behavior -- ha-convergence-recovery's demotion rewrite of `raft_cb_become_follower` → must not resurrect any transitional state; lifecycle test asserts the role set is closed -- Any new role added to `ctrl_role_t` → dispatch gate, vote gates, heartbeat re-arm, and the lifecycle test must be revisited -- New WAL entry types → classified by the barrier-rollout guard (unchanged trigger; WAL_NOOP row is now part of the table) - -## Architecture - -### Existing Architecture Analysis -- Production promotion today: `raft_cb_become_leader` (ha.c:445) truncates to safe LSN, runs phase2, stores `CTRL_PRIMARY_COMMITTING`, resets `repl_ack_count`, promotes vote endpoints, pushes config + ctrlhello, starts heartbeat, arms `committing_timeout_fn` (60s force-promote). Full PRIMARY is reached later by one of three paths: `ha_check_majority_convergence` (heartbeat stateReports), `ha_repl_ack_received` (replCatchUpDone counting), or the watchdog — all three call `ha_promote_to_primary` (term++ via `ha_do_promote`, hook registration, heartbeat re-arm) -- Defects removed (verified, brief + umbrella audit): redirect loop while COMMITTING (`ha_should_redirect` ≠ PRIMARY → redirect to a primary that doesn't exist yet); `ha_is_primary` false on the elected node; watchdog promotion with zero converged standbys violating the commit rule; double term increment (raft-synced term at become-leader + `ha_do_promote`'s `term++` at convergence) -- The replacement guarantee already exists upstream: every confirmation path goes through the commit engine (barrier-rollout), and the engine withholds completion until `committed_lsn >= writable_lsn` once the gate is set — the transitional state's only legitimate purpose (don't confirm until converged) is provided correctly by Raft's own-term rule -- Legacy XCE_TESTING election path (`ha_elect_evaluate_and_signal`, ha.c:2141) mirrors the COMMITTING block (ha.c:2247-2300) and must be updated symmetrically — tests drive lifecycle through it - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph raft_library - RAFT[raft FSM<br/>on_become_leader] - end - subgraph ha_role_lifecycle[ha.c role lifecycle - this spec] - BL[raft_cb_become_leader<br/>rewritten] - HELPER[ha promotion sequence<br/>shared helper] - LEGACY[ha_elect_evaluate_and_signal<br/>XCE_TESTING path] - GATE[ha_dispatch_gate<br/>COMMITTING branch deleted] - PRED[role predicates<br/>is_primary, should_redirect, vote gates] - end - subgraph persist_repl[persist.c / repl.c - consumed] - NOOP[persist_wal_async WAL_NOOP<br/>append + broadcast] - CLS[wal_classify<br/>WAL_NOOP → RETAIN row] - end - subgraph commit_engine[commit engine - consumed] - REG[commit_submit RETAIN] - WL[commit_set_writable_lsn] - end - STBY[standbys ACK] - - RAFT --> BL - BL --> HELPER - LEGACY --> HELPER - HELPER -->|CTRL_PRIMARY + hooks + heartbeat| PRED - HELPER --> NOOP - NOOP --> CLS - HELPER --> REG - HELPER --> WL - NOOP --> STBY - STBY -->|walAck ≥ noop_lsn| commit_engine -``` - -**Architecture Integration** (sliced from umbrella): -- Selected pattern: one shared promotion sequence with two entry points (raft callback, legacy evaluation) — no second promotion flavor survives; write-safety is delegated entirely to the engine gate rather than a role sub-state -- Existing patterns preserved: safe-LSN truncation, phase2 ordering, endpoint promotion, config push + ctrlhello, heartbeat arming, catch-up decision loop, `errnox` early-return style, barrier-rollout's gate allowlist -- Removal rationale (umbrella): the lifecycle diagram has no COMMITTING node; "no other component may observe COMMITTING after this spec" -- Steering compliance: no blocking added on iomgr or in the raft callback; TDD with testsuite/ tests first; ha-async-io.md synchronized as part of this spec - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), stdatomic | promotion rewrite, deletions | no new dependencies, no .capnp changes | -| Data / Storage | existing WAL (persist.c) | `WAL_NOOP = 40` entry, empty payload | on-disk format unchanged (new type value only, below WAL_ENTRY_TYPE_MAX) | -| Build / Test | meson, testsuite/ pattern | test_ha_role_lifecycle.c; two test files deleted | XCE_TESTING seams | - -## File Structure Plan - -### Directory Structure -``` -src/controller/ -├── ha.c # MODIFIED — promotion rewrite, all deletions, predicate collapse -├── ha.h # MODIFIED — enum value, fields, defaults, decls removed; comments -├── cluster.c # MODIFIED — committing_timeout_ms init removed; comment sweep -├── persist.c # MODIFIED — apply_wal_entry WAL_NOOP case; wal_classify row -├── meson.build # MODIFIED — test list: -test_ha_repl_ack.c, -test_ha_evaluate_committing.c, -│ # +test_ha_role_lifecycle.c -└── testsuite/ - ├── test_ha_role_lifecycle.c # NEW — direct promotion, noop+gate, predicate honesty, no watchdog - ├── test_ha_repl_ack.c # DELETED — subject machinery removed - ├── test_ha_evaluate_committing.c # DELETED — subject machinery removed - └── (8 fixtures) # MODIFIED — COMMITTING expectations retargeted (census in research.md) -src/include/wal_types.h # MODIFIED — WAL_NOOP = 40 -src/lib/raft/raft.c # MODIFIED — stale comment only (line 123) -.kiro/steering/ha-async-io.md # MODIFIED — steering sync (4 sites) -src/controller/CLAUDE.md # MODIFIED — role mapping table (line 40) -.claude/skills/xce-deploy-ha/SKILL.md # MODIFIED — expected log line (line 98) -src/docs/cluster-config-split-analysis.md # MODIFIED — config listing (line 34) -``` - -### Modified Files (what and why) -- `src/controller/ha.c` — the promotion rewrite and every deletion (verified anchor table in research.md §3); net-negative LOC -- `src/controller/ha.h` — remove `CTRL_PRIMARY_COMMITTING`, `HA_DEFAULT_COMMITTING_TIMEOUT_MS`, `committing_timeout_ms`, `repl_ack_count`, `committing_timeout_ctx`, decls of the two deleted functions; comment sweep; `standby_reported_*`/`standby_converged` arrays stay -- `src/controller/cluster.c` — drop `.committing_timeout_ms` initializer (cluster.c:899); update the ctrlhello comment (cluster.c:1402-1410) -- `src/controller/persist.c` — `apply_wal_entry` gains `case WAL_NOOP: break;` (LSN-advance/ACK via the existing common path); `wal_classify` gains the WAL_NOOP → `WAL_MODE_RETAIN` row pre-declared by barrier-rollout -- `src/include/wal_types.h` — `WAL_NOOP = 40` with comment (internal, empty payload, replay no-op, promotion-only) -- `src/controller/testsuite/test_config_repl.c` — delete the 11 `cfg.committing_timeout_ms` initializers (compiler-forced once the field is gone) - -## System Flows - -### Promotion (after this spec) — `raft_cb_become_leader` - -```mermaid -sequenceDiagram - participant RF as raft FSM - participant BL as raft_cb_become_leader - participant P as persist - participant CE as commit engine - participant S as standbys - RF->>BL: on_become_leader(term) - BL->>BL: sync term, truncate to safe_lsn, phase2 (existing) - BL->>BL: role = CTRL_PRIMARY (direct), register persist hooks - BL->>BL: promote vote eps, push config, ctrlhello, start heartbeat (existing) - BL->>P: persist_wal_async(WAL_NOOP, len=0) → lsn=N, broadcast - BL->>CE: commit_submit(N, RETAIN, 0) - BL->>CE: commit_set_writable_lsn(N) - S-->>CE: walAck acked_lsn >= N (live or via catch-up) - CE->>CE: committed_lsn >= N → own-term gate open - Note over CE: request waiters complete only when<br/>committed_lsn >= max(their lsn, N) -``` - -**Promotion ordering decisions**: hooks are registered BEFORE the no-op append so the persistence seam is live for any racing client write (their entries land after the no-op in LSN order and wait behind it — 2.1). The gate is set immediately after the append; in the sub-microsecond window before `commit_set_writable_lsn` returns, request-shaped entries still cannot be falsely confirmed — their own commits require per-entry majority of new-term entries, which transitively covers the no-op (single-writer contiguous advancement). Standalone (`n_peers <= 1`): the engine's self-quorum commits the no-op at registration and the gate opens immediately — standalone promotion latency is unchanged (umbrella 7.5). No watchdog of any kind is armed (1.5, 2.3). - -### Role lifecycle (umbrella diagram, the contract this spec realizes) - -```mermaid -stateDiagram-v2 - [*] --> INIT - INIT --> STANDBY: heartbeat seen - INIT --> ELECTING: timeout - ELECTING --> PRIMARY: quorum won, truncate to safe_lsn, append noop - ELECTING --> STANDBY: lost or higher term - PRIMARY --> RESTART: higher term seen (ha-convergence-recovery) - STANDBY --> ELECTING: primary lost - RESTART --> INIT: process self restart via boot recovery -``` - -No COMMITTING node exists; the `PRIMARY → RESTART` demotion edge is owned by ha-convergence-recovery (this spec only deletes the committing-timeout cancel block inside `raft_cb_become_follower`, leaving its current demote-to-STANDBY behavior otherwise as-is). - -### Pre-own-term request handling (2.2) -A state-changing request arriving after the role store but before the no-op commits takes exactly one of two paths, both engine-owned: (a) through the barrier — the handler's `commit_submit(WAIT)`/`persist_commit_barrier` blocks until `committed_lsn` covers both the entry and `writable_lsn`, confirming only on real majority; or (b) `-XCE_ERR_NOT_READY` fast rejection (admission cap, or gate-unmet engine policy), surfaced to the requester via the existing errnox reply path. No third path exists because the dispatch gate's COMMITTING rejection branch is gone and handlers have no other completion mechanism (barrier-rollout sweep). Status queries and replication traffic are unaffected: the PRIMARY gate branch (including the fact allowlist) admits them as today (1.3, 1.4). - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1 | election winner → PRIMARY directly | Promotion Rewrite, Legacy Path Symmetry | promotion | -| 1.2 | single leader role; zero residual references | Transitional-State Deletions, Reference Sweep | — | -| 1.3 | primary processes via barrier, never redirects | Role Predicate Simplification (`ha_should_redirect` fixed by state removal), Transitional-State Deletions (gate branch) | pre-own-term handling | -| 1.4 | status reports primary pre-own-term-commit | Role Predicate Simplification (`ha_is_primary` honest by construction) | promotion | -| 1.5 | no wall-clock leadership change | Transitional-State Deletions (watchdog + counters) | — | -| 2.1 | no confirmation before own-term entry commits | WAL_NOOP Entry, Promotion Rewrite (`commit_set_writable_lsn`) | promotion | -| 2.2 | pre-own-term requests: barrier or NOT_READY only | Promotion Rewrite (gate armed), engine contract (consumed) | pre-own-term handling | -| 2.3 | no timeout bypassing the own-term gate | Transitional-State Deletions (watchdog deleted without replacement) | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| WAL_NOOP Entry | wal_types.h, persist.c | own-term commit vehicle: empty entry, replay no-op | 2.1 | persist append/broadcast (P0), wal_classify (P0) | State | -| Promotion Rewrite | controller/ha.c | direct PRIMARY + hooks + noop + gate, one shared sequence | 1.1, 1.4, 2.1, 2.2 | raft callback (P0), commit engine (P0), WAL_NOOP (P0) | State | -| Transitional-State Deletions | controller/ha.c, ha.h, cluster.c | remove COMMITTING and all machinery serving it | 1.2, 1.3, 1.5, 2.3 | — | State | -| Role Predicate Simplification | controller/ha.c | collapse compound role checks; honest predicates | 1.3, 1.4 | — | Service | -| Legacy Path Symmetry | controller/ha.c (XCE_TESTING) | `ha_elect_evaluate_and_signal` mirrors production promotion | 1.1, 1.5 | shared promotion sequence (P0) | State | -| Reference Sweep | tests, docs, steering | grep-verified zero COMMITTING references | 1.2 | census (research.md §3) | — | -| Role Lifecycle Tests | testsuite/test_ha_role_lifecycle.c | pin the new lifecycle contract | all | XCE_TESTING seams | — | - -### WAL_NOOP Entry (`src/include/wal_types.h`, `src/controller/persist.c`) - -| Field | Detail | -|-------|--------| -| Intent | A current-term log entry with no state-machine effect, existing purely to satisfy Raft §5.4.2 | -| Requirements | 2.1 | - -**Responsibilities & Constraints** -- `WAL_NOOP = 40` in `wal_entry_type` (wal_types.h:44-58): below `WAL_ENTRY_TYPE_MAX` (100) so existing standby apply advances `current_lsn` and ACKs it through the common path (persist.c:719-721); the value gap 31–99 is otherwise free -- `payload_len = 0`; appended only by the promotion sequence via `persist_wal_async(p, WAL_NOOP, NULL, 0)` — one call appends, fsync-schedules, and broadcasts (repl_broadcast, persist.c:358) -- Replay/apply: explicit `case WAL_NOOP: break;` in `apply_wal_entry`'s type switch (persist.c:728-765) — no entity restore, no hooks; boot replay and standby apply are byte-identical no-ops -- Classification: `wal_classify(WAL_NOOP, *) → WAL_MODE_RETAIN` — the row barrier-rollout pre-declared ("guard forces classification"); the enum value and the row land in the same change so the totality test never breaks -- One entry per election win; no compaction concern (checkpointing treats it like any committed entry) - -**Contracts**: State [x] — the entry's LSN is the term's `writable_lsn`; `committed_lsn >= noop_lsn` ⇔ own-term condition met - -### Promotion Rewrite (`ha.c` — `raft_cb_become_leader` + shared sequence) - -| Field | Detail | -|-------|--------| -| Intent | One promotion sequence: ELECTING → PRIMARY directly with hooks, heartbeat, no-op, and gate | -| Requirements | 1.1, 1.4, 2.1, 2.2 | - -**Responsibilities & Constraints** -- `raft_cb_become_leader` (ha.c:445-617) keeps unchanged: term sync from raft, `ha_compute_safe_committed_lsn` + `persist_truncate_wal_above`, `last_known_committed_lsn`/`committed_config_tag` stores, first-boot `cluster_init_phase2`, per-term convergence-array memsets (still feed the catch-up loop), vote-endpoint promotion, config push, ctrlhello, `ha_start_primary_heartbeat` -- Replaces the COMMITTING store (ha.c:505-508) with a direct `CTRL_PRIMARY` store + `INFO("HA FSM (raft): → CTRL_PRIMARY ...")`; deletes the `repl_ack_count` reset (ha.c:511) and the entire watchdog-arm block (ha.c:593-616) -- **Hook registration moves here** (from `ha_promote_to_primary`, ha.c:1691-1698): extracted into a static helper (`ha_register_persist_hooks(cluster_t *c)` with the existing `hooks_registered` idempotency guard) called by both the rewrite and the surviving `ha_promote_to_primary`; registration precedes the no-op append so the persistence seam is live before any client write can race -- Appends the no-op and arms the gate (production, `c != NULL`): `persist_wal_async(WAL_NOOP)` → derive `noop_lsn = current_lsn - 1` under `p->lock` (same pattern as persist.c:1019-1021) → `commit_submit(&c->commit, noop_lsn, COMMIT_MODE_RETAIN, 0)` → `commit_set_writable_lsn(&c->commit, noop_lsn)`. Under `XCE_TESTING` with no cluster/persist context, the append and gate are skipped (gate unset = no gate, engine default) — lifecycle tests assert role/predicate behavior; gate behavior is asserted with a real engine instance -- No `ha_do_promote` call in this path: the raft-owned term was already synced at entry (ha.c:451); the legacy double-increment wart dies with the deleted convergence callers -- `ha_promote_to_primary` (ha.c:1686-1706) survives for its non-raft callers (cluster.c:1121 init-time, legacy path, tests): body becomes `ha_do_promote` + `ha_register_persist_hooks` + `ha_start_primary_heartbeat` + the same noop+gate tail (guarded on persist/engine availability) so every road to PRIMARY arms the own-term gate - -**Dependencies** -- Inbound: raft FSM `on_become_leader` (P0); legacy evaluation path (P0, XCE_TESTING) -- Outbound: persist append/broadcast (P0); commit engine registration + gate (P0) - -**Contracts**: State [x] -- Postconditions: role == CTRL_PRIMARY; persistence hooks registered; heartbeat armed; (production) a WAL_NOOP of the current term is appended+broadcast and `writable_lsn == noop_lsn` -- Invariants: no path sets any role value other than {PRIMARY, STANDBY, INIT, ELECTING}; promotion never blocks on the no-op committing (1.4 — status honesty does not wait) - -**Implementation Notes** -- Integration: `cluster_init_phase2` call point unchanged; the noop append happens after endpoint promotion so the broadcast reaches the freshly promoted standby endpoints; standbys not yet in the replicator registry receive the entry via the existing uncommitted-suffix catch-up streaming -- Validation: lifecycle test asserts no observable role transition through any intermediate value (atomic store — single transition ELECTING→PRIMARY) -- Risks: none new at runtime — the window "role==PRIMARY, gate not yet set" cannot falsely confirm (see §Promotion ordering decisions) - -### Transitional-State Deletions (`ha.c`, `ha.h`, `cluster.c`) - -| Field | Detail | -|-------|--------| -| Intent | Remove the state and every mechanism that existed only to serve it — no replacements, no shims | -| Requirements | 1.2, 1.3, 1.5, 2.3 | - -**Deletion list** (verified anchors in research.md §3; grep before editing): -- `CTRL_PRIMARY_COMMITTING` enum value (ha.h:64) — final removal step, compiler-enforced sweep -- Dispatch-gate COMMITTING branch (ha.c:1594-1612) — the `case CTRL_PRIMARY_COMMITTING:` block ONLY; the PRIMARY branch, including barrier-rollout's fact allowlist (`OPCode_notify/update/setpreason/clrpreason` under no-quorum), is preserved byte-for-byte -- `committing_timeout_fn` (ha.c:3080-3121), `committing_timeout_fn_for_test` (ha.c:2950-2990), `committing_timeout_ctx_t` (ha.c:64-69, fwd decl :83), watchdog arming in become-leader (ha.c:593-616) and in the legacy path (ha.c:2275-2299), the cancel block in `raft_cb_become_follower` (ha.c:640-648), `committing_timeout_ctx` field (ha.h:187), `committing_timeout_ms` knob (ha.h:131, :141; cluster.c:899; test_config_repl.c ×11) -- `ha_check_majority_convergence` (ha.c:1295-1353; decl ha.h:396-412) and its call site in the heartbeat-reply path (ha.c:2710-2715) -- `repl_ack_count` (ha.h:180; resets ha.c:511, :835, :2255) and `ha_repl_ack_received` (ha.c:3123-3141; decl ha.h:626-630); its `OPCode_replCatchUpDone` call site (ha.c:2659-2661) becomes a logged no-op — the opcode remains gate-allowed (standbys still send it; `repl_complete_catchup` on the replicator path is unrelated and unchanged) -- KEPT deliberately: `standby_reported_tag/lsn`, `standby_converged`, `ha_process_standby_state` (catch-up decision loop + observability); the per-term memsets in become-leader - -**Contracts**: State [x] — after this spec, `ctrl_role_t` has exactly four values and no component, test, or document observes a fifth - -### Role Predicate Simplification (`ha.c`) -Summary-only: collapse every `role == CTRL_PRIMARY || role == CTRL_PRIMARY_COMMITTING` compound to the PRIMARY check — `raft_cb_vote_eligible` (ha.c:210-211), `handle_request_vote` XCE_TESTING gates (ha.c:1984, 2004), `vote_ep_err_handler` (ha.c:3159), `peer_ctrl_msg_handler` case labels (ha.c:2789), and the heartbeat re-arm/run condition in `primary_heartbeat_fn` (ha.c:3956 — drops the COMMITTING term, satisfying the umbrella's "heartbeat re-arm condition drops COMMITTING"). `ha_is_primary` (ha.c:917-920) and `ha_should_redirect` (ha.c:1655-1664) need **zero code change** — both already test PRIMARY only; deleting the state makes them honest by construction (1.4: the elected node's role IS primary; 1.3: a live primary can never satisfy the redirect condition). Vote denial semantics are preserved exactly: a node that won the election denies same-term challengers as PRIMARY (it previously denied as COMMITTING). Comments updated at each site. - -### Legacy Path Symmetry (`ha.c` — `ha_elect_evaluate_and_signal`, XCE_TESTING) -Summary-only: the legacy election evaluation's CTRL_PRIMARY branch (ha.c:2247-2300) is updated symmetrically — direct `CTRL_PRIMARY` store (replacing the COMMITTING store at 2248-2252), `repl_ack_count` reset deleted (2255), config-push/ctrlhello test hooks kept, watchdog-arm block deleted (2275-2299). Promotion side effects route through the same shared sequence (`ha_promote_to_primary` / `ha_register_persist_hooks`) so test-driven and raft-driven promotion cannot diverge; the noop+gate tail is guarded on persist/engine availability as in production. Existing tests that drove COMMITTING through this path are retargeted in the sweep (1.1, 1.5). - -### Reference Sweep (tests + documentation) -Summary-only: the census in research.md §3 is the authoritative population. Test fixtures: **delete** `test_ha_repl_ack.c` and `test_ha_evaluate_committing.c` (their subjects are the deleted machinery; surviving concerns — promotion on quorum, no-promotion-without-majority — are re-pinned in `test_ha_role_lifecycle.c` against the new mechanism); **update** `test_ha_role.c`, `test_ha_startup.c`, `test_ha_rdhup_integration.c`, `test_ha_repl_push.c`, `test_ha_connection_reuse.c`, `test_ha_start_election_async.c`, `test_ha_vote_gate.c` (COMMITTING expectations → PRIMARY), and `test_config_repl.c` (drop the deleted config field); update `meson.build`. Documentation: `ha-async-io.md` (heartbeat re-arm :51, two-tier/promotion :57, dispatch-gate table :68, Raft mapping :79 — rewritten to the three-state model + own-term gate), `src/controller/CLAUDE.md`:40, `.claude/skills/xce-deploy-ha/SKILL.md`:98 (expected log becomes `→ CTRL_PRIMARY` once), `src/docs/cluster-config-split-analysis.md`:34. Completion is grep-verified: zero matches for `CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence` outside `.git/` and `.kiro/specs/` history (1.2). - -## Data Models - -### Domain Model (changes only) -- **ctrl_role_t**: 5 values → 4 (`CTRL_PRIMARY`, `CTRL_STANDBY`, `CTRL_INIT`, `CTRL_ELECTING`); enum numbering of survivors unchanged (no persisted role values exist — role is volatile) -- **WAL entry**: new type `WAL_NOOP = 40`, header-only (payload_len 0); wire/disk format otherwise unchanged -- **ha_state_t**: − `repl_ack_count`, − `committing_timeout_ctx`; `ha_config_t` − `committing_timeout_ms` -- **writable_lsn** (engine-owned, set by this spec): volatile, never persisted (Raft-conformant; a restarted node re-promotes or rejoins) -- Invariants: `committed_lsn >= writable_lsn` ⇔ the current term has a committed entry; exactly one leader role value exists - -## Error Handling - -### Error Strategy -- Promotion no-op append failure (`persist_wal_async` < 0, e.g. disk error): ERROR log; no LSN exists, so the gate stays unset for this term and promotion completes. This is degraded-but-safe: with the gate unset the engine applies no own-term hold, but every confirmation still requires per-entry majority of a new-term entry (universal barrier) — the first state-changing entry of this term becomes the de-facto own-term vehicle (2.2's "process through the barrier" arm). A WARN documents the condition; a node that cannot write its WAL at all fails operations through the existing persist error paths -- Pre-own-term request rejection: `-XCE_ERR_NOT_READY` via the engine (distinct log message, retriable) — unchanged engine contract, no new codes -- No new timeout semantics anywhere: the only deadline that can fail an operation is the commit timeout owned by the barrier (upstream) -- Deletion regressions are compile-time by design: removing the enum value last forces the compiler to surface every missed reference - -### Monitoring -- Promotion INFO logs: `→ CTRL_PRIMARY (term=N)` plus `own-term noop appended lsn=N` and gate-set confirmation; commit latency of the promotion no-op is observable from existing engine counters (umbrella monitoring item) - -## Testing Strategy - -### Unit Tests (`testsuite/test_ha_role_lifecycle.c`, XCE_TESTING) -- Direct promotion: driving the become-leader path (raft callback seam) yields role == CTRL_PRIMARY with no intermediate observable state; `ha_is_primary` true immediately; `INFO` transition logged once (1.1, 1.4) -- No redirect from a live primary: `ha_should_redirect` == 0 the instant promotion returns, before any ACK arrives (1.3) -- Own-term gate: with a real engine instance (n_peers=3 stub registry), promotion appends WAL_NOOP (entry observable in the WAL with payload_len 0) and sets `writable_lsn` to its LSN; a request-shaped `commit_submit(WAIT)` for a later entry does not complete while ACKs cover the entry but not the no-op — completes only when `committed_lsn >= noop_lsn` (2.1) -- Pre-own-term behavior: with the gate unmet, WAIT submission resolves only via barrier success or NOT_READY/timeout — never via any promotion-side path; no delaymgr task is armed by promotion (assert via delaymgr task count seam) (2.2, 2.3, 1.5) -- Standalone: n_peers=1 promotion → gate opens immediately (noop self-committed), WAIT submissions complete as before -- Replay: boot replay over a WAL containing WAL_NOOP applies cleanly with no entity side effects (entry count advances, jobmgr untouched) -- Vote honesty: elected node denies same-term votes as PRIMARY (gate parity with the old COMMITTING denial) (1.1) - -### Integration Tests (testsuite/) -- Legacy path symmetry: `ha_elect_evaluate_and_signal` quorum-win drives the same sequence — role PRIMARY directly, hooks registered, no watchdog armed (1.1, 1.5) -- Gate-branch deletion safety: barrier-rollout's gate tests stay green — PRIMARY fact allowlist intact under no-quorum; COMMITTING-specific gate cases removed with the branch (1.3) -- Updated fixtures (sweep): all 8 surviving HA test files green with PRIMARY-retargeted expectations; deleted files absent from meson - -### Regression -- Full controller testsuite + ASAN (`/xce-build controller`) green; grep verification: zero matches for the deleted-symbol pattern set across the repo (excluding `.git/`, `.kiro/specs/`) - -### E2E (deferred to ha-convergence-recovery's 3-node validation, which assumes this lifecycle) -- Kill-primary failover with the promoted standby confirming writes only after its no-op commits — owned downstream; this spec's testsuite seams (promotion hook, engine stub) are designed so that test can drive them - -## Migration Strategy - -This spec is phase 3 of the umbrella's hard-ordered landing ("barrier before state removal") and is itself ordered so the tree compiles and tests stay green at every step: -1. WAL_NOOP entry type + replay/classification (additive, no behavior change) -2. Promotion rewrite + lifecycle tests (COMMITTING no longer *entered* in production; symbols still exist) -3. Legacy-path symmetry (COMMITTING no longer entered anywhere) -4. Machinery deletions (watchdog, convergence checker, ack counters, gate branch, predicates) with their dedicated test files deleted/retargeted in the same step -5. Fixture sweep, then enum-value removal last (compiler enforces completeness), then docs/steering sync and grep verification -Rollback trigger per step: controller testsuite regression or failover E2E failure — reverting restores the prior (transitional-state) lifecycle, which the umbrella tolerates as "no less consistent than before". After step 5 there is no partial state: the grep gate guarantees all-or-nothing completion. diff --git a/.kiro/specs/ha-single-leader-role/requirements.md b/.kiro/specs/ha-single-leader-role/requirements.md deleted file mode 100644 index 6d8825d9..00000000 --- a/.kiro/specs/ha-single-leader-role/requirements.md +++ /dev/null @@ -1,56 +0,0 @@ -# Requirements Document - -> Sub-spec 3 of 4 implementing the **ha-consistency-invariants** umbrella contract. -> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 1.1, 1.2, 1.3, 1.4, 1.5, 3.1, 3.2, 3.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Locked umbrella decisions (own-term commit via an explicit no-op log entry appended at promotion; no wall-clock timeout may change leadership or confirm operations; the universal commit barrier must be in force before the transitional state is removed) are restated, never re-derived. -> **HARD ORDERING**: this spec must land AFTER ha-commit-barrier-rollout. Removing the transitional state before the universal barrier is in force opens a window where a fresh primary accepts unreplicated writes. - -## Project Description (Input) - -`CTRL_PRIMARY_COMMITTING` is a transitional leader state that Raft does not have. It causes a redirect loop (`ha_should_redirect` points clients at a primary that does not exist yet), `committing_timeout_fn` force-promotes after 60 seconds even with zero converged standbys (violating the commit rule outright), and `ha_is_primary` reports "not primary" during the window even though this node won the election and no other primary exists. Cluster operators need Raft's three-state model: an election winner becomes `CTRL_PRIMARY` directly; post-election write safety comes from committing a no-op log entry of the current term (Raft §5.4.2 own-term commit rule) through the commit engine's writable-LSN gate; and no wall-clock path ever changes leadership or confirms operations. The transitional state, its force-promotion watchdog, its convergence counter, and its acknowledgment counters are deleted outright, with every reference across the controller, its tests, and the steering documentation swept to zero. - -## Introduction - -This spec delivers the single-leader-role slice of the umbrella contract. After this spec, leadership in the XCE controller cluster is a single role: a controller that wins an election is the primary immediately — it reports itself as primary, accepts client traffic subject to the commit barrier, and is never the target of a redirect loop. The weaker guarantee that the transitional state was trying to provide ("don't confirm anything until the cluster has converged") is provided correctly instead by the own-term commit rule: at promotion the new primary appends a no-op entry of its current term and withholds confirmation of every operation until that entry is committed on a majority. Because the entry commits only through real majority acknowledgment, there is no timeout that can force the gate open — the 60-second force-promotion watchdog and the convergence/acknowledgment counting that fed it are removed without replacement. The dispatch-gate behavior contributed by upstream specs (including acceptance of reported facts during quorum loss) is preserved unchanged. - -## Boundary Context - -- **In scope**: Direct election-winner promotion to the primary role; the promotion-time no-op log entry of the current term and its use as the own-term confirmation gate; removal of the transitional leader state, its force-promotion timeout, its convergence checker, and its acknowledgment counters; honest primary self-reporting and redirect behavior; symmetric update of the legacy (testing-only) election evaluation path; exhaustive sweep of every transitional-state reference across controller code, test fixtures, and steering/contributor documentation, verified to zero. -- **Out of scope**: Election and vote correctness — vote RPCs, term management, freshness gates (raft-library); commit-engine internals — per-entry majority evaluation, the writable-LSN gate mechanics, admission cap (ha-commit-engine, consumed as-is); operation classification and reply ordering — which operations wait and which are retained (ha-commit-barrier-rollout, must already be in force); demotion behavior — truncate-and-restart, rejoin reconciliation, egress filtering, catch-up driving, quorum-transition logging (ha-convergence-recovery); replication wire format and standby apply mechanics (ha-wal-replication). -- **Adjacent expectations**: Upstream, this spec consumes the commit engine's writable-LSN gate (`commit_set_writable_lsn`; the engine withholds waiter completion until the commit index covers the writable LSN) delivered by ha-commit-engine, and requires the universal commit barrier plus the dispatch-gate fact allowlist delivered by ha-commit-barrier-rollout to be in force — the gate simplification here must preserve that allowlist. Downstream, ha-convergence-recovery's end-to-end failover validation relies on the simplified lifecycle, and the HA steering document (`ha-async-io.md`) — whose dispatch-gate table and replication-consistency section still describe the transitional state — is synchronized by this spec. - -## Requirements - -### Requirement 1: Unified Primary Role - -**Objective:** As a cluster operator, I want an election winner to act as primary immediately under a single leader role, so that clients are never redirected to a primary that does not exist and the cluster has no transitional mode with weaker consistency guarantees. - -#### Acceptance Criteria - -1. When a controller wins an election, the controller shall assume the primary role directly, without entering an intermediate transitional leader role. *(umbrella 1.1)* -2. The controller shall represent leadership as a single primary role; it shall not define a leader sub-state that accepts a different set of operations than the primary role, and no transitional leader state shall remain observable anywhere in the controller, its tests, or its operator-facing documentation. *(umbrella 1.2)* -3. While a controller is primary, when a client request requiring the primary arrives, the controller shall process the request subject to the commit barrier rather than redirecting the client to another node. *(umbrella 1.3)* -4. While a controller is primary, the controller shall report itself as the primary in status queries, regardless of whether it has yet committed an entry in its current term. *(umbrella 1.4)* -5. The controller shall not assume or change leadership based on a fixed wall-clock timeout that bypasses majority acknowledgment. *(umbrella 1.5)* - -### Requirement 2: Post-Election Write Safety - -**Objective:** As a cluster operator, I want a newly elected primary to confirm operations only after an entry of its own term is committed, so that a leadership change never exposes uncommitted or divergent state from prior terms. - -#### Acceptance Criteria - -1. When a controller becomes primary, the primary shall not confirm any cluster-state-changing operation until at least one log entry created in its current term has been acknowledged by a majority of cluster members; the primary shall create such an entry at promotion so that an idle cluster's new primary becomes writable without waiting for client traffic. *(umbrella 3.1)* -2. While the own-term commit condition of criterion 2.1 is unmet, if a state-changing request arrives, the primary shall either process it through the commit barrier (its entry satisfying criterion 2.1 upon majority acknowledgment) or reject it with an error indicating the cluster is not yet ready, and shall not confirm it by any other path. *(umbrella 3.2)* -3. While the own-term commit condition of criterion 2.1 is unmet, the primary shall not be subject to any timeout that confirms operations or advances the commit index without majority acknowledgment. *(umbrella 3.3)* - -## Umbrella Traceability - -| This spec | Umbrella ID | Notes | -|-----------|-------------|-------| -| 1.1 | 1.1 | Election winner → primary directly; transitional state never entered | -| 1.2 | 1.2 | Single leader role; transitional state deleted and swept to zero references (code, tests, docs) | -| 1.3 | 1.3 | No redirect from a live primary; the redirect-loop window cannot exist once the state is gone | -| 1.4 | 1.4 | Status honesty: primary self-reports as primary before its own-term entry commits | -| 1.5 | 1.5 | Force-promotion watchdog deleted without replacement; no wall-clock leadership change | -| 2.1 | 3.1 | Own-term commit rule (Raft §5.4.2) via explicit promotion no-op entry (umbrella-locked decision) | -| 2.2 | 3.2 | Pre-own-term requests: through the barrier or not-ready rejection — never a third path | -| 2.3 | 3.3 | No timeout substitutes for majority acknowledgment while the own-term gate is unmet | diff --git a/.kiro/specs/ha-single-leader-role/research.md b/.kiro/specs/ha-single-leader-role/research.md deleted file mode 100644 index 9d177153..00000000 --- a/.kiro/specs/ha-single-leader-role/research.md +++ /dev/null @@ -1,193 +0,0 @@ -# Research & Discovery Log: ha-single-leader-role - -Date: 2026-06-11. Discovery type: **light/integration-focused** — the umbrella spec (`ha-consistency-invariants`) completed full discovery (research.md, Option C phased hybrid, locked decisions); this log records the slice-specific verification pass: file:line anchors re-verified by grep against the working tree, the COMMITTING reference census that drives the sweep, and the upstream interface alignment (ha-commit-engine, ha-commit-barrier-rollout designs). - -## 1. Locked Umbrella Decisions Consumed (not re-derived) - -- **Own-term commit vehicle**: explicit `WAL_NOOP` entry appended at promotion (umbrella research item 4, DECIDED 2026-06-11) — bounded post-election write-unavailability independent of traffic; an idle cluster's new primary becomes writable as soon as the no-op commits. -- **No wall-clock leadership/confirmation path**: `committing_timeout_fn` force-promotion deleted without replacement (umbrella 1.5/3.3). -- **HARD ORDERING**: this spec lands only after ha-commit-barrier-rollout — the universal barrier must be in force before the transitional state is removed (umbrella migration phase 3; barrier-rollout design: "ha-single-leader-role must not land until the sweep is complete and audited"). -- **Commit state is volatile** (Raft-conformant): `writable_lsn` lives in the commit engine, never persisted; a restart re-runs promotion or rejoins as standby. - -## 2. Upstream Interface Alignment (verified against upstream designs) - -### ha-commit-engine (sub-spec 1) — consumed as-is -- `commit_set_writable_lsn(commit_engine_t *ce, uint64_t noop_lsn)` ships in `include/commit.h` with **no caller** in that spec ("API ships here; no caller in this spec — unset = no gate"). This spec is the designated caller. -- Engine invariant: "no waiter completes successfully while `committed_lsn < its lsn` or (gate set) `committed_lsn < writable_lsn`"; `commit_submit` returns `-XCE_ERR_NOT_READY` when "admission cap exceeded **or own-term gate unmet at entry**" — so umbrella 3.2's "barrier or NOT_READY" disjunction is engine-implemented; this spec only sets the gate. -- `n_peers <= 1` self-quorum: self-ACK constitutes majority — a standalone node's promotion no-op commits immediately and the gate opens with no peer traffic (standalone behavior preserved). -- Engine instance is `cluster_t.commit` (`c->commit`), wired at cluster init; RETAIN registration mode exists for non-blocking entries. - -### ha-commit-barrier-rollout (sub-spec 2) — must not be disturbed -- Classification: `wal_classify(wal_type, detail)` totality is enforced by a fail-fast guard and a totality unit test over every `wal_entry_type` value. Their design pre-declares the row this spec adds: "WAL_NOOP (future) | promotion | internal | RETAIN | added downstream; guard forces classification". **Adding the enum value without the classification row breaks their totality test** — the row lands in the same task as the enum value. -- Dispatch gate: barrier-rollout adds a **fact allowlist** to the `CTRL_PRIMARY` no-write-quorum branch (`OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason`). Their design: "The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then)." This spec deletes ONLY the COMMITTING branch; the PRIMARY branch including the fact allowlist is preserved byte-for-byte. -- Reply ordering: every request-shaped handler already commit-first-then-reply; this spec adds no reply sites. - -## 3. Verified Code Anchors (grep pass 2026-06-11 against working tree) - -Audit numbers from the brief mostly held; drift noted inline. **Re-verify with grep before editing — barrier-rollout landing first WILL shift ha.c line numbers** (it inserts into the PRIMARY gate branch above the COMMITTING branch). - -### ha.h -| Anchor | Verified | Notes | -|---|---|---| -| `CTRL_PRIMARY_COMMITTING = 4` enum value | ha.h:64 (enum block 59-65) | brief said 59-65 ✓ | -| `HA_DEFAULT_COMMITTING_TIMEOUT_MS 60000` | ha.h:131 | delete | -| `committing_timeout_ms` config field | ha.h:141 | delete | -| `repl_ack_count` field | ha.h:180 | delete | -| `committing_timeout_ctx` field | ha.h:187 | delete | -| `ha_check_majority_convergence` decl + comment | ha.h:396-412 | delete | -| `ha_repl_ack_received` decl + comment | ha.h:626-630 | delete | -| Stale comments mentioning COMMITTING | ha.h:214-230, 244-262, 373, 503-519, 607-614, 673-700, 791 | comment sweep | -| KEEP: `standby_reported_tag/lsn`, `standby_converged` arrays | ha.h:191-193 | still feed `ha_process_standby_state` catch-up decision loop + observability | - -### ha.c (23 CTRL_PRIMARY_COMMITTING refs + machinery) -| Anchor | Verified | Action | -|---|---|---| -| `committing_timeout_ctx_t` type + fwd decl | ha.c:64-69, 83 | delete | -| `raft_cb_vote_eligible` role gate `PRIMARY \|\| COMMITTING` | ha.c:210-211 | collapse to PRIMARY | -| `raft_cb_become_leader` | ha.c:445-617 (brief said ~445-505; body extends through watchdog arm at 593-616) | rewrite (see design) | -| — sets COMMITTING | ha.c:505-508 | → direct CTRL_PRIMARY | -| — `repl_ack_count` reset | ha.c:511 | delete | -| — convergence-array memsets | ha.c:517-519 | KEEP (per-term reset still feeds catch-up loop) | -| — heartbeat start + watchdog arm | ha.c:590, 593-616 | heartbeat stays; watchdog block deleted | -| `raft_cb_become_follower` committing-timeout cancel block | ha.c:640-648 | delete block | -| `ha_init` repl_ack_count init + COMMITTING comment | ha.c:835, 843 | delete/update | -| `ha_do_promote` (term increment + PRIMARY store) | ha.c:887-897 | unchanged; production rewrite does NOT call it (raft owns term — avoid double increment; existing behavior already syncs term at become_leader top, ha.c:451) | -| `ha_is_primary` | ha.c:917-920 ✓ | no code change — honest by construction once the state is gone; comment notes | -| heartbeat-reply convergence tracking comment | ha.c:1257, 1279-1282 | `ha_process_standby_state` KEPT; comment sweep | -| `ha_check_majority_convergence` | ha.c:1295-1353 (brief said 1313-1353; comment starts 1295) | delete (incl. `ha_promote_to_primary` call at 1347, `ha_do_promote` test branch at 1351) | -| dispatch-gate COMMITTING branch | ha.c:1594-1612 ✓ (pre-barrier-rollout numbering) | delete branch only; PRIMARY branch + fact allowlist untouched | -| `ha_should_redirect` | ha.c:1655-1664 ✓ | no code change (PRIMARY→0 already); fixed by state removal; comment | -| `ha_promote_to_primary` hook registration | ha.c:1686-1706 ✓ | registration extracted to shared helper; fn retained for legacy/test callers (cluster.c:1121, test_ha_cluster.c, test_p3_manual_failover.c) | -| `handle_request_vote` (XCE_TESTING) role gates | ha.c:1984, 2004 (comments 1940, 1971) | collapse to PRIMARY | -| stale type/comment block | ha.c:2057-2060 | comment sweep | -| `ha_elect_evaluate_and_signal` legacy COMMITTING block | ha.c:2247-2300 (brief said ~2248 ✓) | symmetric rewrite: direct PRIMARY; delete watchdog arm + repl_ack_count reset | -| `repl_ack_decode_and_dispatch` replCatchUpDone → `ha_repl_ack_received` | ha.c:2625-2627 (comment), 2659-2661 (call) | branch becomes INFO no-op (opcode still gate-allowed; standbys still send it) | -| heartbeat-reply `ha_check_majority_convergence` call | ha.c:2710-2715 | delete call; rest of decision loop unchanged | -| `peer_ctrl_msg_handler` COMMITTING case label | ha.c:2773, 2789 | drop label (falls to PRIMARY case) | -| `committing_timeout_fn_for_test` | ha.c:2950-2990 (region 2945-2995) | delete | -| `vote_ep_err_handler` role check | ha.c:3159 | collapse to PRIMARY | -| `committing_timeout_fn` | ha.c:3080-3121 (brief said 3082-3125 ≈) | delete | -| `ha_repl_ack_received` | ha.c:3123-3141 (brief said 3126-3139 ≈) | delete | -| test-helper comment | ha.c:3044-3045 | sweep | -| `primary_heartbeat_fn` re-arm/run role check | ha.c:3956 | drop COMMITTING term | - -### Other code -| Anchor | Verified | Action | -|---|---|---| -| `cluster.c` config init `.committing_timeout_ms = HA_DEFAULT_COMMITTING_TIMEOUT_MS` | cluster.c:899 | delete line | -| `cluster.c` ctrlhello comment "COMMITTING path" | cluster.c:1402-1410 | comment update | -| `raft.c` comment "HA FSM stays in CTRL_PRIMARY_COMMITTING" | src/lib/raft/raft.c:123 | comment update (raft-library code untouched otherwise) | -| `wal_types.h` enum | src/include/wal_types.h:44-58 | add `WAL_NOOP = 40` (gap 31-99 free; < WAL_ENTRY_TYPE_MAX=100, config.h:80) | -| `apply_wal_entry` type switch | persist.c:691-770 (switch at 728; default:break at 765) | explicit `case WAL_NOOP: break;` — LSN advance happens in common path (persist.c:719-721) so standbys ACK it | -| `persist_wal_async` (append + `repl_broadcast` at persist.c:358) | persist.c:295 | reused as-is for the promotion no-op (rejects only WAL_CHECKPOINT) | -| `wal_classify` table (barrier-rollout) | persist.c (landed by sub-spec 2) | add WAL_NOOP → RETAIN row | -| meson.build `controller_unit_test_src` | src/controller/meson.build:74-131 | remove deleted test files; add test_ha_role_lifecycle.c | - -### COMMITTING reference census (sweep population — grep `CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence`) - -Test fixtures (counts = matching lines at audit time): -| File | Refs | Disposition | -|---|---|---| -| `testsuite/test_ha_repl_ack.c` | 20 COMMITTING + 34 deleted-symbol | **DELETE** — subject is `ha_repl_ack_received`/`repl_ack_count` counting; valid concerns re-covered by test_ha_role_lifecycle.c | -| `testsuite/test_ha_evaluate_committing.c` | 24 + 13 deleted-symbol | **DELETE** — subject is the COMMITTING evaluation path + `committing_timeout_fn`; direct-PRIMARY evaluation asserted in test_ha_role_lifecycle.c | -| `testsuite/test_ha_role.c` | 37 (+16 deleted-symbol) | UPDATE — role-enum/transition expectations retargeted COMMITTING→PRIMARY | -| `testsuite/test_ha_rdhup_integration.c` | 20 (+8) | UPDATE | -| `testsuite/test_ha_repl_push.c` | 14 | UPDATE | -| `testsuite/test_ha_startup.c` | 12 | UPDATE | -| `testsuite/test_ha_connection_reuse.c` | 11 (+12) | UPDATE | -| `testsuite/test_ha_start_election_async.c` | 7 | UPDATE | -| `testsuite/test_ha_vote_gate.c` | 6 | UPDATE (vote-deny: PRIMARY-only deny expectations) | -| `testsuite/test_config_repl.c` | 11 × `cfg.committing_timeout_ms = 10000;` | UPDATE — delete the field initializers (compiler-forced) | - -Documentation: -| File | Site | Disposition | -|---|---|---| -| `.kiro/steering/ha-async-io.md` | :51 (heartbeat re-arm), :57 (two-tier/COMMITTING promotion), :68 (dispatch-gate table row), :79 (Raft-to-code mapping) | steering sync task | -| `src/controller/CLAUDE.md` | :40 (role mapping table) | sweep | -| `.claude/skills/xce-deploy-ha/SKILL.md` | :98 (expected log line "→ CTRL_PRIMARY_COMMITTING then → CTRL_PRIMARY") | sweep | -| `src/docs/cluster-config-split-analysis.md` | :34 (committing_timeout_ms in config listing) | sweep (historical doc — annotate or update listing) | - -## 4. Slice-Specific Findings - -1. **`ha_is_primary` and `ha_should_redirect` need zero code change.** Both already test `role == CTRL_PRIMARY` only (ha.c:917-920, 1655-1664). The brief's "lies during the window"/"redirect loop" defects are properties of the COMMITTING *state existing* (role ≠ PRIMARY while leading); deleting the state fixes both by construction. The "simplification" is the collapse of every `PRIMARY || COMMITTING` compound check (4 production sites: ha.c:210, 1984, 2004, 3159, 3956, 2789) plus comment honesty. -2. **Term double-increment hazard.** `ha_do_promote` (ha.c:887) increments `ha->term`; production `raft_cb_become_leader` already syncs the raft-owned term at entry (ha.c:451). The old flow reached `ha_do_promote` later via convergence/watchdog → `ha_promote_to_primary`, producing a second increment beyond the raft term — a pre-existing wart that dies with the deleted callers. The rewrite stores `CTRL_PRIMARY` directly (no `ha_do_promote`) in the raft path; the legacy XCE_TESTING path keeps `ha_do_promote` semantics (it owns its term there). -3. **`OPCode_replCatchUpDone` survives the deletion.** Standbys still send it on catch-up completion and the dispatch gate still admits it; its only consumer on the promoted-vote-endpoint path was `ha_repl_ack_received` ack counting (ha.c:2661). The handler branch becomes a logged no-op; `repl_complete_catchup` (repl.c:521, replicator-registry path) is unrelated and unchanged. -4. **Convergence tracking arrays stay.** `ha_process_standby_state` (heartbeat stateReport) feeds the per-standby catch-up decision loop (`ha_dispatch_repl_action_for_standby`), which is unchanged by this spec; only its *promotion consumer* (`ha_check_majority_convergence`) is deleted. The per-term memsets in become_leader stay. -5. **WAL_NOOP value/compatibility**: `WAL_NOOP = 40` sits below `WAL_ENTRY_TYPE_MAX` (100, config.h:80) so current standbys advance LSN and ACK it via the common path in `apply_wal_entry`; replay is a no-op via an explicit case. `persist_wal_async` appends AND broadcasts (repl_broadcast, persist.c:358) — one call does both at promotion. Standbys not yet registered in the replicator at promotion receive the entry via catch-up streaming of the uncommitted suffix (existing mechanism). -6. **Engine registration of the no-op**: after append, the promotion path registers the entry with `commit_submit(ce, noop_lsn, COMMIT_MODE_RETAIN, 0)` (non-blocking, observability) then calls `commit_set_writable_lsn(ce, noop_lsn)`. Standalone (`n_peers <= 1`): engine self-quorum commits it immediately — gate opens with no peer traffic (umbrella 7.5 preserved). -7. **meson test list** (src/controller/meson.build:74-131) registers each test file individually — deleting test_ha_repl_ack.c / test_ha_evaluate_committing.c requires list edits, and test_ha_role_lifecycle.c must be added. - -## 5. Risks - -| Risk | Mitigation | -|---|---| -| Line-number drift after barrier-rollout lands | every task instructs grep-before-edit; anchors here are 2026-06-11 pre-rollout | -| Missed COMMITTING reference (silent semantic residue) | dedicated grep-verification task gating completion (zero matches across repo excluding .git and spec history) | -| Deleting the gate branch disturbs barrier-rollout's fact allowlist | deletion task scoped to the `case CTRL_PRIMARY_COMMITTING:` block only; gate tests from barrier-rollout must stay green | -| Legacy XCE_TESTING path divergence from production | symmetric-update task with its own lifecycle assertions; both paths share the promotion helper | -| Election-win window with unset gate (crash between role store and `commit_set_writable_lsn`) | ordering inside the rewrite: gate set before the role store is observable to request traffic is NOT required — the engine rejects/holds via `writable_lsn` only after set; design orders noop-append + gate-set immediately at promotion, and until set the barrier still requires per-entry majority (entries of the new term); documented in design §Promotion Ordering | - -## 6. Design Synthesis Outcomes - -- **Generalization**: one shared promotion sequence (role store → hook registration → heartbeat → noop append/broadcast → engine registration → writable gate) used by both the raft callback and the legacy evaluation path — no second promotion flavor survives. -- **Build vs adopt**: nothing new is built beyond `WAL_NOOP`; the gate, barrier, classification guard, broadcast, and catch-up are all consumed from upstream specs/existing code. Deletions dominate (net-negative LOC in ha.c). -- **Simplification**: `ha_promote_to_primary` survives only as the legacy/test-path wrapper around the shared helper; no compatibility shim for COMMITTING is kept anywhere (tests asserting it are rewritten or deleted, not stubbed). -- Review gate: all 8 requirement IDs traced to components; no orphan components; boundary↔file alignment verified against the census table. - ---- - -## Gap Re-Audit (2026-06-18) - -**Re-audit context**: roadmap.md lists this spec as DONE, but `spec.json.phase` = `tasks-generated` and `tasks.md` shows 6 unchecked `[ ]` against 11 checked `[x]`. This pass determines whether the 6 open items are genuine remaining work or non-work artifacts, and re-verifies every EARS requirement against the live working tree (not the 2026-06-11 pre-implementation anchors in §3). - -### A. Verification method -- Repo-wide grep for the deleted-symbol set (`CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence`), excluding `.git`/`.kiro`/`.wolf`/`.codegraph`/`builddir`. -- Docs/steering grep over the task-6.1 target set. -- codegraph_explore of the promotion path, dispatch gate, role predicates, legacy path; direct grep of `ha.c`, `ha.h`, `wal_types.h`, `persist.c`, `meson.build`. -- git history of `.kiro/specs/ha-single-leader-role/` and the controller tree. - -### B. Requirement-to-Asset map (verified against live code, 2026-06-18) - -| Req | EARS criterion | Asset (file:line) | Verdict | -|-----|----------------|-------------------|---------| -| 1.1 | Election winner → PRIMARY directly, no transitional role | `raft_cb_become_leader` direct store `ha.c:1034` (`→ CTRL_PRIMARY` log `ha.c:1036`); legacy `ha_elect_evaluate_and_signal` direct store `ha.c:~2919` (`→ CTRL_PRIMARY` log) | **MET** | -| 1.2 | Single leader role; zero residual refs (code+tests+docs) | `ctrl_role_t` = 4 values `ha.h:60-64` (no `CTRL_PRIMARY_COMMITTING`); repo-wide deleted-symbol grep = **0**; docs/steering grep = **0** | **MET** | -| 1.3 | Live primary processes via barrier, never redirects | `ha_dispatch_gate` PRIMARY branch `ha.c:2067` (no COMMITTING case); `ha_should_redirect` PRIMARY→0 unchanged; barrier-rollout fact allowlist (notify/update/setpreason/clrpreason) preserved `ha.c:~2099-2105` | **MET** | -| 1.4 | Primary self-reports as primary pre-own-term-commit | `ha_is_primary` `ha.c:1466-1470` tests PRIMARY only; promotion stores PRIMARY before/without waiting on noop commit | **MET** | -| 1.5 | No wall-clock leadership change | `committing_timeout_fn`/`_for_test`/`committing_timeout_ctx_t` deleted (grep 0); no watchdog armed in either promotion path; only `ha_arm_own_term_gate` (no timeout) on the tail | **MET** | -| 2.1 | No confirmation before an own-term entry commits; entry created at promotion | `WAL_NOOP=40` `wal_types.h:57`; `ha_arm_own_term_gate` `ha.c:2259-2286` appends WAL_NOOP → `commit_submit(RETAIN)` → `commit_set_writable_lsn(noop_lsn)`; apply no-op `persist.c:1381`; replay no-op `persist.c:2037` | **MET** | -| 2.2 | Pre-own-term request: barrier or NOT_READY only, no third path | Gate COMMITTING-reject branch gone; PRIMARY branch returns 0 (→ barrier) or `-XCE_ERR_NOT_READY` (`ha.c:~2105`); no promotion-side completion path exists | **MET** | -| 2.3 | No timeout bypassing the own-term gate | watchdog/ack/convergence machinery deleted without replacement (grep 0); gate is `committed_lsn >= writable_lsn` only, engine-owned | **MET** | - -**Requirements met: 8 / 8.** - -Supporting structural confirmations: -- `ha_register_persist_hooks` (`ha.c:2223`) + `ha_arm_own_term_gate` (`ha.c:2259`) are the shared promotion sequence; both the raft path (`ha.c:1122`,`1134`) and `ha_promote_to_primary` (`ha.c:2305`,`2312`) and the legacy path (`ha.c:~2920`) call them — single promotion flavor, no divergence. -- Raft path does NOT call `ha_do_promote` (term synced at entry); `ha_promote_to_primary` keeps `ha_do_promote` (owns term) — double-increment wart resolved per research §4 finding 2. -- Test disposition exact-match to plan: `test_ha_role_lifecycle.c` present + registered (`meson.build:94`); `test_ha_repl_ack.c` and `test_ha_evaluate_committing.c` deleted (absent on disk, absent from meson). -- barrier-rollout gate tests intact and asserting the preserved PRIMARY fact allowlist (`test_commit_barrier_rollout.c` fact_gate_* cases). - -### C. Open-task classification (the 6 unchecked `[ ]`) - -`tasks.md` uses a two-level checkbox scheme: integer rows (`- [ ] 1.`…`6.`) are **phase group headers**; the actual work lives in decimal sub-tasks (`- [x] 1.1`, `1.2`, `2.1`, …). All 11 decimal sub-tasks are `[x]`. The 6 unchecked items are EXACTLY and ONLY the 6 group headers. - -| `[ ]` item | Type | Evidence | -|---|---|---| -| 1. WAL_NOOP entry type | **GROUP-HEADER-ARTIFACT** | sub-tasks 1.1 (`6df1283`), 1.2 (`e6abd7c`) both `[x]` + committed; WAL_NOOP live | -| 2. Promotion rewrite | **GROUP-HEADER-ARTIFACT** | 2.1 (`b7a69af`), 2.2 (`c089fb6`) `[x]` + committed; direct-PRIMARY + gate live | -| 3. Legacy path symmetry | **GROUP-HEADER-ARTIFACT** | 3.1 (`b7b951b`) `[x]` + committed; legacy path symmetric, live | -| 4. Machinery deletions | **GROUP-HEADER-ARTIFACT** | 4.1 (`a5bf861`), 4.2 (`71ce41f`) `[x]` + committed; grep 0 | -| 5. Fixture sweep + symbol removal | **GROUP-HEADER-ARTIFACT** | 5.1 (`7f974e7`), 5.2 (`93dfc02`) `[x]` + committed; enum collapsed | -| 6. Docs sync + grep verification | **GROUP-HEADER-ARTIFACT** | 6.1 (`dbc6316`), 6.2 (`e558a62`) `[x]` + committed; docs grep 0, ASAN-green evidence in commit | - -**Genuine gaps: 0. Deferred: 0. Group-header artifacts: 6/6.** - -The headers are unchecked purely because the kiro task generator does not auto-check parent rows when all children complete; they carry no independent deliverable. Task 6.2's commit (`e558a62`) explicitly records the final zero-match grep gate + ASAN regression green — the spec's own completion gate, which this re-audit independently reproduces (grep = 0 across code and docs). - -### D. Options for the open items (no genuine engineering work exists) - -- **Option A (recommended) — Administrative close-out.** Tick the 6 group-header checkboxes and flip `spec.json.phase` to `implemented`/`completed`. Effort: trivial (~2 min, doc-only). Risk: none. Rationale: every child task is committed and independently verified; roadmap already reflects DONE. -- **Option B — Leave as-is.** Accept the cosmetic header/`spec.json` lag. Effort: zero. Risk: low-but-real — future `/kiro-spec-status` will keep reporting 6 "open" tasks and a `tasks-generated` phase, inviting repeat audits like this one (false-signal cost). -- **Option C — Re-run a clean regression as fresh evidence.** `/xce-build controller` (debug + ASAN + suite) to re-confirm green on the current tree, then close out as Option A. Effort: one build cycle (note research §Implementation-Notes 5.1: meson 300s cap is borderline — prefer the direct binary run; a `Timeout: 1` is a wrapper artifact, not a failure). Risk: none; pure belt-and-suspenders. Recommended only if a green build artifact is required for sign-off beyond the existing task-6.2 commit evidence. - -**Effort/Risk for genuine work: N/A — there is no genuine remaining work.** Closing recommendation: **Option A**. diff --git a/.kiro/specs/ha-single-leader-role/spec.json b/.kiro/specs/ha-single-leader-role/spec.json deleted file mode 100644 index 307caab8..00000000 --- a/.kiro/specs/ha-single-leader-role/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "ha-single-leader-role", - "created_at": "2026-06-11T12:00:00Z", - "updated_at": "2026-06-11T12:45:00Z", - "language": "en", - "phase": "implemented", - "umbrella": "ha-consistency-invariants", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation_status": "implemented — 17/17 tasks complete. Phase reconciled 2026-07-03 (was mislabeled tasks-generated)." -} diff --git a/.kiro/specs/ha-single-leader-role/tasks.md b/.kiro/specs/ha-single-leader-role/tasks.md deleted file mode 100644 index fd2f768f..00000000 --- a/.kiro/specs/ha-single-leader-role/tasks.md +++ /dev/null @@ -1,95 +0,0 @@ -# Implementation Plan: ha-single-leader-role - -> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first; deletion tasks are driven by retargeted test expectations plus compiler enforcement (the enum value is removed LAST so the compiler surfaces every missed reference). New tests live in `testsuite/test_ha_role_lifecycle.c`; two existing test files are deleted (their subject is the deleted machinery) and eight are updated — this shared-ha.c reality is why no `(P)` markers are applied: the plan is sequential. -> **Hard upstream prerequisite**: ha-commit-engine AND ha-commit-barrier-rollout must be implemented and green first — the promotion path calls `commit_set_writable_lsn` on the engine in `cluster_t.commit`, and the universal barrier + dispatch-gate fact allowlist must be in force before the transitional state is removed (umbrella hard ordering). -> Verify exact line numbers with grep before editing — research.md §3 has the 2026-06-11 verified anchor table, but barrier-rollout landing first WILL shift ha.c line numbers. - -- [x] 1. WAL_NOOP entry type (additive foundation) -- [x] 1.1 Write RED tests for the no-op entry - - Create `testsuite/test_ha_role_lifecycle.c` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (src/controller/meson.build:74-131); first cases cover the entry type only - - Tests: `persist_wal_async(p, WAL_NOOP, NULL, 0)` appends a header-only entry (payload_len 0) and broadcasts it (stub replicator observes the entry); `apply_wal_entry` on a WAL_NOOP advances `current_lsn` past the entry's LSN and touches no entity state (jobmgr/queue/worker untouched); boot `replay_wal` over a segment containing WAL_NOOP replays cleanly with the same no-effect guarantee; `wal_classify(WAL_NOOP, 0) == WAL_MODE_RETAIN` (keeps barrier-rollout's totality test green) - - Observable completion: build succeeds with the new test file; all new cases FAIL (red) — `WAL_NOOP` does not exist yet - - _Requirements: 2.1_ - -- [x] 1.2 Implement WAL_NOOP to green - - Add `WAL_NOOP = 40` to `wal_entry_type` (src/include/wal_types.h:44-58) with comment: internal, empty payload, replay no-op, appended only at promotion; value stays below `WAL_ENTRY_TYPE_MAX` (100, src/include/config.h:80) so standbys LSN-advance and ACK it via the existing common path - - Add explicit `case WAL_NOOP: break;` to `apply_wal_entry`'s type switch (persist.c:728-765 anchor); add the `WAL_NOOP → WAL_MODE_RETAIN` row to `wal_classify` (the row barrier-rollout pre-declared) - - Observable completion: all task 1.1 tests PASS; barrier-rollout's classification totality test still PASSES - - _Requirements: 2.1_ - -- [x] 2. Promotion rewrite: direct PRIMARY + own-term gate -- [x] 2.1 Write RED lifecycle tests for direct promotion and the own-term gate - - Tests in `test_ha_role_lifecycle.c`: driving the become-leader path yields `ha->role == CTRL_PRIMARY` directly (no intermediate value observable; `ha_is_primary()` true immediately); `ha_should_redirect()` returns 0 the instant promotion returns, before any standby ACK; persistence hooks are registered by promotion (hooksys observes the three persist hook fns; `hooks_registered == 1`); promotion arms NO delaymgr watchdog (delaymgr task-count seam unchanged across promotion); with a real engine (n_peers=3, stub registry): promotion appends WAL_NOOP at lsn=N, registers it RETAIN, and `commit_set_writable_lsn(N)` is in effect — a `commit_submit(WAIT)` for entry N+1 does NOT complete while ACKs cover N+1 but not N, and completes once `committed_lsn >= N`; with the gate unmet a WAIT submission resolves only as barrier success, `-ETIMEDOUT`, or `-XCE_ERR_NOT_READY` — never via any promotion-side path; n_peers=1: gate opens immediately at promotion (noop self-committed), WAIT returns 0 as before - - Observable completion: new cases FAIL (red) against the current COMMITTING-entering `raft_cb_become_leader` - - _Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3_ - -- [x] 2.2 Rewrite raft_cb_become_leader and share the promotion sequence - - In `raft_cb_become_leader` (ha.c:445-617 anchor): keep term sync, safe-LSN truncation, `last_known_committed_lsn`/`committed_config_tag` stores, phase2, convergence-array memsets, vote-endpoint promotion, config push, ctrlhello, `ha_start_primary_heartbeat`; replace the `CTRL_PRIMARY_COMMITTING` store (ha.c:505-508) with a direct `CTRL_PRIMARY` store + `INFO("HA FSM (raft): → CTRL_PRIMARY (term=...)")`; delete the `repl_ack_count` reset (ha.c:511) and the watchdog-arm block (ha.c:593-616) - - Extract hook registration from `ha_promote_to_primary` (ha.c:1691-1698) into static `ha_register_persist_hooks(cluster_t *c)` (keeps the `hooks_registered` idempotency guard); call it in the rewrite BEFORE the no-op append; `ha_promote_to_primary` body becomes `ha_do_promote` + `ha_register_persist_hooks` + `ha_start_primary_heartbeat` + the same noop+gate tail — every road to PRIMARY arms the gate; the rewrite does NOT call `ha_do_promote` (raft owns the term; no double increment) - - Noop+gate tail (production, guarded on `c != NULL` and engine availability): `persist_wal_async(WAL_NOOP)` after endpoint promotion → `noop_lsn = current_lsn - 1` under `p->lock` → `commit_submit(&c->commit, noop_lsn, COMMIT_MODE_RETAIN, 0)` → `commit_set_writable_lsn(&c->commit, noop_lsn)`; append failure → ERROR + WARN, gate left unset (degraded-but-safe per design Error Strategy), promotion still completes - - Observable completion: all task 2.1 tests PASS; promotion never blocks (returns without waiting for the noop to commit) - - _Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3_ - -- [x] 3. Legacy XCE_TESTING election path symmetry -- [x] 3.1 Update ha_elect_evaluate_and_signal and its direct tests (RED → GREEN in one pass) - - Retarget existing legacy-path expectations first (test_ha_start_election_async.c, the evaluation cases in test_ha_role.c): quorum win → role `CTRL_PRIMARY` directly, no watchdog armed — run to confirm RED against current code - - In the CTRL_PRIMARY branch of `ha_elect_evaluate_and_signal` (ha.c:2247-2300 anchor): replace the COMMITTING store (2248-2252) with the direct `CTRL_PRIMARY` store, delete the `repl_ack_count` reset (2255) and the watchdog-arm block (2275-2299); keep the config-push/ctrlhello XCE_TESTING hooks; route promotion side effects through the same shared sequence (noop+gate tail guarded on persist/engine availability, as in production) - - Observable completion: retargeted legacy-path tests PASS; lifecycle test for symmetry PASSES (both entry points produce identical role/hook/heartbeat/no-watchdog state) - - _Requirements: 1.1, 1.5_ - -- [x] 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates) -- [x] 4.1 Delete the force-promotion watchdog and ack/convergence machinery - - Delete from ha.c: `committing_timeout_fn` (3080-3121), `committing_timeout_fn_for_test` (2950-2990), `committing_timeout_ctx_t` type + fwd decl (64-69, 83), the cancel block in `raft_cb_become_follower` (640-648); `ha_check_majority_convergence` (1295-1353) and its heartbeat-reply call site (2710-2715); `ha_repl_ack_received` (3123-3141) — its `OPCode_replCatchUpDone` branch in `repl_ack_decode_and_dispatch` (2659-2661) becomes an INFO-logged no-op (opcode stays gate-allowed; `repl_complete_catchup` untouched); `repl_ack_count` init at ha.c:835 - - Delete declarations from ha.h: `ha_check_majority_convergence` (396-412), `ha_repl_ack_received` (626-630) - - Delete test files whose subject is this machinery: remove `testsuite/test_ha_repl_ack.c` and `testsuite/test_ha_evaluate_committing.c` and their entries in `controller_unit_test_src` (meson.build:116, 118 anchors); concerns still valid (promotion only on real quorum, no promotion without majority) are already pinned by task 2.1's gate tests - - Observable completion: controller builds; `grep -rn "committing_timeout_fn\|ha_check_majority_convergence\|ha_repl_ack_received" src/controller/*.c src/controller/*.h` returns zero matches; test suite green - - _Requirements: 1.5, 2.3_ - -- [x] 4.2 Delete the dispatch-gate COMMITTING branch and collapse role predicates - - Delete ONLY the `case CTRL_PRIMARY_COMMITTING:` block from `ha_dispatch_gate` (ha.c:1594-1612 pre-rollout anchor — re-grep; the block follows the ELECTING case); the CTRL_PRIMARY branch — including barrier-rollout's no-quorum fact allowlist (`OPCode_notify/update/setpreason/clrpreason`) — is preserved byte-for-byte - - Collapse compound role checks to PRIMARY-only: `raft_cb_vote_eligible` (ha.c:210-211), `handle_request_vote` gates (ha.c:1984, 2004), `vote_ep_err_handler` (ha.c:3159), `peer_ctrl_msg_handler` case labels (ha.c:2789), `primary_heartbeat_fn` re-arm/run condition (ha.c:3956 — heartbeat re-arm condition drops COMMITTING); update the stale comments at each site plus ha.c:1940, 1971, 2057-2060, 2625-2627, 3044-3045 and the `ha_is_primary`/`ha_should_redirect` doc comments (no code change in those two functions) - - Update gate tests: COMMITTING-specific cases in `test_ha_vote_gate.c` and gate fixtures retargeted (elected node denies same-term votes as PRIMARY — denial parity pinned) - - Observable completion: barrier-rollout's gate tests (PRIMARY fact allowlist under no-quorum) still PASS; vote-gate tests PASS with PRIMARY-only denial; suite green - - _Requirements: 1.2, 1.3, 1.4_ - -- [x] 5. Test-fixture sweep and final symbol removal -- [x] 5.1 Retarget the remaining COMMITTING-referencing fixtures - - Update expectations COMMITTING→PRIMARY (census: research.md §3): `test_ha_role.c` (37 refs), `test_ha_rdhup_integration.c` (20), `test_ha_repl_push.c` (14), `test_ha_startup.c` (12), `test_ha_connection_reuse.c` (11), `test_ha_start_election_async.c` (7), `test_ha_vote_gate.c` (6, residual from 4.2); delete the 11 `cfg.committing_timeout_ms = 10000;` initializers in `test_config_repl.c` - - Each updated test asserts the new contract (direct PRIMARY, no transitional observations), not stubbed compatibility - - Observable completion: full controller suite green; `grep -rln "COMMITTING" src/controller/testsuite/` returns zero files - - _Requirements: 1.1, 1.2_ - -- [x] 5.2 Remove the enum value and remaining declarations (compiler-enforced completeness) - - Delete `CTRL_PRIMARY_COMMITTING = 4` from `ctrl_role_t` (ha.h:64); delete `HA_DEFAULT_COMMITTING_TIMEOUT_MS` (ha.h:131), `committing_timeout_ms` (ha.h:141), `repl_ack_count` (ha.h:180), `committing_timeout_ctx` (ha.h:187); delete the `.committing_timeout_ms` initializer (cluster.c:899); sweep remaining ha.h comments (214-230, 244-262, 373, 503-519, 607-614, 673-700, 791), the cluster.c ctrlhello comment (1402-1410), and the raft.c comment (src/lib/raft/raft.c:123) - - Fix everything the compiler then surfaces — this is the completeness gate for code references - - Observable completion: clean build with zero warnings about the removed symbols; full controller testsuite + ASAN green via `/xce-build controller` - - _Requirements: 1.2_ - -- [x] 6. Documentation sync and grep verification -- [x] 6.1 Synchronize steering and contributor docs to the three-state model - - `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING) - - `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob) - - Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches - - _Requirements: 1.2_ - -- [x] 6.2 Final grep verification and full regression - - Run the completion gate: `grep -rEn "CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence" --exclude-dir=.git --exclude-dir=specs --exclude-dir=.wolf --exclude-dir=.codegraph --exclude-dir=builddir .` — zero matches; the only permitted residue is git history and `.kiro/specs/` documents (this spec's own census and the umbrella audit). Note `.kiro/steering/` is deliberately INSIDE the gate — a missed steering reference fails it - - Run `/xce-build controller` (debug + ASAN + full testsuite) — green; record the grep output and build result as task evidence - - Observable completion: zero-match grep output committed as evidence; controller suite + ASAN green - - _Requirements: 1.2, 1.5, 2.3_ - -## Implementation Notes - -- **(2.1/2.2) Two promotion seams in the lifecycle tests**: cases 1-2 (role/redirect) drive `raft_cb_become_leader_for_test` which forces `c==NULL` (the noop+gate tail is `c!=NULL`-guarded and skipped); cases 3-7 (hooks/gate) drive the real-cluster shared seam `ha_promote_to_primary(&g_lc)`. Hook registration cannot be pinned on the `c==NULL` path (the helper needs `&c->hooksys`) — case 3 is a green contract pin via `ha_promote_to_primary`. -- **(2.2) Expected category-(a) failures open from task 2.2 until retargeted**: after the raft path enters `CTRL_PRIMARY` directly, three single-node tests still assert `CTRL_PRIMARY_COMMITTING (4)` and FAIL (`Expected 4 Was 0`): `test_single_node_shortcircuit_evaluates_immediately` + `test_single_node_not_primary_directly` (test_ha_start_election_async.c) → retargeted by **task 3.1**; `test_nonblocking_single_node_election_enters_committing` (test_ha_startup.c) → retargeted by **task 5.1**. Single-node election reaches `raft_cb_become_leader` via the `raft_start_election` shortcircuit firing `on_become_leader`. -- **(2.2) ha.c shared sequence**: raft path stores `CTRL_PRIMARY` directly with NO `ha_do_promote` (term synced at entry — avoids double-increment); `ha_promote_to_primary` keeps `ha_do_promote` (owns its term). Both call static `ha_register_persist_hooks(c)` (BEFORE noop append) and static `ha_arm_own_term_gate(c)` (`persist_wal_async(WAL_NOOP)` → `noop_lsn = current_lsn-1` under p->lock → `commit_submit(RETAIN)` → `commit_set_writable_lsn`); append-failure degrades safe (gate unset, promotion completes); non-blocking. `committing_timeout_fn` + enum retained until tasks 4.x/5.2. -- **Unity runner gotcha**: the aggregated controller runner prints the PREVIOUS passing test's `file:line` as a stale prefix on a FAIL line — attribute failures by test NAME, not the printed file:line prefix. -- **(3.1) Legacy path routes through BARE helpers, not `ha_promote_to_primary`**: `ha_elect_evaluate_and_signal`'s PRIMARY branch calls `ha_register_persist_hooks` + `ha_start_primary_heartbeat` + `ha_arm_own_term_gate` directly (matching the raft path), NOT `ha_promote_to_primary` — because the latter calls `ha_do_promote` (term increment) and the legacy branch historically has NO term store/increment (research §4 finding 2). This is a deliberate deviation from design §283's loose wording ("route through ha_promote_to_primary / ha_register_persist_hooks"); the bare-helper route preserves term semantics and is the correct reading. -- **(4.1) No shims — DELETE, don't stub** (design §Transitional-State Deletions / §Design Synthesis: "rewritten or DELETED, not stubbed"). Machinery-subject test cases (convergence checker, committing watchdog, ack-driven promotion, ack counting) were DELETED, not shimmed: test_ha_role.c (5 majority_convergence + 2 committing_timeout_promotes_degraded), test_ha_rdhup_integration.c (rdhup_committing_to_primary_on_ack, rdhup_3node_one_ack_promotes), test_ha_connection_reuse.c (dispatcher_committing_handles_repl_ack, promoted_ep_repl_ack_promotes_to_primary). Kept state-entry cases had their deleted-fn CALLS excised (not shimmed). `test_ha_connection_reuse.c` was overlooked by the spec's §3 census disposition for machinery cases — confirmed in census (line 97) but its ack cases needed DELETE, not UPDATE. -- **(4.2) Residual stale COMMITTING comments in ha.c** (~lines 443, 826, 1240): documentary prose on the legacy raft_cb_become_leader/convergence path — NOT live branches. Scrub in 5.2/6.x (they fail the task-6.2 repo-wide grep). After 4.2, ZERO live code branches on `CTRL_PRIMARY_COMMITTING` in ha.c; the enum value persists in ha.h:64 until 5.2. -- **(5.2) Code is COMMITTING-free; only DOCS remain for task 6.1**: after 5.2 the grep `CTRL_PRIMARY_COMMITTING|HA_DEFAULT_COMMITTING_TIMEOUT_MS|committing_timeout_ms|repl_ack_count|committing_timeout_ctx` is ZERO across `src/controller/` + `src/lib/`. Remaining tree references (task 6.1 docs-sync targets, then 6.2 verifies): `src/docs/cluster-config-split-analysis.md:34` (committing_timeout_ms) AND `src/controller/CLAUDE.md` (TLA+/role-mapping table still names PRIMARY_COMMITTING — 6.1 must scrub ALL its COMMITTING refs, not just the :40 line) plus the steering/skill docs already listed in 6.1. ctrl_role_t now has 4 values: CTRL_PRIMARY=0, CTRL_STANDBY=1, CTRL_INIT=2, CTRL_ELECTING=3. -- **(5.1) Meson 300s timeout is now routinely hit** — barrier tests' commit-deadline waits push controller-suite wall-time over the 300s meson cap under load (`Fail: 0, Timeout: 1`). The AUTHORITATIVE green check is the DIRECT binary run (`src/xce.build/controller/controller_unit_test`, EXIT=0, 167 Tests/0 Failures aggregated). For task 6.2's "suite + ASAN green" gate: run the direct binary OR re-run meson until under-cap; a `Timeout: 1` is a wrapper artifact, NOT a test failure. `test_committing_timeout_default_is_60s` was DELETED in 5.1 (not 5.2) because its body used the uppercase production macro `HA_DEFAULT_COMMITTING_TIMEOUT_MS` (caught by the grep gate); the knob it tested dies in 5.2 regardless. -- **(4.2) Test-leak / suite-timeout facts**: the controller test target sets `ASAN_OPTIONS=detect_leaks=0` (controller/meson.build:163), so a pre-existing suite-wide LSan leak (~153KB, present at baseline 678b7f8, this spec leaks slightly LESS) is suppressed under the meson gate — LSan exit=1 only on a RAW direct-binary run, not a regression. The controller suite runs ~268s against a 300s meson timeout — borderline; the final task-6.2 full regression may TIMEOUT under machine load — re-run if so (it is not a failure). -- **(4.1) Residual for task 6.2 final grep**: 3 DOCUMENTARY PROSE comments in `test_ha_role_lifecycle.c` (~lines 760/815/825) still mention `ha_repl_ack_received`/`committing_timeout_fn` by name (no live code). The 4.1 gate (`src/controller/*.c|*.h`) excludes testsuite so this passed; the task-6.2 repo-wide grep WILL catch them — scrub these comments in 5.x/6.x. Also a non-tracked editor backup `src/controller/.ha.c.~undo-tree~` holds stale refs (irrelevant; not a build input — do not commit). -- **(3.1) Expected-failure set after task 3.1 (8 category-(a), all owned downstream)**: task 4.1 DELETES `test_ha_evaluate_committing.c` (3 fails: test_winning_quorum_enters_committing, test_committing_resets_repl_ack_count, test_election_timeout_fn_fires_evaluate); task 5.1 retargets `test_nonblocking_single_node_election_enters_committing` (test_ha_startup.c), `test_3node_win_calls_both_pushes` (test_ha_repl_push.c), and 3 RDHUP cases (test_ha_rdhup_integration.c: rdhup_full_path_reaches_committing, rdhup_committing_to_primary_on_ack, rdhup_3node_one_ack_promotes). `committing_timeout_fn` now has a benign `-Wunused-function` warning (no arming site left) until task 4.1 deletes it. -- **Lifecycle fixture (test_ha_role_lifecycle.c)**: `lifecycle_cluster_setup` installs `repl_broadcast_send_hook = lc_drop_send_hook` (drop-only, reset to NULL in teardown) — required once promotion broadcasts the WAL_NOOP, else fake `int` endpoints cause an ASAN overflow. Gate cases prime `current_lsn = noop_lsn` so the post-increment append lands the noop at `noop_lsn` (production derives `noop_lsn = current_lsn-1`). diff --git a/.kiro/specs/ha-term-safety/TLA-gap-analysis.md b/.kiro/specs/ha-term-safety/TLA-gap-analysis.md deleted file mode 100644 index d18c6ae0..00000000 --- a/.kiro/specs/ha-term-safety/TLA-gap-analysis.md +++ /dev/null @@ -1,62 +0,0 @@ -# Gap Analysis: XCE HA implementation vs `raft/raft.tla` - -Reference spec: `raft/raft.tla` (Diego Ongaro's canonical Raft TLA⁺, 471 lines). -Implementation: `src/lib/raft/raft.c` (election library) + the controller log/replication/commit layer (`persist.c`, `repl.c`, `ha.c`, `ha_convergence.c`, `config_repl.c`). -Date: 2026-06-21. Grounds the `ha-term-safety` spec and bug-851. - -## Headline - -`raft.c` implements **only the election half** of the TLA⁺ spec — `currentTerm`, `votedFor`, `state`, `RequestVote`, `BecomeLeader`, `UpdateTerm` (on vote RPCs), `Timeout`. It has **no `log`, no `AppendEntries`, no `AdvanceCommitIndex`, no `commitIndex`**. The replicated log and its replication/commit/reconcile are implemented **separately** in the controller (WAL = `log`, catch-up streaming = `AppendEntries`, commit barrier = `AdvanceCommitIndex`, `ha_convergence.c` = conflict reconcile) and **do not follow the TLA⁺ log protocol**. Every bug-851 defect lives in that split: the election module is faithful, but the log layer omits the term-coupling the TLA⁺ spec uses to guarantee Log Matching and Leader Completeness. - -## Action-by-action conformance - -| TLA⁺ action / invariant | Implementation site | Status | -|---|---|---| -| `Timeout` (term++, clear vote, self-vote) | `raft_start_election` (raft.c:61) — `s->term++`, `voted_for=self`, persist before RPC | **Conformant** | -| `RequestVote` freshness `logOk` = `mlastLogTerm > LastTerm ∨ (= ∧ mlastLogIndex ≥ Len)` | `on_vote_eligible` gate (raft.c:171) → controller LSN floor | **GAP G6** — freshness is **LSN-only**; never compares `lastLogTerm` (the WAL has no usable term) | -| `HandleRequestVoteRequest` grant rules (≤ one vote/term, persist) | `raft_on_vote_request` (raft.c:110): stale-term deny, dup-vote deny, persist-before-grant | **Conformant** (Election Safety primitives correct) | -| `BecomeLeader` (votesGranted ∈ Quorum) | `raft_on_vote_reply` step 5 (raft.c:373) `votes_granted ≥ n/2+1` | **Conformant** | -| `UpdateTerm` — **any** RPC with higher term advances `currentTerm`, steps down | `raft_on_vote_request` step 2 + `raft_on_vote_reply` step 1 (vote RPCs **only**) | **GAP G5** — term adoption is wired to **vote RPCs only**; the AppendEntries/heartbeat-equivalent path does **not** route the leader's term into raft state | -| `ClientRequest` — append `[term ↦ currentTerm, value]` | `persist_wal_write*` (persist.c) — `hdr.term = p->term`, `p->term` never set | **GAP G1** — entry term is **always 0**; the log carries no term | -| `AppendEntries` / `HandleAppendEntriesRequest` `logOk` = `mprevLogTerm = log[prevLogIndex].term` | WAL catch-up stream + `persist_wal_write_replicated` contiguity guard (bug-850) | **GAP G2** — replication matches **LSN contiguity only**; no `prevLogTerm` check | -| `HandleAppendEntriesRequest` conflict branch — delete entry whose `term ≠` leader's + all after | `ha_convergence.c` rejoin reconcile (truncates above committed boundary) | **GAP G3** — truncates by committed-boundary heuristic, **not by `(term,lsn)` conflict**; cannot detect same-LSN/different-term divergence | -| `AdvanceCommitIndex` — commit `Max(agreeIndexes)` **only if** `log[Max].term = currentTerm` (Figure-8 safety) | commit barrier + `ha_compute_safe_committed_lsn` | **GAP G4** — current-term check is unenforceable (`term=0`); the own-term `WAL_NOOP` gate is a **partial** mitigation, but the safe-committed computation itself is term-blind | -| `Restart` — keep `currentTerm`, `votedFor`, `log` | recover WAL + `ha-term.bin`; `raft_init` restores `(term, voted_for, voted_for_term)` | **Mostly conformant**, but feeds G5: a restarted node's raft term is only as fresh as what reached `ha-term.bin`; with G5 it never advanced from heartbeats, so it recovers **stale** | -| Election Safety (≤ one leader/term) | emergent from the above | **VIOLATED in practice** — see below | -| Log Matching (same idx+term ⇒ identical prefix) | emergent from the log layer | **VIOLATED** (bug-851) — G1/G2/G3 | -| Leader Completeness (committed entry survives in all future leaders) | emergent | **VIOLATED** (bug-851) — G4 commits term-blind | -| State Machine Safety (same idx applied ⇒ same command) | emergent | **VIOLATED** (bug-851 repro: divergent jobs at same committed LSN) | - -Extensions beyond the TLA⁺ spec (not gaps): PreVote (`raft_begin_pre_vote`, disruption avoidance), single-node short-circuit, even-cluster-size rejection, priority/jitter staggering. - -## How the gaps compose into bug-851 (observed) - -1. **G5** — a standby never adopts the leader's term from heartbeats/replication (confirmed live: rejoined ctrl-1 reported `term: 2` while serving under a term-3 primary). So its raft `currentTerm` lags. -2. On failover that lagged node runs `raft_start_election`, computing the new term as **stale + 1** → it **reuses a term** a prior, now-dead leader already used (observed: term 3 led by ctrl-3 then ctrl-1; term 4 by ctrl-2 then ctrl-3). Election Safety is bypassed not by a double-vote in one live term, but by **term-number reuse across the restart churn** that the vote-only term plumbing can't see. -3. **G1** — each such leader appends its own-term no-op + jobs with `hdr.term = 0` at overlapping LSNs (both wrote `lsn=45`). -4. **G2/G3** — replication/reconcile compare by LSN only, so `(term=3, lsn=45)` and `(term=4, lsn=45)` look identical (both type+term 0) and the divergent suffix is never truncated. -5. **G4** — `committed` advances term-blind "from 1 granted voter" over entries a majority never held. - -Result: all members report `committedLsn=54` over divergent content at LSN 45–54 — a silent committed-prefix divergence (State Machine Safety violation). - -## Configuration is a second, parallel violation - -Config commits through a **separate** index (`committed_config_tag`) with **no term** and tag-based anti-entropy (`config_repl.c`) — it is entirely outside both the TLA⁺ log and `raft.c`. It is exposed to the same term-reuse divergence, plus a state-materialization hazard: a config upgrade does config-only push + restart + **per-node WAL replay under the new config**, with no snapshot to pin an identical post-upgrade state (the `lost_and_found` re-homing is the tell that replay is config-dependent). The TLA⁺ spec assumes config changes are log entries; XCE's are neither log entries nor snapshot-anchored. - -## Mapping gaps → `ha-term-safety` requirements - -| Gap | Requirement that closes it | -|---|---| -| G1 entry term always 0 | R1 (per-entry WAL term recording) | -| G5 term not adopted on replication path; term reuse | R2 (monotonic, durably-established term) | -| G6 vote freshness LSN-only | R2.4 + R1 (term available to vote gate) | -| G2 no `prevLogTerm` match | R3.1 (`(term,lsn)` comparison) | -| G3 no conflict truncation | R3.2 (truncate divergent `(term,lsn)` suffix) | -| G4 commit term-blind | R4.1/R4.5 (commit only `(term,lsn)`-quorum-corroborated; own-term) | -| Config outside log / no term | R7 (config marker in unified log) | -| Config-upgrade state re-derive | R8 (full snapshot at change boundary) | -| All of the above | R6 (chaotic-failover regression test) | - -## Note on scope vs `raft.c` - -**DECISION (2026-06-21):** rather than patch the gaps in the controller's split architecture, the chosen direction is to **give `raft.c` a real log and route AppendEntries/commit through it — strict `raft.tla` conformance.** This is the `Raft TLA+ Conformance` series on the roadmap (`raft-log-replication` → `raft-commit-safety` → `raft-sim-conformance`, specs generated 2026-06-21). The library owns the protocol decisions (term on entries, `UpdateTerm` on every RPC, `(term,index)` matching+truncation, current-term commit, `(lastLogTerm,lastLogIndex)` freshness); the controller becomes a **thin callback adapter** (WAL = log, repl = transport, the sync commit barrier triggered via `on_commit_advanced`). `ha-term-safety` is reframed accordingly: its protocol invariants R1–R6 are **realized by the library specs** (not re-implemented in `ha_convergence.c`/`ha_compute_safe_committed_lsn`), and it retains only the controller-side wiring, the config marker + upgrade snapshot (R7/R8), and the live chaotic-failover regression. When `ha-term-safety` design is generated, map each requirement onto the library callbacks rather than controller-internal logic. diff --git a/.kiro/specs/ha-term-safety/design.md b/.kiro/specs/ha-term-safety/design.md deleted file mode 100644 index 4a5868bc..00000000 --- a/.kiro/specs/ha-term-safety/design.md +++ /dev/null @@ -1,443 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: `ha-term-safety` is the **controller-side integration + config + live-regression umbrella** for the Raft TLA+ Conformance library series. It wires the XCE controller as a **thin callback adapter** to the protocol that `raft-log-replication`, `raft-commit-safety`, and `raft-sim-conformance` implement in `src/lib/raft/raft.c`, closing bug-851 (silent committed-prefix divergence under chaotic failover) **without re-implementing any Raft protocol logic in the controller**. (The related configuration-upgrade materialized-state divergence, bug-857, is descoped to `config-upgrade-snapshot`.) - -**Users**: The XCE HA controller cluster (as Raft nodes), operators relying on "acknowledged ⇒ never lost across any failover pattern", and maintainers needing a live regression that reproduces and fences bug-851. - -**Impact**: Today the replicated log, term propagation, reconcile, and commit are hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c`/`config_repl.c` and match by LSN only (term-blind). After this feature the controller **delegates** those decisions to the library through five new callbacks and three new entry points, **stamps the real election term** into the existing 40-byte WAL header, **derives committed config from an in-log `(term,lsn)` marker** instead of a separate index, and ships a live chaotic-multi-kill regression. Configuration **bytes** continue to move through the existing out-of-band config-push transport (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. The Raft protocol itself (logOk, `(term,index)` truncation, current-term commit, freshness) is **not** in this spec — it is realized by the three library specs. - -### Goals -- Wire the controller adapter to the upstream callbacks **verbatim** (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), stamping `wal_entry_hdr_t.term` from the library's term (R1) and routing incoming heartbeat/replication through the library so term adoption happens on every RPC (R2/R3). -- Replace the term-blind safe-commit (`ha_compute_safe_committed_lsn`) and committed-boundary reconcile (`ha_rejoin_reconcile`) with the library's `commit_index` (via `on_commit_advanced`) and `(term,lsn)` truncation (via `on_truncate_from`), **preserving** the synchronous commit barrier + persist-before-ACK durability and the clean-failover behaviors of bugs 848/849/850 (R4/R5). -- Order configuration changes through the same `(term,lsn)` log as a lightweight `WAL_CONFIG_MARKER` (version_tag + sha256, no bytes); derive committed config from the latest committed marker; deliver the configuration **bytes** via the existing out-of-band config-push transport, hash-verified against the committed marker before activation (R7). -- Ship a live `xce_test` chaotic-multi-kill regression asserting cross-member `(term,lsn)`-fingerprinted committed-prefix identity (R6). - -### Non-Goals -- **The Raft protocol itself** — logOk/`prevLogTerm` match, `(term,index)` conflict truncation, the Figure-8 current-term commit rule, `(lastLogTerm,lastLogIndex)` freshness, `UpdateTerm`-on-any-RPC — owned by `raft-log-replication` + `raft-commit-safety`; consumed verbatim. -- **The commit-barrier / durability mechanism** — the synchronous barrier, group-commit flusher, persist-before-ACK, reply gate, durable-floor freshness — owned by `ha-commit-durability`; this spec only changes the barrier's *input*, never its mechanism. -- **The in-process conformance oracle** — Election Safety / Log Matching / Leader Completeness / State Machine Safety in `raft_sim.c` — owned by `raft-sim-conformance`; the R6 live regression complements it, it does not duplicate it. -- **Configuration-upgrade materialized-state consistency** — establishing a full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state (former R8) — is **descoped to the independent `config-upgrade-snapshot` spec**, which HARD-depends on `entity-bearing-snapshots`. It is **out of scope here**; this spec delivers the config marker (consensus) plus out-of-band config-push byte delivery, not a full upgrade snapshot. -- Voting-set membership changes / joint consensus (`ha-config-bootstrap`); bugs 854/847/845/828 (tracked separately). - -## Boundary Commitments - -### This Spec Owns -- **The controller callback-adapter bodies** implementing the five upstream callbacks over the WAL / repl transport / commit barrier (and an in-process route for incoming AppendEntries/heartbeat). -- **`wal_entry_hdr_t.term` value stamping**: a `persist_set_term` seam so every append records the library's live term; reading `hdr.term` back on `replay_wal` and feeding it to `on_get_entry_term` + the library's restore (`initial_log_len`, last-log-term). -- **`ha_convergence` delegation**: replacing the committed-boundary truncate with routing the leader's AppendEntries through `raft_on_append_entries_request` + `on_truncate_from`. -- **Commit-source switch**: feeding the library's `commit_index` (via `on_commit_advanced`) into the existing synchronous barrier, retiring the term-blind `ha_compute_safe_committed_lsn` as the commit source. -- **`WAL_CONFIG_MARKER` (R7)**: the new WAL entry type + append-via-`raft_log_append` + committed-config-derived-from-latest-committed-marker, retiring `committed_config_tag` as an authoritative index. -- **Out-of-band config-push byte delivery + hash-verify-before-activate (R7)**: keying the existing config-push transport (`OPCode_replConfigPush`) to the committed marker's version_tag + sha256, verifying held bytes against the marker hash before activation, and fencing a member from serving under mismatched configuration. -- **The live R6 regression**: the chaotic-multi-kill `xce_test` script + the `walcompare` cross-member `(term,lsn,pcrc)` divergence tool. - -### Out of Boundary -- All Raft protocol decisions (library: `raft-log-replication`, `raft-commit-safety`). -- The commit barrier, durability/fsync, persist-before-ACK, durable-floor freshness mechanism (`ha-commit-durability`). -- The conformance oracle / four-invariant checkers (`raft-sim-conformance`). -- **Configuration-upgrade materialized-state consistency** — the full entity-bearing state snapshot at the change boundary (so no member re-derives post-upgrade state) and its `xzarray_freeze`/`thaw` + entity encode/decode payload — is **descoped to the independent `config-upgrade-snapshot` spec** (which HARD-depends on `entity-bearing-snapshots`). This spec ships the config marker (consensus) + out-of-band config-push byte delivery; it does **not** build the upgrade snapshot or entity serialization. -- Membership/quorum transitions (`ha-config-bootstrap`); the on-disk WAL byte layout (unchanged — value fix only). - -### Allowed Dependencies -- `src/lib/raft/raft.{c,h}` after the three upstream specs land: the five callbacks + entry points + `raft_append_result_t` + `commit_index`/`next_index[]`/`match_index[]` + `(lastLogTerm,lastLogIndex)` freshness, consumed **verbatim**. -- The existing controller infrastructure: `persist.c` WAL/recovery, `repl.c` transport, `commit.c` synchronous barrier + durability (`ha-commit-durability`), the out-of-band **config-push** transport (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`), the ten already-wired election callbacks (`s_real_cbs`, `ha.c:1309-1320`). -- The test infrastructure: `waldump.{c,h}`, `xce_test` script primitives, `/xce-deploy-ha`. - -### Revalidation Triggers -Re-check this design if any of the following change: -- The signature/semantics of any upstream callback (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) or entry point (`raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), including `raft_append_result_t`. -- The 1-based-index ↔ WAL-LSN identity or the "term 0 == absent" convention. -- The addition of `last_log_term` to `raft_on_vote_request`/`raft_on_pre_vote_request` (the controller must plumb it onto the wire and read it from `hdr.term` on recovery). -- The `commit_index` restore convention (`initial_commit_index`) or `initial_log_len`. -- The config-push transport contract (`OPCode_replConfigPush`) or the marker's `{version_tag, sha256}` key — the byte delivery is hash-verified against the committed marker before activation. -- The synchronous commit barrier / persist-before-ACK contract (`ha-commit-durability`) — `on_commit_advanced` must continue to *trigger* it unchanged. -- The split of the configuration-upgrade materialized-state work into `config-upgrade-snapshot` — if that spec changes how it consumes the `(term,lsn)` config marker, re-check the marker contract here. - -## Architecture - -### Existing Architecture Analysis - -The controller already drives the **election** half of the library through ten callbacks wired in `s_real_cbs` (`ha.c:1309-1320`): vote/PreVote send, freshness gate (`raft_cb_vote_eligible`, `ha.c:299-360`), promotion/demotion (`raft_cb_become_leader`/`follower`, `ha.c:880-1242`), and term persistence (`raft_cb_persist` → `ha_persist_term`, `ha.c:1251-1294, 2441-2474`). The controller uses **only** `src/lib/raft/raft.c` (no legacy `src/controller/raft.c`). The **log** half is hand-rolled and term-blind: -- Every WAL append stamps `hdr.term = p->term` (`persist.c:588/726/1040`) but `p->term` is **never assigned** → term 0 everywhere (G1). The 40-byte header already has the `term` field (`wal_types.h:29-39`) — a value fix, not a layout change. -- Incoming heartbeats/replication never adopt a higher term (`ha_consume_heartbeat_committed` drops stale only, `ha.c:5013-5036`) (G5). -- Reconcile truncates by committed-boundary, not `(term,lsn)` (`ha_rejoin_reconcile`, `ha_convergence.c:295-366`; `persist_truncate_wal_above` LSN-only, `persist.c:2455-2565`) (G3). -- Safe commit is computed term-blind from voter LSNs (`ha_compute_safe_committed_lsn`, `ha.c:480-509`) (G4); config commits through a separate `committed_config_tag` with no term (`config_repl.c:941-979`), with the bytes pushed out-of-band via `OPCode_replConfigPush` — the marker (consensus) will replace the term-blind index, while the byte transport is reused as-is. - -The durability layer (`ha-commit-durability`) is correct and must be preserved: `majority_candidate_locked` self-caps at `persist_durable_lsn` (`commit.c:79-138`), persist-before-ACK on standbys (`persist_wal_write_replicated`), the pre-promotion eligibility fence (`ha.c:933-951`), advertise-real-durable-floor freshness, and the contiguity guard (`persist.c:691-706`, bug-850). - -This design follows the **same Ports-and-Adapters discipline already established by the election callbacks**: the library decides; the controller performs I/O. - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TB - subgraph Library["src/lib/raft/raft.c — PROTOCOL OWNER (NOT this spec)"] - APP["raft_log_append (ClientRequest)"] - AEREQ["raft_on_append_entries_request<br/>UpdateTerm + logOk + truncate"] - AEREP["raft_on_append_entries_reply"] - ADV["raft_advance_commit<br/>current-term Figure-8"] - VOTE["vote freshness lastLogTerm,lastLogIndex"] - end - subgraph Adapter["src/controller — THIN ADAPTER (this spec)"] - PE["on_persist_entry<br/>persist_set_term + WAL write"] - GET["on_get_entry_term<br/>WAL index / replay read-back"] - SEND["on_send_append_entries<br/>repl envelope prevLogTerm+entryTerm"] - TRUNC["on_truncate_from<br/>persist_truncate_wal_above"] - COMMIT["on_commit_advanced<br/>TRIGGER existing sync barrier"] - ROUTE["heartbeat/repl intake -> raft_on_append_entries_request<br/>(replaces ha_consume_heartbeat + ha_rejoin_reconcile)"] - end - subgraph Config["Config marker + out-of-band byte delivery (this spec)"] - MARK["WAL_CONFIG_MARKER<br/>version_tag + sha256 via raft_log_append"] - CPUSH["config-push byte delivery<br/>OPCode_replConfigPush, hash-verify vs marker before activate"] - end - subgraph Preserved["PRESERVED policy (other specs)"] - BARRIER["sync commit barrier + persist-before-ACK<br/>ha-commit-durability"] - CFGXPORT["existing config-push transport<br/>OPCode_replConfigPush"] - end - subgraph Test["Live regression (this spec)"] - WC["walcompare term,lsn,pcrc"] - SCRIPT["xce_test chaotic multi-kill"] - end - APP --> PE - AEREQ --> GET & PE & TRUNC - ROUTE --> AEREQ - AEREP --> SEND - ADV --> COMMIT - COMMIT --> BARRIER - MARK --> APP - CPUSH --> CFGXPORT - SCRIPT --> WC -``` - -**Architecture Integration**: -- **Selected pattern**: thin callback adapter (Ports-and-Adapters), identical to the election-callback pattern already in `ha.c`. The library owns protocol; the controller owns I/O. -- **Domain boundaries**: protocol decisions (append/truncate/commit/vote) in `raft.c`; durable I/O (WAL write, repl send, barrier, config-push transport) in the controller; the marker + out-of-band byte delivery in this spec; the configuration-upgrade materialized-state snapshot is descoped to `config-upgrade-snapshot`. -- **Existing patterns preserved**: `s_real_cbs` callback wiring, persist-before-ACK, the synchronous barrier, the pre-promotion fence, the contiguity guard, "controller never self-terminates". -- **Steering compliance**: `ha-async-io.md` (no blocking in iomgr/vtask; barrier on its own path), `persistence.md` (durable floor / acknowledged⇒durable preserved; never persist commit index), `error-handling.md` (status returns, fail-loud non-promotable fence, never `exit`/`abort`), `testing.md` (TDD; tests in `testsuite/` before C changes). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Protocol core | `src/lib/raft/raft.c` (gnu17, after 3 upstream specs) | Owns Raft log/commit/freshness | Consumed verbatim; not modified here | -| Adapter / WAL | `src/controller/persist.c` + `wal_types.h` | `hdr.term` stamping, recovery read-back, truncate | 40-byte header unchanged; `persist_set_term` seam added | -| Adapter / transport | `src/controller/repl.c`, `typemap.h` | carry `prevLogTerm` + entry term on AppendEntries | extend repl envelope fields | -| Adapter / commit | `src/controller/commit.c` (`ha-commit-durability`) | barrier triggered by `on_commit_advanced` | mechanism unchanged | -| Config | `src/controller/config_repl.c` + `wal_types.h` | `WAL_CONFIG_MARKER`, marker-derived committed config | retires `committed_config_tag` index | -| Config bytes | existing config-push transport (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`) | deliver bytes out-of-band; hash-verify vs marker before activate | transport reused as-is; no snapshot | -| Test | `xce_test` + `waldump.{c,h}` + `/xce-deploy-ha` | live chaotic regression + `walcompare` | ~90% infra exists | - -## File Structure Plan - -### Modified Files -- `src/include/wal_types.h` — add `WAL_CONFIG_MARKER = 50` to `wal_entry_type`; add the marker payload struct `wal_config_marker_t { uint64_t version_tag; uint8_t sha256[32]; }` (no config bytes). **No change to `wal_entry_hdr_t` (40 bytes preserved, R1.7).** -- `src/include/persist.h` — declare `persist_set_term(persist_t *p, uint64_t term)`; declare recovery accessors that surface per-entry term + restored `log_len`/last-log-term for the library init. -- `src/controller/persist.c` — implement `persist_set_term` (sets `p->term`); make `replay_wal` retain `hdr.term` per LSN (in-memory term index) so `on_get_entry_term` and library restore can read it; `persist_truncate_wal_above` reused unchanged by `on_truncate_from`. -- `src/controller/repl.c` + `src/include/typemap.h` — extend the AppendEntries-carrying envelope (`wal_catchup_entry_t`/`capnp_repl_envelope_t`) to carry `prev_log_index`, `prev_log_term`, and the entry's `term` explicitly, so a standby persists the leader's term (not local `p->term`). -- `src/controller/ha.c` — wire the five new callbacks into `s_real_cbs` (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`); add the heartbeat/replication intake that routes through `raft_on_append_entries_request` (replacing the term-blind `ha_consume_heartbeat_committed` term handling); drive `raft_log_append` on the primary append path; pass `last_log_term` on the vote wire and decode it on intake; retire `ha_compute_safe_committed_lsn` as the commit source (the library's `commit_index` drives the barrier via `on_commit_advanced`). -- `src/controller/ha_convergence.c` — replace `ha_rejoin_reconcile`'s committed-boundary truncate with delegation: route the leader's AppendEntries through the library; `on_truncate_from` performs the truncate. Keep the non-promotable retry-fence behavior (R3.5/R5). -- `src/controller/config_repl.c` — `do_upgrade_config_with_bytes` appends a `WAL_CONFIG_MARKER` via `raft_log_append` (replacing the `committed_config_tag` push/commit); derive committed config from the latest committed marker in the committed prefix; key the existing out-of-band config-push (`OPCode_replConfigPush`) to the marker's `{version_tag, sha256}`, hash-verify the held bytes against the committed marker before activating, and fence a member from serving under mismatched configuration (R7.5/R7.8–R7.10). The full entity-bearing upgrade snapshot is **out of scope** (`config-upgrade-snapshot`). -- `src/controller/commit.c` — no mechanism change; `on_commit_advanced` calls into the existing `commit_advance_locked`/`commit_set_writable_lsn` path with the library-supplied `commit_index`. - -### New Files -- `src/controller/testsuite/walcompare.{c,h}` — cross-member WAL diff by `(lsn → term, pcrc)`; nonzero exit on committed-prefix divergence. Reuses the `waldump.c` reader/decoder (roadmap-planned tooling, `roadmap.md:55`). -- `src/controller/testsuite/test_term_safety_adapter.c` — unit/integration tests for the adapter callbacks (term stamping, `on_get_entry_term` read-back, truncate delegation, `on_commit_advanced` barrier trigger, marker append/derive). -- `src/tools/test-term-safety-chaos.sh` (or an `xce_test` script under `testsuite/`) — the R6 chaotic-multi-kill driver invoking `walcompare`. - -> Each file keeps one responsibility: protocol stays in `raft.c` (untouched); `persist.c` = WAL I/O + term value; `ha.c`/`ha_convergence.c` = callback bodies + intake routing; `config_repl.c` = marker + out-of-band config-push byte delivery; `testsuite/` = regression. The File Structure Plan reflects the boundary: no protocol logic file is created or modified, and no upgrade-snapshot file is added (descoped to `config-upgrade-snapshot`). - -## System Flows - -### Replicated append → standby persist with the leader's term (R1, R2, R3) - -```mermaid -sequenceDiagram - participant P as Primary library (raft.c) - participant PA as Primary adapter (this spec) - participant SA as Standby adapter (this spec) - participant S as Standby library (raft.c) - Note over P: raft_log_append(value) — stamps s->term - P->>PA: on_persist_entry(idx, term, entry) - PA->>PA: persist_set_term(p, term) then WAL write hdr.term=term - Note over P: raft_send_append_entries(peer, commit_index) - P->>PA: on_send_append_entries(peer, prevIdx, prevTerm, leaderCommit, entry+entryTerm) - PA-->>SA: repl envelope (carries prevLogTerm + entryTerm) - Note over S: raft_on_append_entries_request()<br/>UpdateTerm FIRST (G5) then logOk (G2) - S->>SA: on_get_entry_term(prevIdx) %% logOk - alt term conflict at idx (G3) - S->>SA: on_truncate_from(idx) then on_persist_entry(idx, leader_term) - else append - S->>SA: on_persist_entry(idx, leader_term) %% standby persists LEADER term - end - Note over SA: persist-before-ACK preserved -> WalAck only on durable -``` - -Flow-level decisions: -- The standby persists the **leader's** entry term (carried explicitly on the envelope), closing the `persist_wal_write_replicated` "best-effort `p->term`" hazard. `persist_set_term` keeps `p->term` aligned with the library term for the local-append path. -- Incoming heartbeats are 0-entry AppendEntries routed through the **same** `raft_on_append_entries_request`, so a higher-term heartbeat triggers `UpdateTerm` (G5) — the controller no longer needs `ha_consume_heartbeat_committed` to adopt term. -- The contiguity guard (bug-850) and persist-before-ACK (ha-commit-durability) are unchanged and sit beneath `on_persist_entry`. - -### Commit advance → existing barrier; config marker (R4, R7) - -```mermaid -sequenceDiagram - participant L as Leader library (raft.c) - participant A as Adapter (this spec) - participant B as Sync commit barrier (ha-commit-durability, PRESERVED) - Note over L: replication updated match_index[] (library) - Note over L: raft_advance_commit() — current-term Figure-8 (G4) - alt commit_index strictly advances - L->>A: on_commit_advanced(commit_index) - A->>B: feed safe commit position into existing barrier - Note over B: majority self-cap at durable_lsn + persist-before-ACK<br/>+ own-term writable gate (UNCHANGED) - else held (prev-term / no quorum) - Note over L: no callback, barrier untouched - end - Note over A: committed config = latest WAL_CONFIG_MARKER<br/>within committed prefix (derived, not a separate index) -``` - -Flow-level decisions: -- `on_commit_advanced` **triggers** the barrier; it never performs durability itself. `ha_compute_safe_committed_lsn` is retired as the commit source (the library's current-term-gated `commit_index` replaces it), but the post-election truncate-to-safe at promotion and the durable self-cap remain (R5). -- A config change is an ordinary committed log entry: the `WAL_CONFIG_MARKER` commits through the same barrier and reconciles through the same `(term,lsn)` matching (R7.1/R7.3/R7.6/R7.11). The committed config is the latest committed marker (R7.7). - -### Configuration bytes — out-of-band config-push, hash-verified against the committed marker (R7.5, R7.8–R7.10) - -```mermaid -flowchart TB - M["WAL_CONFIG_MARKER commits at term,lsn<br/>(version_tag + sha256, no bytes)"] --> NEED{"local bytes match<br/>marker sha256?"} - NEED -->|no| PUSH["obtain bytes via existing config-push<br/>OPCode_replConfigPush, keyed to version_tag"] - PUSH --> VERIFY["verify obtained bytes sha256 == marker hash"] - NEED -->|yes| VERIFY - VERIFY -->|match| ACTIVATE["activate configuration"] - VERIFY -->|mismatch| FENCE["serve-fence: keep member non-serving<br/>under new config; retry (R7.9)"] -``` - -Flow-level decision: the marker (consensus) is the authority; the **bytes travel out-of-band on the existing config-push transport** (`OPCode_replConfigPush`), never in a WAL payload (R7.4). A member obtains and **hash-verifies the bytes against the committed marker before activating** (R7.8), fences itself on mismatch (R7.9), and on recovery activates the configuration referenced by the latest committed marker (R7.10). The full materialized-state **upgrade snapshot** (former R8) is **out of scope** — see `config-upgrade-snapshot`. - -## Requirements Traceability - -| Requirement | Summary | Adapter component(s) | Library/upstream realization | Flow | -|-------------|---------|----------------------|------------------------------|------| -| 1.1, 1.2 | record live/promotion term in entry | `on_persist_entry` + `persist_set_term`; `raft_log_append` (incl. own-term NOOP) | library stamps `s->term` | Append flow | -| 1.3, 1.4 | transmit + persist leader term on replication | `on_send_append_entries` carries entryTerm; standby `on_persist_entry` writes leader term | `raft_on_append_entries_request` | Append flow | -| 1.5 | recovery reads back per-entry term | `replay_wal` term index; `on_get_entry_term`; restore `initial_log_len`/last-log-term | library init restore | — | -| 1.6 | no zero/placeholder term while term valid | `persist_set_term` keeps `p->term` live | — | — | -| 1.7 | preserve 40-byte WAL layout (value fix) | `wal_entry_hdr_t` unchanged | — | — | -| 1.8 | legacy zero term = lowest-known for matching | `on_get_entry_term` maps legacy 0 → lowest-known | library `(term,index)` match tolerant | — | -| 2.1, 2.2 | persist won term before appending; not committed until majority | `raft_cb_persist`→`ha_persist_term`; own-term `WAL_NOOP` gate (`commit_set_writable_lsn`) | `BecomeLeader` + commit rule | Commit flow | -| 2.3, 2.4, 2.6 | no re-elect at stale term; adopt higher term on any RPC | heartbeat/repl intake → `raft_on_append_entries_request`; `on_persist` | `UpdateTerm` (G5) | Append flow | -| 2.5 | at most one leader advances committed per term | library current-term commit + freshness | `raft-commit-safety` G4/G6 | Commit flow | -| 3.1, 3.2 | reconcile by `(term,lsn)`; truncate divergent suffix | `ha_convergence` delegation → `on_truncate_from` | `raft_on_append_entries_request` (G3) | Append flow | -| 3.3 | never truncate committed prefix | library truncates only above match point; `on_truncate_from` honors it | library | — | -| 3.4 | reconciled prefix byte-identical to leader | standby persists leader entry+term | library | Append flow | -| 3.5, 3.6 | non-promotable retry; no false committed | keep loud non-promotable fence; do not report divergent as committed | `ha-commit-durability` fence preserved | — | -| 4.1, 4.5 | advance committed only on `(term,lsn)` quorum + current-term | `on_commit_advanced` feeds barrier with library `commit_index` | `raft_advance_commit` (G4) | Commit flow | -| 4.2, 4.3, 4.4 | committed entry byte-identical/preserved across leadership | library Leader Completeness + barrier durability | library + `ha-commit-durability` | Commit flow | -| 5.1 | clean single failover converges identically | unchanged promotion truncate-to-safe + library matching | `raft_cb_become_leader` | — | -| 5.2 | recovery-on-promotion + advertise-real-LSN (848/849) | `raft_cb_become_leader` + `raft_cb_vote_eligible` unchanged | preserved | — | -| 5.3 | contiguity guard, no forward gap (850) | `persist_wal_write_replicated` guard unchanged beneath `on_persist_entry` | preserved | — | -| 5.4 | no added steady-state ack latency | barrier path untouched; `on_commit_advanced` only changes source | preserved | — | -| 6.1, 6.2, 6.3 | chaotic regression asserts `(term,lsn,content)` prefix identity | `walcompare` + chaos script | — | — | -| 6.4 | no acked-committed job absent post-run | `expect-job $jobid` per acked submit | — | — | -| 6.5 | pre-fix repro fails, post-fix passes | negative-direction assertion (term=0 pre-fix) | — | — | -| 6.6 | config changes interleaved → same committed config ref | `walcompare` checks committed marker tag+hash | R7 derive | Config flow | -| 7.1–7.4, 7.6, 7.7, 7.11 | config as `(term,lsn)` marker, no bytes, term-stamped, reconciled, derived, majority-committed | `WAL_CONFIG_MARKER` via `raft_log_append`; marker-derived committed config | library log/commit/reconcile | Config marker flow | -| 7.5, 7.8, 7.9, 7.10 | bytes via out-of-band config-push; verify hash before activate; fence on mismatch; activate from latest committed marker on recovery | config-push byte delivery (`OPCode_replConfigPush`) keyed to marker `{version_tag, sha256}` + hash-verify-before-activate + serve-fence | existing config-push transport | Config bytes flow | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|-------|--------|--------------|--------------------------|-----------| -| WAL term adapter | Adapter | Stamp/read-back the library term in `hdr.term` | 1.1–1.8 | library callbacks (P0), `persist.c` (P0) | Service, State | -| AppendEntries intake/route | Adapter | Route heartbeat/repl through the library; term adoption | 2.3–2.6, 3.1–3.4 | `raft_on_append_entries_request` (P0), repl envelope (P0) | Service | -| Reconcile delegation | Adapter | Replace committed-boundary truncate with `on_truncate_from` | 3.1–3.6 | library (P0), `persist_truncate_wal_above` (P0), fence (P1) | Service | -| Commit-source switch | Adapter | Feed library `commit_index` into existing barrier | 4.1–4.5, 5.4 | `on_commit_advanced` (P0), `commit.c` barrier (P0) | Service | -| Clean-failover preservation | Adapter | Keep 848/849/850 behaviors unchanged | 5.1–5.4 | promotion/vote callbacks, contiguity guard (P0) | Service | -| Config marker | Adapter | `WAL_CONFIG_MARKER` in unified log; derive committed config | 7.1–7.4,7.6,7.7,7.11 | `raft_log_append` (P0), `config_repl.c` (P0) | Service, State | -| Config byte delivery | Adapter | Out-of-band config-push keyed to marker; hash-verify-before-activate; serve-fence on mismatch | 7.5,7.8,7.9,7.10 | config-push transport `OPCode_replConfigPush` (P0), `config_repl.c` (P0) | Service | -| Live regression | Test | Chaotic-multi-kill + `walcompare` | 6.1–6.6 | `xce_test`, `waldump`, `/xce-deploy-ha` (P0) | Test | - -### Adapter Callback → Controller Implementation Site (the core seam table) - -| Library callback / entry point (upstream) | Controller implementation site | Behavior | -|---|---|---| -| `on_persist_entry(ctx,index,term,entry,len)` | `ha.c` callback → `persist_set_term(p,term)` + `persist_wal_async`/`persist_wal_write_replicated` (`persist.c:588/726`) | Stamp `hdr.term=term` at LSN=index; persist-before-ACK preserved. | -| `on_get_entry_term(ctx,index)->term\|0` | `ha.c` callback → in-memory WAL term index from `replay_wal`/live append | Return recorded `hdr.term` for LSN; legacy 0 → lowest-known (R1.8); 0 if absent. | -| `on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)` | `ha.c` callback → `repl.c` envelope build (`typemap.h` extended fields) | Enqueue AppendEntries carrying `prevLogTerm` + entry `term`; `leaderCommit = commit_index`. | -| `on_truncate_from(ctx,index)` | `ha.c` callback → `persist_truncate_wal_above(p,index-1)` (`persist.c:2455`) | Delete LSN≥index; replaces `ha_rejoin_reconcile` committed-boundary heuristic. | -| `on_commit_advanced(ctx,commit_index)` | `ha.c` callback → `commit.c` existing barrier (`commit_advance_locked`/`commit_set_writable_lsn`) | Feed safe commit position; **trigger** barrier, do not replace it. | -| `raft_log_append(s,entry,len,&idx)` | primary append path in `ha.c`/`config_repl.c` (incl. `WAL_CONFIG_MARKER`) | Library stamps `s->term`, assigns LSN; drives `on_persist_entry`. | -| `raft_on_append_entries_request(...)` | heartbeat/repl intake in `ha.c` (replaces `ha_consume_heartbeat_committed` term handling) | Library does `UpdateTerm`+logOk+truncate/append via callbacks. | -| `raft_on_append_entries_reply(...)` | repl ACK intake in `ha.c` | Library updates `match_index[]`/`next_index[]`. | -| `raft_advance_commit(s)` | driven after repl ACK in `ha.c` | Library computes current-term-gated `commit_index`; may fire `on_commit_advanced`. | -| `raft_on_vote_request/pre_vote_request(...,last_log_term,...)` | existing vote callbacks in `ha.c`, extended | Controller decodes candidate `last_log_term` from wire; reads own via `on_get_entry_term(log_len)`. | - -### WAL term adapter - -| Field | Detail | -|-------|--------| -| Intent | Make `hdr.term` carry the library's election term on every entry and surface it on recovery | -| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8 | - -**Responsibilities & Constraints** -- `persist_set_term(persist_t *p, uint64_t term)` assigns `p->term` (the missing seam); called by `on_persist_entry` and on term adoption so no append stamps 0 while a valid term is in effect (R1.6). On-disk `wal_entry_hdr_t` is unchanged (40 bytes, R1.7). -- `replay_wal` retains each `hdr.term` in an in-memory per-LSN index so `on_get_entry_term` answers from recovered state (R1.5); a legacy entry with term 0 maps to the lowest-known term for matching (R1.8). -- Standbys persist the **leader's** entry term (carried on the envelope), not local `p->term` (R1.3/R1.4). - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -/* Set the controller's live election term so subsequent WAL appends stamp it - * into hdr.term (R1.1/R1.6). Called from on_persist_entry and on term adoption. - * No on-disk layout change; value fix only (R1.7). */ -void persist_set_term(persist_t *p, uint64_t term); - -/* Recovered per-entry term for an LSN (1-based == index), feeding on_get_entry_term - * and the library's init restore. Returns 0 when absent; legacy-0 is mapped to the - * lowest-known term by the adapter for (term,lsn) matching (R1.8). */ -uint64_t persist_entry_term(const persist_t *p, uint64_t lsn); -``` -- Preconditions: `p` valid; `lsn` within `[1, current_lsn)`. -- Postconditions: `on_get_entry_term(lsn)` returns the recorded term; appends stamp the live term. -- Invariants: `hdr.term` never 0 while a valid term is in effect; 40-byte layout preserved. - -### Config marker - -| Field | Detail | -|-------|--------| -| Intent | Order config changes through the unified `(term,lsn)` log as a lightweight marker | -| Requirements | 7.1, 7.2, 7.3, 7.4, 7.6, 7.7, 7.11 | - -**Responsibilities & Constraints** -- `do_upgrade_config_with_bytes` (`config_repl.c:1001-1163`) appends `WAL_CONFIG_MARKER` (`wal_config_marker_t = {version_tag, sha256[32]}`, **no bytes**, R7.2/R7.4) via `raft_log_append`, so it inherits term-stamping (R7.3), commit (R7.11), and `(term,lsn)` reconcile/truncate (R7.6). -- Committed config is derived from the **latest committed marker within the committed prefix** (R7.7); the separate `committed_config_tag` index is retired as authoritative (it may remain as a derived cache). - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -/* Append a config-change marker (version_tag + sha256, no config bytes) to the - * unified replicated log via raft_log_append. Returns RAFT_OK / the append rc. - * The config bytes travel out-of-band via config-push (R7.5), never the marker. */ -int config_append_marker(cluster_t *c, uint64_t version_tag, - const uint8_t sha256[32], uint64_t *out_lsn); - -/* Derive the committed configuration reference from the latest WAL_CONFIG_MARKER - * at or below committed_lsn (R7.7). */ -int config_committed_marker(const cluster_t *c, uint64_t *version_tag, - uint8_t sha256_out[32]); -``` -- Preconditions: leader for append; valid `sha256`. -- Postconditions: marker committed ⇒ committed config reference advances; reconcile truncates a divergent marker exactly like a job entry. -- Invariants: config bytes never enter a WAL payload (R7.4). - -### Config byte delivery - -| Field | Detail | -|-------|--------| -| Intent | Deliver config bytes out-of-band via the existing config-push transport; hash-verify against the committed marker before activation | -| Requirements | 7.5, 7.8, 7.9, 7.10 | - -**Responsibilities & Constraints** -- Move the configuration bytes through the **existing out-of-band config-push transport** (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`), keyed to the committed marker's `{version_tag, sha256}` — never through a WAL payload (R7.4/R7.5). -- When a committed marker references bytes not yet held locally, obtain them via config-push and **verify the obtained bytes' sha256 against the marker hash before activating** (R7.8). -- If a member's locally held bytes do not match the latest committed marker's hash, **keep the member from serving under the new configuration and keep retrying** rather than activating mismatched config (R7.9) — the "controller never self-terminates" loud non-promotable fence, never `exit`/`abort`. -- On recovery, activate the configuration referenced by the **latest committed marker**, obtaining the bytes from out-of-band storage (R7.10). - -**Contracts**: Service [x] - -##### Service Interface -```c -/* Ensure the bytes for the latest committed config marker are held locally and - * hash-verified against the marker, then activate. Obtains bytes via config-push - * (OPCode_replConfigPush) when absent. Returns RAFT_OK on activate; a non-success - * rc keeps the member fenced (non-serving under the new config) and retrying - * (R7.8/R7.9/R7.10). */ -int config_activate_committed(cluster_t *c); -``` -- Preconditions: a committed `WAL_CONFIG_MARKER` exists in the committed prefix. -- Postconditions: on success the active config matches the marker's `{version_tag, sha256}`; on hash mismatch the member stays fenced and retries. -- Invariants: bytes are never sourced from a WAL payload; activation only after hash-verify against the committed marker. - -> **Note**: the full materialized-state **upgrade snapshot** (embed entities + config in one artifact, install on every member so no member re-derives post-upgrade state — former R8) is **out of scope** here and lives in the independent **`config-upgrade-snapshot`** spec, which HARD-depends on `entity-bearing-snapshots`. This spec delivers config bytes via the interim out-of-band config-push path that `config-upgrade-snapshot` later supersedes for the upgrade case. - -### Live regression (R6) - -| Field | Detail | -|-------|--------| -| Intent | Reproduce bug-851 and assert cross-member `(term,lsn)` committed-prefix identity | -| Requirements | 6.1, 6.2, 6.3, 6.4, 6.5, 6.6 | - -**Responsibilities & Constraints** -- `walcompare <dirA> <dirB> ...` diffs members by `(lsn → term, pcrc)`; nonzero exit on any committed-prefix divergence (R6.1/R6.2/R6.3), reusing `waldump.c`'s decoder. -- The chaos script drives repeated primary kills interleaved with `jobsub` + config upgrades via `/xce-deploy-ha` + `kill_node`/`start_node` + deterministic `--wal-sync-ms`, then asserts `expect-job $jobid` for every acked submit (R6.4) and `walcompare` identity incl. committed config marker (R6.6). -- R6.5: on **pre-fix** code (term=0) the repro must fail (`walcompare` detects divergence / `expect-job` loss); on fixed code it passes. - -**Contracts**: Test [x] - -## Data Models - -### Domain Model -- **Log entry**: `(index==LSN, term, type, payload)` — the unified `(term,lsn)` log for both job state and config markers. Index 1-based == WAL LSN (library convention). -- **Config-change marker**: `wal_config_marker_t = {uint64_t version_tag, uint8_t sha256[32]}` — a reference, never the bytes. -- **Committed config**: derived value = the latest `WAL_CONFIG_MARKER` at or below `committed_lsn`. - -### Logical Data Model -- `wal_entry_type` gains `WAL_CONFIG_MARKER = 50` (between `WAL_NOOP=40` and `WAL_CHECKPOINT=100`). -- `persist_t.term` becomes a written field via `persist_set_term`; a recovered per-LSN term index is added to `persist_t` (in-memory only). -- The repl envelope (`wal_catchup_entry_t`/`capnp_repl_envelope_t`) gains explicit `prev_log_index`, `prev_log_term`, and entry `term` fields on the AppendEntries path. -- **No on-disk WAL layout change** (R1.7): `wal_entry_hdr_t` stays 40 bytes with `term` at offset 0. - -### Data Contracts & Integration -- **AppendEntries wire**: carries `(prevLogIndex, prevLogTerm, leaderCommit, entry, entryTerm)` — the standby persists `entryTerm`, not local `p->term`. -- **Vote wire**: carries `(lastLogTerm, lastLogIndex)`; the controller decodes `lastLogTerm` and passes it to the library; own value via `on_get_entry_term(log_len)`. -- **Config bytes**: delivered out-of-band via config-push (`OPCode_replConfigPush`), keyed to the marker's `{version_tag, sha256}`, hash-verified against the committed marker before activation; never in a WAL payload. The full materialized-state upgrade snapshot is out of scope (`config-upgrade-snapshot`). - -## Error Handling - -### Error Strategy -The adapter follows "controller never self-terminates" (`error-handling.md`, `ha-async-io.md`): callbacks return status; on a non-recoverable condition the controller enters a **loud, non-promotable fence and keeps retrying** — it never `exit`/`abort`. Persist-before-ACK and the durable self-cap are unchanged. - -### Error Categories and Responses -- **Persist failure** (`on_persist_entry` rc≠0) → propagate; library does not advance the local log; standby does not ACK (persist-before-ACK preserved). -- **Truncate failure** (`on_truncate_from` rc≠0) → report a non-success request result so the leader retries; the member stays non-promotable until reconciled (R3.5). -- **Unreconcilable member** → loud permanent fence, non-promotable, keep retrying catch-up (R3.5/R5; never report false convergence, R3.6). -- **Config hash mismatch** (R7.9) → keep the member from serving under the new config; retry; do not activate mismatched config (`config_activate_committed` returns non-success; member stays fenced). -- **Config bytes absent for a committed marker** (R7.8) → obtain via config-push and hash-verify before activate; do not activate on a missing or mismatched payload. - -### Monitoring -Reuse `raft_cb_trace` and existing INFO/ERROR logging for: term-stamp, term-adopt-on-AE, conflict-truncate, commit-advance-trigger, config-marker-commit, config-byte-verify, and the loud fences. - -## Testing Strategy - -TDD-mandatory (`testing.md`): tests in `src/controller/testsuite/` precede C changes; gate on the ANSI-stripped `:FAIL:` count; run final durability/regression validation on a real fsync-able filesystem. - -### Unit Tests (adapter callbacks, in `test_term_safety_adapter.c`) -1. `on_persist_entry` stamps `hdr.term` from the library term (via `persist_set_term`) and `on_get_entry_term` reads it back; legacy-0 maps to lowest-known (R1.1–1.8). -2. AppendEntries intake routes a higher-term heartbeat through `raft_on_append_entries_request`, the library adopts the term, and `ha_persist_term` records it (R2.3–2.6 / G5). -3. `on_truncate_from` deletes the divergent suffix and the standby persists the leader entry+term; committed prefix untouched (R3.1–3.4 / G3). -4. `on_commit_advanced` feeds the library `commit_index` into the existing barrier and the WAIT entry releases only on durable majority; `ha_compute_safe_committed_lsn` no longer the commit source (R4.1/R4.5, R5.4). -5. `config_append_marker` appends `WAL_CONFIG_MARKER` (no bytes) and `config_committed_marker` derives committed config from the latest committed marker; a divergent marker truncates like a job entry (R7.1–7.7). - -### Integration Tests -1. Clean single failover converges to byte-identical WALs incl. terms, preserving recovery-on-promotion + advertise-real-LSN + contiguity (R5.1–5.3). -2. Marker commit + reconcile under a forced leader change keeps every member on the same committed config reference (R6.6, R7.11). -3. Config byte delivery: a committed marker whose bytes are obtained via config-push activates only after the bytes' sha256 matches the marker hash; an injected hash mismatch keeps the member fenced (non-serving under the new config) and retrying (R7.8/R7.9/R7.10). - -### E2E / Live Regression (R6 — the headline) -1. `xce_test` chaotic-multi-kill: 3-node cluster, repeated primary SIGKILL interleaved with `jobsub` and config upgrades under deterministic `--wal-sync-ms`; after the run `walcompare` asserts `(term,lsn,pcrc)` committed-prefix identity across all members and `expect-job $jobid` confirms no acked-committed job is absent (R6.1–6.4, R6.6). -2. Negative direction: the same script on **pre-fix** code (term=0) reproduces bug-851 (committed-prefix divergence / job loss); on fixed code it passes (R6.5). - -## Resolved Scope Note (descope of former R8) - -The former Requirement 8 (full entity-bearing configuration-upgrade snapshot) was a **blocking item** in the prior draft of this design: it required a full materialized-state snapshot at the change boundary, which depends on **entity-bearing snapshots** that do not exist yet (today snapshots are a 16-byte LSN sentinel; the `xzarray_freeze`/`thaw` machinery is built but dormant — the unstarted `entity-bearing-snapshots` backlog item). - -**Resolution (2026-06-21): the materialized-state-consistency-on-upgrade work is descoped into the independent [`config-upgrade-snapshot`](../config-upgrade-snapshot/brief.md) spec, which HARD-depends on `entity-bearing-snapshots`. It is out of scope here.** - -What this spec ships now (no snapshot dependency): -- R1–R6: per-entry term, monotonic term adoption, `(term,lsn)` reconcile, term-safe committed prefix, and the live chaotic-multi-kill regression — all realized on the library + existing transport. -- R7: the config-change **marker** (consensus, `(term,lsn)`, no bytes) in the unified log, with configuration **bytes** delivered via the existing **out-of-band config-push transport** (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. `config-upgrade-snapshot` later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land. - -Constraints preserved by the descope: config bytes are never carried in the WAL marker (R7.2/R7.4), and no member re-derives committed state from a stale separate index — the marker is the consensus authority. diff --git a/.kiro/specs/ha-term-safety/requirements.md b/.kiro/specs/ha-term-safety/requirements.md deleted file mode 100644 index cc79f2fc..00000000 --- a/.kiro/specs/ha-term-safety/requirements.md +++ /dev/null @@ -1,131 +0,0 @@ -# Requirements Document - -## Introduction - -This feature closes a Raft safety gap (bug-851) in the XCE HA controller cluster. Under chaotic controller failover (repeated loss of the primary), members can reach a state where every node reports the same `committedLsn` over a contiguous, CRC-intact WAL, yet the nodes hold **different entries at the same LSNs** — a silent committed-prefix divergence that loses acknowledged jobs. A clean single failover converges correctly today (validated end-to-end: bugs 848, 849, 850), so the defect is specific to leadership churn. - -Root cause has three compounding parts, each addressed by a requirement area below: -1. The election term is not monotonic across the cluster — a leader can act in a term that was never durably propagated to a majority, letting a restarted node re-elect at the same term, so two leaders share one term. -2. The per-entry WAL term is never recorded (always `0`), so the WAL carries no Raft term information. -3. Rejoin reconcile cannot perform `(term, lsn)` log-matching, so divergent same-LSN tails are never truncated and `committed` advances over entries a majority never held. - -The objective is to restore the Raft Log Matching and Leader Completeness guarantees so that any entry reported `committed` is byte-identical on every member, including across arbitrary failover patterns — without regressing clean-failover behavior or the on-disk WAL format. - -The same divergence class latently affects configuration, which today commits through a separate index (`committed_config_tag`) with no per-entry term. Rather than maintain two consistency domains, this feature unifies the **consensus** half: a configuration change is recorded as a **lightweight in-log marker** carrying only a reference to the configuration (version tag + content hash), ordered in the same `(term, lsn)` log as job state, so configuration inherits the same term-safety and reconcile. The **configuration bytes stay out of the WAL records** (the marker carries only a reference); the bytes continue to move through the existing **out-of-band config-push transport**, hash-verified against the committed marker before activation. - -The remaining hazard — a configuration *upgrade* leaving members with divergent *materialized* state because XCE's state machine is interpreted against the configuration and each member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — requires a full entity-bearing state snapshot at the change boundary, which does not exist yet (snapshots are presently a 16-byte LSN sentinel). That work is **descoped to the independent `config-upgrade-snapshot` spec** (depends on entity-bearing snapshots) and is out of scope here. - -## Boundary Context - -- **In scope**: - - Recording the live election term in every WAL entry and carrying it through replication and recovery. - - Making the election term monotonic and durably established before a new leader appends entries (no two leaders per term). - - `(term, lsn)` log-matching during rejoin/reconcile, including truncation of divergent local suffixes. - - Preserving the committed-prefix consistency invariant across arbitrary failover sequences. - - **Unified ordering of configuration changes and job state through the same replicated log**, by recording each configuration change as a lightweight in-log marker that references the configuration (version tag + content hash) rather than carrying its bytes, so configuration inherits the same term-safety and reconcile guarantees. Configuration bytes continue to move through the existing out-of-band config-push transport, hash-verified against the committed marker. - - A regression test that drives chaotic multi-kill failover and asserts cross-member committed-prefix identity (including configuration markers). -- **Out of scope** (tracked separately): - - **Configuration-upgrade materialized-state consistency** (full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state) — **descoped to the independent `config-upgrade-snapshot` spec**, which depends on entity-bearing snapshots (`entity-bearing-snapshots` backlog item; today snapshots are a 16-byte LSN sentinel). This spec ships the config marker (consensus) but not the full-snapshot upgrade. - - Client primary-discovery being positional (bug-854). - - Catch-up payload drop across multiple WAL segments (bug-847). - - Each job producing two `JOB_SUBMIT` WAL entries (bug-845). - - Snapshot-based prefix backfill / empty-rejoin gap (bug-828, `ha-commit-durability`). - - Changes to the controller voting set (adding/removing a controller peer): such membership transitions change the quorum itself and remain governed by `ha-config-bootstrap`; this spec orders the config-change marker uniformly but does not introduce joint-consensus quorum transitions. -- **Adjacent expectations**: - - Relies on the existing election/PreVote machinery (`ha-prevote`, `ha-single-leader-role`) and the commit barrier (`ha-commit-engine`) to provide term and quorum signals; this feature constrains how those signals are recorded and reconciled, and does not redesign the election protocol itself. - - Relies on persistence recovery (`ha-commit-durability`) for the recovered durable floor; this feature adds the term dimension to what recovery and reconcile compare. - - Carries configuration bytes via the existing out-of-band config-push transport, never via WAL entry payloads (the WAL holds only the `(term, lsn)` marker); the bytes are hash-verified against the committed marker before activation. The follow-on `config-upgrade-snapshot` spec later supersedes this byte delivery with a full entity-bearing upgrade snapshot once entity-bearing snapshots exist. - - Realizes its protocol invariants (R1–R6) through the **raft library** callbacks delivered by the `Raft TLA+ Conformance` series (`raft-log-replication`, `raft-commit-safety`); this spec is the controller-side **adapter + config marker + live regression**, not a re-implementation of the Raft protocol. - - Complements `ha-consistency-invariants` and `ha-convergence-recovery` by supplying the per-entry term those invariants assume already exists. - -## Requirements - -### Requirement 1: Per-entry WAL term recording - -**Objective:** As an operator of an HA controller cluster, I want every WAL entry to record the election term under which it was created, so that entries from different terms are distinguishable and Raft log-matching is possible. - -#### Acceptance Criteria -1. When the primary appends a WAL entry, the Controller shall record in that entry the value of the leader's current election term in effect at append time. -2. When the primary appends its own-term no-op on promotion, the Controller shall record the newly established term in that no-op entry. -3. When the primary replicates a WAL entry to a standby, the Controller shall transmit the entry's recorded term so that the standby persists the leader's term, not a substitute value. -4. While applying a replicated entry, the standby shall persist the entry with the same term the leader recorded for that entry. -5. When a member recovers its WAL at startup, the Controller shall read back each entry's recorded term and make it available to the reconcile and promotion logic. -6. The Controller shall not record a fixed or zero placeholder term for entries created while a valid election term is in effect. -7. The Controller shall preserve the existing on-disk WAL entry header size and field layout, treating the term as a value fix rather than a format change. -8. Where a WAL entry predates this feature and carries a legacy zero term, the Controller shall treat it as belonging to the lowest-known term for matching purposes rather than failing recovery. - -### Requirement 2: Monotonic, durably-established election term - -**Objective:** As an operator, I want a node to become an acting leader only at a term that is strictly greater than any term already used in the cluster and durably established, so that two leaders never share a term. - -#### Acceptance Criteria -1. When a node wins an election, the Controller shall durably persist the won term locally before it appends any entry as leader. -2. While a node has not propagated its won term to a majority of members, the Controller shall not treat any entry it appends in that term as committed. -3. If a node recovers a stale election term after restart, then the Controller shall not begin or win a new election at a term less than or equal to any term it can observe from a reachable peer or its own persisted WAL. -4. When a node initiates an election, the Controller shall choose a prospective term strictly greater than the highest term it has observed locally and from reachable peers. -5. If two members would otherwise act as leader in the same term, then the Controller shall ensure at most one of them can advance committed in that term and the other steps down without having committed divergent entries. -6. When a member observes a higher term from a peer than its own, the Controller shall adopt the higher term and abandon any leadership it holds at the lower term. - -### Requirement 3: `(term, lsn)` log-matching reconcile - -**Objective:** As an operator, I want a rejoining member to converge its WAL to the current leader's log, so that divergent tails created during churn do not persist. - -#### Acceptance Criteria -1. When a member rejoins and synchronizes with the current leader, the Controller shall compare its local WAL to the leader's by `(term, lsn)` pairs, not by `lsn` alone. -2. If a member's local entry at a given LSN has a term that does not match the leader's entry at that LSN, then the Controller shall truncate that local entry and every entry after it before accepting the leader's entries. -3. When the Controller truncates divergent local entries, it shall not truncate any entry that is part of the cluster's committed prefix. -4. When reconcile completes, the member's WAL prefix up to the leader's committed point shall be byte-identical to the leader's. -5. If a member cannot reconcile to a committed-consistent state with the leader, then the Controller shall keep that member non-promotable and continue retrying rather than reporting false convergence. -6. While a member holds an unreconciled divergent suffix, the Controller shall not report those divergent entries as committed. - -### Requirement 4: Committed-prefix consistency invariant - -**Objective:** As a client and operator, I want any entry reported committed to be identical on every member, so that an acknowledged job is never lost or replaced after a failover. - -#### Acceptance Criteria -1. The Controller shall only advance `committed` to an LSN whose entry is held with a matching `(term, lsn)` by a majority of members, applying the same rule to job-state entries and configuration-change markers. -2. When any member reports an LSN as committed, the Controller shall ensure the entry at that LSN is byte-identical on every member that reports the same or a higher committed LSN. -3. If an entry was acknowledged to a client as committed, then the Controller shall preserve that exact entry across all subsequent leadership changes. -4. While leadership changes occur, the Controller shall never replace, reorder, or drop an entry within the established committed prefix. -5. When a new leader computes its safe committed point from granted voters, the Controller shall not include an LSN whose `(term, lsn)` is not corroborated by the required quorum. - -### Requirement 5: Clean-failover behavior preserved - -**Objective:** As an operator, I want the validated clean-failover and recovery behavior to keep working unchanged, so that this fix introduces no regression. - -#### Acceptance Criteria -1. When a single clean failover occurs (one primary killed, one standby promoted, old primary restarted), the Controller shall converge all members to a byte-identical WAL as it does today. -2. While a promoted standby recovers and advertises its recovered applied LSN, the Controller shall preserve the existing recovery-on-promotion and advertise-real-LSN behavior (bugs 849 and 848). -3. When a standby falls behind and the primary streams catch-up entries, the Controller shall preserve the existing contiguity behavior that refuses a forward gap without dropping live entries (bug 850). -4. The Controller shall not increase committed-acknowledgment latency for the clean, single-leader steady state beyond its current behavior. - -### Requirement 6: Chaotic-failover regression verification - -**Objective:** As a maintainer, I want an automated test that reproduces the chaotic-failover divergence, so that the fix is proven and protected against regression. - -#### Acceptance Criteria -1. When the regression scenario drives repeated primary kills interleaved with job submissions, the test shall assert that every member's committed WAL prefix is byte-identical. -2. When the test inspects member WALs, it shall compare entries by `(term, lsn)` and content so that a same-LSN divergence is detected, not masked by equal LSN ranges. -3. If any member holds a divergent entry within the committed prefix, then the test shall fail with the diverging LSN and the per-member entry identities reported. -4. When the regression scenario completes, the test shall confirm that no entry acknowledged as committed during the run is absent from any surviving member. -5. The test shall reproduce the original bug-851 failure on the pre-fix code and pass on the fixed code. -6. When the regression scenario interleaves configuration changes with the chaotic kills, the test shall assert that every member converges to the same committed configuration reference (version tag and content hash). - -### Requirement 7: Unified configuration ordering with out-of-band configuration data - -**Objective:** As an operator, I want configuration changes ordered and committed through the same replicated log and term machinery as job state, while the configuration bytes stay out of the WAL records (carried by the existing out-of-band configuration transport), so that configuration inherits term-safety and reconcile without bloating the log. - -> **Scope note (descope 2026-06-21):** the full materialized-state consistency of a configuration *upgrade* — establishing a full state snapshot at the change boundary so no member re-derives post-upgrade state — is moved to the independent **`config-upgrade-snapshot`** spec, which depends on entity-bearing snapshots (a snapshot that carries entity state does not exist yet; today it is a 16-byte LSN sentinel). This requirement keeps the consensus half (the in-log `(term, lsn)` marker) and continues to deliver the configuration *bytes* through the existing out-of-band config-push transport, hash-verified against the marker. - -#### Acceptance Criteria -1. When a configuration change is committed, the Controller shall append a configuration-change marker to the same replicated log used for job state, ordered by the same `(term, lsn)` sequence. -2. The Controller shall record in each configuration-change marker a reference to the configuration — its version tag and content hash — and shall not place the configuration bytes in the marker. -3. The Controller shall record the live election term in each configuration-change marker, identically to job-state entries (Requirement 1). -4. The Controller shall not store configuration bytes in WAL entry payloads. -5. The Controller shall move the configuration bytes through the existing out-of-band configuration transport (config push), keyed to the marker's version tag and content hash, not through the WAL. -6. When reconcile compares logs by `(term, lsn)`, the Controller shall truncate a divergent configuration-change marker exactly as it truncates a divergent job-state entry. -7. The Controller shall derive the committed configuration from the latest configuration-change marker within the committed log prefix, rather than from a separate configuration commit index. -8. While a configuration-change marker is committed but its referenced configuration bytes are not yet present locally, the Controller shall obtain the bytes through the out-of-band transport and verify them against the marker's content hash before activating the configuration. -9. If a member's locally held configuration bytes do not match the content hash of the latest committed configuration marker, then the Controller shall keep that member from serving under the new configuration and continue retrying rather than activating mismatched configuration. -10. When a member recovers, the Controller shall activate the configuration referenced by the latest committed configuration-change marker, obtaining the bytes from out-of-band storage. -11. The Controller shall preserve the existing majority-acknowledgment requirement for a configuration change, now expressed as the configuration-change marker reaching the committed prefix. diff --git a/.kiro/specs/ha-term-safety/research.md b/.kiro/specs/ha-term-safety/research.md deleted file mode 100644 index 8e8cc46e..00000000 --- a/.kiro/specs/ha-term-safety/research.md +++ /dev/null @@ -1,122 +0,0 @@ -# Research & Design Decisions - -## Summary - -- **Feature**: `ha-term-safety` -- **Discovery Scope**: Complex Integration (controller-side adapter + config marker + out-of-band config-push byte delivery + live regression umbrella for the Raft TLA+ Conformance library series; the configuration-upgrade materialized-state snapshot is descoped to `config-upgrade-snapshot`). -- **Key Findings**: - - The WAL on-disk header **already carries** `wal_entry_hdr_t.term` at offset 0 of a 40-byte layout (`src/include/wal_types.h:29-39`). R1.7 ("preserve on-disk layout, value fix only") is satisfied by the format as-is. The defect is purely that `persist_t.term` (`src/include/persist.h:160`) is **never assigned** anywhere — every `hdr.term = p->term` site (`persist.c:588, 726, 1040`) therefore stamps `0`. The fix is a `persist_set_term` seam fed by the library's term, plus reading `hdr.term` back on replay (`replay_wal` reads but discards it today, `persist.c:~1500-1620`). - - The controller uses the **new library** `src/lib/raft/raft.c` exclusively (no legacy `src/controller/raft.c` exists); ten election callbacks are already wired via `s_real_cbs` (`ha.c:1309-1320`). The log/commit callbacks defined by the three upstream specs (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) are **not yet wired** — that wiring is this spec's core deliverable. - - **G5 (term not adopted on the replication/heartbeat path) is confirmed in code**: only `raft_on_vote_request`/`reply` route a higher term into raft state (`raft.c:125-134` → `raft_cb_persist` → `ha_persist_term`, `ha.c:1251-1294, 2441-2474`). The heartbeat consume path (`ha_consume_heartbeat_committed`, `ha.c:5013-5036`) only *drops* stale heartbeats; it never adopts a higher term. With the library now owning `UpdateTerm`-on-any-RPC, the controller must route incoming heartbeat/replication RPCs through `raft_on_append_entries_request`. - - **G3 (reconcile is committed-boundary, not `(term,lsn)`) is confirmed**: `ha_rejoin_reconcile` (`ha_convergence.c:295-366`) clamps the truncation cutoff to `committed_lsn` and truncates the suffix blindly via `persist_truncate_wal_above` (LSN-only, `persist.c:2455-2565`). It cannot detect same-LSN/different-term divergence. The library's `raft_on_append_entries_request` + `on_truncate_from` replace this decision. - - **R8 SNAPSHOT FEASIBILITY → RESOLVED BY DESCOPE (2026-06-21)**: the snapshot-feasibility investigation found that snapshots are a **16-byte LSN sentinel stub** today (`"XCESNAP\0"` + checkpoint_lsn; `write_snapshot` `persist.c:1068-1101`, `load_snapshot` `persist.c:1280-1307`) carrying **zero entity state**; the COW machinery a full upgrade snapshot needs (`xzarray_freeze`/`xzarray_thaw`, `zarray.c:695-757`) is built and unit-tested but **dormant** (`kvmgr.c:297-298` `vfreeze = NULL /* wired in P0-2 */`, zero production callers); the roadmap's **"entity-bearing snapshots (Task 11)"** (`roadmap.md:43`) is unstarted. Because the full configuration-upgrade materialized-state snapshot is a **HARD dependency on entity-bearing-snapshots that does not exist**, the former Requirement 8 was **descoped into the independent `config-upgrade-snapshot` spec** (which HARD-depends on `entity-bearing-snapshots`). `ha-term-safety` ships R1–R7 with **no snapshot dependency**: R7's config **marker** (consensus) is in-log, and the config **bytes** are delivered via the existing **out-of-band config-push transport** (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. The config marker never depended on entity-bearing snapshots; only the descoped upgrade snapshot did. - -## Research Log - -### Upstream library callback contract (the seam this spec wires to) - -- **Context**: This spec must delegate the Raft protocol to the library and wire the controller as a thin adapter, reusing the upstream callback names verbatim. -- **Sources Consulted**: `.kiro/specs/raft-log-replication/design.md`, `.kiro/specs/raft-commit-safety/design.md`, `.kiro/specs/raft-sim-conformance/design.md`, `TLA-gap-analysis.md`. -- **Findings**: - - `raft-log-replication` owns: `raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`, `raft_append_result_t`; callbacks `on_persist_entry(ctx,index,term,entry,len)`, `on_get_entry_term(ctx,index)->term|0`, `on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)`, `on_truncate_from(ctx,index)`; per-peer `next_index[]`/`match_index[]`; **index 1-based == WAL LSN**; **term 0 == absent**; `UpdateTerm` applied on every RPC (G5); conflict branch = truncate-then-append in one handler call (G3). - - `raft-commit-safety` owns: `raft_advance_commit(s)` with the Figure-8 current-term rule (G4); `on_commit_advanced(ctx,commit_index)` fired only on strict advance — **triggers** the existing barrier, never replaces it; vote/PreVote freshness becomes `(lastLogTerm,lastLogIndex)` lexicographic (G6), with the candidate's `last_log_term` added to `raft_on_vote_request`/`raft_on_pre_vote_request` signatures. - - `raft-sim-conformance` owns the in-process oracle (Election Safety / Log Matching / Leader Completeness / State Machine Safety) — the verification cross-reference; the live regression here (R6) **complements** it (the sim proves the library; the live test proves the controller adapter wiring). -- **Implications**: Every R1–R6 protocol invariant is realized by the library; ha-term-safety implements the **adapter callback bodies** + the term plumbing on the wire + the reconcile delegation + the live regression. The controller must NOT re-implement `logOk`, conflict truncation, the current-term commit rule, or freshness — those live in `raft.c`. - -### WAL term stamping & persistence seam (R1, R2) - -- **Sources Consulted**: `wal_types.h:29-59`, `persist.h:160`, `persist.c:588/726/1040/1500-1620/2455-2565`, `typemap.h:338-387`, `repl.c:239-311`. -- **Findings**: - - `wal_entry_hdr_t` is 40 bytes, `_Static_assert`-locked, `term` already present at offset 0 → **R1.7 is free**. - - `persist_t.term` exists but `grep "p->term ="` / `"->term ="` returns **no assignment** in persist.c → every entry stamps term 0 (G1). Fix: a `persist_set_term(p, term)` seam called from the adapter when the library establishes/adopts a term. - - Replication transport: the active wire format is `wal_catchup_entry_t` (`typemap.h:338-351`) which has **no term field**; the term currently rides implicitly inside the serialized header bytes. `capnp_repl_envelope_t.term` exists (`typemap.h:380-387`) but is a test-only mapped type. The adapter's `on_send_append_entries` must carry `prevLogTerm` + the entry's term explicitly so a standby persists the leader's term, not its local `p->term` (closing the `persist_wal_write_replicated` "best-effort p->term" hazard at `persist.c:719-726`). - - `replay_wal` reads `hdr.term` but discards it; recovery must surface per-entry term to feed `on_get_entry_term` and restore `initial_log_len`/last-log-term for the library at init. - - Next free WAL type id: `WAL_NOOP=40`, `WAL_CHECKPOINT=100` → `WAL_CONFIG_MARKER = 50` slots cleanly (R7). - -### Term adoption, reconcile, commit, promotion seams (R2, R3, R4, R5) - -- **Sources Consulted**: `ha.c:113-1320, 480-509, 880-1033, 5013-5036, 5338-5402`, `ha_convergence.c:240-384`, `commit.c:79-395`, `cluster.c:1200-1299`, `myapp.c` standby branch, `persist.c:691-706`. -- **Findings**: - - **R2/G5**: heartbeat/replication never adopt a higher term (`ha_consume_heartbeat_committed` only drops stale). Routing incoming heartbeat/repl through `raft_on_append_entries_request` makes the library do `UpdateTerm`; the adapter's `on_persist` (already `ha_persist_term`) durably records it. Monotonic-term durability before appending as leader (R2.1/2.2) is realized by the library's `BecomeLeader` + the own-term `WAL_NOOP` gate already present (`raft_cb_become_leader` arms `commit_set_writable_lsn`, `ha.c:880-1033` / `commit.c:383-395`). - - **R3/G3**: `ha_rejoin_reconcile` committed-boundary clamp replaced by routing the leader's AppendEntries through `raft_on_append_entries_request`; the library decides match/truncate/append and calls `on_truncate_from` (adapter → `persist_truncate_wal_above`) + `on_persist_entry`. - - **R4/G4**: `ha_compute_safe_committed_lsn` (`ha.c:480-509`) is term-blind (collects voter LSNs/tags, no term). It is superseded by the library's `commit_index` (current-term-gated) delivered via `on_commit_advanced`, which **triggers** the existing synchronous barrier. The barrier and durability (`majority_candidate_locked` self-caps at `persist_durable_lsn`, `commit.c:79-138`; persist-before-ACK; `wait_done_locked` own-term gate) are PRESERVED AS POLICY (ha-commit-durability) — only the *input* (safe commit position) changes source. - - **R5**: clean-failover behaviors to preserve — recovery-on-promotion + truncate-to-safe (`raft_cb_become_leader`, `ha.c:917-919`), pre-promotion eligibility fence (`ha.c:933-951`, bugs 848/849), advertise-real durable floor for freshness (`raft_cb_vote_eligible`, `ha.c:299-360`), and the contiguity guard refusing a forward gap (`persist_wal_write_replicated`, `persist.c:691-706`, bug-850). These remain; the library's `(term,index)` matching is strictly additive on top. - -### Config replication seam (R7) - -- **Sources Consulted**: `config_repl.c:941-979, 1001-1163, 1857-1944`, `ha.c:5338-5402, 4980-5036`, `ha_convergence.c:372-384`. -- **Findings**: config commits today through a **separate** `committed_config_tag` index (`push_config_to_standbys` waits majority ACK then `atomic_store committed_config_tag`, `config_repl.c:965-970`), carried on heartbeats (`ha.c:5343-5345`), with **no term** and tag-based anti-entropy — entirely outside the WAL `(term,lsn)` log. This is the second parallel violation. R7 replaces it: a `WAL_CONFIG_MARKER` entry (version_tag + sha256, **not bytes**) appended via `raft_log_append`, ordered/committed/reconciled in the same `(term,lsn)` log; committed config derived from the latest committed marker within the committed prefix (replaces the separate index). Config **bytes** continue to travel via the **existing out-of-band config-push transport** (`OPCode_replConfigPush`), now keyed to the marker's `{version_tag, sha256}` and hash-verified against the committed marker before activation — never in the WAL record. (The reverted byte-delivery decision: the prior draft routed bytes through a full upgrade snapshot; that snapshot is descoped to `config-upgrade-snapshot`, so the byte transport reverts to the existing config-push path.) - -### Live regression infrastructure (R6) - -- **Sources Consulted**: `src/controller/testsuite/waldump.{c,h}`, `test.c:129-1760`, `.kiro/specs/ha-commit-durability/repro-851/`, `INVESTIGATION-failover-job-loss.md`, `test_commit_barrier_rollout.c`, `src/tools/deploy-ha.sh`, `test-commit-durability.sh`, `roadmap.md:55`. -- **Findings**: Infrastructure is ~90% present. - - `waldump_path()` (`waldump.h:42`) already decodes per-entry `(LSN, TERM, TYPE, PLEN, PCRC)` and validates the CRC chain, with a payload-CRC fingerprint for cross-member comparison and nonzero exit on gaps/crc-break. - - `xce_test` script primitives exist with exit-code verdicts: `jobsub` (captures `$jobid`), `wait-primary`, `expect-jobs N`, `expect-job $jobid`, `ctrlstatus`, `waldump`, `!shell` escape, `$?`/`$jobid` vars, `g_session_failed` → process exit code (`test.c:129-1760`). - - `/xce-deploy-ha` + `deploy-ha.sh`/`start.sh` launch a 3-node cluster; `kill_node`/`start_node` + deterministic `--wal-sync-ms 30000` drive chaotic kills (`test-commit-durability.sh:113-198`). - - **Gaps to build**: (a) `waldump` currently prints `TERM=0` for all entries because of G1 — once R1 lands it will show real terms; the R6 test must assert per-LSN term identity across members; (b) a `walcompare <dirA> <dirB>` cross-member `(term,lsn,pcrc)` divergence diff (already on the roadmap as planned tooling, `roadmap.md:55`) is the missing assertion primitive. R6 builds the chaotic-multi-kill script + `walcompare`, reusing all existing primitives. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| **Thin callback adapter (chosen)** | Controller implements the upstream callback ports over WAL/repl/commit; library owns all protocol decisions | Honors the TLA-gap-analysis DECISION; one consistency domain; no protocol logic duplicated in controller; sim already proves the library | Requires deleting/neutralizing the term-blind controller logic (`ha_compute_safe_committed_lsn`, committed-boundary reconcile) rather than extending it | Matches Ports-and-Adapters already established by the election callbacks | -| Patch the controller's split log (rejected) | Add `(term,lsn)` matching directly into `ha_convergence.c` + `ha_compute_safe_committed_lsn` | No library dependency | Re-implements raft.tla in the controller; two consistency domains; cannot be sim-verified; explicitly rejected by the 2026-06-21 DECISION | Rejected | -| Config bytes in WAL (rejected) | Carry config bytes in `WAL_CONFIG_MARKER` payload | Self-contained log | Bloats the log; R7.2/R7.4 forbid it | Rejected by requirements | - -## Design Decisions - -### Decision: Unify R1–R6 protocol onto the library; controller is a thin adapter - -- **Context**: TLA-gap-analysis DECISION (2026-06-21) mandates the library own the protocol; the controller adapts. -- **Alternatives Considered**: (1) thin adapter (chosen); (2) patch controller split-log (rejected — duplicates protocol, unverifiable). -- **Selected Approach**: Wire five new library callbacks to controller I/O sites; route incoming heartbeat/replication through `raft_on_append_entries_request`; drive append via `raft_log_append`; replace the term-blind safe-commit and committed-boundary reconcile with the library's `commit_index`/`on_truncate_from`. No protocol logic is added to the controller. -- **Rationale**: Single consistency domain; the deterministic sim already verifies the protocol, so the live regression need only prove the *wiring*. -- **Trade-offs**: Must neutralize existing term-blind controller paths (a deletion, not an extension); upstream signature changes are revalidation triggers. -- **Follow-up**: Confirm the upstream callback/entry-point signatures are stable before wiring (Revalidation Triggers below). - -### Decision: Generalize job-state and config-change onto one `(term,lsn)` replicated log (R7) - -- **Context**: Config is a second parallel consistency domain (`committed_config_tag`, no term) exposed to the same divergence. -- **Selected Approach**: Record each config change as a lightweight `WAL_CONFIG_MARKER` (version_tag + sha256, no bytes) appended via `raft_log_append` so it inherits term-safety, reconcile, and commit. Derive committed config from the latest committed marker; retire the separate `committed_config_tag` commit index. -- **Rationale**: One log, one reconcile, one commit rule — the Generalization lens: config change is a special case of "an ordered committed log entry". -- **Trade-offs**: Config bytes are delivered out-of-band via the existing config-push transport (`OPCode_replConfigPush`), hash-verified against the committed marker before activation — no snapshot dependency. During migration, `committed_config_tag` becomes a derived value, not an authoritative index. -- **Follow-up**: R7's marker is implementable now (no entity-bearing snapshots needed); the config bytes ride the existing config-push install path. The full materialized-state upgrade snapshot is descoped to `config-upgrade-snapshot` — see the resolved decision below. - -### Decision (RESOLVED 2026-06-21): former R8 descoped into `config-upgrade-snapshot`; R7 reverts to out-of-band config-push byte delivery - -- **Context**: The former Requirement 8 required a **full entity-bearing state snapshot** embedding the new config, installed on every member, so no member replays pre-upgrade WAL under the new config (the bug-857 hazard). -- **Finding (the prior blocker)**: Snapshots are a 16-byte LSN sentinel (`persist.c:1068-1101/1280-1307`); the entity-serialization machinery (`xzarray_freeze/thaw`) is dormant (`kvmgr.c:297`); roadmap "entity-bearing snapshots (Task 11)" is **unstarted** (`roadmap.md:43`). The full upgrade snapshot is therefore a HARD dependency on entity-bearing-snapshots, which does not exist. -- **Alternatives Considered**: - 1. Make entity-bearing-snapshots a hard prerequisite and sequence the upgrade snapshot after it, inside this spec. - 2. Carry config bytes in the WAL marker — **violates R7.2/R7.4**. - 3. Per-node replay under new config — the exact bug-857 hazard. - 4. **Descope the materialized-state-on-upgrade work into an independent spec (selected)** — split the entity-snapshot blocker out of `ha-term-safety` so R1–R7 can ship now. -- **Selected Approach**: The configuration-upgrade materialized-state consistency (former R8) is **descoped into the independent `config-upgrade-snapshot` spec**, which **HARD-depends on `entity-bearing-snapshots`**. `ha-term-safety` retains R7's consensus half (the in-log `(term,lsn)` config marker) and **reverts** the configuration-byte delivery to the **existing out-of-band config-push transport** (`OPCode_replConfigPush`), keyed to the marker's `{version_tag, sha256}` and hash-verified against the committed marker before activation. No snapshot is built or depended on here. `config-upgrade-snapshot` later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land. -- **Rationale**: Do not paper over an infeasible requirement, and do not block the implementable R1–R7 (protocol term-safety + config marker) on an unstarted entity-serialization backlog. The marker + commit + reconcile (R7) never required entities; only the materialized-state install did. The byte delivery falls back to the already-working config-push path. -- **Trade-offs**: `ha-term-safety` no longer delivers the full upgrade-snapshot guarantee (now owned by `config-upgrade-snapshot`); the interim config-push byte delivery is superseded for the upgrade case once that spec lands. -- **Follow-up**: `config-upgrade-snapshot` brief + requirements written (`../config-upgrade-snapshot/{brief.md,requirements.md}`); it anchors on this spec's `(term,lsn)` config marker and consumes `entity-bearing-snapshots`. - -### Decision: Preserve the commit barrier + durability + clean-failover behaviors as policy - -- **Context**: R4 and R5 must not regress ha-commit-durability (bugs 661/828) or bugs 848/849/850. -- **Selected Approach**: `on_commit_advanced` only **triggers** the existing synchronous barrier; `majority_candidate_locked` self-cap, persist-before-ACK, the pre-promotion fence, advertise-real-durable-floor freshness, and the contiguity guard are unchanged. The library supplies the safe commit position; the durable act stays in the controller. -- **Rationale**: The barrier/durability is verified by tests + the Protocol Verification Checklist (not by raft.tla, which assumes a durable log); changing source-of-truth for the commit position must not change the durability mechanism. -- **Trade-offs**: None functional; R4.x latency for the steady state is unchanged (R5.4) because the barrier path is untouched. - -## Risks & Mitigations - -- **Former R8 infeasible without entity-bearing snapshots** → RESOLVED by descoping the configuration-upgrade materialized-state work into the independent `config-upgrade-snapshot` spec (HARD-depends on `entity-bearing-snapshots`); `ha-term-safety` ships R1–R7 with out-of-band config-push byte delivery and no snapshot dependency. -- **Upstream signature drift** (the three library specs are generated but not yet implemented) → Listed as Revalidation Triggers; the adapter wires to the exact published signatures; if a signature changes, re-check the adapter table. -- **Neutralizing term-blind controller paths could regress clean failover** → R5 acceptance criteria + the existing durability tests are the guard; the live R6 regression asserts both chaotic AND clean convergence. -- **`waldump` shows TERM=0 pre-fix** → expected (G1); R6 asserts non-zero, per-LSN-consistent terms post-fix, and the negative direction (pre-fix repro must fail) per R6.5. -- **Migration of `committed_config_tag` to marker-derived** → derive-not-store; on first boot after upgrade, a cluster with no marker treats config as term-0/lowest-known (R1.8 analogue) until the first marker commits. - -## References - -- `.kiro/specs/ha-term-safety/requirements.md` — R1–R7 (former R8 descoped 2026-06-21). -- `.kiro/specs/config-upgrade-snapshot/{brief.md,requirements.md}` — the descoped configuration-upgrade materialized-state snapshot (HARD-depends on `entity-bearing-snapshots`). -- `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` — gaps G1–G6 + the library-owns-protocol DECISION. -- `.kiro/specs/raft-log-replication/design.md`, `.kiro/specs/raft-commit-safety/design.md`, `.kiro/specs/raft-sim-conformance/design.md` — upstream callback contracts (consumed verbatim). -- `.kiro/steering/persistence.md`, `ha-async-io.md`, `roadmap.md:13-21,43,55` — durability policy, process-lifecycle invariant, Raft conformance series, Task 11 backlog, planned `walcompare` tooling. -- `.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md` + `repro-851/` — bug-851/857 evidence and the regression baseline. diff --git a/.kiro/specs/ha-term-safety/spec.json b/.kiro/specs/ha-term-safety/spec.json deleted file mode 100644 index 4e806aa5..00000000 --- a/.kiro/specs/ha-term-safety/spec.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "feature_name": "ha-term-safety", - "created_at": "2026-06-21T09:34:18Z", - "updated_at": "2026-07-17T11:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "updated_at_tasks": "2026-06-21T18:00:00Z", - "blocked_on": [], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-17T11:00:00Z", - "note": "First post-implementation validation run -- all 19 tasks (1.1-5.5) were already [x] with no prior validate_impl record. Full suite 9/9 green, 46/46 requirement acceptance criteria mapped, fresh 6-round live chaotic-multi-kill soak PASS (zero committed divergence/job loss), boundary/dependency direction clean (src/lib/raft untouched across all 35 tagged commits).", - "known_limitations": [ - "design.md's 'Commit-source switch' section describes the originally-planned architecture (library commit_index replacing the term-blind safe-commit); as shipped (commit 83fbed2, reviewer-approved with a RED->GREEN Figure-8 safety proof), the durable-majority barrier + own-term WAL_NOOP gate remain authoritative and raft_commit_index is purely advisory (commit.c:154). Design doc not updated to match; behavior is correct and safety-proven, this is a documentation-drift note only.", - "design.md's AppendEntries send-side architecture (on_send_append_entries/raft_send_append_entries, task 2.3) is dormant in production per a later cross-spec comment (raft-ha-core-safety task 6.2, ha.c:2530-2536) -- the live send path is the pre-existing repl_broadcast, patched by task 3.4 to carry entry.term/prev_log_term. The receive side IS live and correctly processes these entries -- live-verified safe, just not what the design.md diagram depicts. Documentation-drift note only. RESOLVED 2026-07-29: raft-adapter-conformance has now formally documented and regression-guarded this exact drift -- see src/controller/CLAUDE.md's TLA+ Action-to-Code Mapping table (the dormant nextIndex-driven sender row) and that spec's design.md 'Decision: dormant AppendEntries sender resolution' section for the authoritative disposition; a future reader should land there instead of re-discovering this drift.", - "ha.c:1844 has a stale 'TODO(ha-term-safety task 2.x): REPLACE these stubs' header comment whose own body lines each say '-- IMPLEMENTED'; a companion stale doc-comment sits in ha_convergence.c ~357-360 describing retired behavior. Cosmetic only, self-flagged in tasks.md's own 5.5 cleanup list, never executed." - ] - } -} \ No newline at end of file diff --git a/.kiro/specs/ha-term-safety/tasks.md b/.kiro/specs/ha-term-safety/tasks.md deleted file mode 100644 index c5fd63dc..00000000 --- a/.kiro/specs/ha-term-safety/tasks.md +++ /dev/null @@ -1,229 +0,0 @@ -# Implementation Plan - -> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/controller/testsuite/` (e.g. `test_term_safety_adapter.c`) that fails before the controller change and passes after. Build/run via `/xce-build controller` (builds lib + controller + `controller_unit_test`, ASAN). -> -> **Spec boundary** (applies to ALL tasks): this spec is the **controller-side thin callback adapter + config marker + live regression** over the raft library (`src/lib/raft/raft.{c,h}`), consumed **verbatim**. It does NOT re-implement any Raft protocol decision (logOk, `(term,index)` truncation, current-term commit, freshness, UpdateTerm) — those are owned by `raft-log-replication` + `raft-commit-safety` (both landed, library-side). It does NOT change the commit-barrier / persist-before-ACK durability mechanism (`ha-commit-durability`) — `on_commit_advanced` only TRIGGERS it. The on-disk `wal_entry_hdr_t` (40 bytes) is unchanged — term is a value fix, not a layout change (R1.7). The full entity-bearing config-upgrade snapshot is OUT of scope (`config-upgrade-snapshot`). -> -> **Depends (HARD): `raft-log-replication` + `raft-commit-safety`** — needs the five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`), the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), `commit_index`/`next_index[]`/`match_index[]`, and the extended `(last_log_term, last_log_index)` vote signatures already present in `raft.{c,h}`. -> -> **Build-restoring note:** the upstream vote-signature extension currently breaks the controller build (`cluster.c` vote callers use the old arity). Task 1.2 restores the build; until 1.2 lands, `/xce-build controller` will not compile (`/xce-build lib` stays green). - -## 1. Foundation: wire format, WAL term seam, recovery read-back - -- [x] 1.1 Extend the AppendEntries replication envelope to carry the Raft term fields - - Add `prev_log_index`, `prev_log_term`, and the entry's `term` to the AppendEntries-carrying repl envelope so a standby persists the leader's term, not a local substitute; regenerate the Cap'n Proto bindings - - Observable completion: a replicated entry's envelope round-trips `(prev_log_index, prev_log_term, entry_term)` across send/decode (unit-asserted on a built envelope); existing replication messages still decode - - _Requirements: 1.3, 1.4_ - - _Boundary: repl transport / typemap (Cap'n Proto)_ - -- [x] 1.2 Extend the RequestVote/PreVote wire with `(last_log_term, last_log_index)` and restore the controller vote call sites - - Add `last_log_term` (and `last_log_index`) to the RequestVote/PreVote wire message; regenerate bindings; update the controller's vote and pre-vote intake to decode the candidate's tuple and pass it to the extended library `raft_on_vote_request`/`raft_on_pre_vote_request`, sourcing the voter's own tuple via the entry-term read-back at `log_len` - - Observable completion: `/xce-build controller` COMPILES AND LINKS again (the upstream signature break is resolved); a vote intake decodes a candidate `last_log_term` and the library applies the logOk freshness gate - - _Requirements: 2.3, 2.4, 2.6_ - - _Boundary: repl/vote wire (Cap'n Proto), controller vote intake_ - - _Depends: 1.1_ - -- [x] 1.3 Add the WAL term seam, recovery read-back, and the legacy-zero-term matching rule - - Implement the `persist_set_term` seam (sets the controller's live election term) and a per-LSN recovered term index built during WAL replay, plus an accessor returning the recorded term for an LSN (0 when absent); preserve the 40-byte on-disk header (value fix only) - - Pin the legacy-zero-term rule (REVIEW ISSUE 2): a pre-feature entry carrying term 0 maps to a single, precisely-defined "lowest-known term" for `(term,lsn)` matching so recovery never fails and a legacy entry is never spuriously treated as divergent from another legacy entry; document and test the exact mapping - - Observable completion: after replaying a WAL with mixed real and legacy-0 terms, the accessor returns the recorded term for real entries and the defined lowest-known term for legacy-0 entries; header size unchanged - - _Requirements: 1.1, 1.5, 1.6, 1.7, 1.8_ - - _Boundary: persist (WAL I/O + term value)_ - -## 2. Core: the five callback adapter bodies - -- [x] 2.1 Implement `on_persist_entry` over the WAL write path - - Stamp `hdr.term` from the library-supplied term (via `persist_set_term`) at the given 1-based index == LSN, through the existing async/replicated WAL write; preserve persist-before-ACK and the bug-850 contiguity guard beneath it - - Observable completion: a library append/accept persists an entry whose recovered `hdr.term` equals the library term; persist failure propagates a non-zero rc and does not ACK - - _Requirements: 1.1, 1.2, 1.3, 1.4_ - - _Depends: 1.3_ - -- [x] 2.2 Implement `on_get_entry_term` and `on_truncate_from` - - `on_get_entry_term` answers from the recovered/live per-LSN term index (legacy-0 rule from 1.3); `on_truncate_from` deletes entries at index..end via the existing LSN truncate, replacing the committed-boundary heuristic - - Observable completion: the library reads back a stamped entry term via the callback; a truncate-from call removes the local suffix at and above the index and leaves the prefix intact - - _Requirements: 1.5, 1.8, 3.1, 3.2_ - - _Depends: 1.3_ - -- [x] 2.3 Implement `on_send_append_entries` over the repl envelope - - Build and enqueue an AppendEntries on the repl transport carrying `prev_log_index`, `prev_log_term`, the entry's `term`, and `leader_commit = commit_index`; a heartbeat carries zero entries - - Observable completion: emitting to a peer produces an envelope with the correct prev tuple and entry term; a 0-entry heartbeat is emitted when there is nothing to send - - _Requirements: 1.3, 3.1_ - - _Depends: 1.1_ - -- [x] 2.4 Implement `on_commit_advanced` as a trigger into the existing barrier, and resolve the commit-gate ownership - - Feed the library-supplied `commit_index` into the existing synchronous commit barrier (`commit_advance_locked`/`commit_set_writable_lsn`) as the commit SOURCE; the barrier mechanism (majority self-cap at durable floor, persist-before-ACK) is unchanged - - Resolve double-gating (REVIEW ISSUE 3): state and implement whether the controller's existing own-term `WAL_NOOP` writable gate is now redundant with the library's current-term commit rule (and removed) or retained as a complementary durability gate; ensure the two cannot deadlock (commit held by both) or leave a gap - - Observable completion: a strict library commit advance releases the barrier's WAIT only on durable majority; the term-blind safe-commit is no longer the commit source; a unit test pins the own-term-gate decision - - _Requirements: 2.2, 4.1, 4.5, 5.4_ - - _Depends: 1.3_ - -- [x] 2.5 Wire the five callbacks into the controller callback set and the library init restore - - Add the five callbacks to the controller's `s_real_cbs`; on `raft_init` restore `initial_log_len`, `initial_commit_index`, and the recovered last-log term/index from the WAL so the library boots with the durable floor - - Observable completion: a controller node initializes the raft library with all required callbacks non-NULL and restored indices matching its recovered WAL; existing election callbacks remain wired - - _Requirements: 1.5, 2.1_ - - _Depends: 2.1, 2.2, 2.3, 2.4_ - -## 3. Integration: drive the library entry points on the controller paths - -- [x] 3.1 Drive `raft_log_append` on the primary append path - - Route the primary's job-state append (and the own-term promotion no-op) through `raft_log_append` so the library stamps the term and assigns the LSN, firing `on_persist_entry` - - Observable completion: a primary `jobsub`-driven append goes through the library and lands in the WAL with the live term stamped; the promotion no-op records the newly established term - - _Requirements: 1.1, 1.2, 2.1_ - - _Depends: 2.5_ - -- [x] 3.2 Route heartbeat/replication intake through `raft_on_append_entries_request` (replaces term-blind intake + committed-boundary reconcile) - - Replace the term-handling in `ha_consume_heartbeat_committed` and the committed-boundary truncate in `ha_rejoin_reconcile` with routing every incoming heartbeat/replication (0- or 1-entry AppendEntries) through the library so `UpdateTerm` (G5), logOk (G2), and `(term,lsn)` conflict-truncate (G3) happen via the callbacks; keep the loud non-promotable retry fence for an unreconcilable member - - Observable completion: a higher-term heartbeat makes the node adopt the term and step down (term recorded); a divergent same-LSN tail is truncated and reconverged; an unreconcilable member stays non-promotable and keeps retrying (no false convergence) - - _Requirements: 2.3, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_ - - _Boundary: ha intake + ha_convergence_ - - _Depends: 2.5_ - -- [x] 3.3 Drive `raft_on_append_entries_reply` and `raft_advance_commit` on the ACK path - - On a replication ACK, route it through `raft_on_append_entries_reply` to update `match_index[]`/`next_index[]`, then drive `raft_advance_commit` so the library recomputes the current-term-gated commit and may fire `on_commit_advanced` - - Observable completion: a quorum ACK advances the library `commit_index` only when the agreed entry is current-term, which then triggers the barrier; a stale-term reply is dropped - - _Requirements: 4.1, 4.5_ - - _Depends: 2.5, 3.2_ - -- [x] 3.4 Retire the term-0 `repl_broadcast` live send and drive the library send loop on the live primary (the deferred live-send cutover) - - Make the LIVE primary replicate via the library send (`raft_send_append_entries`/the library's per-peer send) so `on_send_append_entries` (task 2.3) fires on the real replication path, carrying the leader's term; retire/replace the term-0 `repl_broadcast` (`persist.c:747`) as the live replicator so standbys no longer receive term-0 entries. Standbys must intake every replicated entry through `raft_on_append_entries_request` (task 3.2 route) and persist `hdr.term = leader_term` (not 0); retire the legacy direct-apply branch (`ha_apply_wal_catchup_entry` gated `entry->term != 0`) as the live intake. Preserve persist-before-ACK + the commit barrier + 848/849/850 + the bug-850 contiguity guard; keep `next_index[]`/`match_index[]` driven by the ACK path (3.3) so commit advances live. - - Observable completion: on a live cluster a primary `jobsub`-driven append replicates a term-CARRYING AppendEntries; standbys persist the leader's real term (recovered `hdr.term == leader_term`, not 0); a clean failover converges to byte-identical `(term,lsn)` WALs (closes the bug-990 / R5.1 term-identity gap); the library `commit_index` advances from live quorum ACKs. - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.5, 4.1_ - - _Boundary: repl.c / persist.c / ha.c live replication path (send + intake cutover)_ - - _Depends: 3.1, 3.2, 3.3_ - -## 4. Configuration ordering as a unified `(term,lsn)` marker (R7) - -- [x] 4.1 Add the `WAL_CONFIG_MARKER` entry type and reference payload - - Add the new WAL entry type and its marker payload carrying only `{version_tag, sha256}` (no config bytes); no on-disk header layout change - - Observable completion: a config marker can be encoded/decoded with its version tag and 32-byte hash and carries no configuration bytes - - _Requirements: 7.2, 7.4_ - - _Boundary: wal_types / config_repl_ - -- [x] 4.2 Append config markers via the unified log and derive committed config from the latest committed marker - - Append a `WAL_CONFIG_MARKER` through `raft_log_append` (inheriting term-stamp, commit, and `(term,lsn)` reconcile/truncate); derive the committed configuration reference from the latest committed marker within the committed prefix, retiring the separate `committed_config_tag` as the authoritative index - - Observable completion: committing a config change advances the derived committed-config reference; a divergent config marker is truncated by reconcile exactly like a job entry - - _Requirements: 7.1, 7.3, 7.6, 7.7, 7.11_ - - _Depends: 3.1_ - -- [x] 4.3 Deliver config bytes out-of-band, hash-verify before activate, fence on mismatch - - Key the existing out-of-band config-push transport to the committed marker's `{version_tag, sha256}`; obtain bytes when not held locally, verify their hash against the committed marker before activating; keep a member non-serving under the new configuration (loud fence, keep retrying) on mismatch; on recovery activate the configuration referenced by the latest committed marker - - Observable completion: a committed marker whose bytes match the hash activates; an injected hash mismatch keeps the member fenced and retrying (never activates mismatched config); config bytes never enter a WAL payload - - _Requirements: 7.5, 7.8, 7.9, 7.10_ - - _Depends: 4.2_ - -## 5. Verification: clean-failover preservation, tooling, and the live chaotic regression - -- [x] 5.1 Adapter callback unit/integration tests - - Cover: term stamp + read-back incl. legacy-0 mapping; higher-term heartbeat adoption via the intake route; truncate-then-persist of a divergent suffix with committed prefix untouched; `on_commit_advanced` releasing the barrier only on durable majority; config marker append + committed-config derive + divergent-marker truncation - - Observable completion: the adapter test binary exercises every callback and the marker path and passes under ASAN - - _Requirements: 1.1, 1.5, 1.8, 2.3, 3.1, 3.2, 4.1, 7.1, 7.6, 7.7_ - - _Depends: 3.3, 4.2_ - -- [x] 5.2 Clean single-failover preservation integration test - - Assert a single clean failover (one primary killed, one standby promoted, old primary restarted) converges all members to a byte-identical WAL including terms, preserving recovery-on-promotion + advertise-real-LSN (848/849) and the contiguity guard (850); no added steady-state ack latency - - Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green - - _Requirements: 5.1, 5.2, 5.3, 5.4_ - - _Depends: 3.3_ - - _Partially validated on a REAL fs ($HOME/tmp/xce-ha-deploy, ext4) 2026-06-22: bug-990 term propagation CONFIRMED LIVE — PAYLOAD run converged with ctrl-2/ctrl-3 persisting the leader's REAL term (2) for the committed prefix (pre-3.4 they were 0), walcompare OK over the committed prefix. BUT a NEW regression surfaced — see bug-991 below. Harness fixes landed: failover detection (match any post-kill CTRL_PRIMARY, not first_boot=0; 45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept, not commit, so the old harness raced)._ - - _Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept)._ - - _RESOLVED (commit 83fbed2): bug-991 (catch-up applied_lsn off-by-one → last-held) + **Option A** (commit 83fbed2): removed the 2.4 `min(durable-majority, raft_commit_index)` ceiling. Under single-writer the durable-majority watermark is ALREADY (term,lsn)-current-term-safe (term-carrying repl + reconcile-before-durable + own-term gate + truncate-to-safe), so the library commit_index was a redundant, starvation-prone gate. Durable-majority barrier + own-term gate are now authoritative; raft_commit_index is inert-advisory. Reviewer-APPROVED as safe (durable-majority is (term,lsn)-safe; Figure-8 holds via own-term gate, pinned by a new RED→GREEN test; bug-851 cannot recur). 177 unit tests. **LIVE CONFIRMED (commit 9e5e293, $HOME/tmp/xce-ha-deploy/final, ext4): `commit: committed_lsn 0→1→2→3→4→5` — the commit-stall is FIXED; the harness gate logs `jobs committed on a majority (committed_lsn=5 >= 5)`.** committed=5 is backed by a real durable majority (a standby fsync'd LSN 5)._ - - _NEW EXPOSED — bug-828/795 (pre-existing, DOCUMENTED, unfixed; NOT ha-term-safety; NOT introduced by this work): the live STRICT failover now FAILS at walcompare because the FAILOVER LOSES COMMITTED JOBS. Sequence: committed=5 (valid, durable majority), kill primary, a standby promotes at term 3 computing `safe_committed_lsn=1 (from 1 granted voter)` and TRUNCATES the committed jobs (LSN 2-5) → ctrl-2/ctrl-3 end at LSN 1 + term-3 NOOP while ctrl-1 (restarted) holds the committed jobs. This is a Leader-Completeness violation in the ELECTION/safe-commit-from-votes path (`ha_compute_safe_committed_lsn`), which this spec does NOT touch. It was INVISIBLE before because the commit-stall cascade (bug-990/991/2.4-ceiling) kept committed pinned at 1, so the failover never had committed jobs (LSN>1) to lose. Matches the project-memory "Failover job-loss root cause (bug-828/795, root-caused 2026-06-20, NOT fixed)". Evidence: /home/claude/tmp/xce-ha-deploy/final (kept). OWNER: ha-commit-durability / a failover-safety spec, NOT ha-term-safety. 5.2/5.4 cannot be GREEN until bug-828 is fixed._ - - _bug-828 PARTIAL FIX landed (commit 91b9a44, reviewer-verified SAFE, 177 unit tests): (part 1) `vote_reply_dispatch` was discarding the granting voter's decoded peerLastLogLsn (`(void)peer_lsn`), so `ha_compute_safe_committed_lsn` saw only the self-vote → truncated to self_lsn (the "from 1 granted voters" fingerprint). Now routes it into `ctx->votes[]` via a record-only `ha_record_vote_reply` (no double-promotion; raft_on_vote_reply stays the sole promotion driver). (part 2) candidate/voter advertise the real DURABLE (fsync'd) tip; `persist_wal_write_replicated` advances `durable_lsn` after its fail-loud inline fdatasync. Safe-commit is bounded by the durable majority (preserves more committed entries, never marks a non-majority-durable LSN safe). Fingerprint moved "from 1" → "from 2 granted voters"._ - - _REMAINING SLICE (precisely characterized, NOT yet fixed — the documented architectural core): the live clean-failover STILL loses committed jobs because the ELECTION WATERMARK TANGLE persists. A standby that caught up to applied_lsn/durable_lsn=5 during the run still resolves its OWN election position to the STALE BOOT `durable_floor=1` in the vote/PreVote freshness gate + advertise, so (a) it fails to out-rank a job-LESS candidate (freshness gate: job-less candidate's lastLogIndex=1 >= the job-having voter's mis-computed 1 → grants), and (b) a job-less node wins term 3, computes safe_committed=1, and `persist_truncate_wal_above(1)` destroys committed LSN 2-5. FIX DIRECTION: source the voter/candidate election position from the LIVE durable/applied tip (max(persist_durable_lsn/applied_lsn, last_known_committed_lsn)) NOT the boot durable_floor, consistently across ALL FOUR election paths — candidate self-advertise (`ha_candidate_advertise_lsn`), voter floor (cluster.c ~403/495 + what the controller passes as the voter's own last_log_index into `raft_on_vote_request`), the freshness comparison, and safe-commit. SAFETY GUARD: advertise/compare only DURABLE (fsync'd) positions — never a merely-received tip — or a node could win/keep an entry it isn't durable for. This is safety-critical consensus code (a wrong fix = data loss); needs a dedicated, careful, fresh-context effort. Owner: ha-commit-durability / failover-safety. Live evidence: /home/claude/tmp/xce-ha-deploy/{final,fix828,fix828b_r1} (kept)._ - - _**bug-828 FIXED (commit b77d412, reviewer-verified SAFE).** The earlier "watermark tangle" framing was WRONG — instrumentation proved the standbys DID fsync LSN 2-5 (durable_lsn=5; not a durability violation, not an advertisement-staleness bug). ACTUAL ROOT CAUSE: `ha_enter_real_election` (ha.c ~5187) did `persist_truncate_wal_above(&c->persist, ha->last_known_committed_lsn)` BEFORE the vote — a defensive heuristic copied from the RDHUP path that VIOLATES the documented rule "Never use last_known_committed_lsn for truncation — it may be stale". last_known_committed_lsn lags the true commit point (primary committed 2-5 on a durable majority then died before a heartbeat carried committed=5), so the cutoff was 1; the truncate's clamp_durable_lsn lowered durable_lsn 5→1, PHYSICALLY DESTROYING committed entries before the election computed the safe point → candidate advertised 1 → safe_committed=1 → post-election truncate kept only LSN 1. FIX: delete the premature truncate. Only truncates now are the authoritative votes-derived post-election persist_truncate_wal_above(safe_committed_lsn) + the library on_truncate_from (term,lsn) reconcile — both before any commit under the new leader. SAFE because a divergent uncommitted tail can only exist at a stale/lower term (single-writer), and the library's TERM-PRIMARY (lastLogTerm,lastLogIndex) freshness gate denies such a candidate — correct ONLY because task 3.4 fixed the term axis (the whole ha-term-safety cascade was the prerequisite that makes this fix safe). 177 unit tests + RED-proven regression test (`test_repl_catchup_durable_floor_survives_failover`). **LIVE: clean failover preserves committed jobs — walcompare committed-prefix IDENTICAL across all 3 members, both jobs on disk, across multiple runs.**_ - - _RESIDUAL (separate, pre-existing, NOT data-safety): the live harness still EXIT-1s at the POST-walcompare client `wait-primary`/`expect-job` step due to **bug-854** (libxce positional primary-discovery / dead-first-master: the client can't reach the live primary when the killed node is FIRST in the masters list — `ctrl-1,ctrl-2,ctrl-3` with ctrl-1 dead). This is a CLIENT-side discovery defect (libxce), orthogonal to the server-side committed-prefix safety which now PASSES. To get the harness fully GREEN: fix bug-854 (libxce client failover discovery) OR have the harness target a known-live node post-failover. 5.2's DATA-SAFETY assertion (walcompare identical + jobs on disk) now passes; the EXIT-0 is gated on bug-854._ - -- [x] 5.3 Cross-member `walcompare` divergence tool - - Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder - - Observable completion: the tool returns success on identical member WALs and fails (with the diverging LSN) on a hand-divergent pair - - _Requirements: 6.1, 6.2, 6.3_ - - _Boundary: testsuite/walcompare_ - -- [x] 5.4 Live chaotic-multi-kill regression (R6 headline) - - Drive a 3-node cluster through repeated primary kills interleaved with `jobsub` and config upgrades under deterministic WAL sync; after the run assert `(term,lsn,pcrc)` committed-prefix identity across all members via `walcompare`, that every acked-committed job is present on every surviving member, and that every member converges to the same committed config reference; confirm the script reproduces bug-851 on pre-fix code (term 0) and passes on the fixed code - - Observable completion: the regression passes on the fixed code and fails on pre-fix code; no acked-committed job is absent; committed config reference identical across members - - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 7.11_ - - _Depends: 5.2, 5.3, 4.3_ - - _Blocked (ENV-only — code gap CLOSED by 3.4): the R6 headline `(term,lsn,pcrc)` committed-prefix identity is now satisfiable (3.4 fixed bug-990). Remaining blocker is the same real-fs live-run env as 5.2 (bug-924 overlayfs can't run the chaotic-multi-kill harness here). Author/run on a real fs; pass an explicit `--committed N` to walcompare (5.3 advisory) + `expect-job`._ - -- [x] 5.5 Full controller build + suite green under ASAN - - Run `/xce-build controller`; the controller compiles and links (build restored) and all controller unit/integration tests plus the adapter tests pass under ASAN with no regression in election/durability behavior - - Observable completion: controller test binary exits 0; ASAN clean; clean-failover and durability suites still green - - _Requirements: 1.1, 2.3, 3.1, 4.1, 5.1, 6.1, 7.1_ - - _Depends: 5.1, 5.2, 5.4_ - - _Blocked (live portion only — build+unit gate CONFIRMED GREEN): with all of 2.4–4.3 + 3.4 + 5.1 + 5.3 landed, `/xce-build controller` COMPILES + LINKS and the controller_unit_test binary EXITS 0 with **175 Tests 0 Failures** on the reliable gate (`rm -rf /tmp/xce_*` then direct binary, ASAN detect_leaks=0). The `test_wal_catchup_*` FAIL prints (forked-child overlayfs `fdatasync` errno=-11/5) + the meson `test_demotion_replay` flake are the documented bug-924 overlayfs artifacts, present since iter-1, NOT regressions. The ONLY remaining 5.5 acceptance is "clean-failover suite green", which is the LIVE 5.2 harness — env-blocked here (real fs needed). Re-run `/xce-build controller` + the 5.2/5.4 STRICT_TERM harness on a real fs to fully close 5.5._ - -## Implementation Notes - -- **Iter 1 (1.1+1.2) — controller build RESTORED** (compile+link of xce_controller + controller_unit_test). Added RequestVote/PreVote `lastLogTerm@4/lastLogIndex@5` and WalCatchupEntry `term@4/prevLogIndex@5/prevLogTerm@6` (append-only, no field-number collision); fixed cluster.c vote callers to the new library arity; ha.c emits the candidate tuple. -- **5 STUB callbacks wired into s_real_cbs (ha.c) to satisfy raft_init's new non-NULL requirement — tasks 2.x MUST replace:** ha_cb_persist_entry_stub(→2.1), ha_cb_get_entry_term_stub(→2.2, returns 0), ha_cb_send_append_entries_stub(→2.3), ha_cb_truncate_from_stub(→2.4), ha_cb_commit_advanced_stub(→2.5). Freshness logOk is INERT until 1.3+2.2 (candidate last_log_term currently sourced from the unset G1 `persist.term`). -- **ENV BLOCKER (critical):** fresh `nix develop` shells fail — flake pins `c-capnproto` to a mutable `main.tar.gz` whose hash drifted (expected sha256-9dzSc3w+yxYzsF/rZe3qIarKOpjt2zvIyjuS5FXngIM=, got sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk=). Only the pre-cached main worktree builds; no worktree isolation / clean baseline possible until the flake input is repinned to an immutable commit/tag. -- **Pre-existing controller-test failures (NOT this task):** test_repl_unit.c `test_wal_catchup_batch_*` fail with -EAGAIN + ASAN leaks under overlayfs `/tmp` (bug-850 fsync env artifact). The diff touches none of repl.c/persist.c/test_repl_unit.c/ha_apply_wal_catchup_entry; the append-only capnp change cannot affect the @3 entries-list decode. The controller suite needs a real-fsync filesystem for clean validation. - -- **CORRECTION (2026-06-22): env unblocked + iter-1 fully validated.** User refreshed the c-capnproto flake pin (bug-918 resolved). A CLEAN controller build now passes the full controller suite green (Ok:1 Fail:0, ~13s). The earlier `test_wal_catchup_*` failures were a STALE incremental-build artifact (iter-1 capnp schema compiled against the drifted toolchain) — NOT an overlayfs fsync issue and NOT a source regression. Validation gate for tasks 2.x+ is reliable: `/xce-build controller` (clean) or incremental `meson compile -C xce.build xce_controller controller_unit_test && meson test -C xce.build "controller test"`. - -- **1.3 done.** persist_set_term + per-LSN in-memory term index (built in replay_wal, freed in persist_fini) + persist_entry_term accessor; legacy-0 -> WAL_LEGACY_TERM=1 at read-back only (raw 0 kept so present-legacy-0=1 vs absent=0). 40-byte header preserved. 5 new tests in test_term_safety_adapter.c. - - **Task 2.2 MUST document (review MEDIUM):** in a mixed pre/post-feature cluster where the post-feature leader is in term 1, the (term,lsn) check cannot distinguish a legacy-0 entry from a real term-1 entry by term alone; the Raft log-matching invariant + the bounded upgrade window make this safe, but note it in the on_get_entry_term/conflict-truncate comment. - - **Validation env (bug-924):** controller suite creates many /tmp/xce_* WAL dirs; overlayfs fdatasync degrades as they accumulate -> non-deterministic meson timeouts. Reliable gate: `rm -rf /tmp/xce_*` then run the `controller_unit_test` binary directly (all groups 0 failures). Pre-existing ASAN leak in test_commit_barrier_reply.c (not gated by meson detect_leaks=0). - -- **2.1 done.** on_persist_entry body: persist_set_term(p,term) then role-routed write (primary persist_wal_async / standby persist_wal_write_replicated(p,index,...)); failure propagates non-zero, durable_lsn unchanged, persist-before-ACK + bug-850 guard preserved. Added in-flight `raft_entry_blob_t {type,payload_len}` envelope in wal_types.h (NEVER in the 40-byte header). 3 tests. - - **DEFERRED to task 3.1 (review):** primary path uses persist_wal_async's `current_lsn`, NOT the library `index` — holds only while `s->log_len` and `p->current_lsn` stay co-tracked. Task 3.1 MUST retire/disable the hand-rolled primary append so no WAL append bypasses raft_log_append, else current_lsn diverges from s->log_len. Also: persist_set_term + write are two lock regions (safe today, no concurrent appender) — 3.1 should re-verify when real concurrent appends exist. - - **DEFERRED to task 3.2 (review):** a heartbeat / 0-entry AppendEntries must NOT call on_persist_entry (entry==NULL,len==0 currently falls through to WAL_NOOP) — verify the library only calls on_persist_entry when there is an entry. - - **3.1/3.2 producer format:** must pack `raft_entry_blob_t {type,payload_len}+payload` to match the on_persist_entry unpack (single shared def in wal_types.h). - -- **2.2 done.** on_get_entry_term → persist_entry_term (legacy-0 caveat documented per 1.3 review); on_truncate_from → persist_truncate_wal_above(p,index-1) (delete LSN>=index), returns rc, non-promotable fence preserved. NEW `truncate_term_index_locked` (persist.c) shrinks the in-memory term index on BOTH truncate return paths (under p->lock, cutoff-aligned) so on_get_entry_term returns 0 for truncated LSNs — no stale-term hazard. 3 tests. Reviewer APPROVED. - -- **2.3 done.** on_send_append_entries → repl_send_append_entries: builds a WalCatchupEntry AE to standbys[peer_idx].ep carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via ha_read_wal_entry_at); heartbeat = n_entries==0 (zero entries). Async (xce_wal_batch_req_raw), payload freed on all paths. 2 tests via repl_send_ae_capture_hook seam. Reviewer APPROVED. - - **DEFERRED to task 3.2:** leader_commit is accepted+captured but NOT yet wire-encoded — WalCatchupEntry needs a `leaderCommit` capnp field + standby intake of it (3.2). Comment at repl.c corrected to state this. - - **LOW (future refactor):** ha_read_wal_entry_at duplicates WAL segment-file parsing already in persist.c/config_repl.c/waldump.c — a persist_read_entry_at() accessor would consolidate; correct as written. - -- **2.4 done.** on_commit_advanced (ha.c `ha_cb_commit_advanced_stub` body, still named `*_stub` per the 2.1–2.3 convention) routes the library `commit_index` into the barrier via NEW `commit_set_raft_commit_index(ce, commit_index)` (commit.c) → `ce->raft_commit_index` ceiling. `majority_candidate_locked` now caps `committed = min(durable-majority, raft_commit_index)` on BOTH the standalone (n_peers<=1) and multi-peer paths. Own-term WAL_NOOP writable gate **RETAINED** (design 193/201/228) — proven non-conflicting: `writable_lsn` is only a WAIT predicate (`wait_done_locked`), `raft_commit_index` only caps `committed_lsn` — disjoint quantities, no deadlock; `committed=min(...)` ⇒ no gap; ceiling only ever lowers `cap` (starts at durable floor) ⇒ never over-commits; `assert(cand<=persist_durable_lsn)` still holds. Monotonic: setter raises only on `commit_index > raft_commit_index`. 5 tests; reviewer APPROVED (stash-and-rebuild baseline proved test_repl_unit 6-fail + 1 timeout are pre-existing bug-924/working-dir env, not regressions). - - **FEATURE GATE (critical for 3.x):** ceiling is INERT while `raft_commit_index == 0` (no on_commit_advanced fired). Lowest real LSN is 1 (commit_submit rejects lsn==0), so `0` unambiguously means "nothing committed" — no 0-vs-unset window. Tasks 3.1–3.3 wiring `raft_advance_commit` on the ACK path will fire the FIRST `on_commit_advanced(commit_index>=1)` and engage the library source; until then the autonomous barrier is preserved (no steady-state regression, R5.4). - - **DEFERRED to 3.x (decommission the term-blind source):** `ha_compute_safe_committed_lsn` is no longer the commit source but is NOT yet removed — it still drives the promotion-time safe truncate (R5). 3.x must ensure no path still feeds it INTO the barrier as a commit source once `raft_advance_commit` is wired. - - For-test seam: `ha_cb_commit_advanced_for_test` (ha.h, XCE_TESTING) + `commit_set_raft_commit_index` are the hooks 5.1 integration tests should reuse. - -- **2.5 done.** Init restore on the REAL post-recovery boot path `ha_init_raft` (ha.c ~4472, runs after `persist_recover` per myapp.c:237/242) — NOT the pre-recovery `ha_init` placeholder (~1756, raft_c==NULL, left at zero by design). `rcfg.initial_log_len = persist_durable_floor(&c->persist)`; `rcfg.initial_commit_index = committed_lsn` clamped to log_len (preserves library `commit_index<=log_len`, raft.h:61). There is NO `initial_last_log_term` field — the library sources last-log term via the wired `on_get_entry_term(s->log_len)` (raft.c:161). All five callbacks + election callbacks (on_become_leader/on_persist) verified non-NULL + real bodies in s_real_cbs. 2 tests; reviewer APPROVED (independently reverted restore→`Expected 3 Was 0` RED; investigated commit_index concern vs literal §1.5). - - **commit_index-at-recover = 0 is CORRECT-BY-DESIGN, not a gap:** `persist_recover` does NOT re-derive `committed_lsn` from disk (it stays 0 after persist_init); the committed position is re-established post-election from voter replies / via `on_commit_advanced` (tasks 3.x). §1.5/design Traceability row 1.5 mandate restoring `initial_log_len` + last-log-term ONLY. The clamp is just defensive wiring for any future path that sets committed_lsn pre-init. The 2.5 test sets committed_lsn=2 manually to exercise the clamp/pass-through (honestly commented); the load-bearing real-recover assertions are log_len==3 + last-log term==7. - - **PRE-EXISTING STALE COMMENT (cleanup, not 2.5 scope):** ha.c ~1326-1334 still calls the five callbacks "stubs"/"placeholders" that "perform NO persistence" — stale since 2.1–2.4 gave them real bodies. Also the callbacks are still NAMED `ha_cb_*_stub` (real bodies). A rename + comment refresh is a good 5.5 / cleanup-pass item; cosmetic only. - -- **3.1 done.** `raft_log_append` is now the SINGLE primary append entry point. NEW `ha_primary_log_append(c,type,payload,len)` (ha.c) packs the `raft_entry_blob_t{type,payload_len}+payload` envelope → `raft_log_append` → on_persist_entry → persist_wal_async (leaf). Hand-rolled bypass RETIRED: persist.c `wal_hook_append_classified` (~2332) + `persist_wal_sync_append` (~2047) route through `ha_primary_log_append` when `p->cluster != NULL` (standalone persist keeps direct path). `ha_arm_own_term_gate` NOOP also routes through it (stale current_lsn-1 LSN derivation deleted). Surviving direct persist_wal_async callers are ONLY: the leaf writer, on_persist_entry's library sink, and the `-EPERM` legacy fallback (stamps real `ha->term` via persist_set_term — never term 0, only the non-raft/engineless road). 3 tests; reviewer APPROVED (verified retire-not-parallel, no recursion [persist_wal_async is leaf], envelope parity, persist-before-act, stash-baseline proved bug-924 7-fail cluster pre-existing). - - **Anti-divergence invariant now LIVE + tested:** `p->current_lsn == s->log_len + 1` (next-LSN co-tracking) asserted across consecutive appends. Seeded at boot by 2.5 restore (s->log_len = durable_floor = current_lsn-1); preserved because raft_log_append (+1 to log_len) and its on_persist_entry→persist_wal_async (+1 to current_lsn) move in lockstep. - - **DEFERRED to 3.2:** heartbeat / 0-entry AppendEntries intake must route through `raft_on_append_entries_request` (UpdateTerm/logOk/(term,lsn) truncate via callbacks), replacing `ha_consume_heartbeat_committed` term handling + `ha_rejoin_reconcile` committed-boundary truncate; WalCatchupEntry still needs a `leaderCommit` capnp field + standby intake (deferred from 2.3). - - **PROCESS NOTE:** implementer first emitted `STATUS: COMPLETE` (non-parseable); corrected to READY_FOR_REVIEW. (Continue an existing subagent with SendMessage, not a fresh Agent call — a fresh Agent has no context.) - -- **3.2 done.** Heartbeat/replication intake routes through the library. NEW `ha_consume_append_entries` (ha.c) = library-route intake; `ha_consume_heartbeat_committed` (~5691) now sends a 0-entry AE through `raft_on_append_entries_request` for UpdateTerm (G5) BEFORE the monotonic committed-learn — old term-blind `if(hb->term<myterm)return` DELETED; reachable on the LIVE heartbeat path (cluster.c:326). `ha_apply_wal_catchup_entry` (~2034) routes term-carrying single-entry AE through the library (logOk G2 + (term,lsn) truncate G3 via on_truncate_from/on_persist_entry). `ha_rejoin_reconcile` (ha_convergence.c) RETIRED to no-op; dead `ha_rejoin_reconcile_fn`/`_ctx_t` removed (the watermark committed-boundary truncate was the bug-851-unsafe heuristic). Wire gap CLOSED: `WalCatchupEntry.leaderCommit @7` (controller.capnp, append-only, no collision w/ term@4/prevLogIndex@5/prevLogTerm@6) encoded (request.c/repl.c) + decoded into typemap `leader_commit` + passed to the library (library currently `(void)`s it — commit-advance is 3.3). 3 new intake tests + 3 rewritten rejoin tests (test_demotion_convergence.c — preserve safety asserts, pin the retired contract, NOT weakened). 156 Tests 0 Failures; reviewer APPROVED (verified live-reachability, replace-not-parallel, leaderCommit end-to-end, baseline-stash proved bug-924 cluster pre-existing). - - **CRITICAL for 3.3 (the live-path gap):** live steady-state replication still uses TERM-0 `repl_broadcast` (legacy direct-apply in ha_apply_wal_catchup_entry, gated `entry->term != 0`). So the term-carrying single-entry AE RECEIVE route is UNIT-TEST-ONLY in production until 3.3 wires the library SEND loop (`raft_on_append_entries_reply` + `raft_advance_commit`) AND RETIRES the term-0 `repl_broadcast`. The higher-term-heartbeat UpdateTerm IS live (heartbeats are 0-entry → always go through the library route). Live divergent-tail convergence is validated in 5.2/5.4. - - **LOW cleanup (5.5 / cleanup pass):** (1) ha_convergence.c ~292-297 doc-comment on `cluster_apply_ctrlhello_reply` still describes the old "clamped truncate + restart" — stale. (2) `ha.h:197 rejoin_reconciled` atomic is now a dead struct field (once-guard unused). Both harmless/cosmetic. Add to the growing 5.5 cleanup list alongside the `ha_cb_*_stub` rename + stale stub-header comment. - -- **3.3 done (REJECTED round 1, fixed, APPROVED round 2).** ACK path: NEW `ha_raft_on_repl_ack(c, abs_peer_idx, acked_lsn, reply_term)` (ha.c ~2914) calls `raft_on_append_entries_reply` then `raft_advance_commit` → fires `on_commit_advanced` on a strict current-term advance (activates the 2.4 commit gate). Wired into the production walAck success branch (`repl_ack_decode_and_dispatch`, ha.c ~4327); `reply_term = war_raw.term`. §4.1/§4.5 current-term gate owned by library (raft.c G4); stale-term reply dropped library-first. 6 ACK tests; reviewer APPROVED round 2 after a full re-verify. - - **bug-960 (CRITICAL, caught by review — logged in buglog.json):** `ha_raft_on_repl_ack` first applied the votes-index→absolute remap UNCONDITIONALLY, but the walAck call site fires for BOTH promoted-vote eps (`vctx->ctx != NULL`, relative peer_idx) AND heartbeat standby eps (`vctx->ctx == NULL`, ALREADY-absolute peer_idx — see authoritative ha.c ~6172-6189 / ~5808). Over-shift → OOB `match_index[]` write (raft.c:802 has NO bounds check) in a 3-node cluster / wrong-peer FALSE-QUORUM data-loss in larger. FIX: `ha_repl_ack_abs_peer_idx(ha, ep_is_vote, peer_idx)` resolver (vote→remap, heartbeat→passthrough) resolved at the call site via `vctx->ctx != NULL`; `ha_raft_on_repl_ack` now takes ABSOLUTE idx + fail-loud `[0,n_peers)` self-excluding bounds check (logs+returns, never crashes/self-terminates). Unit tests had MASKED it (only exercised votes_idx under my_priority==0). LESSON: any peer-index passed to a library `match_index[]`/`next_index[]` write MUST be resolved per endpoint-type (vote-relative vs heartbeat-absolute) — never blind-remap; test BOTH endpoint types at my_priority 0 AND non-0. - - **DO-NOT-REPEAT (cerebrum):** an implementer ran `clang-format -i` (whole-file reformat) then `git checkout HEAD -- <files>` to undo it, which WIPED the uncommitted task diff (HEAD was at the prior task's commit). It reconstructed the diff from context and re-review confirmed fidelity, but: never `git checkout HEAD --` to undo a reformat on a file holding uncommitted work; reformat only the changed hunks. - - **STILL DEFERRED to 5.2/5.4 (carried 3.2→3.3):** live steady-state send is still TERM-0 `repl_broadcast`; the library send loop (`on_send_append_entries`) is not yet the live sender, so the ACK→commit path is proven end-to-end in UNIT TESTS but the first LIVE `on_commit_advanced(>=1)` requires retiring term-0 `repl_broadcast` + driving the library send. Live chaotic convergence (and this retirement) is validated/owned by 5.2 (clean failover) + 5.4 (chaotic regression). Phase 3 (drive library entry points) is structurally complete; phase 4 (config marker) + phase 5 (verification) remain. - -- **4.1 done.** `WAL_CONFIG_MARKER = 50` (wal_types.h, between WAL_NOOP=40/WAL_CHECKPOINT=100) + `wal_config_marker_t {uint64_t version_tag; uint8_t sha256[32];}` + `WAL_CONFIG_MARKER_PAYLOAD_LEN (8u+32u)`. Reference-only codec `config_marker_encode/decode` (config_repl.c/h): explicit BIG-ENDIAN version_tag + raw 32-byte sha256 memcpy; NO config-bytes parameter on any path (R7.4); `-EINVAL`/`-ENOSPC` guards, no OOB on truncated decode/short encode. 4 tests (test_config_repl.c). 40-byte hdr static_assert intact; src/lib/raft untouched. Reviewer APPROVED. - - **For 4.2 (use, don't rediscover):** (1) the marker version_tag is consistent with the EXISTING config tag: `config_version_tag = be64toh(first 8 bytes of sha256)` (config_repl.c ~171-174/382-386), so a marker's BE version_tag bytes == sha256[0..7]. (2) `wal_classify` (persist.c ~2188) has `default: assert(0)` under XCE_TESTING and `replay_wal` (persist.c ~1499) has `default: break;` — 4.2 MUST add WAL_CONFIG_MARKER classification (and update the hand-maintained `test_classify_is_total_over_wal_entry_type` array in test_commit_barrier_rollout.c ~263-281) BEFORE any live path appends a type-50 entry, or the assert fires. The marker rides the `raft_entry_blob_t {type=WAL_CONFIG_MARKER, payload_len=40}` envelope through `ha_primary_log_append`→raft_log_append→on_persist_entry. - -- **4.2 done (REJECTED round 1, fixed, APPROVED round 2).** LIVE wiring: `do_upgrade_config_with_bytes` step 7 (config_repl.c ~1396, reached via OP_upgradeConfig handler ~1468) calls NEW `config_commit_change` → `config_append_marker` → `ha_primary_log_append(WAL_CONFIG_MARKER)` at the live term, replacing the term-blind authoritative path. `committed_config_tag` DEMOTED to a non-authoritative cache (only advanced in push_config_to_standbys; remaining readers are status/heartbeat-propagation, NOT ordering authority); the authoritative committed-config reference is DERIVED via `config_committed_marker` (latest committed marker within the committed prefix). `wal_classify`+`replay_wal` give WAL_CONFIG_MARKER → `WAL_MODE_RETAIN` with NO entity restore (reference-only). 6 tests; reviewer APPROVED round 2 (RED reproduced by neutering; 162/0 ASAN). - - **Round-1 REJECT lesson (design settles scope):** the live cutover (do_upgrade_config_with_bytes appends the marker + retire committed_config_tag as authoritative) is **4.2's** job per design §306-320 (Config marker component owns R7.1/7.3/7.6/7.7/7.11); task 4.3's component §336-364 is EXCLUSIVELY byte-delivery R7.5/7.8/7.9/7.10. Providing append/derive PRIMITIVES without wiring the live path = incomplete task (dead code). When scoping "is the live wiring this task or the next", read the design COMPONENT→requirement mapping, not just the task title. - - **Compaction hazard FIXED (Finding 2):** `trim_wal_segments` can unlink a committed marker's segment. `persist_checkpoint` now calls `scan_latest_committed_config_ref` (committed-prefix, latest-wins) BEFORE write_snapshot+trim, stamps the 41-byte `{version_tag,sha256}` ref into an OPTIONAL snapshot-sentinel TAIL (backward-compat: load_snapshot needs only n>=16; tail gated on `n>=57 && header[16]!=0`, so OLD 16-byte sentinels still load — no durable-format regression). `config_committed_marker` falls back to the durable `persist.cfgref_*` ONLY on live-scan -ENOENT (newer live marker always wins; durable ref is the last-checkpoint FLOOR, monotonic, never a shadow). This is a 40-byte REFERENCE in the snapshot, NOT the out-of-scope full `config-upgrade-snapshot`. - - **DEFERRED to 4.3:** `config_committed_marker` has no production CONSUMER yet (only tests) — activation/serve-fence/hash-verify-before-activate + the status-from-derived-value consumer is 4.3 (R7.5/7.8/7.9/7.10). 4.2 only requires the derive to ADVANCE on commit (proven). cb5 fixture in test_commit_barrier_rollout.c made persist-backed so the live marker append succeeds (assertions unchanged). - -- **4.3 done (REJECTED ×2, APPROVED round 3). PHASE 4 COMPLETE.** Byte delivery + hash-verify-before-activate + serve-fence. NEW `config_activate_committed(c)` = the production consumer: reads `config_committed_marker`, hashes the locally-held out-of-band bytes (`local_config_sha256`), VERIFIES sha256 == committed marker BEFORE `load_primary_config`, else sets `ha.config_serve_fence` (loud, `-EAGAIN`, stay-alive). `config_serve_fenced` accessor. ENFORCED by `ha_dispatch_gate` (CTRL_PRIMARY denies config-dependent state-changing ops e.g. jobsub while fenced; admits replConfigPush/read/status — no un-fence deadlock) + `ha_config_promotion_fenced` (refuse promotion in raft_cb_become_leader, stay follower). Wired at: (A) `handle_repl_config_push` (save-then-activate, SOLE activation point — no pre-activate), (B) `myapp.c` boot step 7 (recovery activates latest committed marker; -ENOENT→load_primary_config fallback, NO false fence), (D) `ha_cb_commit_advanced_stub` schedules an OFF-IOMGR delaymgr retry. bug-981 logged. Reviewer APPROVED round 3. - - **bug-981 (CRITICAL liveness, caught round 2):** push-before-commit on a 2nd+ upgrade — a standby that saves NEW bytes before its committed prefix advances past the NEW marker fences (config_committed_marker still returns the OLDER marker → hash mismatch), and NOTHING re-activated post-commit (on_commit_advanced is async-I/O-bound so can't do disk reads; primary's config-behind re-push defeated because conf_save_config already advanced the advertised tag) → PERMANENT fence until reboot, violating R7.9 "keep retrying". FIX: `ha_cb_commit_advanced_stub` (iomgr ACK thread) does ONLY an atomic `config_serve_fenced` read + non-blocking `config_schedule_activate_retry` (delay_task_ms→ordq_put, NO disk I/O); the disk-reading `config_activate_retry_fn` runs on the delaymgr worker (mirrors ha_convergence.c delay_task pattern); clears the fence once the NEW marker commits + bytes verify; no-op once unfenced (no busy-spin, single worker serializes, atomic fence). - - **bug (verify-bypass, caught round 1 Finding A):** handle_repl_config_push originally called load_primary_config (activating pushed bytes) BEFORE the hash gate → mismatched config went live. Fixed: gate is the SOLE activation point. - - **PATTERN (Do-Not-Repeat):** for a marker-keyed out-of-band-bytes consumer, (1) verify-before-activate must be the SOLE activation path (no earlier load); (2) the consumer must be wired into ALL production reach paths — byte-arrival AND boot-recovery AND commit-advance — not just one (dead-code-in-production = incomplete task, same trap as 4.2); (3) a fence flag must have a real production READER or it's not a fence; (4) marker (WAL) and bytes (out-of-band) commit on INDEPENDENT transports → always handle push-before-commit with a commit-advance retry. async-I/O: schedule disk work off the iomgr ACK thread via delaymgr. - - **CONSOLIDATED 5.5/cleanup list:** (a) `ha_cb_*_stub` rename + stale stub-header comment ha.c ~1326; (b) ha_convergence.c ~292-297 stale cluster_apply_ctrlhello_reply doc-comment; (c) ha.h rejoin_reconciled dead field; (d) committed_config_tag now cache-only (audit remaining readers). All cosmetic. - -- **5.1 done.** Coverage audit found all 5 scenarios + 5 callbacks already covered by per-task tests; the genuine gap was the task's literal observable ("exercises every callback AND the marker path"). Added ONE integration test `test_every_callback_and_marker_path_end_to_end` (test_term_safety_adapter.c, test-only, empty production diff) driving a leader-lifecycle flow through the REAL seams: ha_primary_log_append→raft_log_append→on_persist_entry (job@1, marker@2, job@3 term-stamped); on_get_entry_term read-back; on_send_append_entries captures the marker entry (lsn2/WAL_CONFIG_MARKER/payload40) via repl_send_ae_capture_hook; on_commit_advanced ceiling caps then real ha_raft_on_repl_ack→raft_advance_commit releases; on_truncate_from drops LSN3, committed job+marker prefix + derived config intact. RED proven by neuter-and-revert. 33 Tests 0 Failures under DEFAULT ASAN (leak-detect ON), no new leak. Reviewer APPROVED (independently rebuilt the coverage map; confirmed real seams + load-bearing assertions). - - **NEXT (5.x live env):** 5.2 (clean single-failover integration) _Depends 3.3_ and 5.4 (live chaotic-multi-kill regression) _Depends 5.2,5.3,4.3_ require a real 3-node deploy (/xce-deploy-ha) on a real-fs (bug-924: overlayfs fdatasync degrades meson + would corrupt timing-sensitive live runs). 5.3 (walcompare tool) is build-only (no live env). The live-send retirement (term-0 repl_broadcast → library send loop) deferred from 3.2/3.3 lands/validates in 5.2/5.4. - -- **5.3 done.** `walcompare` tool: `xce_test walcompare [--committed N] <memberA> <memberB> [...]` + reusable `walcompare()` (walcompare.c/h). Diffs members by `(lsn → term, pcrc)` over the committed prefix; pcrc = payload-only crc32 (NOT header) so same-LSN/diff-content divergences surface; legacy-0→WAL_LEGACY_TERM. Exit 0 identical / WALCMP_DIVERGENT(1) on committed-prefix divergence (reports diverging LSN + per-member term/pcrc/present-or-MISSING for ALL N members) / negative errno on load error. REUSES the canonical waldump decoder via a NEW `waldump_path_visit` per-entry visitor seam (`waldump_path` is now a thin wrapper — existing waldump behavior + tests unchanged, NO 5th WAL parser). 11 tests; reviewer APPROVED (RED empirically reproduced by neutering compare; CLI hand-verified). - - **ADVISORY for 5.4 (reviewer MEDIUM):** the DEFAULT committed boundary = `min(recovered_tip)` across members. A member that diverged then has a CRC-broken/short tail carries a LOWER recovered_tip → lowers the shared boundary → could MASK a divergence above it. 5.4 MUST pass an EXPLICIT `--committed N` (highest acked LSN), and pair walcompare with `expect-job $jobid` (R6.4) which independently catches a lost committed job. - -- **3.4 done — bug-990 CLOSED (the live-send cutover).** ROOT CAUSE: `repl_broadcast` (the LIVE replicator, persist.c:747) built `wal_catchup_entry_t` without setting `entry.term`, so the live broadcast wire-encoded term=0 even though primary `hdr.term=p->term`; standbys took the legacy term-0 direct-apply and persisted term=0. FIX (MINIMAL cutover, reviewer-justified over the fuller raft_send_append_entries rewrite — the requirement set is met by carrying the term so entries auto-route through the already-wired 3.2 intake + 3.3 ACK): `repl_broadcast` now takes `prev_log_term` + sets `entry.term=hdr->term`/`prev_log_index=lsn-1`/`prev_log_term`; `persist_wal_async` sources prev_log_term via NEW lock-free `entry_term_locked(p, lsn-1)` (predecessor term, under p->lock; lsn==1→(0,0)); catch-up streamer `ha_stream_wal_to_standby` (config_repl.c) carries term+prev tuple per entry (ascending (lsn,term) tracker + conservative segment-skip so the FIRST streamed entry's prev references what the standby already holds); `ha_apply_wal_catchup_entry` (ha.c) routes EVERY term≠0 entry (single + each batch sub-entry, in order, guarded raft_c==c) through `raft_on_append_entries_request`→on_persist_entry (leader term stamped) — retiring the term-0 direct-apply for live/catch-up; `wal_entry_to_wire` (request.c) encodes prevLog{Index,Term}+leaderCommit PER SUB-ENTRY. 0-entry heartbeat still routes through the library for UpdateTerm (persists nothing). bug-850 contiguity + persist-before-ACK + commit-barrier + bug-960 peer-resolution all preserved. Live commit chain now genuinely end-to-end (term-carrying send → standby on_persist_entry + ACK → ha_raft_on_repl_ack → raft_advance_commit → on_commit_advanced → barrier). 2 new tests (send: `Expected 7 Was 0`; receive via the REAL `ha_apply_wal_catchup_entry` entry point: `Expected 7 Was 3`), RED independently reproduced by the reviewer. 175 Tests 0 Failures. Reviewer APPROVED. - - **REMAINING (ENV-only, NOT code):** the definitive LIVE proof — clean failover → byte-identical `(term,lsn)` WALs via `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` — could NOT run here: this overlayfs sandbox can't complete a failover on baseline OR the change (split-vote / `fdatasync errno=5`, bug-924). The fix is proven at the integration level (real receive path, real wire shape). 5.2/5.4 + the live half of 5.5 need a REAL (non-overlayfs) fs to run the harness; the harness + STRICT_TERM gate are ready and should flip GREEN there. - -- **bug-1095 + bug-1096 ABSORBED (Requirement 1 / Objective "two leaders never share a term"; bug-851 family) — owned here, fixed cross-boundary in libxce `raft.c` with user authorization.** _Surfaced by the raft-sim-conformance oracle (task 4.2), absorbed into ha-term-safety because the defect is an Election-Safety violation in the controller's SOLE production promotion driver: `vote_reply_dispatch` (ha.c ~4336) → `raft_on_vote_reply(&ha->raft, idx, vote_granted, term)` (ha.c ~4361)._ - - **The defect:** `raft_on_vote_reply` tallied vote grants with a plain counter that ignored `from_idx` and had no stale-term drop. (1) **bug-1095** — a duplicated vote-response (legal `DuplicateMessage`) from ONE peer double-counted toward quorum, so two candidates could each cross majority and become **leader at the SAME term**. (2) **bug-1096** — a grant carrying `mterm < currentTerm` (a stale, prior-election reply) was counted toward the CURRENT election, again admitting a spurious quorum. Both break Election Safety (`raft.tla` `votesGranted` is a SET; `DropStaleResponse` discards `mterm < currentTerm`). - - **The fix (commit `b5b7e01`, libxce `raft.c` only — NOT touched by this task):** added a per-peer `vote_replies[RAFT_MAX_PEERS]` idempotency set (a duplicate `from_idx` is dropped before the tally) + a `term < s->term` `DropStaleResponse` that discards the stale reply WITHOUT stepping the candidate down — mirroring the already-present pre-vote dedup idiom (`pre_vote_replies[]`). - - **Regression coverage (BOTH levels):** protocol unit — libxce `test_vote_reply_duplicate_grant_not_double_counted` + `test_vote_reply_stale_term_dropped` (`src/lib/testsuite/test_raft.c`); controller level — NEW `test_vote_reply_duplicate_grant_does_not_promote` + `test_vote_reply_stale_term_does_not_count_or_promote` (`src/controller/testsuite/test_ha_election.c`). The controller tests drive the REAL `raft_on_vote_reply` against the controller's own `ha->raft` instance (re-inited to a 5-node cluster; `xce_vote_reply_hook` NULL, `raft_c` NULL so the callbacks take their test-mode paths and `raft_cb_become_leader` still sets `ha->role = CTRL_PRIMARY`) and assert on the resulting **role / promotion** (Election Safety), not an internal counter — a duplicate grant must NOT promote at 2-of-5, and a stale grant must NOT count or promote (and must NOT step the candidate down). Both PASS over the fixed library; they would FAIL without the dedup / stale-drop. `test_ha_election` group: 0 failures. diff --git a/.kiro/specs/ha-wal-replication/brief.md b/.kiro/specs/ha-wal-replication/brief.md deleted file mode 100644 index 587607f7..00000000 --- a/.kiro/specs/ha-wal-replication/brief.md +++ /dev/null @@ -1,61 +0,0 @@ -# Brief: ha-wal-replication - -## Problem -WAL entries are written to disk on the primary but never transmitted to standbys. -`repl_broadcast()` is a no-send stub (casts payload to void, only increments counters). -`repl_wait_majority_ack()` returns immediately because the replicator's `standbys[]` -array is never populated from the HA layer's `standby_endpoints`. The result: -job submits succeed locally but are invisible on standbys and not durable across failover. - -## Current State -- `repl.c`: `repl_add_standby()`, `repl_remove_standby()`, `repl_broadcast()`, - `repl_wait_majority_ack()` all exist with correct signatures. `repl_broadcast` - is a stub (line 196-197: `(void)payload; (void)payload_len`). -- `persist.c`: `persist_wal_sync_append()` calls `repl_broadcast` → `repl_wait_majority_ack` - → advances `committed_lsn`. The call chain is wired; the send is what's missing. -- `ha.c` / `config_repl.c`: `ha->standby_endpoints` manages per-standby connections - (promoted vote eps, late-joiner connects). Nobody calls `repl_add_standby()` from here. -- `cluster.c`: `OPCode_walAck` handler exists (`ha_repl_ack_received`) but is unreachable - because standbys never receive WAL entries to ACK. - -## Desired Outcome -1. WAL entries written on the primary are transmitted to all connected standbys. -2. Each standby applies received WAL entries and sends an ACK. -3. The primary advances `committed_lsn` only after a majority of standbys ACK. -4. Job submits, state transitions, and queue changes survive primary failover. - -## Approach -Bridge the replicator and HA layer: -1. **Register**: When the HA layer connects to a standby (promoted vote ep or late-joiner), - call `repl_add_standby()` to register that endpoint with the replicator. -2. **Deregister**: When a standby disconnects (RDHUP / err_handler), call - `repl_remove_standby()`. -3. **Send**: Fill in `repl_broadcast()` to serialize the WAL header + payload into - an `OPCode_walAck` or similar message and enqueue via `xce_request()`. -4. **ACK**: On the standby, when a WAL entry is applied, send a reply carrying the - acked LSN. On the primary, `handle_wal_ack` calls `repl_on_ack()`. - -## Scope -- **In**: repl_broadcast TCP send, HA→replicator standby registration/deregistration, - standby-side WAL ACK reply, integration with existing `repl_wait_majority_ack`. -- **Out**: Changes to the heartbeat-driven decision loop (WAL streaming via NOP replies - is already wired in ha-repl-hardening). Snapshot transfer path. Config replication path. - -## Boundary Candidates -- `repl_broadcast` send implementation (repl.c) -- Standby registration bridge (ha.c / config_repl.c → repl_add_standby) -- Standby WAL ACK reply (cluster.c / ha.c) - -## Upstream / Downstream -- **Upstream**: ha-config-bootstrap (repl_broadcast API, repl_wait_majority_ack), - ha-repl-hardening (WAL catch-up payload, standby-side apply_wal_entry) -- **Downstream**: None — this is the last missing link - -## Existing Spec Touchpoints -- **Reopens**: ha-config-bootstrap task 7.6 (repl_broadcast TCP send deferred) -- **Adjacent**: ha-repl-hardening (WAL payload already wired to standby apply path) - -## Constraints -- Must use the existing `repl_add_standby` / `repl_remove_standby` API -- Must reuse `OPCode_walAck` opcode for the ACK path -- Standby WAL apply must be idempotent (replay-safe) diff --git a/.kiro/specs/ha-wal-replication/design.md b/.kiro/specs/ha-wal-replication/design.md deleted file mode 100644 index 2d31aecb..00000000 --- a/.kiro/specs/ha-wal-replication/design.md +++ /dev/null @@ -1,114 +0,0 @@ -# Design: HA WAL Replication - -## Overview - -Bridge the WAL replicator and HA connection layer so WAL entries written on the primary are transmitted to standbys, ACKed, and committed via majority quorum. The existing `repl_broadcast()` is a no-send stub; this design fills in the send path and wires the standby registration/ACK lifecycle. - -## Architecture - -``` - persist_wal_sync_append (persist.c) - → repl_broadcast(payload) ───→ xce_wal_batch_req_raw(standby ep) - → repl_wait_majority_ack() ──→ blocks until majority acked_lsn >= target - - Standby receive (cluster.c): - OPCode_walCatchupEntry → handle_wal_catchup_entry - → ha_apply_wal_catchup_entry → persist_wal_write_replicated (durable - write; entity apply is now deferred to the commit-driven apply pump, - not immediate — see note below) - → send WalAckReply(lsn) [helper: ha_send_wal_ack_reply, ha.c] - - Primary receive (ha.c): - OPCode_reply + WalAckReply → repl_on_ack → signal ack_cond -``` - -## Components - -### 1. repl_broadcast send (repl.c) - -Fill in the stub at `repl.c:193-234`. For each non-catch-up registered standby, serialize the WAL entry into a single-entry `wal_catchup_entry_t` with `raw_payload` from the caller's buffer, and enqueue via `xce_wal_batch_req_raw(ep, &entry, 1)`. - -- The existing stub already iterates `r->standbys[]` and has the catch-up/pending_ack_count guards. -- Replace the `(void)payload; (void)payload_len;` and `s->pending_ack_count++` with actual transmission. -- The `wal_catchup_entry_t` is stack-allocated; `xce_wal_batch_req_raw` copies the payload via capnp serialization (caller retains ownership). - -### 2. Standby registration bridge (ha.c) - -Wire `repl_add_standby` / `repl_remove_standby` into the HA connection lifecycle: - -- **Register on connect**: In `raft_cb_become_leader` (ha.c:~513-529), when promoting vote eps to `ha->standby_endpoints`, also call `repl_add_standby(&c->persist.repl, ep, host, port)`. In `on_standby_connected` (ha.c), when a late-joiner connects, call `repl_add_standby`. -- **Deregister on disconnect**: In `vote_ep_err_handler` (ha.c) and `standby_ep_err_handler_primary` (ha.c), call `repl_remove_standby(&c->persist.repl, ep)`. - -The replicator accesses `c->persist.repl` (already initialized in `persist_init`). - -### 3. Standby ACK reply (cluster.c / ha.c) - -After the standby applies a WAL entry (in `ha_apply_single_entry` or its caller), send a `WalAckReply` to the primary via `xce_reply`: - -- The `WalAckReply` carries `term = ha->term`, `lsn = applied_entry->lsn`, `success = 1`. -- Use `RequestReply_data_walackreplymsg` union arm. -- Sent from `ha_apply_wal_catchup_entry` after successful application of each entry in the batch. - -### 4. Primary ACK handler (ha.c) - -In `repl_ack_decode_and_dispatch` (ha.c:2506), add handling for `RequestReply_data_walackreplymsg`: - -- Extract the ACKed LSN from `WalAckReply`. -- Map the sending endpoint to a peer index via `repl_ep_t` comparison. -- Call `repl_on_ack(&c->persist.repl, ep, acked_lsn)`. - -### 5. WAL entry format - -Reuse `OPCode_walCatchupEntry` with single-entry mode (`n_entries=0`): - -| Field | Source | -|-------|--------| -| lsn | `hdr->lsn` | -| entry_type | `hdr->type` | -| raw_payload | `payload` buffer from caller | -| raw_payload_len | `payload_len` | - -This is backward-compatible with the existing standby-side handler (`ha_apply_wal_catchup_entry` in ha.c:986) and the raw wire encoder (`xce_wal_batch_req_raw` in request.c). - -## Wire Protocol - -``` -Primary Standby - │ │ - │── OPCode_walCatchupEntry ──────→ │ (single entry, lsn + payload) - │ │ → apply_wal_entry - │ ←── OPCode_reply ──────────── │ (WalAckReply { lsn, success }) - │ → repl_on_ack │ - │ → signal majority_ack_cond │ -``` - -## File Structure Plan - -| File | Action | Responsibility | -|------|--------|----------------| -| `src/controller/repl.c` | MODIFY | Fill in `repl_broadcast()` TCP send | -| `src/controller/ha.c` | MODIFY | Register/deregister standbys with replicator; handle incoming WalAckReply | -| `src/controller/cluster.c` | MODIFY | Send WalAckReply after WAL entry applied | -| `src/controller/testsuite/test_ha_wal_repl.c` | CREATE | Unit tests for broadcast send, ACK round-trip, registration lifecycle | -| `.kiro/specs/ha-wal-replication/tasks.md` | CREATE | Implementation task list | - -## Testing Strategy - -- **Unit (broadcast send)**: Create a mock endpoint, call `repl_broadcast` with a test entry, verify `xce_wal_batch_req_raw` is invoked with correct LSN and payload. -- **Unit (registration)**: Call `ha_discover_primary` path that adds a standby, verify `r->standbys[i].ep != NULL` and `repl_broadcast` iterates it. -- **Unit (ACK round-trip)**: Encode a WalAckReply, decode it on the primary side, verify `repl_on_ack` updates `acked_lsn` and signals the condition. -- **Integration**: Start 3-node cluster, submit a job, verify WAL appears on all standbys, kill primary, verify promoted standby has the job. -- **ASAN**: All new tests pass under ASAN with zero leaks. - -## Out of Boundary - -- Changes to `repl_wait_majority_ack` quorum logic (already correct). -- Changes to the HA heartbeat-driven decision loop (WAL streaming for catching-up standbys is already handled by `ha-repl-hardening`). -- Config replication path (uses `OPCode_replConfigPush`, not `repl_broadcast`). -- WAL entry serialization format changes (reuses `wal_catchup_entry_t`). - -## Boundary Commitments - -- `repl_broadcast` transmits via `xce_wal_batch_req_raw` (single-entry mode), NOT a new custom wire format. -- Standby registration uses the existing `repl_add_standby` / `repl_remove_standby` API — no new replicator functions. -- The standby ACK uses the existing `WalAckReply` capnp struct — no schema changes needed. diff --git a/.kiro/specs/ha-wal-replication/requirements.md b/.kiro/specs/ha-wal-replication/requirements.md deleted file mode 100644 index c1d93e65..00000000 --- a/.kiro/specs/ha-wal-replication/requirements.md +++ /dev/null @@ -1,62 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Bridge the WAL replicator (`repl.c`) and HA connection layer (`ha.c`/`config_repl.c`) so that WAL entries written on the primary are transmitted to all connected standbys, ACKed by a majority, and committed before the caller is acknowledged. Currently `repl_broadcast()` is a no-send stub, `repl_wait_majority_ack()` returns immediately because no standby endpoints are registered with the replicator, and job submits succeed locally but are invisible on standbys and not durable across failover. - -## 1. Replicator Standby Registration - -The primary shall maintain the replicator's standby list in sync with the HA layer's connected standby endpoints. - -**1.1** When the primary establishes a standby control channel (promoted vote endpoint during election victory, or late-joiner outbound connect), the HA layer shall register the connected endpoint with the replicator via `repl_add_standby`. - -**1.2** When a standby disconnects (RDHUP or connection error), the HA layer shall deregister the endpoint from the replicator via `repl_remove_standby`. - -**1.3** If `repl_add_standby` is called with an endpoint already registered, the system shall treat it as idempotent (no-op or update-in-place). - -## 2. WAL Broadcast - -The primary shall transmit new WAL entries to all registered standbys. - -**2.1** When `repl_broadcast` is called with a WAL entry header and payload, the system shall serialize the entry into a capnp message and enqueue it for transmission to every non-catch-up registered standby via `xce_request`. - -**2.2** While a standby is in catch-up mode (`in_catch_up` flag set), the system shall skip that standby in `repl_broadcast` and not queue WAL entries for it. - -**2.3** If a standby's pending ACK count exceeds `MAX_PENDING_ACKS`, the system shall trigger a catch-up reconnect for that standby and skip it in the current broadcast. - -**2.4** If no standbys are registered, `repl_broadcast` shall return 0 immediately without sending any messages. - -## 3. Standby WAL Receipt and ACK - -The standby shall apply received WAL entries and acknowledge them to the primary. - -**3.1** When the standby receives a WAL entry from the primary, it shall apply the entry to local persistent storage via `apply_wal_entry` and advance its `applied_lsn`. - -**3.2** After successfully applying a WAL entry, the standby shall send an ACK reply to the primary carrying the applied LSN. - -**3.3** If `apply_wal_entry` fails for a received WAL entry, the standby shall log the error, skip the entry, and not send an ACK for it. - -**3.4** The standby shall remain in `CTRL_STANDBY` state throughout WAL receipt and application — it shall not accept job dispatch or client traffic while applying replicated entries. - -## 4. Majority Commit - -The primary shall advance `committed_lsn` only after a majority of standbys have acknowledged a WAL entry. - -**4.1** When `repl_wait_majority_ack` is called for a WAL entry (ack_op == OPCode_walAck), the system shall count standbys whose `acked_lsn >= target_lsn` and return 0 when the count reaches quorum (ceil((n_active + 1) / 2) including the primary's self-ack). - -**4.2** If `repl_wait_majority_ack` times out before quorum is reached, the system shall return -ETIMEDOUT and the caller shall not advance `committed_lsn`. - -**4.3** When the primary receives a WAL ACK from a standby, it shall update that standby's `acked_lsn` in the replicator and signal any waiter on the ACK condition variable. - -**4.4** In a single-node cluster (no registered standbys), `repl_wait_majority_ack` shall return 0 immediately and the primary shall advance `committed_lsn` to the written LSN. - -## 5. Correctness and Observability - -**5.1** The system shall apply WAL entries on the standby in LSN order — entries delivered out of order shall be buffered or rejected. - -**5.2** When `repl_broadcast` transmits a WAL entry to a standby, the system shall log at DEBUG3 level the opcode and target LSN. - -**5.3** When `repl_wait_majority_ack` reaches quorum, the system shall log at INFO level the opcode and quorum count. - -**5.4** When `repl_wait_majority_ack` times out, the system shall log at WARN level with the opcode, timeout value, and per-standby ACK state. - -**5.5** All new WAL replication paths shall pass the controller unit test suite under ASAN with zero new leaks or errors. diff --git a/.kiro/specs/ha-wal-replication/research.md b/.kiro/specs/ha-wal-replication/research.md deleted file mode 100644 index 9fa23058..00000000 --- a/.kiro/specs/ha-wal-replication/research.md +++ /dev/null @@ -1,106 +0,0 @@ -# Research: HA WAL Replication - -This document captures the codebase-grounded research and gap analysis for the -`ha-wal-replication` spec — bridging the WAL replicator (`repl.c`) and the HA -connection layer (`ha.c`/`cluster.c`/`config_repl.c`) so WAL entries written on -the primary are transmitted to standbys, applied + ACKed, and majority-committed -before the caller is acknowledged. - -There was no prior `research.md`; this file is created fresh. - ---- - -## Gap Re-Audit (2026-06-18) - -### STATUS: PARTIAL (code-complete; only the live 3-node integration test is open) - -Every code path required by the 5 requirement groups is implemented and unit- -tested in the controller suite under ASAN. The single open item that is genuine -work is **5.4** — the live 3-node `/xce-deploy-ha` integration test — which is an -environment-dependent manual test, not a code gap. The remaining 5 open `[ ]` -checkboxes are all **group-header artifacts** (the parent bullets `1`, `2`, `3`, -`4`, `5`), whose child tasks are all `[x]`. - -### Requirement-to-Asset Map - -Tag legend: **MET** (verified in code), **Constraint** (design boundary, behaves -as specified), **Unknown** (only verifiable on a live cluster), **Missing**. - -| Req | Statement (abbrev.) | Tag | Evidence (file:line) | -|-----|---------------------|-----|----------------------| -| 1.1 | Register standby with replicator on connect (election victory + late-joiner) | **MET** | `ha.c:1060-1074` (become-leader promote loop calls `repl_add_standby`); `ha.c:5035-5049` (late-joiner `on_standby_connected` path) | -| 1.2 | Deregister on disconnect (RDHUP / error) | **MET** | `ha.c:3961-3964` (`vote_ep_err_handler` → `repl_remove_standby`); `ha.c:5089-5091` (`standby_ep_err_handler_primary`); `ha.c:5258` (evicted batch) | -| 1.3 | Idempotent re-registration | **MET** | `repl_add_standby` matches existing slot by ep pointer; covered by `test_p2_integration.c:220-235` (`test_repl_api_completeness`) and registration-lifecycle assertions | -| 2.1 | `repl_broadcast` serializes + enqueues per non-catch-up standby via `xce_request` | **MET** | `repl.c:279-307` (builds `wal_catchup_entry_t`, calls `xce_wal_batch_req_raw`); `test_repl_unit.c:222-231` (send-hook captures payload round-trip) | -| 2.2 | Skip standby while `in_catch_up` set | **MET** | `repl.c:265-266` (`atomic_load in_catch_up` → continue) | -| 2.3 | Pending-ACK over `MAX_PENDING_ACKS` → trigger catch-up + skip | **MET** | `repl.c:268-277` (inline start-catchup + continue); `test_repl_majority.c:282` (over-limit eviction) | -| 2.4 | No standbys → return 0 immediately | **MET** | `repl.c:251-257` (early return, WARN) | -| 3.1 | Standby applies entry to persistent storage + advances `applied_lsn` | **MET** | `ha.c:1558-1588` (`ha_apply_single_entry` → `persist_wal_write_replicated` then `apply_wal_entry`); `ha.c:1676` `ha_advance_applied_lsn` | -| 3.2 | After apply, send ACK carrying applied LSN | **MET** | `ha.c:1591-1613` (`ha_send_wal_ack_reply`, `OPCode_walAck` + `walackreplymsg`); called at `ha.c:1650`, `ha.c:1679` | -| 3.3 | Apply failure → log, skip, no ACK | **MET** | `ha.c:1573-1579`, `1640-1647`, `1666-1672` (return non-zero, no ACK) | -| 3.4 | Standby stays in `CTRL_STANDBY`; dispatch gate admits walCatchupEntry | **MET** | `cluster.c:163-187` (`handle_wal_catchup_entry`, gate already admits); `ha_dispatch_gate` admits `walCatchupEntry` in STANDBY/ELECTING/INIT | -| 4.1 | `repl_wait_majority_ack` counts standbys ≥ target, quorum incl. self-ack | **MET** | `repl.c:422+` (`repl_wait_majority_ack`); `test_repl_majority.c`, `test_commit_engine.c` quorum tests | -| 4.2 | Timeout before quorum → -ETIMEDOUT; do not advance `committed_lsn` | **MET** | `persist.c:1898-1919` (`sync_err = repl_wait_majority_ack(...)`, only advances on success); `test_commit_engine.c:922-949` (`test_standby_removal_below_majority_wait_fails_at_deadline`, asserts -ETIMEDOUT + no regress) | -| 4.3 | On WAL ACK → update `acked_lsn` + signal ack cond | **MET** | `ha.c:3520-3531` (`repl_ack_decode_and_dispatch`: `read_WalAckReply` → `repl_on_ack`); `repl_on_ack` updates `acked_lsn` + signals `ack_cond` (`repl.c:194+`) | -| 4.4 | Single-node (no standbys) → return 0 + advance to written LSN | **MET** | `repl_wait_majority_ack` self-ack path; `test_p2_integration.c:238+` (single-node production-contract test) | -| 5.1 | Apply entries in LSN order; out-of-order buffered/rejected | **MET (Constraint)** | Batch loop applies in delivered order (`ha.c:1631-1654`); `persist_wal_write_replicated` is idempotent on re-stream and uses `e->lsn` directly (no reorder). Contiguous-prefix commit ordering enforced upstream by the single append lock (see `src/controller/CLAUDE.md` Commit Ordering). | -| 5.2 | DEBUG3/TRACE log opcode + target LSN on transmit | **MET** | `repl.c:304-305` (`TRACE("repl_broadcast: sent lsn=%lu type=%u to %s:%u")`) | -| 5.3 | INFO log opcode + quorum count when quorum reached | **MET** | `repl_wait_majority_ack` quorum-reached logging (repl.c); 5.x tests assert quorum path | -| 5.4 | WARN log on timeout w/ opcode, timeout, per-standby ACK state | **MET** | `repl.c` timeout WARN path; verified by majority-timeout unit test | -| 5.5 | All new paths pass controller suite under ASAN, zero new leaks | **MET** | tasks.md 5.5 note: 45 tests / 308 assertions pass; ASAN leaks pre-existing in unrelated tests only | - -**Score: 22/22 EARS requirements MET** (4.x/2.x/3.x/5.x all verified in code + -unit tests). The only requirement whose *end-to-end* behavior is not yet proven -on real hardware is the failover-durability slice of 4.1/5.5 exercised by task -5.4 — that is an Unknown pending the live cluster, not a code Missing. - -### Open Tasks — Classification - -| Task | Text | Classification | Notes | -|------|------|----------------|-------| -| `1` | repl_broadcast TCP send (parent) | **group-header-artifact** | children 1.1, 1.2 both `[x]`; `repl.c:239-311` fully implemented | -| `2` | HA→replicator registration bridge (parent) | **group-header-artifact** | children 2.1–2.3 `[x]`; bridge wired at `ha.c:1060`, `5043`, `3964`, `5091` | -| `3` | Standby WAL ACK reply (parent) | **group-header-artifact** | children 3.1–3.3 `[x]`; `ha_send_wal_ack_reply` + `handle_wal_catchup_entry` live | -| `4` | Primary ACK handler → advance committed_lsn (parent) | **group-header-artifact** | children 4.1, 4.2 `[x]`; `read_WalAckReply`→`repl_on_ack` at `ha.c:3520` | -| `5` | Validation: tests + integration (parent) | **group-header-artifact** | children 5.1, 5.2, 5.3, 5.5 `[x]`; only 5.4 remains | -| `5.4` | Integration: 3-node cluster WAL replication | **env-dependent-test** (the one genuine remaining item) | Requires live `/xce-deploy-ha` cluster; explicitly marked `_Blocked_` / not automatable in the unit suite | - -**Doc-drift note (non-blocking):** `design.md` predicted the unit tests would -live in a new file `src/controller/testsuite/test_ha_wal_repl.c`. That file was -never created; the 5.1/5.2/5.3 unit tests actually landed in existing files — -`test_repl_unit.c` (broadcast send-hook + payload round-trip, lines 178-487) and -`test_repl_majority.c`/`test_p2_integration.c` (ACK round-trip, registration -lifecycle, over-limit eviction). Coverage is present; only the filename in the -design is stale. No action required unless the team wants the design corrected. - -### Genuine Remaining Work — Options - -The only genuine open item is **task 5.4** (live 3-node integration test). It is -a verification gap, not a code gap. - -**Prerequisite for 5.4:** a running multi-node cluster via `/xce-deploy-ha` -(e.g. `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`). Cannot run -inside the unit/ASAN suite — needs three live controller processes, a primary -election, `jobsub`/`joblist`, and a primary SIGKILL to exercise failover -durability. (`ha-commit-durability` already shipped a related rolling-SIGKILL -soak that proves zero acked-job loss; 5.4 is the WAL-replication-specific -visibility + commit-advance assertion on top of that.) - -| Option | Description | Effort | Risk | -|--------|-------------|--------|------| -| **A — Run 5.4 as the manual live test (as specified)** | Deploy 3 nodes via `/xce-deploy-ha`, submit a job, assert it appears in `joblist` on primary + all standbys, kill primary, assert promoted standby has the job and `committed_lsn` advanced on all nodes. Mark 5.4 `[x]` with captured evidence. | **Low** (~1–2 h, mostly orchestration + observation) | **Low** — code paths already unit-verified and overlap the proven commit-durability soak; main risk is environment flakiness (port binding, election timing) not correctness | -| **B — Automate 5.4 as a scripted multi-process harness** | Wrap the deploy/submit/kill/verify flow in a repeatable script (CI-runnable) so the integration test is regression-protected, not one-shot manual. | **Medium** (~0.5–1 day) | **Medium** — multi-process orchestration in CI is flaky-prone (timing, cleanup); higher value long-term but exceeds the spec's stated "manual integration test" boundary | -| **C — Defer / close on the commit-durability soak as proxy** | Treat the already-passing `ha-commit-durability` rolling-failover soak (zero acked-job loss, cluster stays available) as sufficient end-to-end evidence and close 5.4 by reference, without a dedicated WAL-replication run. | **Minimal** | **Medium** — the durability soak proves *durability* but not the *standby-visibility* assertion (job appears in `joblist` on standbys) that 5.4 specifically calls out; closing without that leaves a small explicit coverage hole | - -**Recommendation:** Option A — run 5.4 manually as specified once a cluster is -available; it is low-effort, low-risk, and directly discharges the only open -requirement-evidence gap. Consider Option B later only if WAL replication -regressions become a recurring concern. - -### Bottom line - -`ha-wal-replication` is **functionally code-complete**: all 22 EARS requirements -are implemented and unit-tested under ASAN. The 5 open parent checkboxes are -group-header artifacts. The single substantive open item is the live 3-node -integration test (5.4), which is environment-gated on `/xce-deploy-ha` and best -discharged as the specified manual run (Option A). diff --git a/.kiro/specs/ha-wal-replication/spec.json b/.kiro/specs/ha-wal-replication/spec.json deleted file mode 100644 index 21415154..00000000 --- a/.kiro/specs/ha-wal-replication/spec.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "feature_name": "ha-wal-replication", - "created_at": "2026-06-10T17:00:00Z", - "updated_at": "2026-07-18T02:05:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T02:05:00Z", - "note": "Prior state: all 20 tasks (1.1-5.5) were [x] in tasks.md but phase was stuck at \"tasks-generated\" with no validate_impl record (same administrative-oversight pattern as ha-entity-snapshots / ha-commit-durability). Real implementation commits exist under the ha-wal-replication tag (18e351e repl_broadcast send, 81dbc4d registration bridge, 566c91c standby ACK reply, 861b746 primary ACK handler, ab270fa/e2fba08 tests, plus two later hardening commits: 708f227 durable-before-ACK, 64ab210 CRC-chain ownership fix, 46bd207 test-seed fix) -- this pass focused on cross-task/feature-level checks per kiro-validate-impl scope, run immediately after a sibling spec (raft-drive-async) validated NO-GO on a shared-codebase ASAN crash (bug-1447), so extra weight was given to reproducing or ruling out that crash signature in this spec's own live soak.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) clean, zero compiler errors; meson Ok:3 Fail:0. Standalone controller_unit_test binary run directly (authoritative over meson's aggregate line per this project's own convention): 972 Tests, 0 Failures, 0 Ignored, OK -- zero \":FAIL:\" occurrences. WAL-replication-owned tests explicitly confirmed passing: test_repl_add_standby_max, test_repl_add_standby_idempotent, test_repl_registration_lifecycle, test_broadcast_one_standby_sends_entry, test_wal_ack_reply_updates_acked_lsn_and_signals, test_repl_broadcast_rejects_checkpoint, test_pin_repl_broadcast_full_fanout_skips_empty_and_catchup, test_repl_on_ack_wired_updates_watermark_and_commits. TBD/TODO grep on repl.c/ha.c/cluster.c/test_repl_unit.c: only the pre-existing, already-flagged-unrelated ha-term-safety stub at ha.c:1844. Secrets grep clean.\n\nINTEGRATION (cross-task wiring confirmed by direct code read, matching design.md's architecture end-to-end): standby registration on connect (ha.c:1556 raft_cb_become_leader, ha.c:8374 on_standby_connected) -> repl_add_standby; deregistration on disconnect (ha.c:6787 vote_ep_err_handler, ha.c:8422 standby_ep_err_handler_primary, ha.c:8814 eviction path) -> repl_remove_standby; standby applies entry then sends WalAckReply (ha.c:2601 ha_send_wal_ack_reply, wired from ha_apply_wal_catchup_entry, called from cluster.c:192 handle_wal_catchup_entry with ep+c passed through per task 3.1); primary ACK dispatch (ha.c:5928-5940 repl_ack_decode_and_dispatch's RequestReply_data_walackreplymsg arm -> repl_on_ack -> acked_lsn update + ack_cond signal, repl.c:190-226). repl_broadcast (repl.c:236-316) is a real TCP send via xce_wal_batch_req_raw, not a stub -- catch-up skip and MAX_PENDING_ACKS eviction guards both present and exercised by tests. Note: repl_broadcast's live signature now also carries prev_log_term (a later ha-term-safety-era addition for AppendEntries term-safety) beyond what design.md's Component 1 describes -- additive, not a boundary violation; the core WAL-transmission contract this spec committed to is unchanged.\n\nCOVERAGE: all 5 requirement sections map to completed tasks with no gaps -- Req 1 (Registration) -> tasks 2.1-2.3; Req 2 (Broadcast) -> tasks 1.1-1.2; Req 3 (Standby Receipt/ACK) -> tasks 3.1-3.3; Req 4 (Majority Commit) -> tasks 4.1-4.2; Req 5 (Correctness/Observability) -> tasks 5.1-5.5, with 5.1's in-order-apply guarantee satisfied by the adjacent bug-850 contiguity guard in ha_apply_single_entry (ha.c, returns -EAGAIN on an ahead-of-tail entry, silently deferred to catch-up) -- correctly relied upon rather than re-implemented, matching design.md's Out of Boundary note that catch-up streaming is ha-repl-hardening's territory. Minor logging deviation: repl_broadcast's per-send trace uses TRACE not literally \"DEBUG3\" (Req 5.2) -- cosmetic, not a functional gap.\n\nDESIGN: Architecture matches design.md's wire-protocol diagram exactly (repl_broadcast -> xce_wal_batch_req_raw -> standby apply -> WalAckReply -> repl_on_ack -> majority_ack_cond). File Structure Plan MISMATCH (non-blocking): design.md specified CREATE src/controller/testsuite/test_ha_wal_repl.c; the actual implementation consolidated the same test coverage into the pre-existing test_repl_unit.c (plus test_p2_integration.c, test_commit_engine.c, test_convergence_verification.c for cross-spec-integration pins) instead of a new file -- functionally equivalent, all Boundary Commitments (xce_wal_batch_req_raw single-entry mode, existing repl_add_standby/repl_remove_standby API, existing WalAckReply struct, no schema changes) held. Blocked tasks: none.\n\nRUNTIME LIVENESS (the blocking check, given the sibling bug-1447 finding): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh binaries from this session's build) against a real ext4 deploy dir (/home/claude/tmp/xce-ha-deploy, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, walcompare committed-prefix consistent every round, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied). Explicitly grepped all per-node logs for AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake/io_worker (bug-1447's exact signature): zero matches -- the sibling's netio.c SSL-handshake-race crash did NOT reproduce in this run. This does not prove bug-1447 is fixed or absent (it is a rare, timing-dependent race that took the sibling until round 9/10 to trigger once, and ha-wal-replication's own boundary does not touch the raft-drive-async reconnect-offload path that triggered it) -- but it does establish that ha-wal-replication's own WAL-transmission/ACK/commit path runs clean through a full supported failover lifecycle on the current codebase, which is this spec's own regression contract (Req 4, 5.5).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record); it is noted here only as a risk that was actively checked for and ruled out in this run, not adopted as this spec's blocker.", - "known_limitations": [ - "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered once by raft-drive-async's own reconnect-offload cadence at round 9/10 of its sequential soak. Did not reproduce in this spec's clean 10/10 run, but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-wal-replication; tracked entirely under raft-drive-async's remediation.", - "File Structure Plan drift (non-blocking): design.md specified a new test_ha_wal_repl.c; actual tests landed in test_repl_unit.c (+ cross-spec integration pins in test_p2_integration.c/test_commit_engine.c/test_convergence_verification.c). Coverage is equivalent; a future doc pass could update design.md's File Structure Plan to match, but this is cosmetic.", - "Req 5.2's DEBUG3-level broadcast log is implemented as TRACE level in the current logging framework's naming, not a literal \"DEBUG3\" call site -- functionally satisfies the intent (per-send LSN/opcode visibility at a low log level) but is a naming mismatch against the requirement's literal wording." - ] - } -} diff --git a/.kiro/specs/ha-wal-replication/tasks.md b/.kiro/specs/ha-wal-replication/tasks.md deleted file mode 100644 index d1bcf45a..00000000 --- a/.kiro/specs/ha-wal-replication/tasks.md +++ /dev/null @@ -1,128 +0,0 @@ -# Implementation Plan: HA WAL Replication - -## Task Overview - -5 requirements mapped to 4 implementation groups + 1 validation group. Bridging `repl.c` (replicator) and `ha.c`/`config_repl.c` (HA connection layer) so WAL entries transit from primary to standbys and are majority-committed before the caller is acknowledged. - ---- - -- [x] 1. repl_broadcast TCP send -- [x] 1.1 Fill in repl_broadcast send path - - Remove `(void)payload; (void)payload_len;` and the `pending_ack_count++` counter. - - For each non-catch-up standby (`s->ep != NULL, !in_catch_up`), build a single-entry `wal_catchup_entry_t` with `lsn=hdr->lsn`, `entry_type=hdr->type`, `raw_payload=payload`, `raw_payload_len=payload_len`. - - Enqueue via `xce_wal_batch_req_raw(s->ep, &entry, 1)`. - - If `xce_wal_batch_req_raw` fails, log WARN and continue to next standby. - - Pending-ack-count eviction guard (line 217-225) stays; evicted standbys skip the send. - - Compilation succeeds; `/xce-build controller` green. - - _Requirements: 2.1, 2.2, 2.3, 2.4_ - - _Boundary: Replicator (repl.c)_ - -- [x] 1.2 Verify no standbys → immediate return - - Confirm existing guard: when `r->standbys[]` is all-NULL, loop iterates zero times, returns 0. - - Add unit test: alloc replicator with zero standbys, call `repl_broadcast`, verify returns 0 without crashing or sending. - - _Requirements: 2.4_ - - _Boundary: Replicator (repl.c)_ - ---- - -- [x] 2. HA→replicator standby registration bridge -- [x] 2.1 Register standbys on connect - - In `raft_cb_become_leader` (ha.c), after promoting vote eps to `ha->standby_endpoints`: for each promoted ep, call `repl_add_standby(&c->persist.repl, ep, peer_host, peer_port)`. - - In `on_standby_connected` (ha.c), after ctrlhello is sent: call `repl_add_standby` with the late-joiner's endpoint, host, and port. - - Host/port obtained from `c->myconf->peers_cfg.candidates[i]`. - - Log DEBUG3 on successful registration. - - _Requirements: 1.1_ - - _Boundary: HA connection layer (ha.c, config_repl.c)_ - -- [x] 2.2 Deregister standbys on disconnect - - In `vote_ep_err_handler` (ha.c): before or after removing ep from `ha->standby_endpoints`, call `repl_remove_standby(&c->persist.repl, ep)`. - - In `standby_ep_err_handler_primary` (ha.c): call `repl_remove_standby` before endpoint cleanup. - - `repl_remove_standby` is NULL-safe (returns early if ep NULL). - - _Requirements: 1.2_ - - _Boundary: HA connection layer (ha.c)_ - -- [x] 2.3 Idempotent re-registration - - `repl_add_standby` already handles duplicate ep: it finds the existing slot by ep pointer match, resets fields. - - Unit test: call `repl_add_standby` twice with same ep, verify no double-count and no crash. - - _Requirements: 1.3_ - - _Boundary: Replicator (repl.c)_ - ---- - -- [x] 3. Standby WAL ACK reply -- [x] 3.1 Send WalAckReply after WAL entry applied - - In `ha_apply_wal_catchup_entry` (ha.c), after `ha_apply_single_entry` succeeds for an entry: construct a `WalAckReply { term=ha->term, lsn=ent->lsn, success=1 }` and send via `xce_reply` to the endpoint that delivered the entry. - - The endpoint is not directly available in `ha_apply_wal_catchup_entry` — it's called from `cluster.c:handle_wal_catchup_entry` which has the ep. Pass `ep` as a new parameter: `ha_apply_wal_catchup_entry(ha, p, entry, ep)`. - - _Requirements: 3.1, 3.2_ - - _Boundary: Standby dispatch (cluster.c, ha.c)_ - -- [x] 3.2 Skip ACK on apply failure - - In `ha_apply_single_entry` (ha.c): return distinct error codes for skip-vs-crash. - - In `ha_apply_wal_catchup_entry`: if `ha_apply_single_entry` returns non-zero, skip the ACK for that entry (already logged as ERROR). - - _Requirements: 3.3_ - - _Boundary: Standby dispatch (ha.c)_ - -- [x] 3.3 Standby role guard - - The existing `ha_dispatch_gate` already allows `OPCode_walCatchupEntry` in `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT`. - - Verify with unit test: standby applies WAL entry while in `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT` — all succeed. - - _Requirements: 3.4_ - - _Boundary: Standby dispatch (cluster.c, ha.c)_ - ---- - -- [x] 4. Primary ACK handler → advance committed_lsn -- [x] 4.1 Handle incoming WalAckReply on primary - - In `repl_ack_decode_and_dispatch` (ha.c:2506), add case for `RequestReply_data_walackreplymsg`. - - Extract `lsn` and `success` from `WalAckReply`. If `success`, call `repl_on_ack(&c->persist.repl, ep, lsn)`. - - `repl_on_ack` (repl.c:158) already updates `acked_lsn` and signals `ack_cond` — no changes needed there. - - _Requirements: 4.3_ - - _Boundary: Primary heartbeat handler (ha.c)_ - -- [x] 4.2 Verify majority-ack advances committed_lsn - - `persist_wal_sync_append` already calls `repl_wait_majority_ack` and advances `committed_lsn` on success. - - Integration test: start 3-node cluster, submit a job, verify `committed_lsn > 0` on primary after submit. - - _Requirements: 4.1, 4.2, 4.4_ - - _Boundary: Persistence (persist.c, repl.c)_ - ---- - -- [x] 5. Validation: tests and integration -- [x] 5.1 Unit test: repl_broadcast sends to registered standby - - Alloc replicator, register a mock endpoint via `repl_add_standby`. - - Call `repl_broadcast` with test header + payload. - - Verify `xce_wal_batch_req_raw` was invoked (via test hook pattern). - - Verify payload bytes round-trip through encode→decode. - - _Requirements: 2.1, 5.2_ - - _Boundary: Test (testsuite/test_ha_wal_repl.c)_ - -- [x] 5.2 Unit test: ACK round-trip - - Standby side: encode WalAckReply, decode on primary side, call `repl_on_ack`. - - Verify `acked_lsn` updated, `ack_cond` signaled, `repl_wait_majority_ack` unblocks. - - _Requirements: 4.3, 5.3_ - - _Boundary: Test (testsuite/test_ha_wal_repl.c)_ - -- [x] 5.3 Unit test: registration lifecycle - - Register, verify `nstandbys==1` and `standbys[0].ep == ep`. - - Register duplicate, verify `nstandbys==1` (idempotent). - - Deregister, verify slot freed (`ep==NULL`, `nstandbys==0`). - - _Requirements: 1.1, 1.2, 1.3_ - - _Boundary: Test (testsuite/test_ha_wal_repl.c)_ - -- [x] 5.4 Integration test: 3-node cluster WAL replication <!-- VERIFIED 2026-07-02 on ~/tmp (ext4) — see Implementation Notes --> - - Deploy via `/xce-deploy-ha`; verify primary elected. - - Submit job via `jobsub`; verify job appears in `joblist` on primary AND on all standbys. - - Kill primary; verify promoted standby has the job (failover durability). - - Verify `committed_lsn` advanced on all nodes. - - _Requirements: 4.1, 5.5_ - - _Boundary: Integration (deploy-ha)_ - -- [x] 5.5 ASAN and regression - - Full controller unit suite green under ASAN. - - No new ASAN leaks or errors vs baseline. - - `/xce-build controller` green. - - _Requirements: 5.5_ - - _Boundary: Build (meson)_ - - 45 tests (308 total assertions) pass. ASAN leaks in test_ha_rdhup_integration and test_ha_start_election_async are pre-existing (last touched by election-iomgr-refactor/checkpoint commits, not ha-wal-replication). Zero new leaks introduced by this feature. - -## Implementation Notes -- 5.4 VERIFIED 2026-07-02: deployed a 3-node cluster to `~/tmp/xce-wal-repl-deploy` (real ext4). Submitted a job on primary ctrl-1; `ctrlstatus` on all 3 nodes showed `committedLsn=2` matching (replication confirmed — note: `joblist` itself is gated primary-only, `E_XCE_STANDBY`/10014, by design in `ha.c`'s `CTRL_STANDBY` dispatch switch, so standby replication is verified via matching `committedLsn`/`appliedLsn`, not a direct standby `joblist` read). Killed primary ctrl-1; ctrl-3 was promoted (term 2→4); `joblist` against the surviving pair (masters=all 3, client transparently rotates) showed the job intact; `committedLsn=3` matched on both surviving nodes post-failover. PASS on all 4 Observable bullets. diff --git a/.kiro/specs/job-array-single-wal-record/brief.md b/.kiro/specs/job-array-single-wal-record/brief.md deleted file mode 100644 index 9f495516..00000000 --- a/.kiro/specs/job-array-single-wal-record/brief.md +++ /dev/null @@ -1,74 +0,0 @@ -# Brief: job-array-single-wal-record - -## Problem - -A job-array submission fans out into **one Raft log record per array element**. Confirmed in the controller: - -- `job_submit()` loops `for (i = 0; i < submit->array_size; i++)` calling `create_job()` once per element (`job.c:74`). -- `persist_job_hook_fun` then loops over the resulting job list calling `persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...)` once per element (`persist.c:2556`), and **each** call runs its own `repl_wait_majority_ack` commit barrier (`persist.c:2248`). -- The scheduler receives one push per committed element (`sched.c`). - -So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time **linearly in array size**, while holding the submit thread across every per-element barrier. - -## Current State - -- No batched/atomic array record exists; the array is a loop over the single-job path. -- The commit engine (`ha-commit-engine`) and barrier rollout (`ha-commit-barrier-rollout`) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically. -- Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler. - -## Desired Outcome - -A job-array submission persists, replicates, and pushes as a **single** Raft log record: - -- The controller writes **one** WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier. -- Replication carries the single record; standbys reconstruct the whole array from one committed LSN. -- The scheduler receives the array as one push and expands locally. -- Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id). - -## Approach - -Single atomic array record on top of the post-decoupling apply seam: - -- Define a `WAL_JOB_ARRAY_SUBMIT` record encoding base job spec + index range/count (no per-index overrides — array elements are uniform clones today, distinguished only by jobid; the client submit API has no per-index override field). -- Command handler appends + commits this one record through a single barrier; on commit, the shared `apply_committed_entry()` seam expands it into N in-memory `job_t` entries (reusing `create_job` per element) under one apply step. -- Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence). -- Scheduler push delivers one array descriptor; scheduler expands locally. -- Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements. - -## Scope - -- **In**: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct). -- **Out**: the apply-ordering/hook-decoupling refactor itself (sibling spec `raft-apply-decoupling`); non-array single-job path changes; scheduler policy changes. - -## Boundary Candidates - -- The `WAL_JOB_ARRAY_SUBMIT` record format + encode/decode. -- The single-barrier commit of the array record. -- The expand-on-apply step (online + replay, shared). -- The single scheduler push + scheduler-side expansion. - -## Out of Boundary - -- Per-element override semantics beyond what single-job submit already supports. -- The commit engine / barrier internals (consumed unchanged). - -## Upstream / Downstream - -- **Upstream**: `raft-apply-decoupling` (HARD — builds the array expand step on the clean `apply_committed_entry()` seam rather than the old optimistic-before-commit path); `ha-commit-engine` + `ha-commit-barrier-rollout` (single atomic commit + idempotency); `ha-consistency-invariants` (replicated scheduling state). -- **Downstream**: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits. - -## Existing Spec Touchpoints - -- **Extends**: none (new record type). -- **Adjacent**: `raft-apply-decoupling` (shared apply seam — must land first); `ha-commit-barrier-rollout` (idempotency guards reused for re-apply safety); `scheduler-controller-msg-gap` (scheduler push path). - -## Constraints - -- Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id). -- Atomic commit: either the whole array commits or none of it (no partial array on a standby). -- Idempotent re-apply (replay/retry must not duplicate elements). -- Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design). - -## Provenance - -Promoted 2026-06-24 from the roadmap backlog item `job-array-single-wal-record` (recorded 2026-06-16) via `/kiro-discovery`. diff --git a/.kiro/specs/job-array-single-wal-record/design.md b/.kiro/specs/job-array-single-wal-record/design.md deleted file mode 100644 index 9dc3bd16..00000000 --- a/.kiro/specs/job-array-single-wal-record/design.md +++ /dev/null @@ -1,486 +0,0 @@ -# Design Document: job-array-single-wal-record - -## Overview - -**Purpose**: This feature collapses a job-array submission's persistence, replication, and downstream-notification cost from O(array size) to O(1). Today, `handle_jobsubmit()` builds one WAL payload per array element and appends them through `persist_submit_job_payloads_async()`, registering one commit continuation per element (job.c:1055). A 10,000-element array therefore produces 10,000 WAL appends and 10,000 commit registrations before a single client reply is sent. This design replaces that per-element loop with one `WAL_JOB_ARRAY_SUBMIT` record — base job spec plus `(base_jobid, array_size)` — committed through exactly one majority-commit barrier, and expanded into individual `job_t` entries inside the existing `apply_committed_entry()`/`apply_wal_entry()` seam shared by live apply, restart replay, and standby catch-up. - -**Users**: Cluster operators submitting large job arrays (commit-engine load, replication bandwidth, and failover replay time no longer scale linearly with array size); maintainers of the controller's submit/apply/push code (one array-shaped record instead of N single-job records to reason about). - -**Impact**: Adds one new `wal_entry_type` (`WAL_JOB_ARRAY_SUBMIT`) alongside the unchanged single-job `WAL_JOB_SUBMIT`; adds one new shared expansion primitive (`restore_job_array()`) consumed by both live dispatchers of committed WAL entries; adds a compact push variant to the existing scheduler/API-server notification path; closes a live apply-pump gap (a resource-exhaustion apply failure is currently silently skipped rather than retried); hardens five specific, pre-existing gaps in `src/lib/jobj.c`'s job-construction primitives that this feature's N-way apply-time loop amplifies. - -### Goals -- One WAL record, one majority-commit barrier, one scheduler push, one API-server push per job-array submission — regardless of array size. -- Online apply, restart replay, and standby catch-up expand a committed array record identically, through one shared primitive. -- A committed array is applied deterministically: content-derived failures are impossible post-commit (caught pre-commit); resource-exhaustion failures fence the affected node without rejecting the array or diverging from peers, and self-heal automatically. -- Zero behavior change to non-array (single-job) submission: same WAL format, same commit-barrier count, same wire shape. - -### Non-Goals -- Per-element override semantics (array elements remain uniform clones of one base spec, distinguished only by jobid — no client-facing API change). -- Bulk query/cancel of a whole array as one operation (deferred; array elements can be in heterogeneous states at request time). -- Any change to the commit engine's or commit barrier's internal mechanics (consumed unchanged from `ha-commit-engine`/`ha-commit-barrier-rollout`). -- Scheduler dispatch/allocation policy for array elements (scheduled as ordinary single jobs once expanded). - -## Boundary Commitments - -### This Spec Owns -- The `WAL_JOB_ARRAY_SUBMIT` wire format (reusing the existing `JobSubmit` capnp struct) and its encode/append/commit path in `handle_jobsubmit()`. -- The shared array-expansion primitive (`restore_job_array()`) and its wiring into both live dispatchers of committed WAL entries (`apply_committed_entry()`, `apply_wal_entry()`). -- Atomicity of one array record's apply: all-or-nothing per attempt, idempotent re-apply, no partial/divergent element sets ever observable. -- The compact scheduler/API-server push variant for a committed array and the corresponding local-expansion logic on both receivers. -- The live apply-pump return-value fix (currently discarded) and the new live, self-healing, non-promotable fence for a resource-exhaustion apply failure. -- Hardening `src/lib/jobj.c`'s `job_init`/`makeJobsub`/`job_destroy`/`decode_Job`/`free_Job` for the five specific gaps requirements.md names, since the apply-time N-way loop amplifies each by array size. - -### Out of Boundary -- The apply-ordering/hook-decoupling refactor itself (delivered by `raft-apply-decoupling`, consumed unchanged) — `persist_job_hook_fun`'s `WAL_JOB_SUBMIT` branch is already a no-op and stays untouched. -- The non-array single-job submission WAL format, commit path, or reply plumbing — `array_size == 1` continues through today's exact code, unmodified. -- The commit engine's/commit barrier's internal mechanics — consumed via `commit_submit_async()` unchanged. -- Bulk array cancel/query as one client operation (deferred to a future spec — heterogeneous per-element state at request time is a materially different problem). -- Scheduler placement/dispatch policy for array elements — out of scope once elements are ordinary schedulable jobs. - -### Allowed Dependencies -- `raft-apply-decoupling` (complete): `apply_committed_entry()` and its dispatch contract, `persist_apply_pump_run()`'s LSN-ordered walk, `persist_register_pending_apply()`/`persist_take_pending_apply()` hand-off, `jobsubmit_async_ctx_t`'s batched-reply mechanism (consumed as-is, `pending` set to 1 for the array path instead of N). -- `ha-commit-engine`/`ha-commit-barrier-rollout` (complete): `commit_submit_async()`, per-LSN idempotency guards — consumed unchanged. -- bug-1419's in-flight jobid-range registry (`inflight_range_try_add`/`inflight_range_remove`/`jobid_range_exists`, job.c) — already range-shaped; consumed unchanged, called exactly once per array submission as it is today. -- `idgen()`'s existing `MAX_ARRAY_IDX` (32767) cap on `array_size`, enforced at jobid-minting time in the Go apiserver before a request ever reaches the controller — relied on as the structural bound satisfying Requirement 7, not re-implemented. -- `scheduler-controller-msg-gap` (complete): the controller→scheduler push protocol and its established pattern for adding a new `Data` union variant. - -### Revalidation Triggers -- Any change to `apply_committed_entry()`'s or `apply_wal_entry()`'s dispatch signature or contract (per `raft-apply-decoupling`'s own trigger: a new WAL entry type not added to both switches silently fails to apply). -- Any change to `struct JobSubmit`'s capnp shape (`controller.capnp:533`) — it is now load-bearing for three wire uses (client request, WAL payload, push payload), not one. -- Any change to `jobsub_table`/`makeJobsub()`'s interning contract in `src/lib/jobj.c` — every array element's `job_t` construction depends on it. -- Any change to `ha.c`'s promotion-gate eligibility conditions — the new Live Apply Resource Fence check must stay wired at both existing call sites (periodic release, election-time). -- Any change to `MAX_ARRAY_IDX`/`N_ARRAY_IDX_BITS` (`src/include/config.h`) or to `idgen()`'s `arraySize` validation — Requirement 7's "no chunking needed" conclusion depends on this cap continuing to hold upstream of the controller. - -## Architecture - -### Existing Architecture Analysis - -The brief's original framing (`job_submit()` looping `create_job()` per element) predates `raft-apply-decoupling` task 3.2 and is stale. Verified current state: - -| Aspect | Current state | Gap vs. this design | -|---|---|---| -| Array submission WAL cost | `handle_jobsubmit()` → `build_job_submit_payloads()` builds N plain `xce_job_t` payloads (job.c:221) → `persist_submit_job_payloads_async()` appends N `WAL_JOB_SUBMIT` entries, registers N commit continuations (persist.c:4846) | Requirement 1: N WAL appends / N commit barriers must become 1 | -| `create_job()` call site | Only inside the apply seam, via `restore_job()` (job.c:307) — never in `handle_jobsubmit` (removed by task 3.2) | None — this is already the correct location for this design's N-way expansion loop | -| Client reply batching | Already one reply per request regardless of `array_size`, via `jobsubmit_async_ctx_t` (job.c:754) and `pending` refcounting | None — reused as-is; the array path sets `pending = 1` instead of `plist.count` | -| Live apply dispatch | **Correction (found during task 3.3 review, 2026-07-14):** `apply_wal_entry()` (persist.c:2574) has **zero production call sites** — a 2026-07-10 refactor (`ha_standby_advance_committed_and_apply`, ha.c:8024) moved standby entity materialization onto the SAME `apply_committed_entry()`/`persist_apply_pump_kick` pump the primary uses; the persist.c:3722 comment claiming `apply_wal_entry()` "remains the live standby catch-up dispatch" predates that refactor and was never updated. `apply_committed_entry()` (persist.c:2721) is the **only** live dispatcher — for primary live apply, boot replay, *and* standby catch-up. | Requirement 3/4.1 is fully delivered by task 3.2's `apply_committed_entry()` wiring alone. Task 3.3's parallel wiring into `apply_wal_entry()` is kept as defensive parity with `WAL_JOB_SUBMIT`'s existing (also-dead) case in the same switch, not as a requirement-delivery path. | -| Live apply-pump failure handling | `persist_apply_pump_run()` discards `apply_committed_entry()`'s return value (`(void)apply_committed_entry(...)`, persist.c:3281) and unconditionally advances `last_applied_lsn` | Requirement 9.4/9.5: a resource-exhaustion apply failure would silently be skipped forever, never retried, never fenced | -| Promotion-gate conditions | `ha.c` gates on `truncated`, `durable_floor` vs HWM, `apply_verdict == APPLY_DIVERGENT`, `self_restart_pending` — never on `last_applied_lsn` vs `committed_lsn` | Requirement 9.4: a node stuck on a resource-exhaustion apply failure looks fully promotable today | -| Client-facing array wire format | `struct JobSubmit` (`controller.capnp:533`) already carries `(jobid, arraySize, submit: JobSubmitData)` | None — this is exactly the compact shape Requirement 1/6 need; reused verbatim, not redesigned | -| Array-size bound | `idgen()` already rejects `arraySize > MAX_ARRAY_IDX + 1` (32768) at jobid-minting time, upstream of the controller (`clib.go:165`) | None — Requirement 7 is satisfied by this pre-existing cap plus the record's O(1) payload; no chunking policy needed | -| Scheduler/API-server push | `sched_job_new_fun`/`apiserver_job_new_fun` unconditionally call `xcesched_push_joblist`/`apiserver_push_joblist` with whatever `xlist_t` the hook fire carries — already N-capable by call count | Requirement 6: the wire payload itself still enumerates N full `Job`/`SchedJob` structs; a compact descriptor push is a new wire path, not a byproduct of the existing one | - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - Client[Client array submit] - Handler[handle_jobsubmit] - ArrayAppend[Single array WAL append] - CommitEngine[Commit engine one barrier] - ApplyPumpLive[Apply pump live] - ApplyPumpReplay[Boot replay] - StandbyCatchup[Standby catch up] - RestoreArray[restore_job_array shared helper] - Fence[Live apply resource fence] - HookFire[HOOK_DETAIL_JOB_NEW one fire N jobs] - SchedPush[Compact push to scheduler] - ApiPush[Compact push to apiserver] - SchedExpand[Scheduler local expansion] - ApiExpand[Apiserver local expansion] - - Client --> Handler - Handler --> ArrayAppend - ArrayAppend --> CommitEngine - CommitEngine -->|majority ack| ApplyPumpLive - ApplyPumpLive --> RestoreArray - ApplyPumpReplay --> RestoreArray - StandbyCatchup --> RestoreArray - RestoreArray -->|resource exhaustion| Fence - RestoreArray -->|success| HookFire - HookFire --> SchedPush - HookFire --> ApiPush - SchedPush --> SchedExpand - ApiPush --> ApiExpand -``` - -**Architecture Integration**: -- **Selected pattern**: extend the existing commit-then-apply pipeline (`raft-apply-decoupling`) with one new WAL entry type and one new shared expansion primitive, rather than introducing a parallel array-specific pipeline. The array record travels through the exact same append → majority-commit → apply-pump → hook-fire stages a single-job submission does; only the payload shape and the expansion fan-out differ. -- **Domain/feature boundaries**: `restore_job_array()` is the only place array-record bytes become N `job_t` entries — both live dispatchers call it, neither reimplements it, mirroring how `restore_job()` already anchors single-job construction for both dispatchers today. -- **Existing patterns preserved**: `jobsubmit_async_ctx_t`'s batched single-reply mechanism, `persist_register_pending_apply()`'s command hand-off, the in-flight jobid-range registry, the `t_apply_seam_notifying` thread-local guard, `wal_classify`'s WAIT/RETAIN table, the `Data` union's additive-variant extension pattern. -- **New components rationale**: `restore_job_array()` (no array-aware expansion primitive exists); the Live Apply Resource Fence (no live-path analog of the boot-only `recovery_unrecoverable`/`persist_recover_loud_fence_if_unrecoverable` fence exists); the compact push variant (existing push wire format is per-job, not descriptor-shaped). -- **Steering compliance**: preserves the synchronous-commit-as-policy invariant (`ha-async-io.md`) — the array append still runs off the iomgr thread through the unchanged commit-engine contract; the Live Apply Resource Fence follows the controller's Process Lifecycle invariant exactly (`src/controller/CLAUDE.md`: loud ERROR, non-promotable, stay alive, keep retrying, self-heal — never self-terminate). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Controller core | C (gnu17), existing | All changes land in `src/controller/{job.c,persist.c,sched.c,apiserver.c,ha.c}` and `src/lib/jobj.c` | No new language/runtime | -| Serialization | Cap'n Proto (existing `controller.capnp`) | `WAL_JOB_ARRAY_SUBMIT` payload and the new compact push payload both reuse the existing `struct JobSubmit`; one new `Data` union variant added for the push path | No new struct; `encode_JobSubmit`/`decode_JobSubmit` already generated | -| Scheduler daemon | C (gnu17), existing | `src/scheduler/src/handler.c` gains a sibling case to `Data_data_schedjobs` for local array expansion | Follows `scheduler-controller-msg-gap`'s established extension pattern | -| API server | Go 1.23.4, existing | `src/frontend/apiserver/backend/` gains a sibling decode path to `cJobToModel` for local array expansion | CGO bridge, consistent with existing job-push consumption | -| Test harness | `controller_unit_test`, `xcelib_test`, `/xce-test-ha` | Unit coverage for the new apply path and fence; live soak for non-regression | Extends existing harnesses, no new framework | - -## File Structure Plan - -### Modified Files -- `src/include/wal_types.h` — add `WAL_JOB_ARRAY_SUBMIT` to `wal_entry_type` (a new value distinct from `WAL_JOB_SUBMIT`, keeping the single-job format untouched per Requirement 8.1). -- `src/controller/job.c` — `handle_jobsubmit()`: branch on `submit->array_size == 1` (today's exact code path, unmodified) vs. `> 1` (new single-append path: encode once via a new `capnp_encode_job_array()`-equivalent using `encode_JobSubmit`, append via a new `append_job_array_wal_entry_local()`, register one continuation via a new `persist_submit_job_array_async()`, set `actx->pending = 1`). `jobsubmit_commit_continuation`/`jobsubmit_apply_complete`/`jobsubmit_async_ctx_t` are reused unmodified — both are already parametric on `pending` and keyed by LSN, not by element count. New shared helper `restore_job_array()` (sibling to `restore_job()`, same file) implements atomic N-element construction: scratch-builds all elements first, bulk-inserts into `c->jobmgr` only once every element succeeds, and discards the scratch list directly (never touching `c->jobmgr`) on failure. -- `src/controller/persist.c` — `wal_classify()`: add `WAL_JOB_ARRAY_SUBMIT → WAL_MODE_WAIT` (same classification as `WAL_JOB_SUBMIT`). `apply_committed_entry()` and `apply_wal_entry()`: each gains one new `case WAL_JOB_ARRAY_SUBMIT` calling the shared `restore_job_array()`; `apply_committed_entry()`'s case additionally builds the N-element `xlist_t` and fires `HOOK_DETAIL_JOB_NEW` once (mirroring its existing single-job case); `apply_wal_entry()`'s case stays hook-free (mirroring its existing single-job case). `persist_apply_pump_run()`: capture `apply_committed_entry()`'s return value; on negative rc, stop the walk without advancing `last_applied_lsn`, log loud, engage the Live Apply Resource Fence (mirrors the existing `apply_pump_read_entry` failure branch immediately above it); on success following a prior failure at this LSN, clear the fence. New small primitives: `persist_apply_resource_fenced()`, `persist_mark_apply_resource_fenced()`, `persist_clear_apply_resource_fence()`. -- `src/controller/ha.c` — both promotion-gate eligibility call sites (periodic fence-release check, election-time check) gain one additional required condition: `!persist_apply_resource_fenced(&c->persist)`, threaded alongside the existing `apply_divergent`/`self_restart_pending` conditions. -- `src/controller/sched.c` — `sched_job_new_fun()`: branch on `l->count > 1` (the only way a `HOOK_DETAIL_JOB_NEW` fire ever carries more than one element, per Requirement 3.4) to call a new `xcesched_push_jobarray()` (encodes the compact `Data` union array variant) instead of `xcesched_push_joblist()`. -- `src/controller/apiserver.c` — `apiserver_job_new_fun()`: same `l->count > 1` branch, calling a new `apiserver_push_jobarray()`. -- `src/schema/controller.capnp` — `struct Data`'s union gains one new variant reusing `JobSubmit` (e.g. `jobArray @6: JobSubmit $C.mapname("job_array")`), following the existing additive-variant pattern (`schedjobs`/`jobs`/`workers`/`resources`/`resvals`). -- `src/scheduler/src/handler.c` — sibling case to the existing `Data_data_schedjobs` (line 112) decoding the new array variant and expanding it locally into N schedulable tasks via the same `base_jobid + i` arithmetic used by the controller, logging one confirmation line per array (Requirement 6.4). -- `src/frontend/apiserver/backend/` — sibling decode path to `cJobToModel` (`convert.go:31`) for the new array variant, expanding N `models.JobInfo` entries locally, logging one confirmation line per array (Requirement 6.4). -- `src/lib/jobj.c` — `job_init()`: check `makeJobsub()`'s (and `makeString(src->ckptinfo)`'s) return value; propagate failure instead of storing NULL with a success return (Requirement 10.1). Root-cause and fix the `jobsub_table` initialization-ordering gap behind the previously observed NULL-with-EINVAL failure (Requirement 10.2). `job_destroy()`: fix the `accounts` leak (currently `/* FIXME??? */`, jobj.c:584) by adding the missing free call, following the existing pattern used for `allocs`/`pidinfo`/`run_usage` (Requirement 10.4). -- **`src/lib/request.c` (added during implementation, task 5.3b, discovered 2026-07-15):** `extract_data()` — the shared inbound-push reassembly dispatcher every receiver (scheduler, API server) passes through inside `vtask_entry()` (`src/lib/xcectx.c`) before its own handler ever runs. Missed by this component's original architecture analysis, which examined `sched.c`/`apiserver.c`/`handler.c` but not the transport dispatch layer beneath them. Its switch has no case for `Data_data_jobArray` and falls into `default: rc = -EINVAL`; because `vtask_entry()`'s `if (callnext)` treats any nonzero return (including a negative errno) as "reassembly complete, proceed," this silently frees the real payload and hands the receiver an empty `dataset_t`, with no log anywhere in the chain. Add a `case Data_data_jobArray:` passing the single, unchunked descriptor straight through — no pagination loop needed, unlike the existing LIST-shaped cases. - -### New Test Files -- `src/controller/testsuite/test_job_array_apply.c` — unit tests: single-record N-way expansion, idempotent re-apply (no duplicate elements), all-or-nothing scratch-discard on a simulated construction failure (never touches `c->jobmgr`), identical results from `apply_committed_entry()` vs. `apply_wal_entry()` for the same record. -- `src/controller/testsuite/test_job_array_fence.c` — unit tests: `persist_apply_pump_run()` halts and does not advance `last_applied_lsn` on a negative `apply_committed_entry()` return; Live Apply Resource Fence engages, blocks promotion eligibility, and self-clears on a successful retry (Requirement 9.6's fault-injection scenario). -- `src/lib/testsuite/test_jobj_hardening.c` — unit tests for the five Requirement 10 gaps: `job_init` propagates a `makeJobsub()`/`makeString()` NULL; `jobsub_table` is initialized before first use under the previously-observed ordering; MOVE_PTR fields are confirmed NULL on a freshly-built array-element payload; `job_destroy` no longer leaks `accounts`; `decode_Job`/`free_Job` `ckptinfo` coverage is confirmed correct as-is (no wire field exists, so no leak). -- Extend `/xce-test-ha`'s soak scenario set with a large-array submission case (commit/push count assertions) and a chaotic-failover case submitting an array immediately before killing the primary (Requirement 8.2, 8.3). - -## System Flows - -### Array submission: append, commit, apply, push - -```mermaid -sequenceDiagram - participant C as Client - participant H as handle_jobsubmit - participant CE as Commit engine - participant AP as Apply pump - participant RA as restore_job_array - participant N as HOOK_DETAIL_JOB_NEW - participant S as Scheduler push - participant A as Apiserver push - - C->>H: array submit base spec plus array_size - H->>H: validate, jobid_range_exists, inflight_range_try_add (unchanged, range-shaped) - H->>CE: append ONE WAL_JOB_ARRAY_SUBMIT, register ONE continuation - CE-->>CE: majority ack (one barrier) - CE->>AP: resolve committed, hand off to apply pump - AP->>RA: restore_job_array (base_jobid, array_size, base spec, lsn) - alt all N constructed - RA->>N: fire once, N-element list - N->>S: compact push (base spec + range) - N->>A: compact push (base spec + range) - AP->>H: reply success (base jobid) - else resource exhaustion mid-loop - RA->>RA: discard scratch list (c->jobmgr never touched) - RA-->>AP: negative rc - Note over AP: last_applied_lsn NOT advanced; fence engaged; retried next kick - end -``` - -Key decision not obvious from the diagram: `restore_job_array()` builds every element in a scratch list and only bulk-inserts into `c->jobmgr` once the whole array has successfully constructed — so the array is never observably partial to a concurrent reader on the *success* path either, not only when a retry starts over from index 0 after a failure (Requirement 2.3, 9.4). - -### Live apply resource fence: engage and self-heal - -```mermaid -flowchart LR - Apply[apply_committed_entry fails resource exhaustion] --> Log[Loud ERROR logged] - Log --> Fence[Live Apply Resource Fence engaged] - Fence --> Gate[Promotion gate refuses this node] - Gate --> Retry[Apply pump retries same LSN on next kick] - Retry -->|still failing| Log - Retry -->|succeeds| Clear[Fence cleared by apply pump itself] - Clear --> Eligible[Node eligible for promotion again] -``` - -Key decision: the fence clears from inside the apply pump the moment the retried LSN succeeds — it does not depend on an election or on the unrelated WAL-durability catch-up check ever running, so a long-lived primary that never loses leadership still self-heals (Requirement 9.5). This is deliberately a *new*, narrowly-scoped fence rather than a reuse of `recovery_unrecoverable`, whose existing periodic clear is gated on WAL durability catch-up alone and would wrongly clear before the stuck apply actually resolves (see `research.md`, Design Decisions). - -## Requirements Traceability - -| Requirement | Summary | Components | -|---|---|---| -| 1.1-1.3 | Single WAL record, single commit barrier, no per-element append | WAL Array Submission | -| 2.1-2.3 | Atomic, all-or-nothing commit per record | WAL Array Submission (commit), Array Apply Expansion (no partial construction from an uncommitted record) | -| 3.1-3.4 | Shared expand-on-apply, online = replay = standby catch-up, idempotent re-apply, one hook fire | Array Apply Expansion | -| 4.1-4.3 | Standby reconstruction from one committed position | WAL Array Submission (replication carries one record, unchanged replication mechanics), Array Apply Expansion | -| 5.1-5.3 | Preserved per-element identity/addressing | Array Apply Expansion (unchanged `base_jobid + i` arithmetic — verification-only, no new mechanism) | -| 6.1-6.5 | Single compact push to scheduler and API server, local expansion | Compact Scheduler/API-Server Push | -| 7.1-7.2 | Bounded record size | WAL Array Submission (satisfied by construction — see Architecture table) | -| 8.1-8.4 | No regression to non-array submission or existing tests | Non-Regression & Testing (cross-cutting; `array_size == 1` path untouched) | -| 9.1-9.6 | Deterministic, total apply — pre-commit rejection, live resource fence | WAL Array Submission (9.1, 9.2 — pre-commit validation), Array Apply Expansion (9.3 — total apply post-commit), Live Apply Resource Fence (9.4, 9.5, 9.6) | -| 10.1-10.5 | jobj.c hardening | jobj.c Hardening | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|---|---|---|---|---|---| -| WAL Array Submission | job.c, persist.c, wal_types.h | Encode and commit one array record through one barrier | 1, 4.1-4.2, 7, 9.1, 9.2 | Commit Engine (P0), in-flight range registry (P0) | Service, Batch | -| Array Apply Expansion | persist.c, job.c (`restore_job_array`) | Expand a committed array record into N `job_t` entries, identically for live apply / replay / standby catch-up | 2, 3, 4.3, 5, 9.3 | `apply_committed_entry`/`apply_wal_entry` (P0), jobj.c primitives (P0) | Service, Batch | -| Live Apply Resource Fence | persist.c, ha.c | Fence a node non-promotable on a resource-exhaustion apply failure, self-heal on success | 9.4, 9.5, 9.6 | Apply pump (P0), promotion gate (P0) | State | -| Compact Scheduler/API-Server Push | sched.c, apiserver.c, controller.capnp, scheduler `handler.c`, apiserver `backend/` | One compact push per committed array, expanded locally by each receiver | 6 | `HOOK_DETAIL_JOB_NEW` (P0), `Data` union (P0) | Event, API | -| jobj.c Hardening | src/lib/jobj.c | Reliable, leak-free job construction under N-element reuse | 10 | RTTI object system (P0) | Service | - -### WAL Array Submission - -#### Single array record append and commit - -| Field | Detail | -|-------|--------| -| Intent | Encode the whole array as one `WAL_JOB_ARRAY_SUBMIT` record and commit it through one barrier | -| Requirements | 1.1, 1.2, 1.3, 4.1, 4.2, 7.1, 7.2, 9.1, 9.2 | - -**Responsibilities & Constraints** -- `handle_jobsubmit()` branches on `array_size`: `== 1` runs today's unmodified single-job path (Requirement 8.1); `> 1` runs the new single-append path. -- Pre-commit validation (`jobid_range_exists`, `inflight_range_try_add`) is unchanged and already range-shaped — no new range-claim logic is needed (Requirement 9.1). -- The record's encoded payload is `struct JobSubmit` (`jobid` = base, `arraySize`, one `submit: JobSubmitData`) — O(1) in `array_size` by construction, and `array_size` is already hard-capped at 32768 upstream by `idgen()` (Requirement 7 satisfied without a chunking policy). -- Rejection on failed pre-commit validation happens strictly before any WAL append (Requirement 9.2), identical ordering to today's single-job path. - -**Dependencies** -- Outbound: `commit_submit_async()` via a new `persist_submit_job_array_async()` (Criticality P0) -- Outbound: `inflight_range_try_add`/`jobid_range_exists` (Criticality P0, unchanged) -- Inbound: `handle_jobsubmit()` (Criticality P0) - -**Contracts**: Service [x] / Batch [x] - -##### Service Interface -```c -/* Encodes and appends ONE array-shaped WAL entry (WAL_JOB_ARRAY_SUBMIT, - * payload = encoded JobSubmit), registers ONE commit continuation. Mirrors - * persist_submit_job_payloads_async's per-entry contract but for a single - * array-shaped entry -- no loop over elements at this layer. */ -int persist_submit_job_array_async(persist_t* p, uint64_t base_jobid, - uint64_t array_size, - const xce_job_submit_t* base_spec, - uint64_t* out_lsn, - persist_job_commit_notify_fn fn, void* ctx); -``` -- Preconditions: `array_size > 1`; the `[base_jobid, base_jobid + array_size)` range has already been claimed via `inflight_range_try_add`. -- Postconditions: on success, exactly one WAL entry is durably appended locally and exactly one continuation is registered; `fn` is invoked exactly once with the outcome (mirrors `persist_submit_job_payloads_async`'s per-payload contract, specialized to one payload). -- Invariants: no `job_t` object exists in `c->jobmgr` for any array index until the commit succeeds and the apply seam runs (same non-negotiable ordering as the single-job path). - -##### Batch / Job Contract -- Trigger: `handle_jobsubmit()`'s array-size-branch, after pre-commit validation passes. -- Input / validation: `(base_jobid, array_size, base_spec)`, already range-claimed. -- Output / destination: one WAL entry; one registered commit continuation feeding `jobsubmit_commit_continuation`/`jobsubmit_apply_complete` unmodified (`actx->pending = 1`). -- Idempotency & recovery: a client retry after a commit timeout produces a second, independent `WAL_JOB_ARRAY_SUBMIT` entry for the same range — the array-scoped analog of bug-1419's existing single-job dedup story (see Testing Strategy, Requirement 8.3). - -### Array Apply Expansion - -#### restore_job_array — shared N-way expansion primitive - -| Field | Detail | -|-------|--------| -| Intent | Expand one committed array record into N `job_t` entries, atomically (scratch-build then bulk-insert — never partially visible), identically for every caller | -| Requirements | 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 4.3, 5.1, 5.2, 5.3, 9.3 | - -**Responsibilities & Constraints** -- Decodes `struct JobSubmit` from the committed payload; for `i` in `[0, array_size)`, computes `jobid = base_jobid + i` (the exact arithmetic `build_job_submit_payloads` uses today, Requirement 5.1) and constructs a `job_t` via `create_job()`, sharing the one decoded `submit` pointer across elements (mirroring `restore_job()`'s existing `submit`/`ckptinfo` ownership discipline for a single job). -- Idempotent per index: if `jobid` already exists in `c->jobmgr`, that index is skipped, not treated as an error (handles a re-apply of an already-fully-applied record). If every index already exists, returns `-EEXIST` (mirrors `restore_job()`'s convention), and the caller fires no hook (Requirement 3.3). -- Scratch-construct-then-bulk-insert, not construct-then-rollback: all `array_size` elements are built via `create_job()`-equivalent construction into a local scratch list first, **never touching `c->jobmgr` while the construction loop is running**. Only after every element in the scratch list succeeds does one final pass insert all of them into `c->jobmgr` under a single write-lock acquisition (one atomic visibility step, not per-element lock/unlock cycles — this is what makes the array invisible-then-visible rather than visible-then-rolled-back on the success path). This means there is no window during *construction*, success or failure, where a concurrent reader of `c->jobmgr` (a status query, a cancel request) can observe some array indices present and others absent. If construction fails at index `k` (resource exhaustion only — content-derived failures are impossible here because Requirement 9.1's pre-commit validation already ran), the `0..k-1` scratch objects are destroyed directly (no `c->jobmgr` mutation ever occurred, so no `key_del`/rollback race is possible) and the function returns a negative rc. A retry re-enters the loop from index 0 against a fresh scratch list, not from a saved offset. - - **Residual gap in the bulk-insert step itself** (found during implementation, task 3.1 review round 1): the final insert-into-`c->jobmgr` pass is *not* actually failure-proof — `key_add()` can itself fail on allocation/growth exhaustion (`c->jobmgr`'s underlying table grows via `realloc()`, and no capacity-reservation primitive exists anywhere in this codebase's `kv.c`/`kvmgr.c`/`zarray.c` stack to pre-check this before mutating). Because the underlying table has no `key_del`, an already-inserted prefix `[0, inserted)` cannot be cleanly un-inserted if a later element in the same bulk pass fails. The implemented contract is therefore: `restore_job_array()` never *silently* claims success on a bulk-insert failure (it returns a negative errno, honestly), but a prefix of elements may be left genuinely, permanently visible in `c->jobmgr` while the overall call reports failure — and critically, that prefix is never included in `out_new_jobs`, so `HOOK_DETAIL_JOB_NEW` never fires for it, on this attempt or any later successful retry of the same LSN. Tasks 3.2/4.x must account for this: a bulk-insert-phase failure leaves elements that are individually addressable (status/cancel/completion work) but permanently un-pushed to the scheduler/API server unless something later re-derives and re-notifies them. This is a narrower, real exception to "atomically invisible-then-visible," not the general case — it fires only on late-stage resource exhaustion, not on ordinary construction failures (which remain fully atomic per the paragraph above). -- Called identically from `apply_committed_entry()` (which additionally builds the N-element list and fires `HOOK_DETAIL_JOB_NEW` once, Requirement 3.4) and from `apply_wal_entry()` (hook-free, matching its existing single-job case) — this is the one place expansion logic exists (Requirement 3.1, 3.2). **Correction (2026-07-14, see the Existing Architecture Analysis table's "Live apply dispatch" row):** `apply_wal_entry()` has zero production call sites — `apply_committed_entry()` alone delivers Requirement 3.1/3.2/4.1 for live apply, boot replay, and standby catch-up. The `apply_wal_entry()` wiring is retained as defensive parity with `WAL_JOB_SUBMIT`'s existing (also-dead) case in the same switch, not as a requirement-delivery path. -- Every constructed element is tagged with `submit_lsn = entry_lsn` (mirrors `restore_job()`, feeds the existing egress-watermark predicate unchanged). -- Exposes an `XCE_TESTING`-guarded fault-injection seam (a forceable construction failure at a caller-chosen element index, compiled out in production builds, matching this codebase's existing `#ifdef XCE_TESTING`-guarded test-only-symbol convention per `structure.md`) — the concrete unit `test_job_array_apply.c` drives directly to satisfy Requirement 9.6's fault-injection scenario. - -**Dependencies** -- Outbound: `create_job()` (job.c, unchanged), `jobj.c`'s hardened primitives (Criticality P0) -- Inbound: `apply_committed_entry()`, `apply_wal_entry()` (Criticality P0, both) - -**Contracts**: Service [x] / Batch [x] - -##### Service Interface -```c -/* Expands one committed WAL_JOB_ARRAY_SUBMIT payload into job_t entries. - * Builds every element into a local scratch list first -- c->jobmgr is - * never mutated until every element has successfully constructed, then one - * final pass inserts them all (a pass that cannot fail). Returns the count - * of NEWLY created elements (0..array_size) on success, -EEXIST if every - * index already existed (idempotent no-op, no hook should fire), or a - * negative errno on a genuine construction failure -- in which case the - * scratch list is discarded directly and c->jobmgr was never touched (Req - * 9.4: no partial set ever observable, on the failure path OR the success - * path). */ -int restore_job_array(cluster_t* c, const void* capnp_payload, size_t len, - uint64_t entry_lsn, xlist_t* out_new_jobs /* optional */); -``` -- Preconditions: `lsn` is committed; payload decodes to a valid `JobSubmit` (guaranteed by Requirement 9.1/9.3 — pre-commit validation makes a content-derived decode failure here impossible). -- Postconditions: either all `array_size` elements exist in `c->jobmgr` tagged with `entry_lsn`, or none of this attempt's elements do — and this holds at every instant, not only at return: `c->jobmgr` is untouched until the entire array is ready to insert as one bulk pass, so no concurrent reader can ever observe a partially-constructed array (Requirement 2.3). -- Invariants: applying the same `(WAL_JOB_ARRAY_SUBMIT, lsn, payload)` twice produces the same observable state as applying it once (Requirement 3.3). - -##### Batch / Job Contract -- Trigger: a new `case WAL_JOB_ARRAY_SUBMIT` in `apply_committed_entry()` (live apply pump, boot replay) and in `apply_wal_entry()` (standby catch-up). -- Input / validation: committed WAL payload, already CRC-verified by the point apply is reached (unchanged from the single-job path). -- Output / destination: N `job_t` entries in `c->jobmgr`; one `HOOK_DETAIL_JOB_NEW` fire (live-apply path only) carrying all N. -- Idempotency & recovery: guarded per-index existence check; `last_applied_lsn` is the resumable cursor for restart (unchanged mechanism, now also gated on this function's return value — see Live Apply Resource Fence). - -### Live Apply Resource Fence - -#### Apply-pump return-value gating and live promotion fence - -| Field | Detail | -|-------|--------| -| Intent | Never silently skip a resource-exhaustion apply failure; fence the node non-promotable while retrying; self-heal on success | -| Requirements | 9.4, 9.5, 9.6 | - -**Responsibilities & Constraints** -- `persist_apply_pump_run()` starts checking `apply_committed_entry()`'s return value (currently discarded) — on a negative result, the walk stops at that LSN without advancing `last_applied_lsn`, mirroring the existing `apply_pump_read_entry` failure branch immediately above it in the same function. This is a general fix (any WAL type's apply failure, not array-specific) that this feature's array-expansion failure is the first live trigger for. -- On engaging, logs a loud, operator-visible ERROR (LSN, reason) and sets a new, narrow fence flag — deliberately **not** `recovery_unrecoverable` (its existing periodic clear is gated on WAL-durability catch-up alone and would wrongly clear before the stuck apply actually resolves; see `research.md` Design Decisions). -- The fence clears itself, from inside the apply pump, the instant a retry of the same LSN succeeds — no dependency on an election or on any other periodic check running (Requirement 9.5). -- A fence on one node never blocks any other node's independent apply pump — each node's fence state is local (Requirement 9.5, satisfied structurally: nothing here is cross-node). -- Both existing `ha.c` promotion-gate call sites (periodic fence-release check, election-time eligibility check) gain one additional required condition alongside the existing `apply_divergent`/`self_restart_pending` checks. - -**Dependencies** -- Outbound: none new — a local atomic flag on `persist_t` -- Inbound: `persist_apply_pump_run()` (sets/clears), `ha.c`'s two promotion-gate call sites (reads) (Criticality P0, both) - -**Contracts**: State [x] - -##### State Management -- State model: one new field on `persist_t` (an atomic LSN or bool) recording "an LSN is currently stuck failing to apply." Orthogonal to `recovery_unrecoverable` and `apply_verdict` — no shared state, no risk of one clearing the other's condition. -- Persistence & consistency: in-memory only, like `last_applied_lsn` itself — recomputed by construction (the pump re-derives "stuck" the next time it fails to apply the same LSN after a restart; it is not itself a correctness-critical persisted fact). -- Concurrency strategy: set/cleared only from `persist_apply_pump_run()`'s single-threaded walk (the existing `apply_pump_lock` already serializes this); read from `ha.c`'s promotion-gate paths under their existing locking. - -## Data Models - -No new persistent entity type. `job_t`, `xce_job_t`, and `job_submit_data_t` are unchanged — array elements are ordinary jobs once expanded, addressed exactly as single-submission jobs are today. - -### Data Contracts & Integration - -**WAL payload** (`WAL_JOB_ARRAY_SUBMIT`, new `wal_entry_type` value): the existing `struct JobSubmit` capnp struct, unmodified — - -```capnp -struct JobSubmit $C.mapname("xce_job_submit_t") { - jobid @0: UInt64; # base jobid - arraySize @1: UInt64 $C.mapname("array_size"); - submit @2: JobSubmitData; # one shared spec for every element -} -``` - -**Push payload** (new `Data` union variant, additive): reuses the same struct — - -```capnp -struct Data $C.mapname("xce_data_t"){ - data :union $C.mapuniontag("table") { - schedjobs @0: SchedJobData $C.mapname("jobs"); - schedworkers @1: WorkerData $C.mapname("workers"); - jobs @2: JobData $C.mapname("jobs"); - workers @3: WorkerData $C.mapname("workers"); - resources @4: ResourceData $C.mapname("resources"); - resvals @5: ResourceVals $C.mapname("resvals"); - jobArray @6: JobSubmit $C.mapname("job_array"); # NEW - } -} -``` - -Both reuses are deliberate (Requirement 6's own wire-shape note: "the same compact shape as the WAL record") — one capnp struct, three uses (client request, WAL payload, push payload), zero new encoder/decoder pairs to write or maintain. - -### Compact Scheduler/API-Server Push - -#### One push per committed array, expanded locally - -| Field | Detail | -|-------|--------| -| Intent | Deliver a committed array to the scheduler and API server as one compact descriptor; each expands it locally into N entries | -| Requirements | 6.1, 6.2, 6.3, 6.4, 6.5 | - -**Responsibilities & Constraints** -- `sched_job_new_fun()`/`apiserver_job_new_fun()` (the existing `HOOK_DETAIL_JOB_NEW` subscribers) branch on `l->count > 1` — the only way `apply_committed_entry()` ever fires this hook with more than one element (Requirement 3.4) — to call a new `xcesched_push_jobarray()`/`apiserver_push_jobarray()` instead of the existing per-job `..._push_joblist()`. The `l->count == 1` branch (ordinary single-job submission) is untouched (Requirement 8.1). -- The new push functions derive `(base_jobid, array_size)` from the already-constructed `xlist_t` (first element's jobid, list count) and encode one `Data.jobArray` message — never N individually-encoded `Job`/`SchedJob` structs. -- The scheduler (`handler.c`, sibling case to `Data_data_schedjobs`) and the Go apiserver (`backend/`, sibling path to `cJobToModel`) each expand the descriptor locally into N entries using the identical `base_jobid + i` arithmetic the controller itself uses — the same algorithm at three layers, not three different implementations. -- Redelivery on reconnect/rejoin must not double-expand: both receivers dedupe expanded entries by `jobid` using the exact same key space and lookup an ordinary single-job push already uses downstream (no new dedup mechanism) — a scheduler or API-server reconnect that re-receives an already-expanded array's compact push is a no-op per index, mirroring `restore_job_array()`'s own idempotent-per-index contract at the controller layer. -- Each receiver logs one identifiable confirmation line (base jobid + element count) once local expansion completes, satisfying Requirement 6.4's log-based verification without a new query API. -- Elements are treated as ordinary single jobs for scheduling purposes once expanded (Requirement 6.3) — no new scheduling logic. - -**Dependencies** -- Inbound: `HOOK_DETAIL_JOB_NEW` fire from `apply_committed_entry()` (Criticality P0) -- Outbound scheduler push: direct `xce_request()`/`OPCode_push`/`Request_body_pushmsg` (Criticality P0). **Correction (found during task 5.1 review, 2026-07-15):** `xce_push_data_req()`'s dispatch switch has explicit cases only for jobs/workers/resources/resvals and hard-rejects (`-EINVAL`) anything else via its `default` case — it cannot carry a single non-list `Data.jobArray` descriptor, only chunk LIST-shaped payloads. `xcesched_push_jobarray()` therefore sends directly via `xce_request()`, mirroring `xce_push_req()`'s existing shape for a single non-list `xce_data_t` payload. This is a faithful, wire-compatible substitute but does NOT go through `xce_push_data_req()`'s `checkf`-gated `check_controller`/`c->running` liveness check that every other push path gets — a narrow, low-severity shutdown-race behavioral difference from sibling pushes, not a new safety hazard. Task 5.2's API-server push faces the identical fork and should follow the same direct-`xce_request()` pattern rather than re-deriving it. - -**Contracts**: Event [x] / API [x] - -##### Event Contract -- Published events: `HOOK_DETAIL_JOB_NEW` (unchanged detail, unchanged signature) — the branch on element count lives entirely inside the two existing subscribers, not in the hook system itself. -- Subscribed events: none new. -- Ordering/delivery guarantees: unchanged — one push per committed array, delivered after the array's own commit (same ordering guarantee single-job pushes already have). - -##### API Contract -| Direction | Message | Payload | Notes | -|---|---|---|---| -| Controller → Scheduler | `Data.jobArray` push | `JobSubmit` (base jobid, array_size, one spec) | New variant; sent via direct `xce_request()` (not `xce_push_data_req()` — see Dependencies note above) | -| Controller → API server | `Data.jobArray` push | `JobSubmit` (base jobid, array_size, one spec) | Same variant, same direct-`xce_request()` transport, per connected API server (mirrors existing fan-out) | - -**Implementation Notes** -- Integration: no new opcode — `Data`'s existing union discriminator (`table`) already routes to the new variant on both the scheduler and the API-server receive paths. -- Validation: confirm the scheduler's local expansion treats a partially-received/decode-failed array push the same way a malformed single-job push is handled today (existing error path, not new). -- Risks: the scheduler and API server must never assume `array_size == l->count` implies the array is contiguous starting at any index other than the pushed `jobid` — enforced by construction here since `base_jobid + i` is the only arithmetic either side uses. - -### jobj.c Hardening - -#### Reliable, leak-free construction under N-element reuse - -| Field | Detail | -|-------|--------| -| Intent | Close five specific, pre-existing gaps in job/job-submit construction that this feature's N-way apply-time loop amplifies by array size | -| Requirements | 10.1, 10.2, 10.3, 10.4, 10.5 | - -**Responsibilities & Constraints** -- **10.1** — `job_init()` (jobj.c:532) currently stores `makeJobsub()`'s return value into `dst->job.submit` unconditionally (jobj.c:543), including a NULL on allocation/interning failure, and still returns success. Fix: check the return, propagate failure. Same check for `makeString(src->ckptinfo)`. -- **10.2** — the previously observed `makeJobsub()` NULL-with-EINVAL-for-a-valid-`src`, traced to a likely `jobsub_table` (jobj.c:16) initialization-ordering issue: `job_obj_init()` must complete before the first `makeJob`/`makeJobsub` call on every path that can reach one, including this feature's new N-way `restore_job_array()` loop. Root cause to be confirmed and fixed at the actual ordering dependency, not papered over by 10.1's error propagation alone. -- **10.3** — `job_init()`'s `MOVE_PTR` fields (`pidinfo, allocs, accounts, run_usage, exit_info, pend_reasons`) are confirmed always NULL for a freshly-submitted job: `build_job_submit_payloads()` (job.c:229) and `restore_job_array()`'s equivalent per-element construction both `calloc()` a fresh `xce_job_t` per element and never set these six fields — repeated `MOVE_PTR` from a shared template is benign (NULL→NULL) and, for the array case specifically, each element gets its own independently-zeroed struct in the first place. No code change required; documented as a verified invariant. -- **10.4** — `job_destroy()`'s `accounts` leak (jobj.c:584, `/* FIXME??? */`) is fixed by adding the missing free call, following the existing pattern already used for `allocs`/`pidinfo`/`run_usage` in the same function. Array submission multiplies this leak by element count, making it a correctness requirement here, not a nice-to-have. -- **10.5** — verified: `struct Job`'s capnp schema (`controller.capnp:287`) has no `ckptinfo` field at all, so `decode_Job` never populates it and `free_Job` correctly never frees it — not a gap. Documented as a verified non-issue, no code change. - -**Dependencies** -- Inbound: `create_job()`/`makeJob()` (job.c, jobj.c) — every caller, including this feature's `restore_job_array()` (Criticality P0) - -**Contracts**: Service [x] - -##### Service Interface -```c -/* job_init's contract changes: propagates a construction failure instead of - * silently storing NULL and reporting success. */ -static int job_init(meta_obj_t* obj, va_list ap); /* now fails on - makeJobsub()/makeString(ckptinfo) returning NULL, not only on src==NULL */ -``` -- Preconditions: unchanged (`src != NULL`, `src->submit != NULL`). -- Postconditions: on success, `dst->job.submit` and `dst->job.ckptinfo` are never NULL-with-success; `job_obj_init()` has run before this call on every reachable path. -- Invariants: a `job_t` never exists in a half-initialized state visible to any caller. - -**Implementation Notes** -- Integration: `restore_job_array()`'s all-or-nothing scratch-discard (Array Apply Expansion component) is the safety net if 10.1's now-propagated failure occurs mid-array — it is treated exactly like any other `create_job()` failure in that loop, discarded from the scratch list, never inserted into `c->jobmgr`. -- Validation: `src/lib/testsuite/test_jobj_hardening.c` (File Structure Plan) exercises all five items directly, independent of the array-expansion path, since these are general `jobj.c` fixes the single-job path benefits from too (Requirement 8.1's note explicitly permits this). -- Risks: 10.2's root cause is not yet confirmed at design time — flagged as a task-level investigation, not a pre-decided fix, consistent with Design vs. Implementation (state the contract: "`makeJobsub` must never observe an uninitialized `jobsub_table`," not the exact ordering fix). **Boundary note**: this item's File Structure Plan entry is `src/lib/jobj.c` only, on the assumption the fix is local to `jobj.c`'s own init/call-ordering. If root-causing 10.2 shows the actual gap is elsewhere (e.g. controller boot sequencing, a test-harness call path that skips `job_obj_init()`), that is a signal to split 10.2 into its own task with its own explicit file scope at task-generation time, not to silently widen this component's boundary. - -## Error Handling - -### Error Strategy -- **Pre-commit validation failure** (Requirement 9.2): `handle_jobsubmit()` rejects before any WAL append, identical to today's single-job path — no new error class. -- **Commit failure/timeout for the array record**: the single registered continuation resolves failed; `jobsubmit_reply_failure_once()` fires (unchanged); no job ever constructed (Requirement 2.2, 2.3 by construction — nothing to roll back). -- **Post-commit content-derived apply failure**: structurally impossible per Requirement 9.3 (9.1's pre-commit validation already ran) — if it somehow occurred, it is a programming-invariant violation, handled the same way the Live Apply Resource Fence handles any negative `apply_committed_entry()` return: loud ERROR, fence, retry forever. No `assert()`/abort (`src/controller/CLAUDE.md` Process Lifecycle invariant). -- **Post-commit resource-exhaustion apply failure**: `restore_job_array()` rolls back this attempt's partial construction; `persist_apply_pump_run()` halts the walk at this LSN; Live Apply Resource Fence engages; retried on the next apply-pump kick; self-heals on success (Requirement 9.4, 9.5). -- **`jobj.c` construction failure** (post-hardening): propagates as an ordinary `create_job()` failure into the same rollback path above — no new error class, just a previously-silent failure now made visible. - -### Monitoring -- Loud `ERROR()` on Live Apply Resource Fence engagement, naming the stuck LSN and reason (mirrors `persist_recover_loud_fence_if_unrecoverable`'s existing log shape). -- `INFO()` on fence self-heal (LSN, retry count if tracked). -- One `INFO()` confirmation per array from the scheduler and one from the API server on successful local expansion (base jobid + element count), the log-based verification Requirement 8.4/6.4 rely on. - -## Testing Strategy - -- **Unit Tests**: - - `persist_submit_job_array_async()` appends exactly one WAL entry and registers exactly one continuation for `array_size > 1` (Requirement 1.1-1.3). - - `restore_job_array()`: N-way expansion produces `base_jobid..base_jobid+array_size-1`; idempotent re-apply (all-existing) is a no-op returning `-EEXIST`; a partial-existing edge case skips existing indices without duplicating (Requirement 2, 3.3, 5.1). - - Simulated construction failure at element `k` of `N`, driven via `restore_job_array()`'s `XCE_TESTING`-guarded injection seam (forces a construction failure at a chosen index): elements `0..k-1` exist only in the discarded scratch list, never in `c->jobmgr` (nothing to tear down there); a subsequent successful retry builds a fresh scratch list from index 0 and bulk-inserts once, producing the full, non-duplicated set (Requirement 9.4, fault-injection scenario for 9.6). - - `apply_committed_entry()` vs. `apply_wal_entry()` applied to the same committed array record produce byte-for-byte identical `c->jobmgr` state (Requirement 3.1, 3.2). - - `persist_apply_pump_run()`: a negative `apply_committed_entry()` return halts the walk without advancing `last_applied_lsn`; a subsequent success advances it and clears the Live Apply Resource Fence (Requirement 9.4, 9.5). - - Promotion-gate refusal while the Live Apply Resource Fence is set, at both existing `ha.c` call sites (Requirement 9.4). - - `jobj.c`: all five Requirement 10 items, independent of array submission. -- **Integration Tests**: - - End-to-end array submit: commit success → all N elements visible and individually addressable (status/cancel/completion) via existing per-element APIs, unchanged (Requirement 5.2, 5.3). - - Single push received by both the scheduler and API server, each expanding N local entries and logging one confirmation (Requirement 6.1-6.5). - - Non-array (`array_size == 1`) submission: byte-identical WAL record, identical commit-barrier count, identical reply, before and after this feature (Requirement 8.1). -- **Regression / Live Soak**: - - `/xce-test-ha` extended with a large-array submission scenario asserting WAL/commit-barrier count (one record/one barrier, not N) and failover replay time for the committed array (Requirement 8.3, 4.3). - - A client retrying an identical array submission across a primary failover follows the same cross-primary dedup mechanism as single-job submission (bug-1419), extended to the array's full index range as one unit — asserted not to double-submit or partially duplicate (Requirement 8.3). - - Full sequential and chaotic `/xce-test-ha` soak, including array submission immediately before killing the primary, shows no new job loss, divergence, or liveness stall (Requirement 8.2). - - Non-array (`array_size == 1`) submission WAL format, commit-barrier count, and reply are byte-identical before and after this feature (Requirement 8.1). - -### Performance & Scalability -- WAL/replication cost for an array submission becomes O(1) in `array_size` (one record, one barrier, one replicated entry) versus today's O(N) — the target metric is commit-engine load and replication bandwidth for a large-array submission, measured via the `/xce-test-ha` large-array scenario. -- Apply-time cost (constructing N `job_t` entries inside one `apply_committed_entry()` call) is O(N) CPU inside a single, strictly-ordered apply-pump step — an accepted, bounded cost (N is hard-capped at 32767 by `idgen()`), not a new unbounded-latency risk; noted here as a deliberate trade-off, not re-litigated as a chunking requirement (see Requirement 7's Architecture-table resolution). -- Failover replay time for a committed array is governed by the size of the one array record (O(1)) rather than a per-element replay cost proportional to array size, since replay decodes and expands through the same `restore_job_array()` primitive as live apply (Requirement 4.3) — measured via the `/xce-test-ha` large-array scenario's failover-replay-time assertion. diff --git a/.kiro/specs/job-array-single-wal-record/requirements.md b/.kiro/specs/job-array-single-wal-record/requirements.md deleted file mode 100644 index 4536fe44..00000000 --- a/.kiro/specs/job-array-single-wal-record/requirements.md +++ /dev/null @@ -1,125 +0,0 @@ -# Requirements Document - -## Introduction - -A job-array submission today fans out into one Raft log record per array element: `job_submit()` loops over every element calling `create_job()`, and the persistence layer commits each resulting job through its own independent majority-commit barrier. A 10,000-element array therefore produces 10,000 WAL appends, 10,000 sequential majority-commit round trips, and 10,000 scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time linearly with array size, while holding the submitting thread across every per-element barrier in sequence. - -This feature makes a job-array submission persist, replicate, and push as a single atomic unit: one WAL record describing the whole array (base job specification plus index range/count — every element shares the base spec, distinguished only by jobid), committed through exactly one commit barrier. Standbys reconstruct the entire array from that one committed log position. The scheduler receives the array as a single push and expands it locally. After expansion, every array element retains its existing per-element identity and is addressable exactly as it is today for status, cancellation, and completion. - -This feature builds on the apply seam introduced by `raft-apply-decoupling` (now complete): the single array record is expanded into individual in-memory job entries inside `apply_committed_entry()`, the same seam that already handles single-job submission, so online apply and restart-replay expand identically with no divergence between the two paths. - -> Full discovery context (problem, current state, desired outcome, approach, boundary, upstream/downstream) is in `.kiro/specs/job-array-single-wal-record/brief.md`. - -## Boundary Context - -- **In scope**: a new WAL record format for a job-array submission (base spec + index range/count — every element shares the base spec, distinguished only by jobid); atomic single-barrier commit of that one record; standby reconstruction of the full array from one committed LSN; the expand-on-apply step shared by online apply and restart replay; a single push carrying the full array to each `HOOK_DETAIL_JOB_NEW` subscriber — scheduler (with local scheduler-side expansion) and API server; preservation of per-element identity and state addressing after expansion; idempotent re-application of a committed array record (replay or retry must not double-create elements); a bounded record size or chunking policy so a pathologically large array does not produce a single oversized WAL entry; log-based confirmation on the scheduler and API server that all array elements were received, for harness verification; hardening `src/lib/jobj.c`'s job/job-submit construction primitives (`job_init`, `makeJobsub`) so they are reliable and leak-free under N-element array construction, since Requirement 9's deterministic-apply guarantee depends on them. -- **Out of scope**: the apply-ordering/hook-decoupling refactor itself (delivered by `raft-apply-decoupling`, consumed here unchanged) — `persist_job_hook_fun`'s `WAL_JOB_SUBMIT` branch is already a no-op (task 3.1/6.1: commit is invoked explicitly by `handle_jobsubmit`, never from this hook, for either single-job or today's per-element array submission); this spec replaces the explicit per-element commit loop (`submit_job_wal_entry_and_commit`, called once per array element today) with one array-shaped commit call — the hook plays no role in submission before or after this change, and stays untouched for its other live purpose (job-state facts originating from `job_ctrl`'s direct synchronous path); any change to the non-array single-job submission path; scheduler dispatch/allocation policy; per-element override semantics beyond what single-job submission already supports; the commit engine's or commit-barrier's internal mechanics (consumed unchanged from `ha-commit-engine` / `ha-commit-barrier-rollout`); bulk query/cancel of a whole array as one client operation — deferred to a future spec because array elements can be in heterogeneous states at request time (done/exited elements need no action, running elements need a worker kill signal, pending elements need a direct state transition without ever running), which is a materially different bulk-operation problem from atomic array submission and deserves its own requirements rather than scope creep here. -- **Adjacent expectations**: depends on `raft-apply-decoupling`'s `apply_committed_entry()` seam existing and being the single point where a committed WAL entry becomes in-memory state (now complete); depends on `ha-commit-engine` / `ha-commit-barrier-rollout` for one-record atomic commit and idempotency guards, consumed unchanged; the scheduler-side push path is the one `scheduler-controller-msg-gap` already established, extended here to carry an array descriptor instead of only single-job pushes. - -## Requirements - -### Requirement 1: Single WAL record for a job-array submission - -**Objective:** As a cluster operator submitting a large job array, I want the whole array persisted as one Raft log record, so that submission cost no longer scales linearly with array size. - -#### Acceptance Criteria -1. When a job-array submission is accepted, the Controller shall encode the entire array — base job specification plus index range/count — into a single WAL record. Every array element shares the same base job specification; the only per-element distinction is its jobid (base id + index). -2. The Controller shall commit that single WAL record through exactly one majority-commit barrier, regardless of the array's element count. -3. The Controller shall not append a separate WAL record per array element for a job-array submission. - -### Requirement 2: Atomic, all-or-nothing commit - -**Objective:** As a cluster operator, I want a job-array submission to commit as a single unit, so that a standby can never observe a partial array. - -#### Acceptance Criteria -1. When a single array WAL record is committed on a majority, the Controller shall consider every element described by that record committed as one unit — there shall be no state in which some elements of that record are committed and others are not. (When Requirement 7's chunking policy splits a pathologically large array into multiple records, this atomicity guarantee applies per committed record/chunk, not across the whole array — Requirement 7.2 defines the atomicity unit in that case.) -2. If an array WAL record fails to reach majority commit, then the Controller shall treat that record's elements as not committed and shall not construct any array elements from it. -3. A standby reconstructing committed state shall never observe a single array WAL record in a partially-applied form. - -### Requirement 3: Shared expand-on-apply for online and replay - -**Objective:** As a maintainer, I want array expansion to happen in one shared code path, so that a live apply and a restart replay can never diverge in how they expand the same committed array record. - -#### Acceptance Criteria -1. When a committed job-array WAL record is applied — whether during live/online apply or during restart replay — the Controller shall expand it into individual job entries through the same `apply_committed_entry()` seam used for single-job submission. -2. The Controller shall use identical expansion logic for the online-apply path and the restart-replay path, so that replaying a committed array record produces byte-for-byte the same set of job elements as the original online apply produced. -3. When re-applying an already-applied array record (e.g. during replay after a restart, or a retried commit), the Controller shall not create duplicate job elements for any array index. -4. The array-expand step shall fire `HOOK_DETAIL_JOB_NEW` exactly once per committed array record, carrying all of that record's constructed job entries in one list — not once per element. (This is what Requirement 6's single-push guarantee to the scheduler and API server rests on — both are subscribers of this same hook fire.) - -### Requirement 4: Standby reconstruction from one committed position - -**Objective:** As a cluster operator relying on HA failover, I want a standby to reconstruct a whole job array from one committed log position, so that replication bandwidth and failover replay time no longer scale linearly with array size. - -#### Acceptance Criteria -1. When a standby receives and applies the array's single committed WAL record, the standby shall reconstruct the complete set of array elements from that one record. -2. Replication of a job-array submission shall carry the single array WAL record, not one record per element. -3. Failover replay time for a committed job array shall be governed by the size of the one array record, not by a per-element replay cost proportional to array size. - -### Requirement 5: Preserved per-element identity and addressing - -**Objective:** As a client of the job API, I want every array element to remain individually addressable after expansion, so that existing status, cancellation, and completion behavior is unaffected by this change. - -#### Acceptance Criteria -1. After a job-array WAL record is applied, each resulting job element shall have the same per-element identity scheme (base id + index) that single-per-element WAL records produce today. -2. The Controller shall support querying, cancelling, and completing an individual array element by its per-element identity exactly as it does for job elements created via the current per-element path. -3. This feature shall not change any client-visible identity, status, or addressing behavior for an individual array element. - -### Requirement 6: Single push to scheduler and API server, with local expansion - -**Objective:** As a maintainer of scheduler and API-server load, I want both the scheduler and the API server to receive one push per committed array, so that push volume to either consumer no longer scales linearly with array size. - -**Context:** `HOOK_DETAIL_JOB_NEW` (`HOOK_AFTER_EVENT`) is a single hook fired once per constructed job entry, with two independent subscribers: `sched.c` (`xcesched_push_joblist`) and `apiserver.c` (`apiserver_push_joblist`). Today both fire once per array element, N times per array. Batching the hook fire to once per array (Requirement 3's shared apply step) fixes both consumers by construction — this requirement holds both to the same bar explicitly, so a fix that batches only the scheduler side and leaves the API server side unbatched does not satisfy it. - -**Wire shape note:** "single push" means the push carries the array descriptor (base spec + index range/count + jobid basis) — the same compact shape as the WAL record (Requirement 1) — not N individually-encoded full job structs bundled into one message. Encoding N full duplicate structs would satisfy "one push" by message count while silently reintroducing N-scaling in payload size, since the wire encoder does not know two `Text` fields share the same interned source and will encode each independently. Each receiver decodes one base spec and expands it locally into N entries (Requirement 6.3 / Requirement 6.5), so sharing happens by construction on the receiving side too, with no separate interning step needed there. - -#### Acceptance Criteria -1. When a single array WAL record commits, the Controller shall deliver it to the scheduler as a single push carrying that record's array descriptor (base spec + index range/count), not N individually-encoded job structs. (When Requirement 7's chunking policy applies, this is one push per committed record/chunk, not one push for the whole array — mirroring Requirement 3.4's per-record hook fire.) -2. When a single array WAL record commits, the Controller shall deliver it to the API server as a single push carrying that record's array descriptor (base spec + index range/count), not N individually-encoded job structs. (Same per-record scoping as 6.1 when chunking applies.) -3. The Scheduler shall expand the array descriptor into individual schedulable tasks locally, without requiring one push per element from the Controller. The scheduler shall treat each array element as an ordinary single job for scheduling purposes; scheduling/dispatch outcome for array elements is out of scope for this feature. -4. The Controller shall not send a separate scheduler push, nor a separate API-server push, per array element for a job-array submission. -5. The API server shall expand the array descriptor into individual per-element records locally (mirroring the scheduler's local expansion in 6.3), rather than the Controller sending or the API server requiring one full job struct per element on the wire. - -### Requirement 7: Bounded record size for very large arrays - -**Objective:** As a maintainer of WAL and replication capacity, I want an upper bound on a single array record's size, so that an extremely large array cannot produce a pathologically oversized WAL entry. - -#### Acceptance Criteria -1. The Controller shall define a maximum size or chunking policy for a job-array WAL record so that no single record exceeds a bounded size regardless of array element count. -2. If a job-array submission would exceed the bounded record size, then the Controller shall apply its chunking policy (e.g. multiple atomically-committed records) rather than produce one oversized record, while Requirement 2's atomicity guarantee continues to hold for each committed unit. - -### Requirement 8: No regression to non-array submission or existing tests - -**Objective:** As a maintainer, I want this feature scoped strictly to array submissions, so that ordinary single-job submission and existing HA guarantees are unaffected. - -#### Acceptance Criteria -1. The Controller shall leave the non-array, single-job submission WAL record format and commit-barrier count unchanged by this feature. (This does not prohibit Requirement 10's fixes to `jobj.c`'s shared construction primitives — `job_init`/`makeJobsub` are used by single-job construction too, and Requirement 10's bugfixes are expected to also improve, not regress, that path; "unchanged" here means the WAL wire format and commit-barrier behavior, not a blanket freeze on shared-library code this feature must touch for Requirement 9's correctness.) -2. A full HA soak (sequential and chaotic `/xce-test-ha` regression), including job submission and failover, shall show no new job loss, divergence, or liveness stall attributable to this feature. -3. Regression tests shall cover: WAL/commit-barrier count for a large-array submission (one record/one barrier, not N), failover replay time for a committed array, correct per-element status/cancel/completion after array expansion, single-push delivery to both the scheduler and the API server, and a client retrying an identical array submission across a primary failover — the retry shall follow the same cross-primary dedup mechanism as single-job submission (bug-1419), extended to the array's full index range as one unit, and shall not double-submit or partially-duplicate the array. -4. The Scheduler and the API server shall each log a single, identifiable confirmation that all N elements of a committed array were received and handled correctly (e.g. base jobid + element count), so that push-completeness for the whole array can be verified by log inspection without a new query API. - -### Requirement 9: Deterministic, total apply — content failures rejected pre-commit, resource failures fenced not rejected - -**Objective:** As a maintainer of cluster correctness, I want array expansion at apply time to never fail for a reason the commit already promised was safe, so that a committed array can never be un-appliable and no node can silently diverge from its peers. - -**Context:** once an entry commits, Raft guarantees it is eventually applied on every correct server — but the apply function itself must be deterministic (every replica computes the same outcome from the same entry). Two distinct failure classes require two distinct handlings: (A) content-derived failures (jobid-range collision, malformed spec) are computable identically by every replica from the entry's own bytes plus prior state — these must be caught before the entry is ever appended/committed, never left to fail at apply time. (B) node-local resource exhaustion (e.g. OOM while constructing element 7000 of 10000) cannot be predicted at submit time and must not be allowed to vary apply's outcome across replicas — it is not a commit failure and must never cause element rejection or divergence. - -#### Acceptance Criteria -1. Before appending a job-array WAL record, the Controller shall validate everything that could deterministically prevent successful expansion of every element (jobid-range collision with existing or in-flight jobs, base-spec validity) — reusing the existing in-flight jobid-range claim mechanism (bug-1419), extended to cover the array's full index range in one check. -2. If pre-commit validation fails, the Controller shall reject the submission before the WAL append — the array shall never be committed and shall never reach the apply stage. -3. Once a job-array WAL record is committed, expansion at apply time shall not fail for any content-derived reason — pre-commit validation (9.1) shall make apply total for this class. -4. If a node encounters a resource-exhaustion failure (e.g. allocation failure) while expanding a committed array record, that node shall not reject the array, shall not construct a partial/divergent set of elements, and shall not treat this as a reason to fail the commit. The node shall log the condition loudly, fence itself as behind on that LSN (non-promotable while fenced, per the existing `apply_divergent`/fence pattern), and keep retrying the apply until it succeeds — consistent with the "controller never self-terminates" invariant. -5. A resource-exhaustion apply failure on one node shall not block other nodes that can successfully apply the entry from proceeding; the fenced node shall self-heal (clear its fence) once the apply succeeds. -6. A fault-injection test shall force a resource-exhaustion failure (e.g. an injected allocation failure) during array expansion on one node and verify: the node fences itself as behind on that LSN, constructs no partial or divergent set of elements, other nodes proceed unaffected, and the fenced node self-heals (clears its fence, completes the apply) once the injected failure condition clears. - -### Requirement 10: jobj.c hardening — the shared construction primitive must be reliable under array-scale reuse - -**Objective:** As a maintainer of cluster correctness, I want `src/lib/jobj.c`'s job/job-submit construction primitives to be reliable and leak-free under N-element array construction, so that Requirement 9's "deterministic, total apply" guarantee rests on a sound foundation rather than a primitive with known gaps. - -**Context:** `job_init` (jobj.c) already interns/shares `job_submit_data_t` across identical submissions via `makeJobsub()` (hash+refcount) — for N uniform array elements this collapses to one real allocation plus N cheap refcount bumps, which is the right foundation for Requirement 9. But inspection of the current implementation found gaps that predate this spec and become higher-stakes at array scale. - -#### Acceptance Criteria -1. `job_init` shall check `makeJobsub()`'s return value; on NULL (allocation failure or interning-table error), `job_init` shall fail cleanly (propagate an error) rather than store a NULL `job.submit` and report success. The same check shall apply to `makeString(src->ckptinfo)`. -2. The root cause of the previously observed `makeJobsub()` NULL-with-EINVAL failure (returned even for a valid non-NULL `src`, traced to a likely `jobsub_table` initialization-ordering issue) shall be identified and fixed, not merely papered over by 10.1's error propagation. -3. The array-expansion construction path shall not rely on a pattern that triggers `job_init`'s destructive `MOVE_PTR` fields (`pidinfo, allocs, accounts, run_usage, exit_info, pend_reasons`) more than once against a shared source struct. Either confirm these fields are always NULL for a freshly-submitted (not-yet-started) job — making repeated MOVE_PTR from a shared template benign — or construct each array element from its own independent source struct. -4. The `job_destroy` leak on `accounts` (marked `/* FIXME??? */`, never freed) shall be fixed. Array submission multiplies this leak by element count; it shall not remain unfixed while this spec optimizes everything else for array scale. -5. Whether `decode_Job` ever populates `ckptinfo` for a freshly-submitted job shall be verified; if it does, `free_Job`'s coverage shall be extended to include it (currently undocumented/unclear whether it is freed). diff --git a/.kiro/specs/job-array-single-wal-record/research.md b/.kiro/specs/job-array-single-wal-record/research.md deleted file mode 100644 index 837db4e1..00000000 --- a/.kiro/specs/job-array-single-wal-record/research.md +++ /dev/null @@ -1,172 +0,0 @@ -# Research & Design Decisions: job-array-single-wal-record - ---- -**Purpose**: Capture discovery findings, architectural investigations, and rationale that inform `design.md`. ---- - -## Summary -- **Feature**: `job-array-single-wal-record` -- **Discovery Scope**: Extension (existing system) — the controller's job-submit and apply-seam machinery already exists (`raft-apply-decoupling`, complete); this feature adds an array-shaped WAL record and expansion step on top of it. -- **Key Findings**: - - The brief's premise ("`job_submit()` loops calling `create_job()` per element") is stale. `raft-apply-decoupling` task 3.2 already removed that loop: `handle_jobsubmit()` today builds N lightweight `xce_job_t` WAL payloads (`build_job_submit_payloads`, job.c:221) and appends N separate `WAL_JOB_SUBMIT` entries via `persist_submit_job_payloads_async()` (persist.c:4846), each with its own commit continuation, batched into **one** client reply (`jobsubmit_async_ctx_t`, job.c:754). `create_job()` itself now runs only inside the apply seam (`restore_job()`, job.c:271). This feature's real job is collapsing those N WAL appends into one, not removing a `create_job()` loop that no longer exists. - - The client-facing `JobSubmit` capnp struct (`controller.capnp:533`) already carries exactly `(jobid, arraySize, submitData)` — the compact shape Requirement 1 needs for the new WAL record. No new capnp struct is needed for encoding; a new `wal_entry_type` (`WAL_JOB_ARRAY_SUBMIT`) is needed so the existing single-job `WAL_JOB_SUBMIT`/`struct Job` format stays byte-for-byte unchanged (Requirement 8.1). - - Because array elements are uniform clones (locked earlier in requirements discovery — no per-index overrides), the new record's encoded size is **O(1)** in array size: one `JobSubmitData` regardless of whether `array_size` is 2 or 32767. Combined with the pre-existing hard cap `arraySize <= MAX_ARRAY_IDX + 1` (32768) enforced at jobid-minting time (`idgen()`, idgen.c:31, called from the Go apiserver before a request ever reaches the controller), Requirement 7's "bounded record size" is satisfied by construction — no chunking policy needs to be built. - - `apply_committed_entry()` (persist.c:2721) is not the *only* live dispatcher of `WAL_JOB_SUBMIT`. `apply_wal_entry()` (persist.c:2574) is a second, hook-free dispatcher that remains the live standby catch-up/replicated-apply path (confirmed by an explicit comment at persist.c:3722-3725: "`apply_wal_entry()` ... is NOT retired by this change; it remains the live standby catch-up/replicated-apply dispatch used elsewhere (ha.c)"). Today's codebase avoids divergence between these two dispatchers by sharing one primitive, `restore_job()`, called from both switch statements. This feature must follow the same pattern with a new `restore_job_array()` shared primitive, not add expansion logic to only one dispatcher. - - The scheduler/apiserver push infrastructure already accepts an N-element `xlist_t` in one hook fire (`sched_job_new_fun`/`apiserver_job_new_fun` unconditionally call `xcesched_push_joblist`/`apiserver_push_joblist` with whatever list they receive) — so "one push call" is already achievable for free. But Requirement 6's wire-shape note explicitly demands more: the wire payload itself must stay compact (a descriptor), not N individually-encoded `Job`/`SchedJob` structs, because the existing `List(Job)`/`List(SchedJob)` wire shape re-introduces O(N) payload size even inside a single RPC. This requires a genuinely new compact push message and a new discriminator (`xlist->count > 1`) in the two existing hook handlers, plus new local-expansion logic on the scheduler (`src/scheduler/src/handler.c:112`, `case Data_data_schedjobs:`) and the Go apiserver (`src/frontend/apiserver/backend/convert.go`, sibling to `cJobToModel`). - - `persist_apply_pump_run()` (persist.c:3244) currently **discards** `apply_committed_entry()`'s return value (`(void)apply_committed_entry(...)`, persist.c:3281) and unconditionally advances `last_applied_lsn`. A resource-exhaustion failure mid-array-expansion would, under today's code, be silently skipped forever — the array would never be constructed and never be retried. This is a genuine, load-bearing gap this feature must close (not a hypothetical): the pump must start checking the return value and gate `last_applied_lsn` advancement on it, mirroring the existing `apply_pump_read_entry` failure branch immediately above it. - - No existing promotion-gate condition considers `last_applied_lsn` vs `committed_lsn` at all (`ha.c` gates only on `truncated`, `durable_floor`, `apply_verdict == APPLY_DIVERGENT`, `self_restart_pending`). A node stuck retrying a resource-exhaustion apply failure would look fully promotable today. A new, narrowly-scoped live fence is required — deliberately **not** reusing `recovery_unrecoverable` directly, because that flag's existing periodic auto-clear (`ha.c:1069-1071`, gated on WAL *durability* catch-up, not *apply* progress) would wrongly clear a fence whose underlying condition (stuck apply) is unrelated to durability and has not actually resolved. - - `jobj.c`'s `job_init` (jobj.c:532) does not check `makeJobsub()`'s return value before storing it (jobj.c:543) — a real, pre-existing gap, N×-amplified by this feature's tight per-array `create_job()` loop. `job_destroy`'s `accounts` leak (jobj.c:584, `/* FIXME??? */`) is likewise pre-existing and N×-amplified. Both are already in scope via Requirement 10. - -## Research Log - -### Job-array submission path: brief vs. live code -- **Context**: the brief (written before `raft-apply-decoupling` fully landed) describes a `for` loop over `create_job()` in `job_submit()`. Needed to confirm this is still accurate before designing an insertion point. -- **Sources Consulted**: `src/controller/job.c` (live read, lines 196-1076), git history reference in code comments (commit `6f1fa97`). -- **Findings**: The loop was removed by task 3.2. Current path: `handle_jobsubmit()` → `build_job_submit_payloads()` (builds N plain `xce_job_t` payload structs, no RTTI objects) → `persist_submit_job_payloads_async()` (N WAL appends, N commit continuations sharing one `jobsubmit_async_ctx_t`) → each continuation hands off to `persist_register_pending_apply()` → the apply pump's `apply_committed_entry()` constructs the actual `job_t` via `restore_job()`/`create_job()`, one per committed entry. -- **Implications**: The N-scaling this feature must eliminate is in WAL appends/commit continuations (job.c/persist.c), not in a `create_job()` loop inside `handle_jobsubmit` — that loop is already gone. Design must replace `build_job_submit_payloads` + `persist_submit_job_payloads_async`'s per-element loop with a single array-shaped append, and add the N-way expansion loop to the apply seam instead (where `create_job()` already lives today). - -### WAL wire format and capnp schema -- **Context**: Requirement 1 needs "base spec + index range/count" as one record; needed to know whether new capnp schema is required. -- **Sources Consulted**: `src/schema/controller.capnp` (`struct Job` line 287, `struct JobSubmit` line 533), `src/include/wal_types.h`. -- **Findings**: `struct JobSubmit $C.mapname("xce_job_submit_t")` already has exactly `(jobid, arraySize, submit: JobSubmitData)` — the client request format. `struct Job` (the current WAL_JOB_SUBMIT payload) has no array fields and is single-job only. -- **Implications**: Reuse `struct JobSubmit`'s existing capnp encoder (`encode_JobSubmit`, already generated) as the new WAL_JOB_ARRAY_SUBMIT payload verbatim — zero schema changes for the WAL format itself. Gate on a new `wal_entry_type` value so the single-job `WAL_JOB_SUBMIT`/`struct Job` format is untouched (Requirement 8.1). - -### Record size bound (Requirement 7) -- **Context**: needed to determine whether a chunking policy must be built. -- **Sources Consulted**: `src/include/config.h` (`N_ARRAY_IDX_BITS`, `MAX_ARRAY_IDX`), `src/lib/idgen.c` (`idgen()`), `src/frontend/apiserver/backend/clib.go` (`GenerateJobID`). -- **Findings**: `idgen()` rejects `arraySize > MAX_ARRAY_IDX + 1` (32768) at jobid-minting time, called from the Go apiserver before any submission reaches the controller (clib.go:165) — a request whose `array_size` exceeds this cap never gets a jobid and never reaches `handle_jobsubmit`. Combined with the record's O(1) payload (one `JobSubmitData`, not per-element), the WAL_JOB_ARRAY_SUBMIT record's byte size is already bounded and does not grow with `array_size` at all. -- **Implications**: No new chunking mechanism is needed. Requirement 7.1/7.2 are satisfied by (a) reusing the compact `JobSubmit` shape (Requirement 1's own design decision) and (b) citing the pre-existing `MAX_ARRAY_IDX` cap as the hard upper bound on element count. Building "multiple atomically-committed chunks" machinery for a scenario that cannot occur would be scope creep (design-synthesis Simplification lens). - -### Apply-seam dispatch: two live dispatchers, not one -- **Context**: Requirement 3 demands one shared expansion path for online apply and replay; needed to confirm `apply_committed_entry()` is really the only live dispatcher. -- **Sources Consulted**: `src/controller/persist.c` (`apply_wal_entry`, line 2574; `apply_committed_entry`, line 2721; `persist_recover`'s replay loop, line ~3742), `.kiro/specs/raft-apply-decoupling/design.md`. -- **Findings**: `apply_wal_entry()` was NOT retired by `raft-apply-decoupling` — an explicit in-code comment (persist.c:3722-3725) confirms it "remains the live standby catch-up/replicated-apply dispatch used elsewhere (ha.c)". `persist_recover()`'s own replay loop was switched to call `apply_committed_entry()` (persist.c:3742), but `apply_wal_entry()` is a second, separate switch statement over the same WAL types, still reachable from `ha.c` for standby catch-up. Both dispatchers avoid divergence today only because they both delegate the actual single-job construction to one shared helper, `restore_job()`. -- **Implications**: the array-expansion loop must live in one new shared helper (`restore_job_array()`, sibling to `restore_job()`), called from a new `WAL_JOB_ARRAY_SUBMIT` case added to **both** `apply_committed_entry()` and `apply_wal_entry()`'s switch statements — matching the codebase's existing (not the aspirational) architecture. Recorded as a correction to Requirement 3's framing: the shared seam is the helper function, not either dispatcher alone. - -### Live apply-pump failure handling gap -- **Context**: Requirement 9.4/9.5 need "keep retrying the apply until it succeeds" and "fence non-promotable" for a resource-exhaustion failure during array expansion. -- **Sources Consulted**: `src/controller/persist.c` (`persist_apply_pump_run`, line 3244), `src/controller/ha.c` (promotion-gate call sites, lines ~1069 and ~1344). -- **Findings**: `persist_apply_pump_run()`'s dispatch loop discards `apply_committed_entry()`'s return value (`(void)apply_committed_entry(...)`, persist.c:3281) and unconditionally advances `last_applied_lsn`. The read-failure branch immediately above it (`apply_pump_read_entry` failing) DOES stop the walk and leave `last_applied_lsn` unadvanced for retry — but nothing analogous exists for an `apply_committed_entry()`-internal failure. Separately, `ha.c`'s promotion-eligibility gate never inspects `last_applied_lsn` vs `committed_lsn` at all; it gates on WAL durability/truncation/apply-divergence/self-restart-pending only. -- **Implications**: two concrete, necessary fixes, both required for Requirement 9 to hold, neither optional: (1) the pump must check `apply_committed_entry()`'s return value and mirror the existing read-failure branch (stop, don't advance, retried next kick) on a negative result; (2) a new, narrow live fence must gate promotion on "no LSN is currently stuck failing to apply" — deliberately not reusing `recovery_unrecoverable` (see next entry). - -### Why not reuse `recovery_unrecoverable` for the live apply fence -- **Context**: `recovery_unrecoverable` + `persist_recover_loud_fence_if_unrecoverable()` already implement the exact "loud, non-promotable, stay alive, keep retrying, self-heal" shape Requirement 9.4/9.5 wants — first instinct was to reuse it directly. -- **Sources Consulted**: `src/controller/ha.c` (both promotion-gate call sites), `src/include/persist.h` (`recovery_unrecoverable` field doc comment, `APPLY_DIVERGENT` handling). -- **Findings**: `recovery_unrecoverable` already has two distinct existing consumers with different clear semantics: (a) a periodic/timer-driven clear (`ha.c:1069-1071`) gated purely on WAL *durability* catch-up (`durable_floor >= target_hwm`), unrelated to apply progress; (b) `apply_verdict == APPLY_DIVERGENT`, a **separate** field, explicitly documented as "only ever recomputed by a fresh boot-time `persist_recover()` replay" and never live-cleared. If the new live apply-resource fence reused `recovery_unrecoverable` directly, condition (a)'s periodic clear would fire as soon as WAL durability caught up — which happens independently of whether the stuck array LSN has actually finished applying — silently un-fencing a node that is still failing to construct the array. -- **Implications**: introduce a new, narrowly-scoped field (design.md calls it the Live Apply Resource Fence) instead of overloading `recovery_unrecoverable`. Set by the pump on an `apply_committed_entry()` failure; cleared by the pump itself the moment a retry of that same LSN succeeds — no dependency on the election/promotion-gate call sites ever running, so a long-lived primary that never loses leadership still self-heals (Requirement 9.5). - -### Scheduler/apiserver compact push wire shape -- **Context**: Requirement 6's wire-shape note explicitly rules out "N individually-encoded full job structs bundled into one message" — needed to confirm today's push path does exactly that despite already being list-shaped. -- **Sources Consulted**: `src/controller/sched.c` (`sched_job_new_fun` line 271, `xcesched_push_joblist` line 14), `src/controller/apiserver.c` (`apiserver_job_new_fun` line 127, `apiserver_push_joblist` line 97), `src/schema/controller.capnp` (`struct Data` union, line 355; `SchedJobData`/`JobData`, lines 306-323), `src/scheduler/src/handler.c` (`case Data_data_schedjobs:`, line 112), `src/frontend/apiserver/backend/convert.go` (`cJobToModel`, line 31). -- **Findings**: both `sched_job_new_fun` and `apiserver_job_new_fun` unconditionally call their respective `..._push_joblist(c, l)` regardless of `l->count` — for an N-element list (which the new array apply case will produce) this would encode N full `Job`/`SchedJob` capnp structs on the wire, reintroducing exactly the O(N) payload growth Requirement 6 rules out. `struct Data`'s `data :union $C.mapuniontag("table")` (controller.capnp:356) is an additive discriminated union already carrying 6 variants (`schedjobs`, `schedworkers`, `jobs`, `workers`, `resources`, `resvals`) — adding one more compact variant is the established, low-risk extension pattern (matches `scheduler-controller-msg-gap`'s own precedent). -- **Implications**: `sched_job_new_fun`/`apiserver_job_new_fun` need a branch on `l->count > 1` (the only way `apply_committed_entry`'s `HOOK_DETAIL_JOB_NEW` fire ever produces a multi-element list, per Requirement 3.4) that pushes a new compact `Data` union variant (reusing `JobSubmit`'s shape again — base jobid + array_size + one `JobSubmitData`) instead of the existing per-job list. The scheduler and Go apiserver each need one new local-expansion routine, mirroring the controller's own `base_jobid + i` arithmetic. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| New `WAL_JOB_ARRAY_SUBMIT` type + reused `JobSubmit` capnp shape | Add one new `wal_entry_type`, reuse the existing `JobSubmit` struct as its payload | Zero capnp schema change; keeps `WAL_JOB_SUBMIT`/`struct Job` untouched (Req 8.1) by construction; symmetric with client wire format | New type must be added to two live dispatch switches (`apply_committed_entry`, `apply_wal_entry`) and to `wal_classify` | Selected | -| Overload `WAL_JOB_SUBMIT` with a variable-shape payload (array flag + optional range) | Single WAL type handles both single-job and array | Fewer enum values | Directly violates Requirement 8.1 ("leave the non-array WAL record format... unchanged"); forces every existing single-job decode site to branch defensively | Rejected | -| Chunk large arrays into multiple committed records | Bounds record size explicitly via a chunking policy | Handles unbounded `array_size` | Unnecessary: `array_size` is already hard-capped at 32768 by `idgen()`, and the record is O(1) regardless of count — chunking machinery would have no scenario to ever trigger | Rejected (Requirement 7 satisfied by construction instead) | - -## Design Decisions - -### Decision: Reuse `JobSubmit` as the WAL_JOB_ARRAY_SUBMIT payload and the compact push payload -- **Context**: Requirement 1 needs a compact WAL shape; Requirement 6 needs a compact push shape "the same compact shape as the WAL record." -- **Alternatives Considered**: - 1. Define a new `JobArraySubmit` capnp struct duplicating `JobSubmit`'s fields. - 2. Reuse `struct JobSubmit` verbatim for both the WAL payload and the push payload. -- **Selected Approach**: (2) — one capnp struct, three uses (client request, WAL payload, push payload), one encoder/decoder pair already generated. -- **Rationale**: `JobSubmit` already has exactly the three fields every one of these three uses needs (`jobid` as base, `arraySize`, one `submit: JobSubmitData`); requirements.md's own wire-shape note for Requirement 6 explicitly asks for "the same compact shape as the WAL record," making this the requirement-mandated answer, not just a simplification. -- **Trade-offs**: none identified — no functionality is given up by reuse here. -- **Follow-up**: confirm at implementation time that `encode_JobSubmit`/`decode_JobSubmit` (capnp-generated) round-trip cleanly when `submit` is a shared/interned pointer across all three use sites. - -### Decision: Array expansion lives in a new shared helper, called from both live dispatchers -- **Context**: `apply_wal_entry()` (standby catch-up) and `apply_committed_entry()` (primary live apply + boot replay) are both live and both currently delegate single-job construction to `restore_job()`. -- **Alternatives Considered**: - 1. Add the expansion loop only inside `apply_committed_entry()`, leaving `apply_wal_entry()` unchanged. - 2. Add a new shared `restore_job_array()` helper, called from a new case in both switches. -- **Selected Approach**: (2). -- **Rationale**: Option (1) would silently leave standby catch-up unable to expand an array record, reintroducing exactly the online/replay divergence Requirement 3 exists to prevent — for the array case specifically, on day one. -- **Trade-offs**: one more call site to keep in sync (mitigated: both call sites do nothing but forward to the shared helper, so there is no logic to keep in sync, only a switch-case line). -- **Follow-up**: none — this mirrors `restore_job()`'s existing, already-proven sharing pattern exactly. - -### Decision: A new, narrow live apply-resource fence, not a reuse of `recovery_unrecoverable` -- **Context**: Requirement 9.4/9.5 need a live, self-healing, non-promotable fence for a resource-exhaustion apply failure. -- **Alternatives Considered**: - 1. Set `recovery_unrecoverable` directly from the live apply-pump failure path. - 2. Introduce a new, dedicated field with its own set/clear triggers. -- **Selected Approach**: (2). -- **Rationale**: `recovery_unrecoverable`'s existing periodic clear (ha.c:1069-1071) is gated on WAL durability catch-up, a condition orthogonal to whether the stuck array LSN has actually finished applying — reusing it would risk a premature, incorrect un-fence. -- **Trade-offs**: one more boolean/condition threaded through the two promotion-gate call sites in `ha.c`; justified by correctness. -- **Follow-up**: verify at implementation time that both `ha.c` promotion-gate call sites (periodic release, election-time) are updated together — a design.md Revalidation Trigger. - -## Risks & Mitigations -- **Apply-pump return-value fix is a behavior change on an existing, heavily-tested path** (every WAL type's apply failure, not just arrays, now halts the walk instead of silently skipping) — Mitigation: today's `apply_committed_entry()` implementations never actually return a negative value in practice for any existing type (verified during discovery), so this fix is a no-op for all pre-existing call paths and only activates for the new array-expansion failure case; full HA soak (Requirement 8.2) is the regression gate. -- **Two live dispatchers (`apply_committed_entry`, `apply_wal_entry`) must both gain the new case** — Mitigation: shared helper (`restore_job_array`) means the only per-dispatcher code is one switch-case line; covered explicitly in File Structure Plan and Revalidation Triggers. -- **`jobj.c` hardening (Requirement 10) touches a shared primitive used by every job construction, not just arrays** — Mitigation: scoped narrowly to the five specific gaps requirements.md names (NULL-check, table-init-ordering root cause, MOVE_PTR-reuse audit, `accounts` leak, `ckptinfo` coverage audit); no broader refactor. - -## References -- `.kiro/specs/raft-apply-decoupling/design.md` — the `apply_committed_entry()` seam contract this feature builds on; its own Revalidation Triggers already flag "any new WAL entry type must be added to the `apply_committed_entry()` dispatch table or it silently fails to apply" and "`job-array-single-wal-record` landing — builds its array-expand step directly on `apply_committed_entry()`." -- `.kiro/specs/ha-commit-engine/`, `.kiro/specs/ha-commit-barrier-rollout/` — majority-commit and idempotency primitives consumed unchanged. -- `.kiro/specs/scheduler-controller-msg-gap/` — established precedent for extending the controller→scheduler push protocol with a new `Data` union variant. -- `src/controller/CLAUDE.md` — Process Lifecycle invariant ("a controller process MUST NEVER self-terminate... fail loud, fence, stay alive, keep retrying") governing the Live Apply Resource Fence's behavior. - ---- - -# Gap Analysis: job-array-single-wal-record - -**Note**: `design.md` and `tasks.md`-adjacent work for this spec already exist and have passed design review (GO, zero critical issues, 2026-07-13). This gap analysis is produced retroactively per explicit user request and is used here to confirm/cross-check the design's approach against the framework below, not to precede it. - -## Requirement-to-Asset Map - -| Requirement | Existing Asset | Gap | -|---|---|---| -| 1.1-1.3 (single WAL record/barrier) | `handle_jobsubmit()` (job.c:926), `build_job_submit_payloads()` (job.c:221), `persist_submit_job_payloads_async()` (persist.c:4846) — today's N-append, N-continuation path | **Missing** — no single-array-append path; `wal_entry_type` (wal_types.h:102) has no array variant | -| 2.1-2.3 (atomic commit) | `commit_submit_async()` (commit engine, unchanged), `jobsubmit_async_ctx_t`/`pending` refcount (job.c:754, already generalizes to `pending=1`) | **Constraint** — commit-barrier atomicity is free (existing engine); *apply*-side atomicity (no partial `c->jobmgr` visibility) is not, since `c->jobmgr` insertion isn't natively transactional across N constructor calls | -| 3.1-3.4 (shared expand-on-apply) | `apply_committed_entry()` (persist.c:2721), `apply_wal_entry()` (persist.c:2574) — **two** live dispatchers, both call `restore_job()` for the single-job case today | **Missing** — no `restore_job_array()`-equivalent; neither switch has a `WAL_JOB_ARRAY_SUBMIT` case | -| 4.1-4.3 (standby reconstruction) | Replication streams whatever WAL bytes exist (repl.c, unchanged) — one record replicates as one entry by construction | **None** — falls out of 1+3 once built; no separate replication-layer work | -| 5.1-5.3 (identity preservation) | `base_jobid + i` arithmetic (job.c:238), `idgen()` (idgen.c:22) reserving a contiguous range | **None** — verification-only; new expansion loop reuses this arithmetic identically | -| 6.1-6.5 (compact push) | `sched_job_new_fun`/`apiserver_job_new_fun` (sched.c:271, apiserver.c:127), `Data` union (controller.capnp:355) — already N-capable by *call count*, not by *wire shape* | **Missing** — no compact `Data` union variant; no scheduler-side (`handler.c:112`) or apiserver-side (Go `backend/`) local-expansion logic | -| 7.1-7.2 (bounded record size) | `idgen()`'s `MAX_ARRAY_IDX` cap (config.h:68, idgen.c:31), enforced upstream in the Go apiserver (clib.go:165) before a request reaches the controller | **None** — pre-existing cap plus the record's O(1) payload satisfies this by construction; no chunking policy needed | -| 8.1-8.4 (non-regression) | Existing single-job path (job.c), existing `/xce-test-ha` harness | **Constraint** — `array_size == 1` must stay byte-for-byte identical; verified achievable by branching before any shared state touches | -| 9.1-9.2 (pre-commit validation) | `inflight_range_try_add`/`jobid_range_exists` (job.c:90-212) — already range-shaped from bug-1419 | **None** — consumed unchanged, called once per array submission exactly as today | -| 9.3-9.5 (deterministic apply / live fence) | `persist_recover_loud_fence_if_unrecoverable`/`recovery_unrecoverable` (persist.c:4259) — boot-replay-only fence shape | **Missing** — `persist_apply_pump_run()` (persist.c:3244) discards `apply_committed_entry()`'s return value (a real, load-bearing bug, not array-specific); no live-path fence exists | -| 9.6 (fault injection) | This codebase's `#ifdef XCE_TESTING`-guarded test-only-symbol convention (structure.md) | **Missing** — no injection seam exists yet in any job-construction path | -| 10.1, 10.4 (jobj.c NULL-check, leak) | `job_init()`/`job_destroy()` (jobj.c:532, 565) | **Missing** — both are concrete, located bugs with no fix yet | -| 10.2 (jobsub_table ordering) | `jobsub_table_init()`/`job_obj_init()` (jobj.c:80) | **Unknown** — root cause suspected (init-ordering) but not confirmed; flagged as a task-level investigation in design.md, not pre-decided | -| 10.3, 10.5 (MOVE_PTR safety, ckptinfo coverage) | `job_init()`'s MOVE_PTR fields, `decode_Job`/`free_Job` (generated code) | **None** — both verified as non-issues during design (freshly-calloc'd payloads are always NULL on these fields; `ckptinfo` has no wire field to leak) | - -## Implementation Approach Options - -### Option A: Extend Existing Components Only -Every change lands inside the file that already owns that domain — `job.c` (submit), `persist.c` (apply/commit/pump), `sched.c`/`apiserver.c` (push), `ha.c` (promotion), `jobj.c` (construction) — with no new non-test files. -- ✅ Matches `structure.md`'s "one domain per file" convention exactly; no domain boundary is invented. -- ✅ Reuses `jobsubmit_async_ctx_t`, the in-flight range registry, and the commit engine wiring completely untouched. -- ❌ `persist.c` is already one of the largest files in the component; this adds more surface to it. -- ❌ `ha.c` gets touched for a narrow addition (2 conditions, 2 call sites) but still requires the full TLA+/protocol-verification checklist review (`src/controller/CLAUDE.md`) since it's HA-critical. - -### Option B: New Standalone Component (e.g. a `job_array.c`) -Isolate the array-specific logic (`restore_job_array`, array-submit helpers) into a new file. -- ✅ Cleaner surface for unit-testing `restore_job_array()` in isolation. -- ❌ `restore_job_array()` must still be called from `persist.c`'s two switch statements and share `persist_t`/`cluster_t` internals plus locking discipline (`apply_pump_lock`, `c->jobmgr`'s RTTI lock) — a separate file buys little real isolation given this tight coupling, and risks violating the "No Hidden Shared Ownership" design principle by either exposing currently-internal `persist.c` state or duplicating its lock semantics. -- ❌ Array submission isn't a new *domain* in this codebase's terms — it's a new WAL record shape within the existing job domain — so a new file would be an organizational mismatch, not a genuine boundary. - -### Option C: Hybrid (extend in place, new functions not new domain files) -Extend the six existing files (Option A's list) plus `controller.capnp`, adding clearly-named new functions (`restore_job_array`, `persist_submit_job_array_async`, `xcesched_push_jobarray`, `apiserver_push_jobarray`, `persist_apply_resource_fenced`/`_mark`/`_clear`) rather than modifying existing function bodies destructively. New *files* are created only where isolation is genuinely warranted: test files (`test_job_array_apply.c`, `test_job_array_fence.c`, `test_jobj_hardening.c`). -- ✅ Every new function has one clear, single-file home matching its existing domain — no fragmentation, no fake isolation. -- ✅ Genuine isolation exactly where it earns its cost (tests). -- ❌ Touches 8 existing files across 2 languages (C controller/scheduler, Go apiserver) plus 1 schema file — coordination surface is wide even though each individual touch is small. -- ❌ The "two live dispatchers" fact (`apply_committed_entry` vs. `apply_wal_entry`) must be correctly carried into task generation and implementation by every task that touches either — a real risk if that fact isn't threaded through explicitly (mitigated: design.md's Revalidation Triggers and File Structure Plan both name both dispatchers explicitly). - -## Effort & Risk - -- **Effort: L (1-2 weeks)** — significant functionality (new WAL type, apply-seam expansion, HA promotion-gate change, two new push consumers across two languages, five jobj.c fixes) spanning multiple files and one cross-language boundary, but no single piece requires inventing a new architectural primitive — every piece composes an already-proven existing mechanism (commit engine, hook system, RTTI object system, `Data` union). -- **Risk: Medium** — no unknown technology and no foundational primitive being invented from scratch, but two of the touch points are genuinely HA-sensitive (`ha.c`'s promotion gate, `persist_apply_pump_run()`'s return-value handling) where a subtle mistake has correctness consequences beyond this feature. Mitigated by the design's scratch-build atomicity discipline and by full `/xce-test-ha` regression being a stated, non-optional gate (Requirement 8.2). Not High: nothing here requires new consensus/replication mechanics — those are consumed unchanged. - -## Recommendation - -**Option C (Hybrid)** is the right fit and is what `design.md` already selected — confirmed by this analysis, not contradicted by it. Option A alone would have been viable too (the "new components" in Option C are functions, not files, in every case except tests), so in practice Option C and a disciplined Option A converge here; the distinction matters mainly for the test files, where isolation is unambiguously worth it. - -**Research items carried forward** (already flagged in `design.md`'s own Risk notes, repeated here for gap-analysis completeness): -- Requirement 10.2's `jobsub_table` initialization-ordering root cause — confirmed unknown at design time, scoped to a bounded task-level investigation with an explicit boundary note (if the fix turns out to be outside `jobj.c`, split into its own task rather than silently widening scope). diff --git a/.kiro/specs/job-array-single-wal-record/spec.json b/.kiro/specs/job-array-single-wal-record/spec.json deleted file mode 100644 index 26308404..00000000 --- a/.kiro/specs/job-array-single-wal-record/spec.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "feature_name": "job-array-single-wal-record", - "created_at": "2026-07-09T17:20:00Z", - "updated_at": "2026-07-17T10:45:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-17T09:47:00Z", - "known_limitations": [], - "resolved_limitations": [ - "bug-1445 (first-boot promotion-ordering gap) fixed 2026-07-17 in src/controller/ha.c (raft_cb_become_leader), outside this spec's own task list -- see .wolf/buglog.json for full detail. TDD-verified (RED/GREEN) with a dedicated test plus a full controller_unit_test regression pass." - ], - "prior_decision": { - "decision": "GO", - "validated_at": "2026-07-16T02:10:00Z", - "reopened_at": "2026-07-16T06:15:00Z", - "reopen_reason": "User directed all of this feature's own open residual gaps be fixed (2026-07-16): bug-1443 (bulk-insert partial-visibility), task 4.2's boot-replay fence gap, and task 5.3b's bounded leak. Added as Phase 7 (tasks 7.1-7.3) in tasks.md. Unrelated pre-existing bugs 1437/1438 explicitly excluded from this reopening per user's scope decision.", - "note": "bug-1443 was closed by task 7.1 (commit dcf9f39); superseded by this validation's fresh GO." - } - } -} diff --git a/.kiro/specs/job-array-single-wal-record/tasks.md b/.kiro/specs/job-array-single-wal-record/tasks.md deleted file mode 100644 index 82720ecb..00000000 --- a/.kiro/specs/job-array-single-wal-record/tasks.md +++ /dev/null @@ -1,245 +0,0 @@ -# Implementation Plan: job-array-single-wal-record - -- [ ] 1. Foundation: WAL/schema additions and jobj.c hardening -- [x] 1.1 (P) Add the array WAL record type - - Add a new `WAL_JOB_ARRAY_SUBMIT` value to the WAL entry type set, distinct from `WAL_JOB_SUBMIT` - - Classify it as a WAIT-mode (requester-awaiting) commit, same as the existing single-job type - - Observable: the new type is recognized by the WAL commit-mode classifier and compiles cleanly; the existing single-job type's classification is unchanged - - _Requirements: 1.1, 1.2, 1.3_ - - _Boundary: WAL Array Submission_ - -- [x] 1.2 (P) Add the compact array push wire variant - - Add one new additive variant to the existing controller-to-scheduler/apiserver push message, reusing the client submit request's existing (base jobid, array size, one spec) shape verbatim — no new struct - - Observable: the schema compiles and generates encode/decode functions for the new variant without touching any existing variant's wire layout - - _Requirements: 6.1, 6.2_ - - _Boundary: Compact Scheduler/API-Server Push_ - -- [x] 1.3 (P) jobj.c hardening - - Make job construction check the interning/string-construction return value and propagate failure instead of silently storing a NULL success - - Root-cause and fix the previously observed construction-table initialization-ordering gap; if root-causing shows the actual gap is outside this file's own init/call ordering (e.g. controller boot sequencing), stop and flag it as a separate follow-up rather than forcing a fix into this task's boundary - - Verify (with a test) that the destructive-move fields on a freshly-built job payload are always NULL, so repeated use against a shared template is provably benign - - Fix the known leak on job teardown for the one field that was never freed - - Verify (with a test) that the field claimed to be an undocumented leak risk actually has no wire representation to leak in the first place - - Observable: a new construction-failure path returns an error instead of a silently-broken success; the leak-fix is visible under a leak-detection test run; all five items have a passing test each - - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5_ - - _Boundary: jobj.c Hardening_ - -- [ ] 2. WAL Array Submission: encode, append, commit -- [x] 2.1 Implement the single-record array append and commit path - - Encode one compact payload carrying (base jobid, array size, one shared job spec) — the same shape as the client's own submit request - - Append it as exactly one `WAL_JOB_ARRAY_SUBMIT` WAL entry and register exactly one commit continuation for it, regardless of array size - - Observable: submitting an array produces one WAL append call and one registered commit continuation in a controlled test, not one per element - - _Requirements: 1.1, 1.2, 1.3, 4.2, 9.2_ - - _Boundary: WAL Array Submission_ - -- [x] 2.2 Wire array submissions into the request handler - - Branch the submission handler on array size: exactly 1 continues through today's existing single-job code path completely unmodified; greater than 1 routes to 2.1's single-append path - - Preserve the existing pre-commit range validation (collision check plus in-flight claim) unchanged and in the same order, still rejecting strictly before any WAL append - - Reuse the existing batched-reply mechanism, setting it to expect exactly one outcome instead of one per element - - Observable: a single-job submission (array size 1) is byte-identical in behavior to before this feature; an array submission returns exactly one reply after exactly one outcome resolves - - _Requirements: 1.1, 1.2, 1.3, 8.1, 9.1, 9.2_ - - _Boundary: WAL Array Submission_ - - _Depends: 1.1_ - -- [x] 2.3 Unit tests for the array append/commit path - - Verify exactly one WAL entry and one commit continuation are produced for an array submission, for several array sizes - - Verify the encoded payload's size does not grow with array size (stays constant regardless of element count) - - Verify a single-job (array size 1) submission produces a byte-identical WAL record, commit-barrier count, and reply compared to the pre-feature baseline - - Observable: a passing test suite demonstrating all three properties above - - _Requirements: 1.1, 1.2, 1.3, 7.1, 7.2, 8.1_ - - _Boundary: WAL Array Submission_ - -- [ ] 3. Array Apply Expansion: shared N-way expansion and dispatch wiring -- [x] 3.1 Implement the shared array expansion primitive - - Decode the committed array payload and, for each index in the array's range, build a job entry without yet making it visible to any reader - - Skip building an entry for any index that already exists (idempotent re-apply); if every index already exists, treat the whole record as an already-applied no-op - - Once every index has successfully built, make the whole set visible in one atomic step; if any index fails to build, discard everything built in this attempt without ever making any of it visible — a retry starts over from the first index - - Include a test-only, forceable failure point (compiled out in production) that a test can use to force a failure at a chosen index - - Observable: a test can force a mid-array failure and confirm zero elements are visible afterward, then force success and confirm the full set becomes visible in one step - - _Requirements: 2.1, 2.2, 2.3, 3.3, 4.1, 5.1, 5.2, 5.3, 9.3, 9.6_ - - _Boundary: Array Apply Expansion_ - - _Depends: 1.3_ - -- [x] 3.2 Wire array expansion into live apply - - Call 3.1's primitive from the live-apply/boot-replay dispatch path for the new WAL record type - - After a successful expansion, fire the existing job-created notification exactly once, carrying every newly created element in one list — never once per element - - Observable: applying one committed array record produces exactly one notification event carrying all N elements - - _Requirements: 3.1, 3.4_ - - _Boundary: Array Apply Expansion_ - -- [x] 3.3 Wire array expansion into standby catch-up (defensive parity — see note below) - - Call 3.1's same primitive from the separate standby catch-up/replicated-apply dispatch path for the new WAL record type, without firing any notification (matching that path's existing hook-free behavior for the single-job case) - - Observable: a test applies the same committed array record through both the live-apply dispatch and the standby catch-up dispatch and confirms byte-identical resulting job state either way - - _Requirements: 3.1, 3.2, 4.1 — delivered by task 3.2 alone (see Implementation Notes); this task is defensive parity, not an independent delivery path_ - - _Boundary: Array Apply Expansion_ - -- [x] 3.4 Unit tests for array expansion - - Verify N-way expansion produces the correct contiguous set of element identities for several array sizes - - Verify idempotent re-apply of an already-fully-applied record is a no-op that creates nothing new - - Verify, using 3.1's forceable failure point, that a failure partway through leaves nothing visible, and that a subsequent successful retry produces the complete, non-duplicated set - - Observable: a passing test suite covering all three properties above - - _Requirements: 2.1, 2.2, 2.3, 3.3, 5.1, 5.2, 5.3, 9.4, 9.6_ - - _Boundary: Array Apply Expansion_ - -- [ ] 4. Live Apply Resource Fence -- [x] 4.1 Make the apply pump respect apply failures - - Stop discarding the array-expansion (and every other WAL type's) apply outcome; on a failure, halt the apply walk at that position without advancing past it, and log a loud, operator-visible error naming the stuck position - - Observable: a simulated apply failure at a given position causes the walk to stop exactly there and not proceed to later entries - - _Requirements: 9.4_ - - _Boundary: Live Apply Resource Fence_ - -- [x] 4.2 Add the live, self-healing fence - - Add a new piece of state (distinct from the existing boot-only unrecoverable-fence flag) recording "a position is currently stuck failing to apply" - - Engage it when 4.1's halt condition triggers; clear it automatically, from inside the apply walk itself, the moment a retry of that same stuck position succeeds - - Observable: after a forced failure the new state reads "fenced"; after the same position later succeeds on retry, the state reads "clear" without any external action - - _Requirements: 9.4, 9.5_ - - _Boundary: Live Apply Resource Fence_ - -- [x] 4.3 Gate promotion on the live fence - - Add the new fence state as a required condition at both existing points where a node's eligibility for promotion is decided (the periodic release check and the election-time check) - - Observable: a node with the fence engaged is refused promotion at both points; once the fence clears, the node is eligible again without any other change - - _Requirements: 9.4, 9.5_ - - _Boundary: Live Apply Resource Fence_ - -- [x] 4.4 Fault-injection test for the live fence - - Using 3.1's forceable failure point, force a resource-exhaustion-style failure during array expansion and verify: the node fences itself and is refused promotion, constructs no partial or divergent set of elements, does not affect any other (simulated) node's independent progress, and self-heals — clearing the fence and completing the apply — once the forced condition is removed and the retry succeeds - - Observable: a single passing test exercising the full engage-refuse-clear-recover sequence - - _Requirements: 9.4, 9.5, 9.6_ - - _Boundary: Live Apply Resource Fence_ - - _Depends: 3.1_ - -- [ ] 5. Compact Scheduler/API-Server Push -- [x] 5.1 (P) Build the compact scheduler push - - Add a function that encodes the compact (base jobid, array size, one spec) message for the scheduler and branch the existing job-created push handler to call it, instead of the existing per-job push, whenever the notification carries more than one element - - Leave the single-element (ordinary single-job) branch completely unmodified - - Observable: a synthetic multi-element notification produces exactly one compact push call, not N individually-encoded job structs; a synthetic single-element notification is unaffected - - _Requirements: 6.1, 6.3, 6.4_ - - _Boundary: Compact Scheduler/API-Server Push (controller-side scheduler push)_ - -- [x] 5.2 (P) Build the compact API-server push - - Same construction as 5.1, for the API-server push handler and its own existing per-job push - - Observable: a synthetic multi-element notification produces exactly one compact push per connected API server, not N individually-encoded job structs; the single-element branch is unaffected - - _Requirements: 6.2, 6.4_ - - _Boundary: Compact Scheduler/API-Server Push (controller-side apiserver push)_ - -- [x] 5.3 (P) Scheduler-side local expansion - - Decode the compact push and expand it locally into N ordinary schedulable tasks using the same base-jobid-plus-index arithmetic the controller itself uses - - Deduplicate by element identity so a redelivered push (e.g. after a scheduler reconnect) does not create duplicate tasks - - Log one identifiable confirmation line (base jobid plus element count) once expansion completes - - Observable: feeding the same compact push twice to the receiver produces the task set once, not twice, and exactly one confirmation log line per successful expansion - - _Requirements: 6.3, 6.4, 6.5, 8.4_ - - _Boundary: Compact Scheduler/API-Server Push (scheduler daemon receive path)_ - - _Depends: 5.1_ - -- [x] 5.3b Fix the shared wire dispatcher to carry the compact array push - - `extract_data()` (`src/lib/request.c`) is the shared reassembly step every inbound push (scheduler and API-server alike) passes through inside `vtask_entry()` (`src/lib/xcectx.c`) before a receiver's handler ever runs. Its dispatch switch has no case for the `Data_data_jobArray` variant (added task 1.2) and falls into `default: rc = -EINVAL`. Because `vtask_entry()`'s `if (callnext)` treats ANY nonzero return — including a negative errno — as "reassembly complete, proceed," this doesn't surface as an error: the real payload is freed and the receiver is silently handed an empty, zeroed `dataset_t` (`rows=NULL, nrows=0`), with no log anywhere in the chain (`vtask_entry()` discards the handler's own return value unconditionally) - - Add a `case Data_data_jobArray:` to `extract_data()` that passes the single, unchunked `JobSubmit` descriptor straight through to the caller's `dataset_t` — no pagination/accumulation loop is needed (unlike the existing LIST-shaped cases), since this push is always exactly one descriptor per commit, matching the "wire shape note" both `xcesched_push_jobarray()` (task 5.1) and `apiserver_push_jobarray()` (task 5.2) already establish - - Confirms both the scheduler (task 5.3, already landed) and the future API-server Go receiver (task 5.4) share this exact fix — this task only needs to land once - - Observable: a synthetic `Data_data_jobArray` push driven through the REAL `vtask_entry()`/`extract_data()` path (not task 5.3's white-box `sched_handle_dataset()`-direct test) reaches the scheduler's `sched_on_job_array()` with the genuine payload intact, not an empty dataset - - _Requirements: 6.1, 6.2, 6.4, 8.4_ - - _Boundary: shared wire transport (`src/lib/request.c`, `src/lib/xcectx.c`) — out of the Compact Scheduler/API-Server Push component's original File Structure Plan; discovered during task 5.3's review (2026-07-15), design.md corrected to record it_ - - _Depends: 5.1, 5.2_ - -- [x] 5.4 (P) API-server-side local expansion - - Decode the compact push and expand it locally into N per-element records using the same base-jobid-plus-index arithmetic - - Deduplicate by element identity so a redelivered push does not create duplicate records - - Log one identifiable confirmation line (base jobid plus element count) once expansion completes - - Observable: feeding the same compact push twice to the receiver produces the record set once, not twice, and exactly one confirmation log line per successful expansion - - _Requirements: 6.2, 6.4, 6.5, 8.4_ - - _Boundary: Compact Scheduler/API-Server Push (apiserver receive path)_ - - _Depends: 5.2_ - -- [x] 5.5 Integration test: end-to-end compact push - - Commit one array record end-to-end and confirm the scheduler and the API server each receive exactly one push and each expand it into N correctly-addressable entries - - Confirm a simulated redelivery (reconnect) to either receiver does not double-expand - - Observable: a passing integration test covering both receivers' one-push/N-entries/no-double-expand behavior, including both receivers' confirmation log lines - - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 8.4_ - - _Boundary: Compact Scheduler/API-Server Push_ - -- [ ] 6. Integration & non-regression soak - -- [x] 6.0a Fix replay_wal()'s job-count tally for WAL_JOB_ARRAY_SUBMIT - - `replay_wal()` (`src/controller/persist.c` ~3902) tallies `*new_job_submits_out` only for `hdr.type == WAL_JOB_SUBMIT`. `WAL_JOB_ARRAY_SUBMIT` (task 1.1) is dispatched to `apply_committed_entry()` and correctly materializes N real jobmgr entries, but contributes 0 to this tally. `persist_verify_apply_correctness()` then compares real jobmgr growth (correct, N-sized) against the undercounted expectation, produces a false `APPLY_DIVERGENT` verdict, and engages `persist_recover_loud_fence_if_unrecoverable()` — a PERMANENT, non-self-healing fence (distinct from task 4.2's Live Apply Resource Fence, which self-heals; this one has no live-clear path short of a later restart landing after a checkpoint advances past the array's LSN) - - Live-reproduced during task 6.1's soak attempt (2026-07-15) on 2 of 3 nodes in a single small-array (N=2) sub-round, via two different trigger paths (ordinary boot-time `persist_recover()`, and a standby's election-path recovery fallback) — confirmed deterministic, not racy, not size-dependent. A pre-existing comment at persist.c ~3863-3874 (predates this spec, from `raft-apply-decoupling` task 2.3) already flagged this exact code region as needing re-verification "before tasks 4.1/6.1/6.2/6.3 make this path broadly live" — this is that re-verification, and it failed - - Fix: add a `WAL_JOB_ARRAY_SUBMIT` branch to the tally that adds `apply_rc` (the genuine new-element count `apply_committed_entry()`'s array case already returns) to `*new_job_submits_out` when `apply_rc > 0`, mirroring the existing dedup-safe pattern used for `WAL_JOB_SUBMIT` - - Observable: the exact live-soak reproduction (small array submit, then restart/re-elect a node) no longer produces `APPLY_DIVERGENT`; the node boots/rejoins cleanly and remains promotable - - _Requirements: 3.1, 3.2, 4.1, 9.4_ - - _Boundary: Array Apply Expansion (`persist.c`'s replay-side job-count accounting) — discovered during task 6.1's soak (2026-07-15), inserted per explicit user decision; blocks 6.1/6.2/6.3, all of which require a cluster that survives a restart after an array submission_ - -- [x] 6.1 Large-array soak scenario - - Extend the existing HA soak harness with a large-array submission scenario asserting: one WAL record and one commit barrier regardless of array size, failover replay time governed by the record's size rather than element count, and every array element individually addressable (status, cancel, completion) after a failover - - Observable: a soak run reporting the one-record/one-barrier assertion, a replay-time measurement independent of array size, and successful per-element status checks post-failover - - _Requirements: 4.3, 5.2, 5.3, 8.2, 8.3_ - - _Boundary: Non-Regression & Testing_ - - _Depends: 6.0a_ - -- [x] 6.2 Chaotic-failover array retry scenario - - Extend the soak harness with a scenario submitting an array immediately before killing the primary, then retrying the identical submission against the new primary - - Assert the retry follows the same cross-primary dedup mechanism as single-job submission, extended to the array's full index range as one unit, and does not double-submit or partially duplicate - - Observable: a soak run showing the retried array resolves to exactly one committed instance of the array, with no partial duplication - - _Requirements: 8.2, 8.3_ - - _Boundary: Non-Regression & Testing_ - -- [x] 6.3 Full regression pass - - Run the complete existing unit, integration, and soak suite (sequential and chaotic) and confirm zero new job loss, divergence, or liveness stall attributable to this feature - - Confirm the non-array (array size 1) WAL format, commit-barrier count, and reply are byte-identical to the pre-feature baseline - - Observable: a clean full-suite run with the byte-identical single-job baseline check passing alongside it - - _Requirements: 8.1, 8.2, 8.4_ - - _Boundary: Non-Regression & Testing_ - -- [x] 7. Residual gap closure (post-GO follow-up, user-directed 2026-07-16) -- [x] 7.1 Close the bulk-insert-phase partial-visibility gap (bug-1443) - - Add a new, purely-additive capacity-reservation primitive (`kvmgr_reserve(var kvmgr, int n)` in `kvmgr.c`, following the existing non-vtable `kvmgr_capture_keys()` template exactly — same "caller MUST hold the kvmgr write lock" contract, same bare-extern-function style rather than a new RTTI vtable slot) that pre-grows `c->jobmgr`'s backing `xzarray` for `n` additional entries via a new small non-static wrapper in `zarray.c` calling the existing, unmodified `xzarray_ensure_items(a, a->count + n)` — zero changes to any existing function in either file - - Call `kvmgr_reserve(c->jobmgr, array_size)` in `restore_job_array()` immediately before the bulk-insert pass begins; on reservation failure, treat exactly like today's existing pre-mutation failure paths (discard the scratch list, `c->jobmgr` never touched, return negative) — this eliminates the dominant, N-scaling failure trigger (the large `realloc()` growing the value store) structurally, before any `key_add()` call in the bulk-insert loop can run - - Document honestly (in the function's doc comment) that this narrows, not eliminates, the residual gap: `key_add()`'s own tiny per-entry hash-chain-node allocation and a content-based `-EEXIST` collision remain theoretically possible even after reservation, both far rarer and non-N-scaling than the closed failure mode - - For that now-much-narrower residual case: make `restore_job_array()` populate `out_new_jobs` with whatever prefix was genuinely, successfully inserted before a mid-pass failure, even though the function's overall return is still negative — and make `apply_committed_entry()`'s `WAL_JOB_ARRAY_SUBMIT` case fire `HOOK_DETAIL_JOB_NEW` for that non-empty partial prefix even when `restore_job_array()`'s return is negative, while still propagating the negative return so the LSN correctly halts and the Live Apply Resource Fence still engages for the remaining un-constructed indices. Net effect: a genuinely-inserted element is never left un-pushed/un-scheduled again, even in the narrow residual failure case — closing Requirement 2.3/9.4's literal text, not just narrowing the trigger condition - - Observable: a fault-injection test forcing the bulk-insert-phase failure (reusing task 3.1's existing bulk-insert-phase seam) shows the already-inserted prefix IS included in a fired `HOOK_DETAIL_JOB_NEW` (not silently orphaned), while the overall LSN still correctly halts/fences for the remainder; a separate test proves `kvmgr_reserve()` genuinely pre-grows capacity (no realloc occurs during the subsequent bulk-insert loop when reservation succeeded) - - _Requirements: 2.3, 9.4, 9.6_ - - _Boundary: Array Apply Expansion (`src/lib/kvmgr.c`, `src/lib/zarray.c`, `src/controller/job.c`'s `restore_job_array()`, `src/controller/persist.c`'s `apply_committed_entry()` `WAL_JOB_ARRAY_SUBMIT` case) — touches shared library code (`kvmgr.c`/`zarray.c`) used far beyond this feature; new functions only, zero changes to existing function bodies_ - -- [x] 7.2 Extend the Live Apply Resource Fence's protection across a boot/restart replay (task 4.2's self-flagged gap) - - `replay_wal()` (boot/restart replay path) currently captures `apply_committed_entry()`'s return value only for the job-count tally (task 6.0a) — it does not halt or engage the Live Apply Resource Fence on a negative return the way `persist_apply_pump_run()` (the live path, task 4.1) already does - - Add the same halt-and-fence behavior to `replay_wal()`: on a negative `apply_committed_entry()` return during replay, stop the replay walk at that LSN without advancing past it, log loud, and call the SAME `persist_mark_apply_resource_fenced()` primitive the live path uses — reuse, do not reimplement - - Verify (and document, not just assume) that this engages BEFORE the node could ever be considered for promotion during its own boot sequence — trace the actual boot ordering to confirm `persist_recover()`/`replay_wal()` completes before `ha.c`'s promotion-gate logic could evaluate this node as a candidate - - `persist_init()`'s existing unconditional fence-reset on every fresh process construction remains correct and unchanged: if the same resource-exhaustion condition recurs during a later boot replay, this task's new check will immediately re-engage the fence before promotion is possible, so starting from a clear state on each fresh process is safe by construction, not a gap - - Observable: a test simulating a node that crashes while fenced and restarts into a boot replay hitting the identical resource-exhaustion condition (reuse task 3.1's fault-injection seam) shows the node re-engages the fence during replay and is refused promotion, rather than silently booting clean and looking promotable while actually behind - - _Requirements: 9.4, 9.5_ - - _Boundary: Live Apply Resource Fence (`src/controller/persist.c`'s `replay_wal()`) — reuses existing task 4.1/4.2 primitives verbatim, no new fence mechanism_ - - _Depends: 7.1 is not a prerequisite (independent root cause), but should land first since both touch `persist.c`_ - -- [x] 7.3 Fix the bounded per-commit leak on the scheduler receive path (task 5.3b's self-flagged gap) - - `sched_handle_dataset()`'s `Data_data_jobArray` case never frees `dataset->rows` (the decoded `xce_job_submit_t*`) after `sched_on_job_array()` consumes it — `free_Dataset()` is an empty no-op for every table type, so this is the one call site responsible for freeing this specific variant's payload - - Free `dataset->rows` (and its embedded submit spec, following whatever the existing `xce_job_submit_t`/`JobSubmitData` teardown convention already is elsewhere in this file) after `sched_on_job_array()` returns, in both the success and any early-return/error path through that case - - Observable: a leak-detection test run (ASan) around a real array push shows zero growth attributable to this path, where it previously showed one bounded leak per delivered array push - - _Requirements: 8.4_ - - _Boundary: Compact Scheduler/API-Server Push (`src/scheduler/src/handler.c`'s `sched_handle_dataset()`)_ - -## Implementation Notes - -- **Task 7.3**: `sched_handle_dataset()`'s `Data_data_jobArray` case now calls `free_JobSubmit_ptr((xce_job_submit_t**)&dataset->rows)` + `dataset->nrows = 0` after `sched_on_job_array()` returns (success and error paths alike), mirroring the sibling list-shaped cases' ownership-transfer convention. `free_JobSubmit_ptr()` is the existing capnp-generated teardown for this exact struct shape (frees `submit`'s 7 strings + `start_info`'s 10 strings + both nested structs + the top-level struct) — no new free logic was needed, just the missing call site. Fixing this exposed that `test_job_array_expand.c`'s white-box fixture (`init_job_array()`) built `dataset->rows` from stack-allocated structs with string-literal fields; freeing those crashes (SEGV in `free_JobSubmitData`). The fixture was rewritten to heap-allocate + `strdup` (matching the real `decode_JobSubmit_ptr()` wire-decode shape), and `test_job_array_redelivery_does_not_duplicate` now builds a fresh descriptor per delivery (the first is consumed+freed by the handler, so reusing one pointer across two calls is no longer valid — this also matches production, where a redelivery is always a freshly-decoded wire message, never a reused pointer). Verified via direct ASAN runs of all 4 scheduler job-array test binaries (leak detection is disabled by default in the meson test harness): zero leak growth traceable to the jobArray path in any of them (confirmed by grepping leak stack traces, not just byte counts, since fixture-init debt is a separate pre-existing, unrelated leak source in the same binaries). - -- **Task 1.3 (FIXED 2026-07-17, upstream)**: `meson test --suite xce`'s `libxce test` target used to SIGABRT under ASAN leak detection whenever any code path exercised a `job_init()` failure — root cause was the vendored `rtti` library's `makeInstance()` not freeing its allocation when a type's `.init()` returns negative (no `.c` source available, headers-only in the nix store). Pre-existing, independently reproduced against an unmodified pre-task `jobj.c`, not introduced by this feature. **Resolved by a `librtti` repin** (`src/flake.lock`: rev `7212ba170d8db040ac99d0fb47c7b1df70df6e11` → `53cf11e6df4e2f0bbcdf8e26feed74d011b0e9c1`, revCount 20→21) — verified via a genuinely fresh rebuild (`nix flake metadata` confirms the new rev resolved) plus a full green `libxce test` run and a standalone `test_jobj_hardening` run under `ASAN_OPTIONS=detect_leaks=1:abort_on_error=1`: `test_job_init_propagates_makeJobsub_failure` (the trigger test) now passes clean, zero LeakSanitizer output, no `makeInstance`/`jobj.c:768` leak stack. Any future task whose tests exercise a `create_job()`/`makeJob()` failure path should now see a normal PASS/FAIL Unity result, not a SIGABRT — if the SIGABRT reappears, treat it as a genuine regression (possibly a librtti downgrade), not this known-and-now-closed issue. -- **Task 1.3**: `job_obj_init()`/`job_obj_fini()` are now idempotent (atomic_flag-guarded) — safe to call more than once without an intervening `fini()`. Any future test construction can rely on this instead of coordinating a single call site. -- **Task 1.3**: `makeJobsub()`/`makeJobstartinfo()` now return `-ENODEV` (not a generic `-EINVAL`) when the interning table isn't initialized — if a later task's error handling branches on `job_init()`'s return value, `-ENODEV` is a distinct, checkable signal for "table not ready" vs. other construction failures. -- **Task 3.3**: `apply_wal_entry()` (persist.c:2574) is dead code in production — confirmed via exhaustive grep (zero call sites outside its own declaration/definition/comments) and via `ha.c:8024`'s `ha_standby_advance_committed_and_apply` (dated 2026-07-10), whose own doc comment explicitly says standby entity materialization now goes through the SAME `apply_committed_entry()`/`persist_apply_pump_kick` pump the primary uses, not `apply_wal_entry()`. The persist.c:3722 comment claiming otherwise predates that refactor and was never updated (design.md corrected accordingly). Requirement 3.1/3.2/4.1 are fully delivered by task 3.2 alone. Task 3.3's code was kept (not discarded) as defensive parity with `WAL_JOB_SUBMIT`'s existing, also-dead case in the same switch — it is tested and correct, just not currently reachable in production. If `apply_wal_entry()` is ever revived or removed entirely in a future spec, this task's case should move or be deleted with it. -- **Task 3.1 (superseded by task 7.1, see that entry below)**: `restore_job_array()`'s atomicity originally had one narrow, documented exception: the final bulk-insert-into-`c->jobmgr` pass could itself fail on allocation/growth exhaustion (`c->jobmgr` grows via `realloc()`; no capacity-reservation primitive existed anywhere in `kv.c`/`kvmgr.c`/`zarray.c` to pre-check this before mutating), and an already-inserted prefix `[0, inserted)` could be left permanently visible in `c->jobmgr` but never included in `out_new_jobs`, so `HOOK_DETAIL_JOB_NEW` never fired for it — tracked as bug-1443. **This gap is now closed** (task 7.1, commit dcf9f39, 2026-07-16): `kvmgr_reserve()`/`xzarray_reserve()` pre-grow `c->jobmgr` before the bulk-insert pass starts, structurally closing the dominant N-scaling trigger; for the narrower residual case that remains theoretically possible (a tiny per-entry hash-chain-node allocation or a content collision, non-N-scaling), `restore_job_array()` now populates `out_new_jobs` with the genuinely-inserted prefix even on failure, and `apply_committed_entry()` fires `HOOK_DETAIL_JOB_NEW` for that prefix while still propagating the negative rc so the LSN halts and the Live Apply Resource Fence engages for the remainder. See task 7.1 below for full detail. - **UPDATE (post-4.4): this gap was NOT closed by Phase 4.** Tasks 4.1-4.4 correctly deliver the generic negative-rc halt/fence/gate/self-heal machinery — a bulk-insert-phase failure DOES still engage the fence and block promotion, same as any other failure — but none of them add the "no partial/divergent set" guarantee specifically for this seam, because that guarantee is structurally false on this path with current APIs (no `key_del`). Task 4.4's own capstone test deliberately used the (fully atomic) construction-phase seam instead, precisely because asserting "zero partial elements" against the bulk-insert seam would be dishonest. **This residual gap remains open and unowned by any task in this spec's current scope** — flag at `/kiro-validate-impl` time; closing it would require either a `key_del`/rollback primitive in `kv.c`/`kvmgr.c`, or a re-derive-and-re-notify recovery path for the orphaned prefix, either of which is design-level work beyond this spec's approved task list. -- **Task 4.1**: `persist_apply_pump_run()` now halts the walk (no `last_applied_lsn` advance) on ANY negative return from `apply_committed_entry()`, not just the new array type — this is a general fix. Investigated whether this newly exposes a hazard for the pre-existing job-state fact entries (`WAL_JOB_START/FINISH/FAIL/UPDATE`), which have their own real `-ENOENT` branch when the target jobid isn't in `c->jobmgr`: confirmed NOT a hazard, because `c->jobmgr` (`xkvmgr_type`) has no `key_del` (jobs are never removed for the process lifetime, corroborated by `job.c:428,455,645`), and this task's own strict sequential-no-skip apply means a fact's LSN is only reached after its causally-prior submit already applied. So a live `-ENOENT` on a fact can now only mean genuine corruption/divergence — halting is correct, not a new footgun. Independent reviewer confirmed this via `test_term_safety_adapter.c:2039-2043`'s synthetic (non-production-representative) trigger. design.md's "Existing Architecture Analysis" table row overclaims ("every existing WAL type... returns 0 or positive in practice") — technically inaccurate re: the fact-ENOENT branch — left as-is since the persist.c code comment at the site already documents the exception correctly; low-severity, non-blocking. -- **Task 4.2**: New `_Atomic uint64_t apply_resource_fenced_lsn` field on `persist_t` (0 = clear, nonzero = the stuck LSN) plus `persist_apply_resource_fenced()`/`persist_mark_apply_resource_fenced()`/`persist_clear_apply_resource_fence()`, wired into `persist_apply_pump_run()`'s existing halt branch (engage) and unconditionally after every successful advance (clear). The unconditional clear is provably equivalent to "clear only on the fenced LSN's own successful retry" because `last_applied_lsn` has exactly 3 writers codebase-wide, the other two (`replay_wal`, `persist_recover`) are boot-only and temporally disjoint from the live pump, and a failed walk never advances past the stuck LSN — so the next success by construction IS that LSN's retry. **Known residual gap (out of this task's boundary, flagged by review, NOT yet fixed)**: `replay_wal()` (boot/restart replay, persist.c:3875-3888) still discards `apply_committed_entry()`'s failure return the same way the live pump did before task 4.1, and `persist_init()` unconditionally zeroes the fence field on every restart. A node that crashes while fenced and re-hits the identical resource-exhaustion condition during boot replay will silently pass it and look fully promotable while actually divergent — task 4.1/4.2's protection does not yet extend to the boot-replay path. This is a pre-existing gap (not introduced by this feature) but is now inconsistent with the live path's new guarantee; worth a dedicated follow-up task before anything downstream places full trust in this fence across a restart. -- **Task 5.1**: New `xcesched_push_jobarray()` (`sched.c`) sends via direct `xce_request()`/`OPCode_push`/`Request_body_pushmsg` rather than the existing `xce_push_data_req()` helper — `xce_push_data_req()`'s dispatch switch hard-rejects (`-EINVAL`) anything outside jobs/workers/resources/resvals, so it cannot carry a single non-list `Data.jobArray` descriptor. design.md corrected to match (Dependencies + API Contract for the Compact Scheduler/API-Server Push component). Task 5.2's API-server push will hit the identical fork and should follow the same direct-`xce_request()` pattern rather than re-deriving it. One accepted, low-severity trade-off: the direct path does not go through `xce_push_data_req()`'s `checkf`-gated `check_controller`/`c->running` liveness check that every other push gets — a narrow shutdown-race behavioral difference from sibling pushes, not a new safety hazard; not fixed here since threading a liveness check into a single new call site duplicates rather than reuses the shared gate. `sched.c`/`dataex.c`/`egress.c` were never linked into `controller_unit_test` (two existing test files there ship their own fake `on_scheduler_join()`/`notify()`/`collect_confvars()` precisely because of this, and linking the real ones would collide) — task 5.1's tests live in a new standalone `sched_push_unit_test` binary instead; task 5.2 will likely face the same fork for `apiserver.c` and should check for an analogous existing-fakes collision before choosing a test-binary strategy. -- **Task 5.2**: `apiserver_push_jobarray()` (`apiserver.c`) mirrors task 5.1's `xcesched_push_jobarray()` exactly — same field derivation, same direct-`xce_request()` transport, same `l->count > 1` discriminator — and additionally, correctly replicates `apiserver_push_joblist()`'s existing per-connected-server fan-out loop (`N_APISERVER_MAX` iteration, same lock discipline), independently verified against 2 simulated connected servers each receiving exactly one push. Confirmed the same `apiserver.c`/`dataex.c` link-collision task 5.1 hit for `sched.c` (a fake `notify()` already exists in `test_commit_barrier_reply.c`) — new standalone `apiserver_push_unit_test` binary added, correctly omitting `egress.c` since `apiserver_push_joblist()`, unlike `sched.c`'s push path, never references `job_submit_committed()`/`collect_jobs_committed()`. One low-severity, non-blocking nit (not fixed): the new test file's comment justifying its `apiserver_init()` idempotency guard is factually inaccurate — it claims a second `apiserver_init()` call would double-register hooks, but `xce_add_hook()` has deduped by `(fn,data)` since bug-845, so that specific claim is wrong; the guard itself is still correct/necessary for a different, real reason (avoids re-`mylock_init()`-ing an already-initialized mutex, UB). -- **Task 5.3**: `sched_on_job_array()` (`handler.c`) decodes `Data_data_jobArray`, derives each element's jobid via `base_jobid + i`, and routes constructed elements through the unmodified per-job pipeline (`sched_on_jobs()`). Each element gets its OWN deep clone (`clone_job_submit_data`/`clone_job_startinfo`) of the shared submit spec — required because `sched_pend_update_killer()`'s per-element teardown (`free_Job_ptr`) frees each element's `submit` independently and a shared pointer across N elements would double-free; independently verified. Dedup reuses `sched_get_job()` against the same `sched->jobs` kvtab `sched_add_job()`'s `key_add` authoritatively guards — sound at the data-integrity level, but review found a narrow, non-corrupting TOCTOU: the dedup pre-check runs synchronously at receive time while the actual insert only happens later when `sched_update()` drains `pend_updates[0]`, so a redelivery landing before that drain can re-construct/re-enqueue a duplicate (harmless) object and double the confirmation log line — the same pre-existing, task-5.3-unrelated pattern as `sched_update_jobs()`'s already-ignored `key_add` `-EEXIST` return on the ordinary single-job path. Not fixed (pre-existing class of gap, narrow window, non-corrupting). - **CRITICAL discovery during this task's review (2026-07-15), NOT a defect in task 5.3 itself**: the shared wire dispatcher `extract_data()` (`src/lib/request.c`) has no case for `Data_data_jobArray` and silently drops every real array push before ANY receiver's handler runs (see new task 5.3b, inserted immediately after this task per explicit user decision). Confirmed to affect task 5.4's future API-server receive path identically (both share the same `xcecli_ctx_init()` → `vtask_entry()` → `extract_data()` pipeline). Task 5.3's own tests are unaffected by this gap since they use the same established white-box convention as every sibling `handler.c` test (calling `sched_handle_dataset()` directly, bypassing the transport layer) — this is not a new testing shortcut, it just means task 5.3's passing tests do not, and were never intended to, prove real-wire delivery. -- **Task 5.3b**: Added `case Data_data_jobArray:` to `extract_data()` (`request.c`) — sets `dataset->rows`/`nrows=1` directly (no pagination, matching the always-one-shot wire shape) and nulls `getdata->data.job_array` after transferring it. The null is NOT cosmetic — independently verified `free_Request()`'s `Request_body_pushmsg` case calls `free_Data()`, which has a live `Data_data_jobArray` branch that would double-free/UAF without it. Proven end-to-end with a real `xce_encode()`/`xce_decode()` round-trip through `extract_data()` → `sched_handle_dataset()` → `sched_on_job_array()` (task 5.3, unmodified), not just a unit-level check. Also independently confirmed the alternate hypothesis some earlier notes floated — that `scheduler_req_handler()` itself has a `pushmsg`-rejecting branch — is dead code on the live wire path, since `vtask_entry()` always converts `body_which` to `datamsg` before invoking any handler. **New residual gap, self-flagged, NOT fixed (pre-existing since task 5.3, not introduced here, genuinely out of this task's `request.c`-only boundary)**: `dataset->rows` for the jobArray case is never deep-freed downstream — `free_Dataset()` is an empty no-op for all table types, and `sched_handle_dataset()`'s jobArray case (task 5.3) doesn't null it after consuming it. Net effect: one bounded `xce_job_submit_t` + embedded spec leaks per successfully-delivered array push commit (not per element, not unbounded) — real but small; worth a dedicated follow-up before this spec is considered fully closed, alongside the other self-flagged residual gaps (task 3.1's bulk-insert orphan prefix, task 4.2's boot-replay fence gap). -- **Task 5.4**: Go/CGO receive path — `rawJobArrayToModels()` (pure-Go N-way expansion), `cJobArrayToModels()` (CGO adapter sibling to `cJobToModel`), `dispatchJobArray()` (dedup via the SAME `store.Jobs.Has()` the ordinary `Data_data_jobs` path writes through, one confirmation log only on ≥1 newly-dispatched element), new `case C.Data_data_jobArray:` in `handleDataset`. Elements route through `OnJobUpdated` (not `OnJobCreated`), matching the ordinary single-job push's own routing — verified consistent, not a divergence. CGO boundary needed zero changes — `C.xce_job_submit_t` and `C.Data_data_jobArray` were already exposed/generated from prior tasks. A structural TOCTOU exists between the `Has()` dedup check and the eventual store write (two separate lock acquisitions, no compare-and-set primitive) but is narrower than task 5.3's accepted C-side gap (no intervening async hop) and strictly safer than the pre-existing baseline (the ordinary `Data_data_jobs` path has NO dedup check at all today) — not fixed, judged acceptable. `cJobArrayToModels()` has zero direct unit coverage at the CGO-struct level, but this exactly mirrors the pre-existing, repo-wide precedent for `cJobToModel()` itself (also never unit-tested with a real C struct) — reviewer independently reconstructed a full `CGO_ENABLED=1` build+link+test run against the real generated headers to confirm correctness beyond a compile-only check. `store.ArrayJobs` (array-level rollup aggregate) deliberately left unpopulated — confirmed genuinely out of this spec's scope per requirements.md's Boundary Context, which explicitly defers whole-array bulk query/cancel to a future spec; zero code anywhere in the codebase populates it today, including this diff. -- **Task 5.5**: Cross-daemon integration test, test-only (zero net production diff, confirmed on `request.c` and every task 5.1-5.4 file). Drives real `xce_encode()`/`xce_decode()` wire bytes through the REAL production dispatch chain on both daemons — `extract_data()` → `scheduler_req_handler()` → `sched_handle_dataset()` → `sched_on_job_array()` (C, via `#include "../src/handler.c"` in the test) and `go_push_handler()` → `handleDataset()` → `cJobArrayToModels()`/`dispatchJobArray()` (Go/CGO) — skipping only `vtask_entry()`'s ~10 lines of `static` glue, which the test mirrors verbatim (independently verified line-by-line accurate on the one behavior that mattered: `if (callnext)` treating any nonzero `extract_data()` return, including a negative errno, as "proceed"). Redelivery is tested via genuinely independent fresh decodes of separately-encoded bytes, not a cheaper decode-once-call-twice shortcut. Bug-catching capability independently proven twice (implementer, then reviewer): temporarily reverting task 5.3b's fix makes both the C and Go tests fail meaningfully; restoring returns both to green. Go/cgo toolchain constraint discovered: `import "C"` is disallowed in any `_test.go` file, forcing a two-file split (`wire_dispatch_support.go` for cgo-preamble helpers, `wire_dispatch_test.go` for the actual tests) — logged as bug-1440/1441/1442. One low-severity, non-blocking nit (not fixed): the C test file's own header comment misdescribes which assertion fails first under the RED-phase revert (claims the `callnext != 0` check; actually a downstream `>= 0` check) — cosmetic only, the test's real bug-catching property is independently confirmed accurate. -- **Task 6.0a**: Fixed `replay_wal()`'s job-count tally (persist.c ~3902) to add a `WAL_JOB_ARRAY_SUBMIT` branch that tallies `apply_rc` (the genuine per-record element count `apply_committed_entry()`'s array case already returns) when `apply_rc > 0`. Independently traced the FULL return-value space of `restore_job_array()`/`apply_committed_entry()`'s array case (6 distinct cases: 3 negative pre-mutation failures, `-EEXIST`→`apply_rc=0` idempotent no-op, the task-3.1 bulk-insert partial-failure case (stays negative, correctly still NOT tallied — a pre-existing, out-of-boundary undercount that still surfaces as a detectable anomaly rather than masking one), and positive `created` on genuine success) and confirmed every positive `apply_rc` this fix tallies is PROVABLY equal to real jobmgr growth for that record — the fix cannot overcount, and therefore cannot weaken `persist_verify_apply_correctness()`'s ability to catch a genuine divergence. Verified end-to-end via a from-scratch live 3-node HA soak re-run (`--array-soak`, both by the implementer and independently by the reviewer): zero `APPLY_DIVERGENT` across all three of the original bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-N=10000-array), one-WAL-record/one-commit-barrier confirmed for both N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. This soak run also directly satisfies task 6.1's own Observable — its results should be reused rather than re-run from scratch. **New non-blocking finding (pre-existing, NOT introduced by this task)**: `controller_unit_test`'s single outer `UNITY_BEGIN()` is silently reset by ~19 of its called test-group functions' OWN nested `UNITY_BEGIN()/UNITY_END()` calls, making the process exit code / meson `Ok:/Fail:` summary line unreliable for detecting a regression in any group running before the last such nested call — both the implementer and reviewer worked around this by grepping raw `testlog.txt` for `:FAIL:` lines directly rather than trusting the summary, and this should be the standard practice going forward for this binary until a dedicated follow-up fixes the harness itself. -- **Task 6.1**: Extended `.claude/skills/xce-test-ha/run-ha-soak.sh`/`soak-verdict.sh` with a new `--array-soak` mode (small N=2 + large N=10000 sub-rounds, one clean kill/failover per sub-round — deliberately single-kill, not chaotic, matching this task's correctness-not-chaos scope; chaos-under-array-submission is task 6.2's separate job). `waldump.c` gained a `WAL_JOB_ARRAY_SUBMIT` case in `wal_type_name()` so the scenario can grep an exact type name for its live before/after WAL-record-count delta check. **Round 1 REJECTED**: the replay-time measurement used whole-second (`date +%s`) resolution — both sub-rounds' timings collapsed to `0s`/`n/a` on a real run — and, more fundamentally, nothing ever asserted on the comparison (`verdict_emit_arraysoak()` always returned PASS regardless of measured values), leaving Requirement 4.3 ("replay time governed by record size, not element count") with zero actual evidence and unable to catch a genuine O(N) regression. **Round 2 fixed**: switched to `$EPOCHREALTIME` (millisecond resolution; discovered and worked around a genuine environment trap — this sandbox's BusyBox `date` precedes GNU coreutils on `$PATH` and silently passes `%N`/`%3N` through as a literal string suffix rather than expanding or erroring, which would have poisoned any `date +%s%3N`-based approach silently); added `verdict_check_arraysoak_replay_bound()` with a real `bound = max(t_small*ratio_max, floor_ms) + margin_ms` assertion that `run_array_soak_round()` hard-`fail()`s on when exceeded — independently verified to have real teeth (both the implementer's and reviewer's own hand-picked regression cases correctly trip `BOUND_EXCEEDED`). Live-reproduced 3 times total (once round 1, twice round 2) with genuinely discriminating millisecond numbers each time (final: small=210ms, large=163ms — large array replay was NOT slower, live evidence of the feature's O(1) claim). Also fixed opportunistically: status/cancel/completion checks spread across low/mid/high indices (previously clustered at 1,2,3) matching the already-good basic-addressability spread; selftest extended 17→28 cases covering the new bound-check logic; `SKILL.md` documented. "One commit barrier" remains an inference from the WAL-record-count delta (not independently observed live) — defensible given task 2.1/2.3's unit-tested 1:1 append↔continuation coupling, but flagged as an inference, not a direct check, should a future task want to close that gap. -- **Task 6.2**: Extended the same harness with `--array-chaos-retry` (`submit_array_bg()`, `run_array_chaos_retry_round()`, `verdict_emit_arrayretry()`). Traced the full cross-primary dedup chain end-to-end (bug-1419's mechanism): client-side `idgen()` mints the base jobid once, `xcecli_jobsubmit()`'s `XCECLI_FAILOVER_CALL` retry loop resends the IDENTICAL submit (same jobid, never re-minted); server-side `handle_jobsubmit()` checks `jobid_range_exists()`/`inflight_range_try_add()` — both genuinely RANGE-shaped over `[base,base+array_size)`, run BEFORE the array-append branch, so the whole array is dedup-checked as one unit, not per-element; `xce_failover_classify()`'s "idempotent-replay rule" reclassifies a resulting `-EEXIST` as success on retry. Reviewer went further and confirmed a subtle apply-lag race (freshly-promoted primary's process-local in-flight registry starting empty) is closed by a PRE-EXISTING, unrelated mechanism (`raft_cb_become_leader`'s synchronous apply-pump drain before the promotion callback returns, added 2026-07-10 for an unrelated bug) — not something this task needed to build, just confirmed already-sound. No production code touched or needed — the existing single-job dedup mechanism generalizes correctly to arrays with zero changes. No mutation-test RED phase was performed against production code (correctly declined per this task's explicit constraint); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators and confirming all four scenarios (clean/silent-loss/double-submit/partial-duplication) correctly trigger or don't trigger `fail()` as expected. Live-verified across 4 independent fresh cluster deployments total (2 by implementer, 2 by reviewer) spanning array sizes 6, 10, and 15, 17 total chaotic kill-during-submit rounds, zero double-submits, zero silent losses, zero partial duplications, genuine election/term diversity confirmed each time. `walcheck_count`'s divergence check is informational-only here, matching task 6.1's own established convention exactly (not a new decision) — judged acceptable since `wait_for_full_convergence` already requires matching `committedLsn` across all 3 nodes before this scenario's checks run. -- **Task 6.3**: Pure verification task, zero file changes. Ran the complete suite: `controller_unit_test` (299 tests, 0 failures, verified via raw-log `:FAIL:` grep, not the exit code/summary — per task 6.0a's nested-`UNITY_BEGIN`/`END` counter-reset finding), all scheduler binaries, `xcelib_test` (0 failures with the registered `detect_leaks=0` config; the one pre-existing task-1.3 rtti leak still reproduces under leak-detection-on, unchanged), Go apiserver tests under both `CGO_ENABLED=0` and `=1`, and every soak mode this spec added (`--selftest` 28/28, sequential baseline, `--chaotic`, `--array-soak`, `--array-chaos-retry`) — all clean. Designed and executed a genuine empirical byte-identical baseline comparison for the non-array path (Requirement 8.1): built commit `895a204` (immediately pre-dating this spec's first commit) in an isolated, cleanly-removed `git worktree add --detach`, deployed independent clusters from both the baseline and current-tree binaries, submitted one identical single-job request to each, and did a byte-for-byte `cmp -l` of the raw on-disk WAL entries. Independently reproduced end-to-end by the reviewer (separate build, separate clusters, separate comparison) with a consistent result: every differing byte mapped cleanly to `timestamp`/`crc`/`prev_crc` (the WAL header's inherently run-specific fields) or the payload's `jobid`/`sub_time`/`queue_time` fields — zero differences anywhere in `term`/`lsn`/`type`/`payload_len` or the remaining ~6485 payload bytes. This was further corroborated by a structural code-diff argument (the entire single-job WAL-encode path is untouched between `895a204` and HEAD, purely-additive diff only) and by the pre-existing task 2.3 unit test asserting the same invariant. Confirmed the three self-flagged residual gaps (task 3.1 bulk-insert orphan prefix, task 4.2 boot-replay fence, task 5.3b bounded leak) are the complete and only set of open items across all 24 completed tasks — no new regression, job loss, divergence, or liveness stall found anywhere. **Overall verdict: GO.** -- **`/kiro-validate-impl` feature-level gate (2026-07-16)**: Ran 4 parallel validation dimensions. Requirements coverage: 38 ACs, 33 FULL, 5 PARTIAL (2.3, 8.3, 9.4, 9.5, 9.6 — all tracing to task 3.1's already-known bulk-insert partial-visibility gap plus 8.3's admitted commit-barrier-count inference), 0 outright unclaimed gaps. Design/boundary alignment: ALIGNED, no drift, no boundary violations, all 5 Out-of-Boundary items confirmed untouched, task 5.3b/6.0a's mid-run insertions confirmed surgically scoped to their own narrow fixes. Cross-task integration: SEAMS_SOUND — wire format, `base_jobid + i` arithmetic, `apply_rc` contract, and fence naming all independently verified consistent across every producer/consumer in the chain (5 wire consumers, 3 independent arithmetic implementations in C controller/C scheduler/Go apiserver, 3 tasks sharing the `apply_rc` contract). Mechanical checks: full suite clean, TBD/secrets grep clean, and a genuine live smoke boot (fresh single-node deploy, real `jobsub -a 5` via `xce_test`, exactly one `JOB_ARRAY_SUBMIT` WAL record, all 5 elements individually queryable) — PASS. - The requirements-coverage pass sharpened task 3.1's known gap: it is a literal, code-proven violation of Requirement 2.3 and 9.4 (not just a soft nuance) — a bulk-insert-phase resource-exhaustion failure can permanently orphan array elements that exist and are individually addressable but never reach `HOOK_DETAIL_JOB_NEW`, so they are never scheduled or pushed, not on this attempt or any later successful retry of the same LSN. User decision on final GO/NO-GO: **GO, with the gap filed as a tracked bug (`bug-1443`, `.wolf/buglog.json`)** rather than blocking the spec on a fast-follow fix or holding NO-GO — the trigger condition is narrow (a resource-exhaustion failure at the exact moment of the bulk-insert pass) and every other validation dimension came back clean. Closing `bug-1443` requires either a new `key_del`/capacity-reservation primitive in `kv.c`/`kvmgr.c` (shared library, its own small design effort) or a re-derive-and-renotify recovery path — explicitly out of this spec's approved task list, not a quick follow-up task. -- **Task 7.1**: Added `kvmgr_reserve()`/`xzarray_reserve()` (new, purely-additive primitives in shared `kvmgr.c`/`zarray.c`, following the existing `kvmgr_capture_keys()` template — bare extern function, no RTTI vtable slot, "caller must hold write lock" contract), wired a reservation call into `restore_job_array()`'s bulk-insert phase, and made a genuinely-inserted partial prefix always get notified (`out_new_jobs` populated + `apply_committed_entry()` fires `HOOK_DETAIL_JOB_NEW`) even when the function still fails overall. **Genuine off-by-one found in the existing, unmodified `xzarray_ensure_items()`**: its sufficiency check is strict less-than (`mincap < array->caps`), so reserving exactly `array_size` and then performing `array_size` real stores still triggers one more growth cycle on the LAST store (`mincap == caps` fails the strict check). Independently re-derived and confirmed by the reviewer with concrete arithmetic: reserving `array_size + 1` (not literally `array_size` as this task's text above says) is exactly the minimal correct compensation — not over- or under-provisioning. This is a deliberate, verified deviation from this task's literal wording, documented at the call site. Confirmed the fix narrows (does not fully eliminate) the residual gap as designed: `kvtab_add()`'s own small per-entry hash-chain-node allocation and a content-based `-EEXIST` collision remain theoretically possible even after reservation — both non-N-scaling and far rarer than the closed failure mode, honestly documented as such. Retry-double-notify risk (does a retry's own `out_new_jobs` ever re-include an already-notified prefix) independently traced and confirmed NOT present — the pre-existing per-index idempotent-skip logic excludes already-inserted indices from any subsequent attempt's `built` list. `apply_wal_entry()`'s dead-code standby path confirmed genuinely untouched (zero behavior change, still passes `NULL` for `out_new_jobs`). New build-architecture finding: `libxce.a` is compiled exactly once without `-DXCE_TESTING` and shared by every consumer, so the new test-support definitions in `zarray.c` had to be left unconditionally compiled (only their declarations in `zarray.h` stay `#ifdef XCE_TESTING`-gated) — confirmed zero production risk (force-fail flag defaults false/inert, no production translation unit can reach the ungated symbols without the gated declaration). One non-blocking coverage gap: no single test composes the full engage→notify-prefix→clear-seam→retry→no-double-notify flow end-to-end (the property is provably true by combining two already-passing tests, but not directly demonstrated in one). -- **Task 7.2**: Restructured `replay_wal()`'s per-entry loop into a three-way branch (`apply_halted`/`apply_rc<0`/success) mirroring `persist_apply_pump_run()`'s (task 4.1) halt-and-fence behavior — on a negative `apply_committed_entry()` return during boot/restart replay, logical apply (tally, `last_applied_lsn` advance, `replayed++`) halts at that LSN and `persist_mark_apply_resource_fenced()` engages (reused verbatim, no new mechanism), while PHYSICAL scan bookkeeping (`current_lsn`, CRC chain) deliberately continues unconditionally for every entry — verified safe against `persist_recover()`'s own pre-existing "Round-2 fix" precedent, which already establishes `current_lsn` and `last_applied_lsn` are expected to diverge. This closes the boot-replay half of task 4.2's self-flagged gap: a node that crashes while fenced and re-hits the identical condition on restart now correctly re-engages the fence during replay instead of silently booting clean. - **Went through one remediation round.** Round 1 was REJECTED not on the `persist.c` fix's correctness (verified sound both rounds) but because task 7.2's own explicit deliverable — "verify and document" that the fence engages before any promotion decision — was left unmet, and the new test exercised the wrong `ha.c` gate (the periodic re-alert timer, not the actual election-time check). Round 2 fixed both: the test now drives the real election-time gate (`raft_cb_become_leader_with_cluster_for_test()`, the same seam task 4.3's own test uses for a live-pump-engaged fence — this is the first test to drive it for a fence engaged by boot replay specifically), with teeth independently reproduced twice (temporarily clearing the fence before the gate call reliably flips the test RED, both in isolation and in the full suite). - **CONFIRMED, NOT FIXED, real first-boot promotion-ordering gap (independently verified twice — implementer, then two separate reviewers)**: `raft_cb_become_leader()`'s election-time eligibility gate (the `if (c != NULL) {...}` block, `ha.c:1303-1408`, reading `persist_apply_resource_fenced()` at `ha.c:1337`) runs strictly BEFORE the `is_first_boot` block (`ha.c:1427-1439`) that calls `cluster_init_phase2()` → `persist_recover()` → `replay_wal()` (`cluster.c:1482`) — the function this task modified. There is no re-check of `persist_apply_resource_fenced()` between `cluster_init_phase2()` returning and the unconditional `CTRL_PRIMARY` role assignment a few lines later. Net effect: this task's fix protects every promotion evaluation AFTER a node's first one in a given process's life (a later election, or the periodic re-alert timer) — but a node's very FIRST-EVER promotion evaluation, on its first election win in a freshly-started process, is decided before its own first `persist_recover()`/`replay_wal()` call has even run, so a resource-exhaustion condition that same first replay is about to hit is invisible to that one gate check. This is exactly the "all nodes restart simultaneously" race this whole task exists to guard against, and it is not fully closed by task 7.2 alone. Trigger requires BOTH (a) a node winning its very first election in a fresh process AND (b) that same node's first-ever boot replay hitting the identical resource-exhaustion condition that previously fenced it — narrow but real. Documented via a code comment in `persist.c` (citing these exact file:line references) plus this entry; fixing it requires an `ha.c` change (out of this task's `persist.c`-only boundary) — either re-checking `persist_apply_resource_fenced()` immediately after `cluster_init_phase2()` returns inside `raft_cb_become_leader()`, or reordering recovery to run and be checked before the promotion decision finalizes. Recommend a small dedicated follow-up task against `ha.c` before anything downstream places full trust in this fence across a first-ever-boot promotion race specifically (restarts/re-elections after that first one are already safe). diff --git a/.kiro/specs/libxce-failover/brief.md b/.kiro/specs/libxce-failover/brief.md deleted file mode 100644 index 807d8c4a..00000000 --- a/.kiro/specs/libxce-failover/brief.md +++ /dev/null @@ -1,150 +0,0 @@ -# Brief: libxce-failover - -## Problem - -Client applications using libxce do **not** survive controller primary failover. When the -primary controller is killed and a new one is elected, in-flight and subsequent libxce calls -fail with an opaque `Invalid argument` (EINVAL) instead of either (a) retrying transparently -against the new primary or (b) reporting a clear "cluster is not ready" during the election -window. - -Reproduction (3-node HA cluster, `set masters localhost:9991,localhost:9992,localhost:9993`): -1. Primary elected; `jobsub` / `joblist` work. -2. Kill the primary (`ctrl-1`). -3. Before a new primary is elected, `joblist` should report **cluster not ready**. -4. After a new primary is elected, all commands should transparently route to the new - primary with no error. - -Observed instead: `joblist` keeps returning stale cached output, then `xce_joblist: Invalid -argument` indefinitely — the client never fails over. - -## Current State - -- The requirement already exists: **`ha-config-bootstrap` Requirement 7 — "Client-Side - Transparent Retry for Transient Controller Errors"** (acceptance criteria 7.1–7.6), and its - tasks 10 / 10.1 / 10.2 are marked complete. -- Only the **standalone primitive** was delivered: - - `xce_request_with_retry()` is implemented at `src/lib/request.c:1956` and unit-tested in - `src/lib/testsuite/test_libxce_retry.c` (NOT_READY backoff, STANDBY+redirect reconnect, - STANDBY-without-redirect fallback, non-retriable surface-immediately, max-retries, - 50-client concurrency). `xce_retry_config_t` defaults: `max_retries=20`, - `initial_retry_ms=50`, `max_retry_ms=5000`, `backoff_multiplier=2.0`. -- **Gap 1 — wiring:** every production API path bypasses the wrapper. `xce_joblist` → - `xce_get_req` (`src/lib/request.c:1373`) → plain `xce_request()` (`request.c:1385`). The - same is true for ~18 other `xce_*_req` wrappers — none call `xce_request_with_retry()`. -- **Gap 2 — multi-master discovery:** the retry primitive operates on a single `endpoint_t` - and only follows an explicit `redirect_host/redirect_port`. A *killed* primary cannot - redirect, so the client must **rotate through its configured master list** (9991/9992/9993) - to discover the new primary. That orchestration does not exist; the client connects to - `masters[0]` once in `xcecli_start()` (`src/lib/client.c:122`) and never re-selects. -- Connections are long-lived and cached in `api_ctx_t->ep`; there is no dead-primary - detection or automatic reconnect at the API level. - -## Desired Outcome - -Failover is transparent to client applications: -- Short failovers are hidden: libxce retries across the configured masters with backoff and - succeeds against the new primary once elected. -- If the election outlasts the retry budget, the call returns a clear `E_XCE_NOT_READY` - ("cluster not ready") rather than `EINVAL`. -- Both read and write requests fail over (writes rely on existing server-side commit-barrier + - idempotency guards to dedup re-sends). -- No per-daemon ad-hoc retry loops — the behavior lives entirely in libxce. - -## Approach - -**Central chokepoint (Approach A).** Make failover the default behavior at the single point -all wrappers already funnel through: - -1. Give the request layer access to the cluster/master list + a "current primary" index - (design choice: a back-reference on `endpoint_t`, or a small `xce_cluster_t` the endpoint - references — resolved in design phase). -2. Upgrade `xce_request_with_retry()` to, in addition to its existing redirect/backoff cases: - - rotate through the master list on connection failure (`ECONNREFUSED`/`ETIMEDOUT`/RDHUP) or - `STANDBY`-without-redirect, reconnecting to each candidate to find the new primary; - - apply the bounded backoff budget and, on expiry, surface `E_XCE_NOT_READY`. -3. Route the production `xce_*_req` wrappers through the retry/failover path (e.g. make - `xce_request()` the failover wrapper over an inner `xce_request_raw()`), wiring every API - at once. - -Chosen because it is the only approach that covers the entire client surface at one -integration point, reuses the already-tested primitive, treats reads and writes uniformly, and -keeps the requirement out of caller code — matching "transparent to the client application." - -## Scope - -- **In:** - - Wire all production libxce request paths through transparent retry/failover. - - Multi-master rotation to discover the new primary after the connected one dies. - - Bounded-budget semantics: transparent retry, then `E_XCE_NOT_READY` ("cluster not ready"). - - Failover for both read and write requests (rely on server idempotency for writes). - - Tests: extend `test_libxce_retry.c` for master-list rotation + budget→NOT_READY; a 3-node - live failover integration test (kill primary, assert NOT_READY window then transparent - recovery). -- **Out:** - - Server-side election, WAL, commit barrier, idempotency guards (already owned by the HA - server specs; this spec consumes them). - - Changing the `xce_test` REPL behavior beyond what surfaces the new error semantics. - - Per-request idempotency/retry opt-in flags (writes retry by default per decision). - -## Boundary Candidates - -- Master-list + current-primary selection state and rotation policy. -- The retry/failover wrapper (backoff budget, error classification, NOT_READY surfacing). -- Integration seam: how production `xce_*_req` wrappers reach the failover path. -- Error-surface mapping: connection/EINVAL vs. `E_XCE_NOT_READY` vs. `E_XCE_STANDBY` vs. - non-retriable (auth/perm/invalid/noent). - -## Out of Boundary - -- Any controller/server-side change (election timing, redirect content, idempotency). -- New transport/probe opcodes unless design shows rotation cannot work with existing replies. - -## Upstream / Downstream - -- **Upstream:** `ha-config-bootstrap` (Req 7 retry primitive + `xce_retry_config_t` + error - codes `E_XCE_NOT_READY` / `E_XCE_STANDBY` + `redirect_host/port` reply fields); - `ha-commit-barrier-rollout` (server idempotency guards that make write retries safe); - `ha-single-leader-role` / election specs (define when a standby/primary answers). -- **Downstream:** worker, scheduler, and apiserver clients — all gain transparent failover and - can drop any ad-hoc retry handling; the `/xce-deploy-ha` live test harness. - -## Existing Spec Touchpoints - -- **Extends:** consumes and completes the intent of `ha-config-bootstrap` Req 7 (the primitive - was delivered but never integrated); this spec owns the integration + multi-master failover. -- **Adjacent:** `ha-repl-hardening`, `ha-convergence-recovery` (server-side failover behavior); - `worker-rejoin-resync` (worker-side rejoin, distinct from client-library failover). - -## Protocol Correction (2026-06-16, requirements phase) - -The discovery notes above assumed the controller signals non-primary via `E_XCE_STANDBY` with -`redirect_host/redirect_port` fields. Code review found the real taxonomy (`errnox.h:22-25`): -`E_XCE_REDIRECT` (standby knows a primary exists; `join.c:260-268`), `E_XCE_NOMASTER` (standby, -no primary known yet; `join.c:270-278`), `E_XCE_NOT_READY` (INIT/ELECTING), `E_XCE_STANDBY` -(primary-only op on a standby). The controller's redirect path sends `E_XCE_REDIRECT` with -`data_which = nodata` — **status only, no address** — so the existing `xce_request_with_retry()` -condition (`E_XCE_STANDBY` + populated redirect fields, `request.c:2010`) never matches a real -controller. Decisions (confirmed): classify on the actual statuses — `E_XCE_REDIRECT` → -fast-rotate the master list to find the existing primary (use an address if the reply carries -one, else rotate); `E_XCE_NOMASTER`/`E_XCE_NOT_READY` → backoff → cluster-not-ready; -`E_XCE_STANDBY` → rotate. The existing retry unit tests (keyed on the non-emitted shape) are -corrected to the real statuses. Tracked as bug-608. - -**Scope expansion (confirmed):** the redirect address is absent today only because `join.c` -never fills the reply's `redirect_host/redirect_port` fields — those fields already exist in the -wire schema (`controller.capnp:568-569`) and the reply struct (`typemap.h:225-226`), and the -controller already knows the address (`join.c:263`). One small controller change to populate them -is now **in scope** (Requirement 8), making redirect-based failover deterministic (direct connect -to the primary, O(1)); client-side master-list rotation remains as a defensive fallback for an -older controller that omits the address. - -## Constraints - -- C library (libxce); follow project error-handling (`errnox`, early-return) and async-I/O - rules — `xce_request`/`xce_reply` non-blocking; never block the iomgr thread. -- TDD mandatory: failing tests in `src/lib/testsuite/` before implementation. -- Touches HA/failover behavior → full Kiro workflow (requirements → design → tasks), not the - quick path, per CLAUDE.md. -- Reuse the existing `xce_request_with_retry()` primitive and its config struct; do not fork a - second retry implementation. diff --git a/.kiro/specs/libxce-failover/design.md b/.kiro/specs/libxce-failover/design.md deleted file mode 100644 index 97b80545..00000000 --- a/.kiro/specs/libxce-failover/design.md +++ /dev/null @@ -1,540 +0,0 @@ -# Design Document — libxce-failover - -## Overview - -**Purpose**: This feature delivers transparent controller-primary failover to every client -application built on libxce (worker, scheduler, apiserver via cgo, and the `xce_test` harness), -so that a controller failover is hidden when short and reported clearly when not. - -**Users**: libxce callers issue normal requests (`xce_joblist`, `xce_jobsub`, …). When the -connected primary dies, libxce locates the newly elected primary and completes the request there, -or — if no primary exists yet — returns a distinct cluster-not-ready error instead of an opaque -`EINVAL`. No caller code changes. - -**Impact**: Today the client connects to one master (`client.c:122`), routes every request through -the fire-and-forget `xce_request()`, and has no reply-status-driven retry on the production path -(`xce_request_with_retry()` only retries in test-hook mode). This design places failover at the -**`client.c` (`xcecli_*`) layer** — the universal client surface that every consumer goes through -(worker, scheduler, apiserver-via-cgo, `xce_test`) and the only layer whose context (`api_ctx_t`) -co-locates the master list (`conf`), the endpoint (`ep`), and the reply context (`ctx`). Each -`xcecli_*` op already calls its internal `xce_*` op, which **returns the controller status**; the -failover loop wraps that call, classifies the status, and reconnects/rotates over `api_ctx.conf`. -It adds master-list rotation, redirect-address following, and a small controller change so redirect -replies carry the primary address. `xceapi.c` ops are unchanged. - -### Goals -- Route all libxce request/reply round trips through one transparent failover path (1.1, 1.3). -- Discover the new primary via redirect address, falling back to master-list rotation (2.x, 8.x). -- Hide short failovers via bounded backoff; surface a distinct cluster-not-ready on exhaustion (3.x). -- Fail over reads and writes uniformly, relying on controller idempotency for writes (5.x). -- Reuse the existing retry config/defaults; stagger retries with jitter (6.x). - -### Non-Goals -- Controller-side election, WAL, commit barrier, write-idempotency logic (consumed, not changed). -- Per-daemon adoption (master-list wiring, liveness-safe blocking) — `component-failover-adoption`. -- New transport opcodes or new reply fields (redirect fields already exist). - -## Boundary Commitments - -### This Spec Owns -- The client-side failover loop: classify status → reconnect/rotate/backoff → resume, including the - cluster-not-ready terminal mapping, for **reply-bearing** `xcecli_*` ops. -- Client-side master-list cursor and current-primary stickiness within a context. -- Capturing the redirect address from a reply as a best-effort hint (no correctness dependency). -- A **reconnect/rejoin hook**: a consumer-registered callback the failover machinery invokes after - it reconnects to a new primary and *before* it resumes the interrupted op. libxce owns the hook - mechanism, its ordering, and invoking it; it does **not** own the callback's content. -- A per-ctx concurrency lock so `xcecli_*` failover is safe when one ctx is shared across threads. -- The single controller change that populates `redirect_host/port` on `E_XCE_REDIRECT` replies. -- Realignment of the libxce retry unit tests to the real controller status taxonomy. - -### Out of Boundary -- How the controller decides primary/standby/redirect/no-primary (election, `ha_should_redirect`). -- The **content** of the rejoin callback: the join/state-exchange a daemon performs on reconnect - (owned by `component-failover-adoption`) and the worker's running-jobs re-sync (owned by - `worker-rejoin-resync`). This spec provides and invokes the hook; the daemons fill it in. -- Redelivery of fire-and-forget notifications lost across a failover (also `worker-rejoin-resync`). -- Worker/scheduler/apiserver adoption and their threading/liveness handling of bounded blocking. -- Any reply payload beyond the already-defined `redirect_host/redirect_port`. - -### Allowed Dependencies -- Existing controller status codes (`errnox.h:22-25`) and redirect reply fields (`typemap.h:225`). -- `endpoint_connect()` / endpoint I/O (`netio.*`), request encode/enqueue (`request.c`), and the - context wait channel (`cbuf`, `xcectx.c`). -- `xce_retry_config_t` + `XCE_RETRY_DEFAULTS` (request.h) and the cluster/peer config (`cluster.*`). - -### Revalidation Triggers -- Change to the reply `status`/`redirect_*` contract → re-check client classification + controller. -- Change to the async reply-dispatch model (`handle_reply` → `cbuf` signalling) → re-check round trip. -- Change to `api_ctx_t`/endpoint ownership → re-check reconnect-reuses-same-endpoint assumption. -- New non-primary status code added to the controller → extend client classification. -- Change to the reconnect/rejoin hook signature or its invocation ordering → re-check daemon rejoin - registration (`component-failover-adoption`) and worker resync (`worker-rejoin-resync`). - -## Architecture - -### Existing Architecture Analysis -- **Layering — `xcecli_*` is the universal surface.** Every consumer calls `client.c`'s `xcecli_*` - ops, never `xceapi.c`'s `xce_*` directly: worker (`controller_connector.c:184,272`), scheduler - (`scheduler.c:457,463`), apiserver via cgo (`clib.go:34,36`), `xce_test`. Each `xcecli_*` is a thin - wrapper that calls its `xce_*` counterpart, which **returns the controller status** as its rc. -- **Only `api_ctx_t` reaches the master list.** `api_ctx_t = { conf (masters), ep, ctx }` - (client.c:23). The inner `xce_ctx_t` (xceapi) holds `ep`/`wbuf`/`state` but **no** `conf` - (xcectx.h) — so failover belongs at the client layer, where the master list is already in scope. -- **`xce_*` fail fast; `xcecli_*` retries.** The async send/wait stays inside `xce_*` unchanged: a - single attempt that returns the controller status (or transport error). The `xcecli_*` wrapper - classifies that rc and, on a retriable status, reconnects/rotates `api_ctx.ep` over `api_ctx.conf` - and re-invokes `xce_*`. No async internals, thunks, or master-list plumbing are introduced. -- **Reconnect reuses the endpoint.** `endpoint_connect(ep,host,port)` re-points the same - `endpoint_t`, preserving the `msg_handler` wiring that routes replies to `ctx->wbuf`. -- **Stateful consumers need a rejoin on reconnect.** `xce_test` is a pure client — reconnect + - resend suffices. But the apiserver, worker, and scheduler hold a *session* with the primary - established via `xcecli_join` (role + attrs); a freshly elected primary has no live session for a - reconnecting daemon until it re-joins, and a worker must additionally re-sync its running-jobs - state. So failover for these consumers is **reconnect → rejoin/state-exchange → resume**, not just - reconnect → resend. libxce models this with a consumer-registered reconnect hook; the hook content - lives in `component-failover-adoption` (join) and `worker-rejoin-resync` (job state). -- **Two op shapes.** Reply-bearing ops (`xce_joblist`/`xce_jobsubmit`/`xce_hello`/…) do - `xce_*_req` + `cbuf_get` → return a controller status the loop can classify. Fire-and-forget ops - (`xce_notify*`, heartbeat) do `xce_*_req` only (no `cbuf_get`, xceapi.c:282) → return the enqueue - result, never a controller status. Failover classification applies only to the former. -- **Patterns preserved**: `xce_*` ops, `xce_*_req` enqueue wrappers, and `xce_request`/`xce_reply` - are all unchanged; failover is additive in `xcecli_*`. Trade-off: ~33 `xcecli_*` wrappers each gain - a small retry loop (more lines), but the policy lives in one shared decision helper (fewer moving - parts) — an explicit simplicity-over-LOC choice. - -### Architecture Pattern & Boundary Map - -```mermaid -graph LR - Caller[consumer worker scheduler apiserver xce_test] --> Cli[xcecli op in client.c] - Cli --> Loop[do while retry loop] - Loop --> Decide[xcecli_retry_next decision helper] - Loop --> Xce[xce op in xceapi one shot] - Xce --> Ctrl[controller] - Ctrl --> Reply[handle_reply] - Reply --> Capture[capture redirect hint into ctx] - Decide --> Rotate[reconnect or rotate over api_ctx conf and ep] - Decide --> Backoff[jittered backoff] -``` - -- **Selected pattern**: a thin retry loop in each `xcecli_*` wrapper, driven by one shared decision - helper (`xcecli_retry_next`). Rationale: `client.c` is the universal surface AND the only layer - whose `api_ctx_t` co-locates master list + endpoint + ctx; `xce_*` already returns the status. -- **Dependency direction**: `types → cluster/config → endpoint(netio) → xceapi(xce_*) → client(xcecli_*)`. - Each layer imports only leftward. Failover logic is added at the `client` layer (rightmost), - depending on `xce_*` and `cluster/config`; `xceapi` does not depend on it. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Client wrappers | C (gnu17), libxce `client.c` | `xcecli_*` retry loop + decision helper + rotate | Reuses `xce_*`, `endpoint_connect`, `cluster` | -| Client API | C (gnu17), libxce `xceapi.c` | **Unchanged** — one-shot `xce_*` returns status | No edits | -| Controller | C (gnu17) | Populate redirect address on `E_XCE_REDIRECT` | ~2 lines in `join.c`; fields already in schema | -| Tests | GoogleTest (testsuite) | Realign retry tests to real status taxonomy | Test the `xcecli_*` retry path | - -## File Structure Plan - -### New Files -``` -src/lib/client_failover.c # xcecli_retry_next decision helper, StatusClassifier, rotate/reconnect over api_ctx -src/lib/client_internal.h # private: api_ctx_t + xce_failover_state_t shared by client.c and client_failover.c -``` -> `api_ctx_t` is currently file-local to `client.c`; move its definition to `client_internal.h` so the -> failover helper can operate on it without exposing it in the public API. - -### Modified Files -- `src/lib/client.c` — wrap each **reply-bearing** `xcecli_*` op body in a `do { rc = - xce_<op>(...); } while (xcecli_retry_next(ac,&rc,is_write,&st));` loop (per OpCategory); - fire-and-forget ops get the transport-reconnect-only variant; add `xce_failover_state_t` to - `api_ctx_t` (lock + cursor + retry config + rejoin hook); add `xcecli_set_reconnect_handler()`; - init + take `failover.lock` per op; init in `xcecli_start()`; keep initial connect to `masters[0]`. -- `src/include/client.h` — declare `xcecli_set_reconnect_handler()` + `xcecli_reconnect_cb`. -- `src/lib/xcectx.c` — `handle_reply()` copies the redirect-hint address (`reply.redirect_host/port`, - when present) into `xce_ctx_t` before the reply is freed; the existing error path (`XCE_CTX_ERR` + - `wbuf` error inject, xcectx.c:137-154) is the wake hook reused on reconnect. -- `src/include/xcectx.h` — add the fixed-size redirect-hint `host/port` fields to `xce_ctx_t`. -- `src/include/request.h` / `src/lib/request.c` — keep `xce_retry_config_t` + `XCE_RETRY_DEFAULTS`; - **remove** the orphaned no-op `xce_request_with_retry()` (superseded by the `xcecli_*` loop). -- `src/lib/xceapi.c` — **unchanged** (one-shot `xce_*` ops already return the controller status). -- `src/controller/join.c` — populate `reply.redirect_host = c->ha.primary_host` / - `reply.redirect_port = c->ha.primary_port` on the `E_XCE_REDIRECT` path (audit for other sites). -- `src/lib/testsuite/test_libxce_retry.c` — retarget from the retired primitive to the `xcecli_*` - retry path + `StatusClassifier`; cover `E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`, - master rotation, and budget→not-ready. - -## System Flows - -### Failover loop (per `xcecli_*` op) - -```mermaid -sequenceDiagram - participant C as Caller - participant F as xcecli op do while loop - participant E as Endpoint - participant K as Controller - C->>F: xcecli op - loop until success, terminal, or budget exhausted - F->>E: xce op one shot - E->>K: request - K-->>E: reply status plus optional redirect addr - E-->>F: xce op returns status - alt status success - F-->>C: ok - else redirect or standby or transport error - F->>E: rotate (hint address first if set, else next master) then invoke rejoin hook - else no-primary or not-ready - F->>F: backoff jittered then rotate - else non-retriable - F-->>C: surface error - end - end - F-->>C: cluster-not-ready on budget exhaustion -``` - -After any reconnect to a new endpoint, the failover machinery invokes the consumer-registered -reconnect/rejoin hook (if set) **before** re-sending the interrupted op, so a stateful daemon -re-establishes its session (and a worker re-syncs job state) against the new primary. Pure clients -(`xce_test`) register no hook and skip straight to re-send. - -Key decisions: redirect/standby/transport rotate **without** the no-primary backoff delay (4.2); -only no-primary/not-ready consume the backoff budget (3.1, 4.1); on success the served master -becomes the sticky current primary (2.4). - -### Status classification - -```mermaid -stateDiagram-v2 - [*] --> Classify - Classify --> Success: status==0 or DUP on re-sent write - Classify --> RetryNow: REDIRECT or STANDBY or transport error - Classify --> RetryBackoff: NOMASTER or NOT_READY - Classify --> Terminal: AUTH PERM INVALID NOENT - RetryNow --> Rotate - RetryBackoff --> Rotate - Rotate --> Classify: hint first else next master - Success --> [*] - Terminal --> [*] -``` - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1, 1.3 | All ops through failover; preserve API | XcecliRetryLoop (all `xcecli_*`) | Failover loop | -| 1.2 | Complete next request against new primary | XcecliRetryLoop, EndpointRotator | Failover loop | -| 2.1 | Unreachable → try other masters | EndpointRotator | Classification | -| 2.2, 8.3, 8.4 | Redirect addr if present else rotate | RedirectCapture, EndpointRotator | Classification | -| 2.3, 4.3 | Standby → rotate | StatusClassifier, EndpointRotator | Classification | -| 2.4 | Sticky current primary | FailoverState | Failover loop | -| 2.5 | Redirect/standby/no-primary ≠ success | StatusClassifier | Classification | -| 3.1, 4.1 | No-primary → backoff to budget | XcecliRetryLoop, FailoverState | Failover loop | -| 3.2 | Success before budget | XcecliRetryLoop | Failover loop | -| 3.3, 3.4 | Budget exhausted → distinct cluster-not-ready | XcecliRetryLoop | Failover loop | -| 4.2 | Redirect → prompt switch, no backoff | EndpointRotator | Classification | -| 4.4 | Transport → reconnect/rotate | EndpointRotator, ReplyWaitWake | Classification | -| 4.5 | Non-retriable → surface | StatusClassifier | Classification | -| 1.2 | Re-establish session against new primary | ReconnectRejoinHook | Failover loop | -| 5.1, 5.3 | Reads+writes; no silent drop (reply-bearing) | XcecliRetryLoop, OpCategory | Failover loop | -| 5.2 | Write retry not double-applied | StatusClassifier (DUP→success) + controller idempotency | — | -| 6.1, 6.2, 6.3 | Configurable params, defaults, jitter | FailoverState, XcecliRetryLoop | Failover loop | -| 7.1, 7.2 | Log failover target + exhaustion | XcecliRetryLoop | — | -| 8.1, 8.2 | Controller populates redirect addr | ControllerRedirectAddress | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| XcecliRetryLoop | client (`client.c`) | `do/while` wrapper around each reply-bearing `xce_*` call | 1.1,1.2,3.x,5.x,7.x | xcecli_retry_next(P0) | Service | -| xcecli_retry_next | client (`client_failover.c`) | Decision helper: classify, rotate, rejoin, backoff, budget | 1.2,3.x,4.x,7.x | EndpointRotator(P0), StatusClassifier(P0), ReconnectRejoinHook(P0), FailoverState(P0) | Service | -| ReconnectRejoinHook | client (`client.c`) | Invoke consumer callback after reconnect, before resume | 1.2 | api_ctx_t(P0) | Service, State | -| OpCategory | client (`client.c`) | Per-op shape: reply-bearing / write / fire-and-forget | 1.1,5.1,5.2 | — | State | -| EndpointRotator | client (`client_failover.c`) | Reconnect to addr or next master over api_ctx | 2.1,2.2,2.3,4.2,4.4 | endpoint_connect(P0), FailoverState(P0) | Service | -| StatusClassifier | client (`client_failover.c`) | Map status → action | 2.5,4.1,4.2,4.3,4.5,5.2 | errnox codes(P0) | Service | -| FailoverState | client (`api_ctx_t`) | Master cursor, retry cfg, sticky primary | 2.4,6.1 | xce_client_config_t(P0) | State | -| RedirectCapture | ctx (`xcectx.c`) | Copy redirect-hint address into ctx (if present) | 2.2,8.3 | xce_reply_t(P0) | State | -| ReplyWaitWake | ctx (`xcectx.c`) | Release cbuf_get on endpoint error | 4.4 | endpoint err_handler(P0) | Service | -| ControllerRedirectAddress | controller | Fill redirect_host/port | 8.1,8.2 | ha_state primary_host/port(P0) | State | - -### client (libxce) - -#### XcecliRetryLoop — the wrapper pattern in each `xcecli_*` - -| Field | Detail | -|-------|--------| -| Intent | Turn each one-shot `xce_*` call into a transparent failover loop with no change to `xce_*` | -| Requirements | 1.1, 1.2, 3.1, 3.2, 3.3, 3.4, 5.1, 5.3, 7.1, 7.2 | - -**Responsibilities & Constraints** -- Each `xcecli_*` op runs its existing `xce_*` call inside a `do/while` driven by `xcecli_retry_next`. -- `xce_*` ops are unchanged: each is a single attempt that returns the controller status (fail fast). -- Never drops a request silently (5.3): the loop returns the op's success value or a distinct error. - -**Contracts**: Service [x] -```c -/* Pattern applied to every xcecli_* op (joblist shown; is_write=false for reads). */ -int xcecli_joblist(xcecli_ctx_t c, xce_job_t ***out) { - api_ctx_t *ac = (api_ctx_t *)c; - xcecli_retry_t st = XCECLI_RETRY_INIT(ac); /* attempt=0, delay/seed from ac->failover.retry */ - int rc; - do { - rc = xce_joblist(ac->ctx, out); /* unchanged one-shot xceapi op */ - } while (xcecli_retry_next(ac, &rc, /*is_write=*/false, &st)); - return rc; /* >=0 ok; -E_XCE_NOT_READY = cluster not ready */ -} -``` -- Postconditions: `>=0` success (sticky primary updated, 2.4); `-E_XCE_NOT_READY` on no-primary - budget exhaustion (3.3); terminal errors surfaced as-is (4.5). For writes, pass `is_write=true`. - -#### xcecli_retry_next — decision helper - -| Field | Detail | -|-------|--------| -| Intent | Decide whether to retry after an `xce_*` returned `rc`; perform reconnect/rotate/backoff | -| Requirements | 1.2, 3.1, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 5.2, 7.1, 7.2 | - -**Responsibilities & Constraints** -- Owns the retry budget, backoff progression, sticky-primary update, and cluster-not-ready mapping. -- After a transport error the existing path left `ctx->state = XCE_CTX_ERR`; on a successful reconnect - this helper restores `XCE_CTX_RUN` so the next `xce_*` passes its guard (CI2). -- Drains any stale entry from `ctx->wbuf` after reconnect, before the next attempt (CI1); reconnect - closes the old fd first, so the dead connection yields no further replies. - -**Contracts**: Service [x] -```c -typedef struct { int attempt; int delay_ms; unsigned int rand_seed; } xcecli_retry_t; - -/* Returns true → caller re-invokes its xce_* op; false → stop, *rc is final. - * On DUP write replay it sets *rc = 0 (success) and returns false (5.2). */ -bool xcecli_retry_next(api_ctx_t *ac, int *rc, bool is_write, xcecli_retry_t *st); -``` -- Decisions: SUCCESS/TERMINAL → return false; RETRY_NOW → `EndpointRotator` (hint-first) then true, - no backoff (4.2); RETRY_BACKOFF → jittered sleep then `EndpointRotator` then true (4.1); attempts - beyond `retry.max_retries` while no-primary → set `*rc = -E_XCE_NOT_READY`, return false (3.3, 3.4). -- Validation: INFO log target on rotation (7.1); WARN log attempt count on exhaustion (7.2). - -#### EndpointRotator - -| Field | Detail | -|-------|--------| -| Intent | Re-point `ac->ep` to a redirect address or the next master in `ac->conf` | -| Requirements | 2.1, 2.2, 2.3, 4.2, 4.4, 8.4 | - -**Responsibilities & Constraints** -- On retry, if a redirect-address **hint** was captured, try it first (then clear it); otherwise - advance the master cursor and reconnect to the next configured master (8.3, 8.4). A failed hint - just falls through to rotation on the next retry — correctness never depends on it. -- Reuse the same `endpoint_t` (reply routing to `ctx->wbuf` preserved); close stale fd/ssl first. - -**Contracts**: Service [x] -```c -/* Reconnect ac->ep to the redirect-hint address if one is set (consumes it), else advance to the - * next configured master in ac->conf and reconnect. Returns 0 or -errno. */ -int xce_failover_rotate(api_ctx_t *ac); -``` - -#### StatusClassifier - -| Field | Detail | -|-------|--------| -| Intent | Map a reply status / transport rc to a failover action | -| Requirements | 2.5, 4.1, 4.2, 4.3, 4.5, 5.2 | - -**Contracts**: Service [x] -```c -typedef enum { - XCE_FO_SUCCESS, /* status == 0, or idempotent-replay success (DUP on a re-sent write) */ - XCE_FO_RETRY_NOW, /* E_XCE_REDIRECT / E_XCE_STANDBY / transport error — rotate, no backoff */ - XCE_FO_RETRY_BACKOFF, /* E_XCE_NOMASTER / E_XCE_NOT_READY — rotate after jittered backoff */ - XCE_FO_TERMINAL /* AUTH/PERM/INVALID/NOENT and other non-retriable — surface */ -} xce_fo_action_t; - -/* attempt == 0 for the first send; >0 for a re-send after failover. - * is_write distinguishes mutating ops (job submit) from reads. - * The redirect ADDRESS is NOT a classifier concern — it is a hint consumed by EndpointRotator. */ -xce_fo_action_t xce_failover_classify(int status, bool is_write, int attempt); -``` -- Invariants: `E_XCE_REDIRECT`/`STANDBY`/`NOMASTER`/`NOT_READY` are never reported as success (2.5). -- **Idempotent-replay rule (5.2):** `-E_XCE_DUP_JOB` / `-EEXIST` on a **re-sent write** (`is_write && - attempt > 0`) maps to `XCE_FO_SUCCESS` — the controller's client-supplied-`jobid` replay guard - (`job.c:56,95`) means "already applied" = our write succeeded. On `attempt == 0` the same status - is a genuine duplicate and maps to `XCE_FO_TERMINAL` (surfaced unchanged). - -#### OpCategory — per-op shape (CI1) - -| Field | Detail | -|-------|--------| -| Intent | Classify each `xcecli_*` op so its wrapper applies the right failover behavior | -| Requirements | 1.1, 5.1, 5.2 | - -Three categories drive the wrapper generated for each of the ~33 ops: - -| Category | Ops (examples) | `xce_*` shape | Failover behavior | -|----------|----------------|---------------|-------------------| -| Reply-read | `joblist`, `reslist`, `workerlist`, `clusterconf`, `resval`, `hello`, `jobinfo` | `*_req` + `cbuf_get` → status | Full failover; `is_write=false` | -| Reply-write | `jobsubmit`, `jobctrl`, `queuectrl`, `hostctrl`, `update_resvals`, `runtask`, `join` | `*_req` + `cbuf_get` → status | Full failover; `is_write=true` (DUP-replay→success) | -| Fire-and-forget | `notify_*` (jobstart/finish/fail/stop/chkpt), `heartbeat`, `wakeup`, `nop` | `*_req` only, **no status** | Best-effort: reconnect on a known-dead connection only; **no** standby/redirect detection, **no** no-primary backoff. Redelivery deferred to `worker-rejoin-resync` | - -- The exact per-op assignment is enumerated during task generation; `join` is reply-write and is the - op that triggers `E_XCE_REDIRECT` at reconnect time (it carries the daemon's role/attrs). - -#### ReconnectRejoinHook - -| Field | Detail | -|-------|--------| -| Intent | Let a stateful consumer re-establish its session on the new primary after a reconnect, before the interrupted op resumes | -| Requirements | 1.2 | - -**Responsibilities & Constraints** -- libxce owns the hook *mechanism* and *ordering* (reconnect → hook → resume); the callback *content* - (join handshake, worker job re-sync) belongs to `component-failover-adoption` / `worker-rejoin-resync`. -- Invoked once per successful reconnect to a *new* endpoint, before the next `xce_*` attempt. If the - hook fails (e.g. the new primary rejects the join), the failover loop treats it like a failed - attempt (rotate/backoff per the returned status). Pure clients register no hook (no-op path). - -**Contracts**: Service [x] / State [x] -```c -/* Registered via the client API; cb runs on the calling thread after reconnect. - * Return 0 to proceed with the interrupted op; <0 to treat the rejoin as failed. */ -typedef int (*xcecli_reconnect_cb)(xcecli_ctx_t ctx, void *arg); -int xcecli_set_reconnect_handler(xcecli_ctx_t ctx, xcecli_reconnect_cb cb, void *arg); -``` - -### client / ctx (libxce) - -#### FailoverState (on `api_ctx_t`) - -**Contracts**: State [x] -```c -/* On api_ctx_t — persists across calls (sticky primary + tunables). */ -typedef struct { - mylock_t lock; /* serializes xcecli_* per ctx (CI2 — shared-ctx safety) */ - int cur_master; /* index into conf->masters[] (sticky primary) */ - xce_retry_config_t retry; /* defaults from XCE_RETRY_DEFAULTS unless overridden */ - xcecli_reconnect_cb rejoin_cb; /* consumer rejoin hook; NULL for pure clients */ - void *rejoin_arg; -} xce_failover_state_t; - -/* On xce_ctx_t — written by handle_reply (which only sees xce_ctx_t), read by the rotator via ac->ctx. */ -/* char redirect_hint_host[XCE_MAXHOSTNAME_LEN]; hint from last redirect reply; "" if none */ -/* uint16_t redirect_hint_port; */ -``` -- State model: `cur_master` updated on a successful served-as-primary reply (2.4); retry config from - `XCE_RETRY_DEFAULTS` unless overridden (6.1, 6.2). The redirect hint is read by the rotator on the - next retry and cleared on consumption; it is purely advisory. -- **Concurrency (CI2):** the apiserver shares one ctx across goroutines (a heartbeat goroutine plus - request handlers, `clib.go:182`). Each `xcecli_*` op takes `failover.lock` for the duration of its - `do/while` loop, so reconnect/rotate (which closes `ac->ep` and drains `wbuf`) cannot race another - in-flight call on the same ctx. The single-slot `wbuf` already implies one in-flight request per - ctx; the lock makes that explicit and covers the reconnect window. - -#### RedirectCapture (in `handle_reply`) - -**Contracts**: State [x] -- On a redirect reply that carries an address, copy `reply.redirect_host` (capnp-owned, freed with - the reply — see request.c:243-250 hazard) + `redirect_port` into the fixed-size `xce_ctx_t` hint - fields before the reply is freed; leave the hint empty otherwise. Because the hint is advisory, the - copy needs no sequence correlation or lifetime guarantees beyond "copy now, use-or-discard next - retry." The status continues through the existing `cbuf_tryput(&ctx->wbuf,&status)` path - (xcectx.c:303), so `xceapi.c` is unchanged. - -#### ReplyWaitWake (endpoint error path) - -**Contracts**: Service [x] -- On endpoint error/RDHUP for `ctx->ep`, release a pending `cbuf_get` with a transport error - (`-ECONNRESET`/`-ETIMEDOUT`) so the `xce_*` call returns and the loop rotates instead of hanging - (4.4, F7). The existing error path already injects an error into `wbuf` and sets `ctx->state = - XCE_CTX_ERR` (xcectx.c:148-154) — this component reuses it. -- **State reset (CI2):** that error path leaves `ctx->state = XCE_CTX_ERR`, which the per-op guard - rejects as `-EINVAL` (xceapi.c:14,38,…) — the reported bug. `xcecli_retry_next` restores - `ctx->state = XCE_CTX_RUN` after a successful reconnect, before the next `xce_*` attempt, so the - guard (re-checked per attempt) passes. **CI1 drain:** the helper also drains any stale `wbuf` entry - after reconnect so a late reply from the dead connection is never consumed by the next attempt. - -### controller - -#### ControllerRedirectAddress (`join.c`) - -**Contracts**: State [x] -- On the `E_XCE_REDIRECT` path, set `reply.redirect_host = c->ha.primary_host` and - `reply.redirect_port = c->ha.primary_port` (8.1). When the primary is unknown, keep returning - `E_XCE_NOMASTER` rather than an addressless redirect (8.2 — already the case at join.c:270). -- Audit during implementation: confirm `join.c` is the sole `E_XCE_REDIRECT` emitter; apply the - same populate to any other site found. - -## Error Handling - -### Error Strategy -Classify every round-trip outcome (StatusClassifier) into success / rotate-now / backoff / terminal, -and bound the loop by the retry budget. Repair the context on transport failure rather than leaving -it in a non-RUN state (F7) so failover is repeatable. - -**Context state model (CI2):** the existing transport-error path sets `ctx->state = XCE_CTX_ERR` -(xcectx.c:148); the per-op `XCE_CTX_RUN` guard then rejects with `-EINVAL` (xceapi.c:14,…) — the -reported bug. `xcecli_retry_next` owns the transition back: after a successful reconnect it sets -`ctx->state = XCE_CTX_RUN` before the next `xce_*` attempt, so the per-op guard (re-checked each -attempt) passes. The guard otherwise stays exactly as today — `xceapi.c` is unchanged. - -**Stale-reply avoidance (CI1):** `ctx->wbuf` is a single-slot, status-only channel -(`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). Reconnect closes the old fd **first**, so the dead -connection produces no further replies; `xcecli_retry_next` then drains any entry already sitting in -`wbuf` before the next attempt. This drain-after-close is sufficient — no per-reply sequence -correlation is needed (a deliberate simplification). - -### Error Categories and Responses -- **Transport (refused/timeout/reset)**: reconnect/rotate, retry; wake any pending wait (4.4). -- **No primary (NOMASTER/NOT_READY)**: jittered backoff; on budget exhaustion return the distinct - `-E_XCE_NOT_READY` cluster-not-ready code (3.3, 3.4) — never `-EINVAL`. -- **Redirect/standby**: rotate (address-first) without backoff (4.2, 4.3). -- **Terminal (AUTH/PERM/INVALID/NOENT)**: surface immediately, unchanged (4.5). - -### Monitoring -INFO log on each rotation with the target endpoint (7.1); WARN log on budget exhaustion with the -attempt count and last status (7.2). Reuse existing libxce logging. - -## Performance & Scalability - -Rejoin-on-failover cost is **accepted as-is** (decision 2026-06-16): failover is a rare, exceptional -event, so the cost is not on any hot path, and *some* handshake is unavoidable — a freshly elected -primary has no live connection to a reconnecting daemon until it re-joins. Characteristics: - -- Scheduler/apiserver join carry trivial payloads (`NULL` attrs); worker join carries a small - `hostattrs_t`. All reuse the daemons' existing startup join path — no new heavy machinery. -- On rejoin, apiserver/scheduler receive **full** state (e.g. the full job list), not a delta; for a - large cluster with many workers this is sizable but infrequent. This is simpler and correct, and - still cheaper than a full-history replace with no snapshot (cf. IBM LSF). -- **Herd control**: the jittered backoff (6.3) staggers simultaneous reconnects after a failover. - -**Deferred optimization (YAGNI):** a lightweight "re-attach" (session-token hello letting the primary -skip re-pushing already-replicated state, or delta-only state transfer) is intentionally **not** -designed now — delta reconciliation is non-trivial and full-state-on-rejoin is acceptable. Revisit -only if failover-rejoin latency is *measured* as a problem at scale. Recorded in `roadmap.md` backlog. - -## Testing Strategy - -### Unit Tests (`test_libxce_retry.c`, realigned) -- `xce_failover_classify`: REDIRECT+addr→ROTATE_ADDR; REDIRECT/STANDBY no-addr→ROTATE; - NOMASTER/NOT_READY→BACKOFF; AUTH/PERM/INVALID/NOENT→TERMINAL; 0→SUCCESS (2.5, 4.1-4.5). -- `xcecli_retry_next` loop: NOMASTER until budget → returns `-E_XCE_NOT_READY` (3.3); becomes primary - mid-loop → success, sticky cursor advanced (2.4, 3.2); non-retriable → immediate surface (4.5). -- Stale-reply avoidance (CI1): after a reconnect (old fd closed first), a stale entry left in `wbuf` - is drained before the next attempt, so no prior-attempt status is consumed; no sequence matching (1.2, 5.3). -- Context state (CI2): after a transport error drives `XCE_CTX_ERR`, a successful reconnect restores - `XCE_CTX_RUN` and the retry completes — never surfaces `-EINVAL` (1.2, 3.2). -- Idempotent write (CI3): `DUP_JOB` on a re-sent `jobsub` (attempt>0) → success; on attempt 0 → - surfaced as duplicate (5.2). -- EndpointRotator: redirect address used first (8.3); absent → next master (8.4); rotation reuses - same endpoint object. -- RetryConfig: NULL cfg → defaults (6.2); jitter staggers concurrent retries (6.3, reuse 50-client). - -### Integration / E2E (3-node, `/xce-deploy-ha` + `xce_test`) -- Submit jobs, kill primary, `joblist`: returns cluster-not-ready during the election window, then - transparently succeeds against the new primary with no error (1.2, 3.2, 3.3) — the bug reproduction. -- Redirect path: contact a standby directly → client lands on primary in one hop via the populated - redirect address (8.1, 8.3). -- Write failover: `jobsub` across a failover → exactly one job created (5.1, 5.2 via controller idempotency). - -### Controller Unit Test -- `handle_join` redirect path populates `redirect_host/port` from `ha.primary_host/port` (8.1); - unknown primary still yields `E_XCE_NOMASTER` (8.2). diff --git a/.kiro/specs/libxce-failover/requirements.md b/.kiro/specs/libxce-failover/requirements.md deleted file mode 100644 index f59ba6fe..00000000 --- a/.kiro/specs/libxce-failover/requirements.md +++ /dev/null @@ -1,179 +0,0 @@ -# Requirements Document - -## Introduction - -This feature makes XCE controller primary failover transparent to client applications built on -libxce (the XCE client library), including the worker, scheduler, and API server daemons and -the `xce_test` REPL. - -Today, when the connected controller primary is killed in an HA cluster, libxce keeps using the -dead connection and returns an opaque invalid-argument error indefinitely: it never fails over -to the newly elected primary and never reports a clear "cluster not ready" during the election -window. The upstream requirement (`ha-config-bootstrap` Requirement 7) delivered only a -standalone retry primitive that no production request path invokes and that cannot discover a -new primary after the connected one dies. This spec completes that intent on the client side: -it routes all client requests through transparent retry/failover, rotates the configured master -list to locate the new primary, and surfaces a distinct cluster-not-ready error only when an -election outlasts the retry budget. - -## Boundary Context - -- **In scope**: client-side transparent failover for all libxce request types; discovery of the - new primary by rotating the configured master list, using the redirect address (when present) as - a best-effort hint to try first; bounded transparent retry that surfaces a distinct - cluster-not-ready error on budget exhaustion; failover for both read and write requests; - configurable retry/backoff parameters with safe defaults; operator-visible logging of failover - events; one small controller change so that redirect responses carry the primary's host and port - (the wire schema and reply struct already define these fields); end-to-end verification with the - `xce_test` client (the controller test harness, and the original reproduction case for this bug). -- **Out of scope**: controller-side election, log replication, commit barrier, and write - idempotency (consumed, not changed); introducing new probe or transport message types or new - reply fields (the redirect address fields already exist); per-request retry opt-in flags (writes - fail over by default); **adoption by the worker, scheduler, and apiserver daemons** — their - master-list configuration and liveness-safe handling of bounded blocking are owned by the - follow-on spec `component-failover-adoption` (depends on this spec). -- **Adjacent expectations**: libxce classifies controller responses by the distinct statuses the - controller actually emits — a *redirect* status (a primary exists but this node is a standby), a - *no-primary* status (this standby knows no primary yet), a *not-ready* status (the controller is - still initializing or electing), and a *standby* status (the operation is primary-only). With - the in-scope controller change, a redirect response carries the primary's host and port; libxce - treats this as a **best-effort hint** — it tries the hinted address first, and rotates its - configured master list when the hint is absent, stale, or unreachable. Correctness never depends - on the hint. libxce relies on controllers to deduplicate re-sent - writes through their commit-barrier idempotency guards, and on the configured master list (e.g. - `set masters host:port,host:port,host:port`) being the full set of controller candidates. -- **Stateful consumers**: a pure client (`xce_test`) needs only reconnect + resend. Stateful daemons - (apiserver, worker, scheduler) hold a session established via join; after a failover they must - re-establish it on the new primary (and a worker re-syncs its running jobs). libxce provides a - reconnect/rejoin hook for this; the hook *content* is owned by `component-failover-adoption` and - `worker-rejoin-resync`. Fire-and-forget notifications have no reply, so libxce cannot detect a - standby for them or guarantee delivery across a failover — redelivery is `worker-rejoin-resync`. - -## Requirements - -### Requirement 1: Transparent Failover for All Client Requests - -**Objective:** As a developer of a libxce-based daemon, I want every controller request to fail -over automatically inside the library, so that I do not write ad-hoc retry or reconnect loops in -caller code. - -#### Acceptance Criteria -1. The libxce client library shall route every controller request, both read/query and - mutating/write requests, through the transparent retry-and-failover path. -2. When controller failover occurs and a new primary becomes available, the libxce client - library shall complete the next request against the new primary without requiring any change - to caller code. -3. While failover handling is active, the libxce client library shall preserve the existing - public API call signatures and success/error return conventions so that existing callers - continue to compile and run unchanged. - -### Requirement 2: New-Primary Discovery Across the Configured Master List - -**Objective:** As a client application, I want libxce to find the new primary after the connected -one dies, so that my requests continue to reach an authoritative controller. - -#### Acceptance Criteria -1. If the connected controller becomes unreachable through connection refused, connection - timeout, or peer reset, the libxce client library shall attempt the other controllers in its - configured master list to locate the current primary. -2. When a contacted controller responds with a redirect status indicating that a primary exists - elsewhere, the libxce client library shall locate the primary by rotating its configured master - list, trying the redirect address (when the response carries one) first as a best-effort hint. -3. When a contacted controller responds that the requested operation is primary-only (a standby - status), the libxce client library shall treat it as a redirect and attempt the other - controllers in its configured master list. -4. When a controller accepts and serves the request as primary, the libxce client library shall - treat it as the current primary and direct subsequent requests to it. -5. The libxce client library shall not consider a request successful when it was answered by a - controller reporting a redirect, standby, no-primary, or not-ready status. - -### Requirement 3: Election-Window Behavior and Cluster-Not-Ready Reporting - -**Objective:** As a client application, I want short failovers hidden and longer outages reported -clearly, so that I can distinguish "retry later" from a real failure. - -#### Acceptance Criteria -1. While every reachable controller in the configured master list reports no primary exists (a - no-primary or not-ready status), the libxce client library shall retry across the configured - controllers with exponential backoff until the retry budget is exhausted. -2. When a primary becomes available before the retry budget is exhausted, the libxce client - library shall complete the request successfully and return no error to the caller. -3. If the retry budget is exhausted while no controller has become primary, the libxce client - library shall return a distinct cluster-not-ready error to the caller. -4. The cluster-not-ready error shall be distinguishable by the caller from transport connection - errors and from request-validation errors, so that a no-primary window is never reported as an - opaque invalid-argument error. - -### Requirement 4: Error Classification — Retriable vs. Terminal - -**Objective:** As a client application, I want transient controller errors retried and permanent -errors surfaced immediately, so that I neither hang on unrecoverable errors nor fail on recoverable ones. - -#### Acceptance Criteria -1. If a controller returns a no-primary or not-ready status, the libxce client library shall retry - the request with exponential backoff. -2. If a controller returns a redirect status (a primary exists elsewhere), the libxce client - library shall locate and switch to the primary per Requirement 2 and retry promptly, without - waiting for the full backoff delay used for the no-primary case. -3. If a controller returns a standby status (the operation is primary-only), the libxce client - library shall switch to another configured controller per Requirement 2 and retry. -4. If a transport-level connection refused, timeout, or peer reset occurs, the libxce client - library shall reconnect or rotate to another configured controller and retry. -5. If a controller returns a non-retriable error, specifically authentication failure, permission - denied, invalid request, or not found, the libxce client library shall surface that error to - the caller immediately without retry. - -### Requirement 5: Read and Write Failover Semantics - -**Objective:** As a client application, I want both queries and mutating operations to survive -failover without duplicating side effects, so that failover is safe as well as transparent. - -#### Acceptance Criteria -1. The libxce client library shall apply transparent failover to both read/query requests and - mutating/write requests, such as job submission. -2. When a write request is retried after a failover, the libxce client library shall not cause the - operation to be applied more than once, relying on the controller's idempotency handling for - re-sent writes. -3. While failing over, the libxce client library shall not silently drop a request; it shall - either complete the request against a primary or surface a distinct error to the caller. - -### Requirement 6: Configurable Retry Parameters and Thundering-Herd Avoidance - -**Objective:** As an operator, I want retry behavior tunable with safe defaults and protection -against synchronized retries, so that failover does not overwhelm a freshly elected primary. - -#### Acceptance Criteria -1. The libxce client library shall provide configurable retry parameters: maximum retries, - initial delay, maximum delay, and backoff multiplier. -2. Where no retry configuration is supplied, the libxce client library shall apply safe - production defaults. -3. When many clients fail over simultaneously, the libxce client library shall stagger their - retry attempts through randomized backoff so that they do not retry in lockstep against the new - primary. - -### Requirement 7: Failover Observability - -**Objective:** As an operator, I want failover activity visible in logs, so that I can confirm -and diagnose client failover during a controller outage. - -#### Acceptance Criteria -1. When the libxce client library fails over to a different controller, it shall log the failover - event including the target controller endpoint. -2. When the libxce client library returns a cluster-not-ready error after exhausting its retry - budget, it shall log the exhaustion including the number of attempts made. - -### Requirement 8: Redirect Responses Carry the Primary Address (used as a hint) - -**Objective:** As a client application, I want a standby's redirect to *hint* where the primary is, -so libxce can usually reach it in one hop while still working correctly without the hint. - -#### Acceptance Criteria -1. When a standby controller redirects a client because it knows a primary exists elsewhere, the - controller shall include the primary's host and port in the redirect response. -2. While a standby controller does not know the primary's address, the controller shall return a - no-primary status rather than a redirect that omits the address. -3. When the libxce client library receives a redirect response that includes a primary address, it - shall try that address first as a best-effort hint. -4. If the redirect carries no address, or the hinted address is unreachable or not the primary, the - libxce client library shall fall back to rotating its configured master list; correctness shall - not depend on the hint. diff --git a/.kiro/specs/libxce-failover/research.md b/.kiro/specs/libxce-failover/research.md deleted file mode 100644 index f9882d89..00000000 --- a/.kiro/specs/libxce-failover/research.md +++ /dev/null @@ -1,248 +0,0 @@ -# Research Log — libxce-failover - -## Discovery Scope - -Light, integration-focused discovery (extension of existing libxce). Goal: find the exact -seams to wire transparent failover into the client request/reply path. Verified by reading the -endpoint, client-config, request-core, reply, and controller-redirect code paths. - -## Key Findings - -### F1 — The async request/reply model determines the chokepoint (DECISIVE) -`xce_request(ep, req)` (request.c:317) only **encodes + enqueues** a packet — fire-and-forget. -The controller's reply (and its `status`) is delivered asynchronously: the endpoint's -`msg_handler` runs `handle_reply()` (xcectx.c:30) which signals the caller's wait channel, and -public ops block on `cbuf_get(&ctx->wbuf, &status)` (xceapi.c:116). Therefore the controller -status (`E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`) is observable only at the -**send-and-wait round trip**, not at `xce_request`. Failover orchestration must wrap that round -trip, where the `api_ctx_t` is in scope. - -### F2 — `xce_request_with_retry()` production branch is a no-op stub -request.c:1972-1981: with no test hook installed, it does `xce_request(ep,req); return 0;`. The -entire retry/classify/backoff loop executes only under `xce_retry_send_hook` (tests). So the -"done" primitive cannot fail over against a real controller. Its hook contract -(`int hook(ep, req, reply_out)`) fakes a synchronous reply that production never produces. - -### F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced -- Wire schema has the fields: `controller.capnp:568-569` (`redirectHost`/`redirectPort`). -- Reply struct has them: `typemap.h:225-226` (`char *redirect_host; uint16_t redirect_port;`). -- Controller never sets them: every redirect site (`join.c:264`) sends `E_XCE_REDIRECT` with - `data_which = nodata`; grep finds **zero** production assignments to `reply.redirect_*`. -- Client never surfaces them: `handle_reply()` (xcectx.c:34) returns only `reply.status`; the - redirect fields are dropped before any caller sees them. - -### F4 — Context already holds everything failover needs -`api_ctx_t` (client.c:23): `{ xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }`. -`xce_client_config_t` (client.h:16): `masters[N_MAX_MASTERS]` + `master_ports[]` + `nmasters`. -So the master list, the endpoint to reconnect, and the reply channel are all reachable from one -context object. Missing: a current-master cursor, the retry config, and captured redirect target. - -### F5 — Endpoint reconnect is straightforward and reusable -`endpoint_connect(ep, host, port)` (netio.c:659) does socket connect + SSL handshake on the -**same** `endpoint_t`. Reconnecting to a different master reuses the same endpoint object, so the -`msg_handler`/`owner` wiring that routes replies to `ctx->wbuf` stays intact. - -### F6 — `xcecli_start()` only ever connects `masters[0]` (client.c:122) -No primary discovery at startup; single endpoint, never re-selected. Confirms the gap. - -### F7 — Original EINVAL source -`xce_get_with_condition()` returns `-EINVAL` when `ctx->state != XCE_CTX_RUN` (xceapi.c:104). -After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent -call short-circuits to `-EINVAL` — the opaque error in the bug report. Failover must repair the -endpoint/ctx state instead of leaving it dead. - -## Decisions (synthesis) - -- **D1 (final — client-layer, supersedes the earlier xceapi round-trip idea):** failover lives in - the **`client.c` `xcecli_*` layer**, not at `xce_request` and not at `xceapi.c`. Evidence: every - consumer (worker `controller_connector.c`, scheduler `scheduler.c`, apiserver `clib.go`, xce_test) - calls `xcecli_*`, never `xce_*` directly; and only `api_ctx_t` (client.c) holds the master list — - `xce_ctx_t` (xceapi) does not. So `client.c` is both the universal surface and the only layer with - conf+ep+ctx co-located. **`xce_*` stay one-shot/fail-fast and unchanged**; each `xcecli_*` wraps - its `xce_*` call in a `do/while` driven by a shared `xcecli_retry_next` helper - (`src/lib/client_failover.c` + `client_internal.h`). Trade-off accepted (user): ~33 wrappers gain - a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler - and lower-risk. The orphaned `xce_request_with_retry()` is removed, not wired. -- **D2:** retire the no-op production branch of `xce_request_with_retry()`. Keep `xce_retry_config_t` - + `XCE_RETRY_DEFAULTS`. Realign the unit tests (and hook contract) to the real status taxonomy - and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608). -- **D3:** `handle_reply()` captures `reply.redirect_host/port` + status into the ctx (deep-copy the - host string before the decoded reply is freed) so the failover loop can prefer the address. -- **D4:** on endpoint error/RDHUP, the err path must release the pending `cbuf_get` with a - transport error so the loop reconnects instead of hanging. -- **D5:** controller populates `reply.redirect_host/port` from `ha.primary_host/port` at every - `E_XCE_REDIRECT` site (join.c confirmed sole site; audit during implementation). -- **D6:** budget-exhaustion mapping — no-primary cause → return distinct `-E_XCE_NOT_READY` - ("cluster not ready"); transport-only cause → return last transport error. Never `-EINVAL`. - -### Decisions added after design review (validate-design pass) -- **D7 (CI1 — reply correlation):** `ctx->wbuf` is single-slot, status-only - (`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). `handle_reply` records `reply.seq` into the ctx; - `xcecli_retry_next` drains any stale `wbuf` entry after reconnect (and reconnect closes the old fd - first → no further old replies), so the next `xce_*` attempt cannot consume a prior-attempt reply. -- **D8 (CI2 — context state reset):** the existing transport-error path sets `ctx->state = - XCE_CTX_ERR` (xcectx.c:148), which the per-op guard rejects as `-EINVAL` (xceapi.c:14,…) — the - reported bug. `xcecli_retry_next` restores `XCE_CTX_RUN` after a successful reconnect, before the - next `xce_*` attempt, so the unchanged per-op guard passes. `xceapi.c` is not modified. -### Simplification pass (goal: keep libxce easy to use) -- **D13 (redirect = hint, not contract):** the controller still populates the redirect address - (forward-compatible, kept), but the client treats it as a **best-effort hint** — try it first if - present, else rotate the master list; correctness never depends on it. This removes the heavy - machinery the address-as-contract demanded: **drop the `reply_seq` correlation** (CI1 becomes - drain-after-close only — reconnect closes the old fd first, then drain `wbuf`), and **collapse the - classifier from 5 actions to 4** (`SUCCESS` / `RETRY_NOW` / `RETRY_BACKOFF` / `TERMINAL`) — the - hint is consumed by `EndpointRotator`, not a separate classifier branch. Net: fewer fields, no - lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook - and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is - cheap; the hint is a latency nicety, not a correctness mechanism. - -### Decisions added after 2nd design review (validate-design pass 2) -- **D10 (CI1 — op categories):** the ~33 `xcecli_*` ops split into reply-read (full failover, - `is_write=false`), reply-write (full failover, `is_write=true`, DUP-replay→success), and - fire-and-forget (`notify_*`/`heartbeat`/`wakeup` — `xce_*_req` only, no `cbuf_get`/status). The - failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect - best-effort, and redelivery of lost notifies is deferred to `worker-rejoin-resync`. -- **D11 (CI2 — concurrency):** apiserver shares one ctx across goroutines (`clib.go:182`); add a - `failover.lock` taken per `xcecli_*` op so reconnect/rotate cannot race another in-flight call. -- **D12 (rejoin hook — user requirement):** stateful daemons (apiserver/worker/scheduler) need a - rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds - `xcecli_set_reconnect_handler()`; the failover loop invokes it after reconnect, before resume. The - hook *content* is owned by `component-failover-adoption` (join) and `worker-rejoin-resync` (job - state). Pure clients (`xce_test`) register none. - -- **D9 (CI3 — idempotent-write replay):** job submit carries a **client-generated** `jobid` - (`job.c:56`) with a server `-EEXIST`/`E_XCE_DUP_JOB` replay guard (`job.c:95`). The classifier maps - `DUP_JOB` on a re-sent write (`is_write && attempt>0`) to success; on attempt 0 it is a genuine - duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed. - -## Risks - -- **R-async-blocking:** the round trip blocks longer under retry. Acceptable for CLI/apiserver - (already blocking); for async C daemons it's the `component-failover-adoption` spec's concern - (out of boundary here). -- **R-reply-routing on reconnect:** re-sending after reconnect must correlate the reply to the - same ctx wait channel; mitigated by reusing the same `endpoint_t` (F5) and matching on `seq`. -- **R-redirect-lifetime:** `redirect_host` is a decoded capnp string freed with the reply; must be - copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the `init_reply_by_req` - memset hazard already documented at request.c:243-250). - ---- - -## Gap Re-Audit (2026-06-18) - -**Method:** Forward-looking requirements-vs-codebase verification. Each requirement traced to a -real symbol/file:line in `src/lib`, `src/include`, `src/controller`. Every `[x]` task confirmed -against actual code (not just the checkbox). The only genuinely-open leaf task (6.2) re-scoped for -the remaining design/exec work. - -### Verdict - -**STATUS: PARTIAL — code-complete, live-E2E-incomplete.** All implementation tasks (1.1–6.1 + 5.3) -are present and wired in real code; the sole open leaf is **6.2 (live 3-node failover E2E)**, which -Implementation Notes flag as *partial* (server-side failover proven + 2 live bugs found/fixed, but -clean within-session client transparent-recovery never visually demonstrated in the sandbox env). -The 6 unchecked `[ ]` items besides 6.2 are **phase-group headers** (tasks 1/2/3/4/5/6), not work -units — all their leaf sub-tasks are `[x]` and verified below. - -### Requirement-to-Asset Map - -| Req | Summary | Tag | Evidence (file:line) | -|-----|---------|-----|----------------------| -| 1.1 | All ops (read+write) through failover | **MET** | `client.c` — 21× `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL`; macro in `client_internal.h:152` | -| 1.2 | Next request completes on new primary | **MET** (unit) / **Unknown** (live) | `client_failover.c:xcecli_retry_next` + `xce_ctx_reconnect` (`xcectx.c:320`); live not visually confirmed (6.2) | -| 1.3 | Public API signatures unchanged | **MET** | Notes 5.1 "zero signature changes"; macro wraps op bodies, no proto edits | -| 2.1 | Unreachable → try other masters | **MET** | `xce_failover_next_target` (cursor advance+wrap over `conf->masters[]`), `client_failover.c` | -| 2.2 | Redirect → rotate, hint-first | **MET** | hint-first/consume then cursor; `xce_failover_rotate` | -| 2.3 | Standby → treat as redirect/rotate | **MET** | classifier `client_failover.c:59` (`E_XCE_STANDBY`→retry-now) | -| 2.4 | Sticky current primary | **MET** | `cur_master` in `xce_failover_state_t` (`client_internal.h`) | -| 2.5 | Redirect/standby/no-primary ≠ success | **MET** | classifier never maps these to SUCCESS; unit `test_libxce_retry.c` | -| 3.1 | No-primary → backoff to budget | **MET** | `client_failover.c:64` (NOMASTER/NOT_READY→backoff) | -| 3.2 | Success before budget | **MET** (unit) | `test_libxce_retry.c` become-primary-mid-loop | -| 3.3/3.4 | Budget exhausted → distinct cluster-not-ready (not EINVAL) | **MET** | `*rc = -E_XCE_NOT_READY` (`client_failover.c:282,317`) | -| 4.1–4.5 | Retriable vs terminal classification | **MET** | `xce_failover_classify`, 11-row taxonomy in `failover_harness.h`; tests pass | -| 5.1 | Reads+writes failover | **MET** | OpCategory: writes=true for jobsub/jobctrl/join/… (Notes 5.1) | -| 5.2 | Write retry not double-applied (idempotent DUP→success) | **MET** (client) / **Constraint** | classifier DUP-on-resent-write→success; relies on controller `job.c:56,95` jobid replay guard | -| 5.3 | No silent drop | **MET** | loop returns success or distinct error | -| 6.1/6.2 | Configurable params + safe defaults | **MET** | `xce_retry_config_t` + `XCE_RETRY_DEFAULTS` (`request.h:183-193`, max_retries=20) | -| 6.3 | Jitter / anti-thundering-herd | **MET** | `test_decide_jitter_staggers_many_clients` (64-carrier, Notes 6.1) | -| 7.1/7.2 | Log rotation target + exhaustion count | **MET** | `tlog_info`/`tlog_warn` in `xcecli_retry_next` (Notes 3.1) | -| 8.1 | Redirect carries primary host/port | **MET** | `fill_redirect_reply` (`join.c:28-31`) ← `ha.primary_host/port` (`ha.h:149-150`) | -| 8.2 | Unknown primary → no-primary, not addressless redirect | **MET** | `join.c:289` `-E_XCE_NOMASTER` path unchanged | -| 8.3/8.4 | Client uses hint best-effort, falls back to rotation | **MET** | `xce_ctx_capture_redirect_hint` (`xcectx.c:30,57`); rotate consumes-then-falls-through | - -**Orphan cleanup (task 1.1):** confirmed removed — `xce_request_with_retry()` gone, only a -tombstone comment remains (`request.c:1927`). -**Controller audit (task 5.3):** `join.c` confirmed sole `E_XCE_REDIRECT` emitter; controller test -`test_join_redirect.c` registered (`controller_test.c:155`, `meson.build:97`). - -Tally: **24/24 acceptance criteria MET in code** (1.2 live-unconfirmed; 5.2 carries one external -Constraint = controller idempotency, which is in-scope-consumed not in-scope-built). - -### Open Tasks — Forward Gap Analysis - -Only one leaf task is open. The remaining `[ ]` are phase headers that auto-close when their -sub-tasks are `[x]` (all are) — recommend marking 1/2/3/4/5 complete; keep 6 open until 6.2 lands. - -#### Task 6.2 — Live 3-node failover end-to-end *(the real PENDING work)* - -**What is missing (vs. what is done):** -- DONE in-env: 3-node `deploy-ha.sh` cluster elected a primary; killing it triggered server-side - failover (ctrl-2 → term=2); the run REPRODUCED the original `Invalid argument` bug and the failover - loop now ENGAGES (`xcecli_joblist`→`xcecli_retry_next`→`xce_failover_rotate`). Two live-only bugs - found+fixed with regression unit tests: **bug-609** (fresh-call-after-disconnect `-EINVAL` - classified terminal → known-dead-ctx override in `decide`) and **bug-610** (`xce_failover_rotate` - SEGV — reconnect never re-bound `ep->iomgr`/handlers; fixed via new `xce_ctx_reconnect()`). -- MISSING: a clean, *visually-captured* within-session demonstration of the three acceptance - outcomes against a stable runtime: (a) `joblist` reports cluster-not-ready during the election - window then transparently recovers with NO `EINVAL`; (b) a request to a standby lands on the - primary in one hop via the populated redirect hint (and via rotation when hint absent); (c) a - `jobsub` spanning the failover creates **exactly one** job (idempotency end-to-end). - -**Integration points to exercise:** `/xce-deploy-ha localhost:9991 9992 9993` → `xce_test` -(`set masters`, `client`, `jobsub`, `joblist`) → kill primary → re-`joblist`. NOTE the build hazard -recorded twice in Notes: **building `lib` removes the controller binary from the shared build dir** — -must `xce-build.sh controller` (relinks `xce_test`) before each deploy. - -**Options:** -- **Option A (recommended): stable-env manual E2E.** Run the reproduction in a non-sandbox env where - backgrounded controller procs survive between steps (the sandbox reaps them). Capture the 3 outcomes. - *Effort S–M, Risk Low* — code already proven to engage; this is confirmation + evidence capture. -- **Option B: scripted deterministic E2E harness** (a `testsuite` shell/expect script that boots the - cluster, kills the primary on a signal, asserts on `xce_test` output). *Effort M, Risk Med* — gives - a repeatable regression gate but adds harness maintenance; process-lifecycle flakiness is the risk. -- **Option C: in-process integration test** simulating the controller side with fakes (no real procs). - *Effort M–L, Risk Med* — avoids deploy flakiness but re-tests what unit tests already cover (low - marginal value vs. real-cluster confidence); does NOT satisfy 6.2's "original reproduction" intent. - -**Recommendation:** Option A to *close* 6.2 (intent = the original bug reproduction passes live), -optionally backstopped by Option B as a CI regression gate in a follow-on. - -#### Phase headers 1–5 (and 6) — administrative - -*Gap:* checkbox-only. *Action:* flip 1/2/3/4/5 to `[x]` (all sub-tasks verified `[x]` above); leave -6 `[ ]` until 6.2 closes. *Effort S, Risk None.* - -### Research-Needed (for any remaining design/exec) - -1. **Stable runtime for live E2E** — identify an env where backgrounded `xce_controller` procs - persist across orchestration steps (sandbox reaps them; nix runtime env needed). Blocks 6.2-A. -2. **Build-order guard** — codify "rebuild controller after any `lib` build" (Notes 3.1/6.2 twice). - Consider a `xce-build.sh` ordering note or a make dependency so `xce_test` is never stale-linked. -3. **Idempotency end-to-end proof (5.2)** — confirm the controller's client-`jobid` replay guard - (`job.c:56,95`) actually returns `E_XCE_DUP_JOB`/`-EEXIST` on the *re-sent* write across a real - failover (unit-mocked today; live single-job-created assertion is the open evidence). -4. **Downstream readiness (out-of-boundary, flag only)** — `component-failover-adoption` (HARD dep) - still owes: per-daemon **full master-list** wiring (today even CLI uses `masters[0]`, - `client.c:131`) + **liveness-safe** bounded-blocking verification for async C daemons (worker - `controller_connector.c`, scheduler) so backoff never stalls the heartbeat/iomgr thread. Not 6.2's - scope, but it is the gate before this failover mechanism delivers value to real daemons. - -### Integration Risks (remaining) - -- **Process-lifecycle flakiness** in any live/scripted E2E (controller procs, port reuse, election - timing) — the dominant risk for 6.2; mitigated by Option A in a stable env + jittered election. -- **Stale `xce_test` link** if controller isn't rebuilt after a `lib` build — would mask a real fix - or surface a phantom one. Mitigation = Research-Needed #2. -- **Idempotency dependency** (5.2) is external (controller `job.c`) — a regression there breaks - exactly-once across failover without touching libxce. Mitigation = Research-Needed #3. - diff --git a/.kiro/specs/libxce-failover/spec.json b/.kiro/specs/libxce-failover/spec.json deleted file mode 100644 index d03d414b..00000000 --- a/.kiro/specs/libxce-failover/spec.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "feature_name": "libxce-failover", - "created_at": "2026-06-16T10:30:55Z", - "updated_at": "2026-07-18T00:35:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T00:35:00Z", - "note": "First post-implementation validation run -- all 19 tasks were already [x] since 2026-06-18 with ready_for_implementation still false and no blocked_on/explanation; confirmed this was purely an administrative oversight (the flag was never flipped after completion), not a real blocker or supersession. Git log shows 11 dedicated feature commits (foundation through bug-1436 count-aware macro fix) forming the foundational client-failover layer that later HA specs (ha-term-safety, raft-library, controller-status, etc.) build on top of, not a duplicate of them. Full libxce test suite green (123 tests total, 34 libxce-failover-specific: StatusClassifier, EndpointRotator, retry-decision budget/backoff/jitter, ctx-recovery, rejoin-hook, redirect-hint capture, cross-component integration), controller suite green (3/3), build clean (192 targets, zero errors), 8/8 requirement sections mapped to completed tasks, design.md's 5-component architecture (StatusClassifier, EndpointRotator, retry-decision core, ctx-recovery, rejoin-hook) verified wired end-to-end via call-graph trace, all 18 reply-bearing + 3 fire-and-forget libxce ops confirmed wrapped in the failover macros. Live E2E already recorded 2026-07-02 (3-node real-fs cluster, zero job loss). No LOCAL bugs found during this pass; bug-609 (known-dead ctx EINVAL), bug-610 (endpoint_connect SEGV on reconnect), bug-854 (dead-first-master initial discovery), and bug-1436 (row-count misclassified as transport error) were all found and fixed in prior follow-up commits, already present in the validated build.", - "known_limitations": [ - "Per-daemon adoption (worker, scheduler, apiserver master-list config and liveness-safe bounded blocking) is explicitly out of scope by design -- owned by the follow-on specs component-failover-adoption and worker-rejoin-resync, both of which depend on this spec.", - "Fire-and-forget notifications (XCECLI_FF_CALL: notify/heartbeat/nop) have no reply and get only a single-shot best-effort reconnect attempt, not the full retry/backoff loop -- a fact genuinely lost in a failover window is accepted-by-design here; redelivery is owned by worker-rejoin-resync." - ] - } -} diff --git a/.kiro/specs/libxce-failover/tasks.md b/.kiro/specs/libxce-failover/tasks.md deleted file mode 100644 index 3454d0e3..00000000 --- a/.kiro/specs/libxce-failover/tasks.md +++ /dev/null @@ -1,118 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: failover state, captured-reply fields, and cleanup -- [x] 1.1 Establish shared failover state and reply-capture fields - - Expose the client context to a new client-internal header so the failover helper can operate on it - - Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot - - Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation) - - Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive - - Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable - - _Requirements: 6.1, 6.2_ -- [x] 1.2 Realign the retry test harness to the new model (RED) - - Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path - - Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness - - Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist - - _Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4_ - -- [x] 2. Core: status classification, endpoint rotation, reply-path capture -- [x] 2.1 Implement status classification - - Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal - - Never classify redirect, standby, no-primary, or not-ready as success - - Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller - - Done: classification unit tests pass for every status and for both duplicate-replay cases - - _Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2_ - - _Boundary: StatusClassifier_ -- [x] 2.2 Implement endpoint reconnect and master rotation - - On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect - - Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master - - Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint - - _Requirements: 2.1, 2.2, 2.3, 4.4, 8.4_ - - _Boundary: EndpointRotator_ - - _Depends: 1.1_ -- [x] 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error - - In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation) - - On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging - - Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged - - Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched - - _Requirements: 2.2, 4.4, 8.3_ - - _Boundary: RedirectCapture, ReplyWaitWake_ - - _Depends: 1.1_ - -- [x] 3. Core: the retry/failover decision helper -- [x] 3.1 Implement the decision loop (classify, rotate, backoff, budget) - - Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff - - Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code - - Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion - - Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff - - _Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2_ - - _Depends: 2.1, 2.2, 2.3_ - - _Boundary: xcecli_retry_next_ -- [x] 3.2 Implement reconnect recovery: context-state reset and stale-reply drain - - After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error - - Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply - - Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned - - _Requirements: 1.2, 4.4_ - - _Depends: 3.1_ - - _Boundary: xcecli_retry_next_ - -- [x] 4. Core: reconnect/rejoin hook -- [x] 4.1 Add the reconnect/rejoin hook mechanism - - Provide a client API to register a reconnect callback and argument, stored in the failover state - - Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result - - Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly - - _Requirements: 1.2_ - - _Depends: 3.1, 3.2_ - -- [x] 5. Integration: wrap the client ops; controller redirect address -- [x] 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock - - Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration - - Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention - - Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized - - _Requirements: 1.1, 1.3, 2.4, 5.1, 5.3_ - - _Depends: 3.1, 3.2, 4.1_ - - _Boundary: XcecliRetryLoop, FailoverState_ -- [x] 5.2 Apply best-effort reconnect to fire-and-forget ops - - Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff - - Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope - - _Requirements: 1.1, 5.3_ - - _Depends: 3.1_ - - _Boundary: XcecliRetryLoop_ -- [x] 5.3 (P) Populate the primary address in controller redirect replies - - On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown - - Audit for any other redirect-emitting sites and apply the same population - - Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status - - Done: the controller test passes and a standby redirect reply carries the primary address - - _Requirements: 8.1, 8.2_ - - _Boundary: ControllerRedirectAddress_ - -- [x] 6. Validation: unit suite and live failover -- [x] 6.1 Bring the realigned unit suite green - - Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use - - Done: the full libxce retry unit suite passes under the project's test and ASAN build - - _Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4_ - - _Depends: 5.1, 5.2_ -- [x] 6.2 Live 3-node failover end-to-end <!-- VERIFIED 2026-07-02 on ~/tmp (ext4) — see Implementation Notes --> - - On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary - - Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation - - Verify a job submission spanning a failover creates exactly one job - - Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created - - _Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3_ - - _Depends: 5.1, 5.3_ - -## Implementation Notes -- 6.2 VERIFIED 2026-07-02: deployed a 3-node cluster to `~/tmp/xce-failover-e2e-deploy` (real ext4). Raced a `jobsub` call (masters=all 3) against a background `kill -9` of the primary (ctrl-1, term 2) fired 0.4s in — the call spanned the primary's death and returned `jobsubmit ok` with no error surfaced to the caller; ctrl-3 was promoted (term 2→4). Post-failover `expect-jobs 1` confirmed exactly ONE job exists — no duplicate from the in-flight retry. Standby-lands-on-primary verified via master-list rotation (masters=all 3 transparently found the new primary via `wait-primary`); the redirect-hint path (`E_XCE_REDIRECT` w/ address) is specific to `OPCode_join` (join.c `fill_redirect_reply`) and not exercised by job ops, which instead get `E_XCE_STANDBY` (classified `XCE_FO_RETRY_NOW`, rotates via master list) — confirmed by a control probe: pointing masters at ONLY a standby with no other candidates correctly surfaces the standby error (nowhere to rotate to), matching the classifier's documented Invariant 2.5. PASS on all 3 Observable bullets and the Done criteria. - -## Implementation Notes -- 1.1: design said `XCE_MAXHOST_LEN`; real constant is `XCE_MAXHOSTNAME_LEN` (config.h, =256) — design corrected. `api_ctx_t` now in private `client_internal.h`; `xcecli_reconnect_cb` typedef lives there until task 4.1 adds the public setter. Build: `cd src && nix develop --command bash tools/xce-build.sh lib`. -- 1.2 (ADAPTED): RED-against-undefined-symbols would break linking of the shared lib test binary across tasks, so 1.2 builds a GREEN reusable harness (`src/lib/testsuite/failover_harness.h`): fake multi-master config+cursor, an 11-row controller-status taxonomy table (resolves to real errnox.h codes), and an N-pthread runner. Component RED→GREEN tests live in 2.1/2.2/2.3/3.1/3.2. The harness defines `fo_expected_action_t` (FO_ACT_*) mirroring the design `xce_fo_action_t` — 2.1 asserts the real classifier against these rows. -- 2.2: EndpointRotator `xce_failover_next_target` (pure, socket-free: hint-first/consume, else cursor advance+wrap over conf->masters[]) + thin `xce_failover_rotate` (close-old/reconnect same endpoint). Live connect path is E2E (6.2). LAYERING: functions taking the private `api_ctx_t` are declared in `src/lib/client_internal.h`, NOT the public `xce/client_failover.h` (which keeps only the pure int-typed classifier) — per src/lib/CLAUDE.md (public headers in src/include/xce/ must not pull private state). -- 2.3: RedirectCapture via non-static seam `xce_ctx_capture_redirect_hint()` called in handle_reply BEFORE the capnp free AND before the `status<0` early return (so negative E_XCE_REDIRECT still captures). Bounded strncpy into redirect_hint_host. ReplyWaitWake needed NO production change — existing `handle_disconnect` (err_handler, xcectx.c) already injects -ECONNRESET into wbuf + sets XCE_CTX_ERR; full err_handler path is covered by E2E (6.2). -- 3.1: pure `xcecli_retry_decide` (no I/O — classify+budget+jittered-backoff calc, D6 exhaustion: no-primary→-E_XCE_NOT_READY, transport→keep last rc) + thin `xcecli_retry_next` (decide + RETRY_BACKOFF nanosleep + xce_failover_rotate + tlog_info/tlog_warn). Logger = utils.h tlog_* macros. State-reset/drain deferred to 3.2; rejoin hook to 4.1. -- 3.2: `xcecli_ctx_recover` (socket-free): flips ac->ctx.state ERR→RUN ONLY, drains stale wbuf via non-blocking cbuf_tryget (-EAGAIN on empty); wired into xcecli_retry_next after successful rotate, before return true. THIS is the -EINVAL bug fix. Added missing prototype `extern int cbuf_tryget(cbuf_t*,void*)` to mylock.h (function already defined mylock.c:166, just undeclared). -- 4.1: public `xcecli_reconnect_cb` typedef + `xcecli_set_reconnect_handler` (client.h/client.c, stores cb/arg in failover); socket-free `xcecli_invoke_rejoin` wired into xcecli_retry_next AFTER rotate+recover, BEFORE resume. hook<0 = failed attempt → bounded tail-recursion (consumes attempt budget, capped at max_retries). typedef MOVED from client_internal.h to client.h. API additive. -- 5.1: `XCECLI_FAILOVER_CALL(ac,is_write,EXPR)` macro (client_internal.h) = lock(failover.lock) → retry init → do{rc=EXPR}while(xcecli_retry_next) → unlock → yield rc. Wrapped 18 reply-bearing ops; writes=true for jobsubmit/jobdispatch/jobctrl/queuectrl/hostctrl/update_resvals/update_jusage/runtask/join/set_pend_reason/clear_reason, reads=false otherwise. R1.3: zero signature changes. DESIGN CORRECTION: `xce_nop` is fire-and-forget (no cbuf_get, xceapi.c:30) — moved nop reply-read→fire-and-forget in design OpCategory; nop wraps in 5.2 not 5.1. -- 5.2: `XCECLI_FF_CALL` macro (single-shot, NO retry loop/backoff/budget): lock → if known-dead (xcecli_ff_needs_reconnect = ctx.state==XCE_CTX_ERR) one rotate+recover → one send → unlock. Applied at the 3 wire chokepoints xcecli_notify/heartbeat/nop (notify_* delegate to xcecli_notify). wakeup/try_wakeup are LOCAL-only (cbuf_tryput own wbuf, no wire send) so correctly NOT wrapped. -- 5.3: `fill_redirect_reply(xce_reply_t*, const ha_state_t*)` (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build. -- 6.1: full lib suite green (161 tests). All 14 reqs (2.4,2.5,3.2,3.3,4.1-4.5,5.2,6.2,6.3,8.3,8.4) map to genuine passing tests. Gap-filled 6.3 with `test_decide_jitter_staggers_many_clients` (64 distinct-seed carriers → staggered delays, anti-thundering-herd). No production bugs found. -- 6.2 (LIVE E2E — partial): Live 3-node cluster (deploy-ha.sh) was brought up and DID elect a primary; killing the primary triggered server-side failover (ctrl-2 self-elected term=2). The run REPRODUCED the original `Invalid argument` client bug → root-caused + FIXED (bug-609, known-dead-ctx override) + regression unit test added. The clean within-session client transparent-recovery could NOT be visually demonstrated in this sandbox (background controller processes are reaped between tool calls; nix runtime env needed for backgrounded procs; lib build cleans the controller binary). RECOMMEND final visual confirmation in a stable env: build all, `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`, run xce_test (set masters / client / jobsub / joblist), kill the primary, confirm joblist reports cluster-not-ready then transparently recovers (no EINVAL). -- 6.2 (LIVE, round 2 — your env): failover now ENGAGES (xcecli_joblist→xcecli_retry_next→xce_failover_rotate). Two live-only bugs found+fixed: bug-609 (fresh-call-after-disconnect -EINVAL classified terminal → known-dead-ctx override in decide) and bug-610 (xce_failover_rotate SEGV in endpoint_connect — reconnect never re-bound ep->iomgr nor re-armed owner/msg/err handlers + iomgr_monitor; fixed via new `xce_ctx_reconnect()` in xcectx.c that mirrors xcecli_start establishment without restarting iomgr/vtaskmgr or touching ctx->state). REBUILD controller before retest: `cd src && nix develop --command bash tools/xce-build.sh controller` (relinks xce_test). NOTE: building `lib` removes the controller binary from the shared build dir — always rebuild controller before deploy. diff --git a/.kiro/specs/raft-adapter-conformance/design.md b/.kiro/specs/raft-adapter-conformance/design.md deleted file mode 100644 index a5492e3d..00000000 --- a/.kiro/specs/raft-adapter-conformance/design.md +++ /dev/null @@ -1,385 +0,0 @@ -# Design Document: raft-adapter-conformance - -## Overview - -This feature closes the wire-level reply-integrity gaps between the XCE controller's Raft adapter (`src/controller/ha.c`, `repl.c`, `config_repl.c`) and the protocol guarantees `raft/raft.tla` assumes always hold, and produces a durable, action-by-action conformance record so the next gap is found by inspection rather than by a live incident. It extends the work already landed for bug-1479 (the WAL catch-up reject-reply fix and the catch-up streaming-ceiling fix) to the two other places the same class of defect was found live during this feature's discovery pass: the configuration-push reply channel and the vote/PreVote endpoint failure-routing path. - -**Users**: HA cluster operators (who experience the failure mode as a stalled or slow-to-converge cluster) and XCE maintainers (who consume the conformance record when reasoning about future HA changes). - -**Impact**: Two adapter functions gain previously-missing reply/routing behavior (`handle_repl_config_push`, `vote_ep_err_handler`); one Cap'n Proto struct gains an additive field (`ReplConfigPushReply.term`); `src/controller/CLAUDE.md`'s existing TLA+ mapping tables are extended and reconciled against production behavior; no other subsystem changes. - -### Goals -- Every reject of a replication-class message (WAL catch-up entry, configuration push) produces an explicit reply distinguishable from a lost message. -- Every vote/PreVote request outcome, including a failure arriving after connect, reaches the production tally. -- Every reply channel that can carry a peer's observed term can trigger term adoption. -- A single, explicit, written decision resolves which `AppendEntries` send mechanism is production-authoritative. -- A durable, precise mapping exists from each `raft.tla` action to the adapter code that realizes it, with a conformance verdict. -- The four already-conformant guarantee areas (`ClientRequest`, `AdvanceCommitIndex`/commit-source-ownership, restart/recovery, the two already-fixed term-adoption vectors) are protected by regression coverage so a future change cannot silently reopen them. - -### Non-Goals -- No change to `src/lib/raft/raft.c`'s protocol logic (election, `logOk`, `(term,index)` truncation, `UpdateTerm`) — validated by `raft-log-replication`/`raft-commit-safety`/`raft-sim-conformance`. -- No configuration-*upgrade* materialized-state work (`config-upgrade-snapshot`'s scope). -- No controller voting-set/membership change support (`ha-config-bootstrap`'s scope). -- No change to `committed_lsn` advancement semantics beyond confirming the existing durable-majority-authoritative behavior is preserved. -- No wiring-in of `raft_send_append_entries` as the production send path (see Decision in Architecture below) — it remains dormant, now formally documented as such. - -## Boundary Commitments - -### This Spec Owns -- Reply-integrity behavior on the standby → primary direction for WAL catch-up rejects (regression-lock only, already shipped) and configuration-push rejects (new fix). -- Failure-routing behavior for vote/PreVote endpoints, across all failure timings (connect-time — regression-lock, already shipped; post-connect — new fix). -- The `term` field addition to `ReplConfigPushReply` and its use for term-adoption. -- The written conformance record (`CLAUDE.md` extension) mapping `raft.tla` actions to adapter code, including the explicit "documented dormant" resolution for the unused nextIndex-driven `AppendEntries` sender. -- Regression tests that lock in both the already-shipped guarantees and the guarantees this feature adds. - -### Out of Boundary -- The Raft protocol library itself (`src/lib/raft/raft.c`/`raft.h`) — this feature never modifies it, only verifies the adapter drives it faithfully. -- Configuration-upgrade materialized-state consistency (`config-upgrade-snapshot`). -- Controller membership/quorum-set changes (`ha-config-bootstrap`). -- Client-side primary-discovery (bug-854) and multi-segment catch-up payload handling (bug-847). -- Any change to `committed_lsn` advancement authority (`ha-commit-engine`/`ha-term-safety` own that decision; this feature only confirms it is unchanged). - -### Allowed Dependencies -- Raft library callback contract and entry points (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`, `raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`, vote/pre-vote handling) — consumed as-is, never modified. -- The existing out-of-band configuration-push transport (`ha-term-safety`) — this feature adds reply integrity to it without changing what it carries. -- The existing Cap'n Proto codegen build step (`src/lib/meson.build`) — consumed as-is for the additive schema field. -- Constraint: no dependency direction change — the adapter continues to depend on the library, never the reverse. - -### Revalidation Triggers -- Any change to `vote_ep_ctx_t`'s layout or to how vote/PreVote endpoints are registered. -- Any change to `ReplConfigPushReply`'s wire shape (this feature is itself one such change; a future one must re-check `config_activate_committed`'s `-EAGAIN` handling described below). -- Any change that reintroduces a hard coupling between `raft_commit_index` and `committed_lsn` advancement (would violate a guarantee this feature regression-locks). -- Any change that gives `raft_send_append_entries` a production caller (would invalidate the "documented dormant" resolution and require revisiting Requirement 3's decision). - -## Architecture - -### Existing Architecture Analysis - -The controller is a thin adapter over the Raft library (`ha-term-safety`'s framing, confirmed still accurate): the library owns election/commit/truncate decisions; the adapter's job is to persist what the library decides (`on_persist_entry`), source terms for it (`on_get_entry_term`), transmit its decisions on the wire, and turn wire replies back into library calls (`raft_on_append_entries_reply`, `raft_on_vote_reply`, `raft_on_pre_vote_reply`). This feature's defects are all instances of one pattern: an adapter code path receives or is meant to send a wire message, but a specific branch of that path (a reject, a late failure) returns without ever calling the corresponding "turn this into a library-visible outcome" step. The fix pattern established by bug-1479 (`ha_send_wal_ack_reply_ex`, plus its `XCE_TESTING` send-hook for testability) is reused verbatim for the two new fixes rather than inventing a new mechanism. - -Integration points that must be maintained: the `ha_dispatch_gate` opcode allowlist (unaffected — no new opcode), the five-callback contract (unaffected — no callback signature changes), the two-tier replication model (unaffected — this feature only makes existing tiers report failure explicitly). - -Technical debt addressed: three silent-drop-on-reject sites (WAL catch-up — done pre-feature; config-push — this feature) and one silent-drop-on-late-failure site (vote/PreVote post-connect). Technical debt formally reconciled rather than removed: the dormant nextIndex-driven `AppendEntries` sender (documented, not deleted — see Decision below). - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph Library - RaftLib[Raft library raft.c] - end - subgraph Adapter - HaC[ha.c callbacks and dispatch] - ReplC[repl.c WAL replication] - ConfigReplC[config_repl.c config push] - end - subgraph Wire - WalAck[WalAckReply term lsn success] - ConfigPushReply[ReplConfigPushReply accepted versionTag term] - VoteReply[RequestVoteReply and PreVoteReply] - end - subgraph Peer - Standby[Standby controller] - Primary[Primary controller] - end - - RaftLib --> HaC - HaC --> ReplC - HaC --> ConfigReplC - ReplC --> WalAck - ConfigReplC --> ConfigPushReply - HaC --> VoteReply - WalAck --> Primary - ConfigPushReply --> Primary - VoteReply --> Primary - Standby --> WalAck - Standby --> ConfigPushReply - Standby --> VoteReply - Primary --> RaftLib -``` - -**Architecture Integration**: -- Selected pattern: extend-in-place (see `research.md` Architecture Pattern Evaluation) — reply-integrity fixes land inside the existing functions that already own each reply, no new cross-cutting abstraction introduced. -- Domain boundaries: `ha.c` owns election/vote wiring and top-level dispatch; `repl.c` owns WAL replication send/ack; `config_repl.c` owns out-of-band configuration push/ack — unchanged by this feature. -- Existing patterns preserved: the `XCE_TESTING` send-hook pattern (`ha_wal_ack_send_hook`), the `raft_on_higher_term_observed` single term-adoption entry point, the `vote_reply_dispatch`/`pre_vote_reply_dispatch` production tally paths. -- New components: none introduce a new boundary; all four code changes (config-push reply, config-push term field, vote endpoint handler split, CLAUDE.md conformance record) extend existing ownership. -- Steering compliance: preserves the async-I/O thread-role contract (all changes are in already-non-blocking callback/handler code, no new blocking calls introduced) and the "controller never self-terminates" invariant (no new terminal paths added). - -### Decision: dormant `AppendEntries` sender resolution (Requirement 3) - -No architecture change. `raft_send_append_entries`/`ha_cb_send_append_entries_stub` remains in the codebase, unused in production, and is now explicitly documented as such (see `research.md` Design Decisions for the full rationale). `repl_broadcast` + `ha_stream_wal_to_standby` remain the sole production `AppendEntries`-equivalent send/retry/backoff mechanism. Per Requirement 3.3 (amended after design review to permit this third resolution alongside retire/wire-in), this documentation-only resolution must additionally be backed by a non-regression guard: an `XCE_TESTING`-only call-counter on `ha_cb_send_append_entries_stub` (mirroring the `ha_wal_ack_send_hook` idiom) exercised by a test that drives the production heartbeat/replication path end-to-end and asserts the counter stays at zero while confirming `repl_broadcast`/`ha_stream_wal_to_standby` did fire — so a future change that accidentally starts calling this path is caught, not silently accepted. This adds one small `ha.c`/test diff beyond the pure documentation change described above (see File Structure Plan and Testing Strategy). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Wire serialization | Cap'n Proto (existing `c-capnproto`) | `ReplConfigPushReply` gains an additive `term @2: UInt64` field | Regenerated automatically by the existing `capnp compile` Meson `custom_target` (`src/lib/meson.build:1-10`); no build/Nix change needed (see `research.md`) | -| Controller adapter | C (gnu17), existing `src/controller/` | All behavior changes | No new dependency | - -No other stack layer is touched. - -## File Structure Plan - -This is a small, surgical extension of an existing subsystem — files are listed individually rather than by directory pattern. - -### Modified Files -- `src/schema/controller.capnp` — add `term @2: UInt64 $C.mapname("term")` to `struct ReplConfigPushReply` (Requirement 7.4). -- `src/controller/config_repl.c` — `handle_repl_config_push`: construct and send `ReplConfigPushReply{accepted=0, term}` on each of its four reject early-returns (malformed request, missing bytes/sha256, sha256 mismatch, `conf_save_config` failure); populate `term` on the existing success reply too; preserve the existing `-EAGAIN`-from-`config_activate_committed` behavior unchanged (still sends `accepted=1` — activation-deferred is not a rejection). Add an `XCE_TESTING`-only send-hook (e.g. `ha_config_push_reply_send_hook`), mirroring `ha_wal_ack_send_hook` (`ha.c:3085-3086`), so tests can observe the reply payload without a live endpoint. (Requirements 2.2, 2.3, 2.5, 7.4, 7.5) -- `src/controller/cluster.c` — the existing primary-side `ReplConfigPushReply` handler (`myreq_handler`, `cluster.c:745-762`) already branches on `cack->accepted` (ACKs via `repl_on_config_ack` on true, WARN-and-ignores on false — the false branch has simply never fired in production before this feature's fix). Add a term check alongside the existing `accepted` check: when `cack->term` exceeds the primary's current term, call `raft_on_higher_term_observed`, the same single adoption entry point the heartbeat-reply and WAL-NACK vectors already use. (Requirement 7.4, 7.5) -- `src/controller/ha.c`: - - Split `vote_ep_err_handler` into `vote_ep_err_handler` (real-vote endpoints, registered at the existing `ha.c:688-689` call site) and `pre_vote_ep_err_handler` (PreVote endpoints, registered at the existing `ha.c:827-828` call site), each calling `vote_reply_dispatch` or `pre_vote_reply_dispatch` respectively for the election-phase branch instead of the legacy `handle_request_vote_reply`. Extract the identical `CTRL_PRIMARY` cleanup branch (standby-endpoint removal, `repl_remove_standby`, quorum-transition logging) into a shared static helper called by both. (Requirements 4.2, 4.3, 4.4) - - Extend the `on_send_append_entries` doc-comment at `ha.c:2520-2532` to state plainly that this callback is reachable only via `raft_send_append_entries`, which has no production caller, and point to the CLAUDE.md conformance record for the authoritative production mechanism. Add an `XCE_TESTING`-only call counter to `ha_cb_send_append_entries_stub` (e.g. `ha_send_append_entries_stub_call_count`) as the non-regression guard the amended Requirement 3.3 requires. (Requirement 3.3) -- `src/controller/CLAUDE.md` — extend the "TLA+ Action-to-Code Mapping" table with corrected/added rows for `AppendEntries` reply integrity (WAL catch-up and config-push), vote/PreVote failure routing, and the dormant-sender resolution; extend the "Protocol Verification Checklist" with a checklist item for "every reject path sends an explicit reply." (Requirement 1.1-1.6) -- `.kiro/specs/ha-term-safety/spec.json` — append a one-line pointer on the existing `known_limitations` entry (index 30, the dormant `on_send_append_entries` note) referencing this spec's resolution, so a future reader lands on the authoritative decision. (Requirement 1.2, 1.6) - -### New Test Additions (existing files extended, no new test files needed) -- `src/controller/testsuite/test_config_repl.c` — new tests: each of the four config-push reject paths sends `accepted=0` with the current term; the `-EAGAIN` activation-defer path still sends `accepted=1`; a higher term in an inbound config-push reply triggers adoption. (Requirements 2.2, 2.3, 2.5, 7.4, 7.5) -- `src/controller/testsuite/test_ha_vote_send.c` — new tests: a post-connect endpoint failure on a real-vote endpoint reaches `vote_reply_dispatch`'s tally; the same on a PreVote endpoint reaches `pre_vote_reply_dispatch`'s tally; reuses/extends `vote_ep_ctx_build_for_test` (`ha.c:7337`) as the test-construction seam. (Requirements 4.2, 4.3, 4.5) -- `src/controller/testsuite/test_ha_core_safety.c` — new test asserting `ha_send_wal_ack_reply_ex(success=0)` is actually invoked (via `ha_wal_ack_send_hook`) on a WAL catch-up reject, closing the reply-construction-boundary coverage gap found during design discovery. (Requirement 2.1, regression-lock) -- `src/controller/testsuite/test_commit_barrier_reply.c` or `test_commit_barrier_rollout.c` — new test: one failed payload in a submitted batch propagates failure to every payload in that batch. (Requirement 5.3) - -### Unmodified, Regression-Locked Files - -These files back the four already-conformant components (`ClientLogAppendGuarantee`, `CommitSourceGuarantee`, `TermDiscoveryCompleteness`, `RestartRecoveryGuarantee`). No source changes; cited here so each component in the table above resolves to a concrete file, and so a future diff touching them is recognized as revalidation-worthy rather than incidental. -- `src/controller/persist.c` — `ha_primary_log_append`, `persist_submit_job_payloads_async` (Requirement 5); confirmed conformant, existing tests in `test_commit_barrier_rollout.c`/`test_commit_barrier_reply.c`/`test_term_safety_adapter.c` cover it (plus the one new batch-propagation test above). -- `src/controller/commit.c` — `majority_candidate_locked`, `commit_set_raft_commit_index` (Requirement 6); confirmed conformant, `test_term_safety_adapter.c:646-780` already pins both directions of the advisory-vs-authoritative split, including the `XCE_TESTING` assert at `commit.c:222`. -- `src/controller/ha.c` — heartbeat-reply and WAL-NACK term-adoption call sites (Requirement 7.1, 7.2), `ha_sync_applied_lsn_from_recovery`/`ha_sync_raft_log_from_recovery`/phase2 retry helper/JIT log-length resync (Requirement 8); confirmed conformant, existing coverage in `test_persist_recovery.c`/`test_config_repl.c`. - -No files are created; no files outside `src/controller/` (and its one schema dependency) are touched. `src/lib/`, `src/frontend/`, `src/worker/`, `src/scheduler/` are untouched. - -## System Flows - -### Configuration-push reject reply (Requirement 2, 7) - -```mermaid -sequenceDiagram - participant Primary - participant Standby - - Primary->>Standby: ReplConfigPushRequest bytes sha256 versionTag - alt Malformed or missing fields - Standby-->>Primary: ReplConfigPushReply accepted false term - else SHA-256 mismatch - Standby-->>Primary: ReplConfigPushReply accepted false term - else conf_save_config fails - Standby-->>Primary: ReplConfigPushReply accepted false term - else Save succeeds, activation deferred EAGAIN - Standby-->>Primary: ReplConfigPushReply accepted true term - Note over Standby: Activation retried later on commit advance - else Save and activation succeed - Standby-->>Primary: ReplConfigPushReply accepted true term - end - Primary->>Primary: distinguish reject from timeout, adopt term if higher -``` - -Key decision not obvious from the diagram: `accepted` means "bytes saved out-of-band," not "activated" — the activation-deferred branch and the fully-succeeded branch both reply `accepted=true` by design, since activation is gated separately on the committed marker (`ha-term-safety`) and is expected to retry. - -### Vote/PreVote endpoint failure routing (Requirement 4) - -```mermaid -sequenceDiagram - participant Candidate - participant Peer - - Candidate->>Peer: connect and monitor - alt Connect or monitor fails - Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch denied - else Connect succeeds, later RDHUP or SSL error before reply - Candidate->>Candidate: vote_ep_err_handler or pre_vote_ep_err_handler - Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch denied - else Peer replies before any failure - Peer-->>Candidate: RequestVoteReply or PreVoteReply - Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch - end - Candidate->>Candidate: fast quorum decision once every peer outcome is counted -``` - -Key decision not obvious from the diagram: today only the "connect or monitor fails" and "peer replies" branches reach the production tally; this feature adds the "later RDHUP" branch to the same tally instead of the legacy no-op-counting path. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|---|---|---|---|---| -| 1.1 | Action-by-action mapping produced | ConformanceRecord | CLAUDE.md table | — | -| 1.2 | Live-vs-dormant path recorded on drift | ConformanceRecord | CLAUDE.md table, ha-term-safety spec.json pointer | Dormant sender decision | -| 1.3 | Gap detail sufficient to re-verify | ConformanceRecord | design.md + research.md | — | -| 1.4 | Fix vs documentation-drift classification | ConformanceRecord | design.md Decisions | — | -| 1.5 | Code-fix findings delivered under Req 2-8 | ConformanceRecord | (cross-reference) | — | -| 1.6 | Documentation-drift findings recorded for reconciliation | ConformanceRecord | ha-term-safety spec.json pointer | Dormant sender decision | -| 2.1 | WAL catch-up reject reply (regression-lock) | WalCatchupReplyIntegrity (existing) | `ha_send_wal_ack_reply_ex`, `ha_wal_ack_send_hook` | — | -| 2.2 | Config-push reject reply (new) | ConfigPushReplyIntegrity | `ReplConfigPushReply{accepted=0}` | Config-push reject flow | -| 2.3 | Reject distinguishable from transport loss | ConfigPushReplyIntegrity | `ReplConfigPushReply` vs no reply/timeout | Config-push reject flow | -| 2.4 | Unknown outcome not inferred as success | ConfigPushReplyIntegrity, WalCatchupReplyIntegrity | timeout handling (existing) | — | -| 2.5 | No regression to success-reply behavior | ConfigPushReplyIntegrity, WalCatchupReplyIntegrity | existing `accepted=1` / `success=1` paths | Config-push reject flow | -| 3.1 | Retry on rejection | Existing (`ha_raft_on_repl_nack`, `repl_start_catchup`) | — | — | -| 3.2 | Multi-round convergence | Existing (catch-up subsystem) | — | — | -| 3.3 | Single authoritative send mechanism | ConformanceRecord (documentation decision) | CLAUDE.md table, `ha.c:2520` doc-comment | Dormant sender decision | -| 3.4 | Transport-level send failure not treated as caught-up | Existing (`repl_broadcast`, heartbeat re-check) | — | — | -| 4.1 | Connect-time failure counted (regression-lock) | VoteEndpointFailureRouter (existing behavior) | `vote_reply_dispatch`, `pre_vote_reply_dispatch` | Vote failure routing | -| 4.2 | Post-connect failure counted (new) | VoteEndpointFailureRouter | `vote_ep_err_handler`, `pre_vote_ep_err_handler` | Vote failure routing | -| 4.3 | Same behavior for PreVote | VoteEndpointFailureRouter | `pre_vote_ep_err_handler` | Vote failure routing | -| 4.4 | No regression to connect/monitor routing | VoteEndpointFailureRouter | shared `CTRL_PRIMARY` cleanup helper | Vote failure routing | -| 4.5 | Fast quorum decision reachable | VoteEndpointFailureRouter | tally functions (existing) | Vote failure routing | -| 5.1 | Append persists or fails explicitly | ClientLogAppendGuarantee (existing) | `raft_log_append`, `ha_primary_log_append` | — | -| 5.2 | No log-length advance on failed persist | ClientLogAppendGuarantee | `raft_log_append` | — | -| 5.3 | Batch failure propagates to whole batch | ClientLogAppendGuarantee | `persist_submit_job_payloads_async` | — | -| 5.4 | No regression to failure propagation | ClientLogAppendGuarantee | existing tests | — | -| 6.1 | Durable-majority sole authority | CommitSourceGuarantee (existing) | `majority_candidate_locked` | — | -| 6.2 | Advisory index informational only | CommitSourceGuarantee | `commit_set_raft_commit_index` | — | -| 6.3 | No hard-ceiling reintroduction | CommitSourceGuarantee | `commit.c:139-155` invariant | — | -| 6.4 | Advance proceeds despite lagging advisory index | CommitSourceGuarantee | existing `XCE_TESTING` assert | — | -| 7.1 | Heartbeat-reply term adoption (regression-lock) | TermDiscoveryCompleteness (existing) | `raft_on_higher_term_observed` | — | -| 7.2 | WAL-NACK term adoption (regression-lock) | TermDiscoveryCompleteness (existing) | `raft_on_higher_term_observed` | — | -| 7.3 | Vote/PreVote reply term adoption (regression-lock) | TermDiscoveryCompleteness (existing) | `raft_on_vote_reply` Step 1 | — | -| 7.4 | Config-push reply carries term (new) | ConfigPushReplyIntegrity | `ReplConfigPushReply.term` | Config-push reject flow | -| 7.5 | No regression to existing term-adoption vectors | TermDiscoveryCompleteness, ConfigPushReplyIntegrity | — | Config-push reject flow | -| 8.1 | Recovery raises applied LSN/log length/term | RestartRecoveryGuarantee (existing) | `ha_sync_applied_lsn_from_recovery`, `ha_sync_raft_log_from_recovery` | — | -| 8.2 | Failed recovery step retries, no false-latch | RestartRecoveryGuarantee | phase2 retry helper (`ha.c:415-501`) | — | -| 8.3 | JIT log-length resync beyond boot | RestartRecoveryGuarantee | `ha_apply_term_entry_via_library` resync | — | -| 8.4 | No regression to clean-failover convergence | RestartRecoveryGuarantee | existing tests | — | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|---|---|---|---|---|---| -| ConfigPushReplyIntegrity | Controller/Replication | Send an explicit accept/reject reply, carrying term, for every configuration push | 2.2, 2.3, 2.5, 7.4, 7.5 | `ReplConfigPushReply` schema (P0) | Event | -| WalCatchupReplyIntegrity | Controller/Replication | Existing bug-1479 fix; regression-locked, not modified | 2.1, 2.4, 2.5 | `WalAckReply` schema (P0, existing) | Event | -| VoteEndpointFailureRouter | Controller/Election | Route every vote/PreVote endpoint failure, at any timing, to the production tally | 4.1-4.5 | `vote_reply_dispatch` (P0), `pre_vote_reply_dispatch` (P0), `vote_ep_ctx_build_for_test` (P1, test-only) | Service | -| ConformanceRecord | Documentation | Durable action-by-action mapping and dormant-path resolution | 1.1-1.6, 3.3 | `CLAUDE.md`, `ha-term-safety/spec.json` | State | -| ClientLogAppendGuarantee | Controller/Persistence | Existing conformant append-or-explicit-failure behavior; regression-locked | 5.1-5.4 | `persist.c` (existing) | State | -| CommitSourceGuarantee | Controller/Commit | Existing conformant durable-majority-authoritative behavior; regression-locked | 6.1-6.4 | `commit.c` (existing) | State | -| TermDiscoveryCompleteness | Controller/Election | Existing conformant term-adoption vectors (heartbeat, WAL-NACK, vote reply); regression-locked | 7.1-7.3, 7.5 | `raft_on_higher_term_observed` (existing) | Event | -| RestartRecoveryGuarantee | Controller/Recovery | Existing conformant restart/recovery reconcile; regression-locked | 8.1-8.4 | `ha.c` recovery functions (existing) | State | - -Only the two components introducing new code (`ConfigPushReplyIntegrity`, `VoteEndpointFailureRouter`) warrant full detail blocks below; the regression-lock components need no new interface — their existing interfaces are cited in the table above and exercised by the tests in File Structure Plan. - -### Controller / Replication - -#### ConfigPushReplyIntegrity - -| Field | Detail | -|-------|--------| -| Intent | Every configuration-push outcome (accept or reject) produces an explicit `ReplConfigPushReply` carrying the replying member's current term | -| Requirements | 2.2, 2.3, 2.5, 7.4, 7.5 | - -**Responsibilities & Constraints** -- Owns the reply-construction boundary inside `handle_repl_config_push`; does not own configuration activation (`config_activate_committed` remains the sole activation gate, unchanged). -- Must not conflate an activation-deferred outcome (`-EAGAIN`) with a rejection — that path continues to reply `accepted=1`. -- Must include the current term on every reply, success or failure. - -**Dependencies** -- Outbound: `ReplConfigPushReply` schema field (P0) — the additive `term` field this component populates. -- Outbound: `raft_on_higher_term_observed` (P0) — invoked by the primary's inbound-reply handler when this component's reply carries a higher term (existing entry point, no change to it). -- External: none. - -**Contracts**: Event [x] - -##### Event Contract -- Published: `ReplConfigPushReply {accepted: bool, versionTag: uint64, term: uint64}` — one per `ReplConfigPushRequest` received, always sent (never silently dropped). -- Subscribed: none (this component only replies). -- Ordering/delivery: request/reply pair over the existing outbound control channel (`ha-async-io.md` connection ownership); no new ordering guarantee introduced. - -**Implementation Notes** -- Integration: reuses the existing `xce_reply`/`init_reply_by_req` pattern already present in `handle_repl_config_push`'s success path; the four reject branches gain the same call shape. -- Validation: `XCE_TESTING`-only send-hook (mirroring `ha_wal_ack_send_hook`) lets unit tests assert the reply payload without a live endpoint. -- Risks: none material — narrow, single-function change with a directly analogous shipped precedent (bug-1479). - -### Controller / Election - -#### VoteEndpointFailureRouter - -| Field | Detail | -|-------|--------| -| Intent | Every vote/PreVote endpoint failure, regardless of when in the round it occurs, is counted through the same production tally as a delivered denial | -| Requirements | 4.1-4.5 | - -**Responsibilities & Constraints** -- Owns endpoint-failure handling for outgoing vote and PreVote connections (`on_err` callbacks); does not own the tally logic itself (`vote_reply_dispatch`/`pre_vote_reply_dispatch` are consumed, not modified). -- Must preserve the existing `CTRL_PRIMARY` cleanup behavior (standby-endpoint deregistration, quorum-transition logging) identically for both endpoint kinds. -- Must not introduce a new discriminant field on `vote_ep_ctx_t`; the split-handler registration already distinguishes endpoint kind by call site. - -**Dependencies** -- Outbound: `vote_reply_dispatch` (P0) — real-vote tally, already correct for connect/monitor failures. -- Outbound: `pre_vote_reply_dispatch` (P0) — PreVote tally, already correct for connect/monitor failures. -- Outbound: `vote_ep_ctx_build_for_test` (P1, test-only) — reused/extended as the unit-test construction seam. -- Inbound: iomgr `on_err` callback invocation (existing, unchanged trigger). - -**Contracts**: Service [x] - -##### Service Interface -```c -/* vote_ep_err_handler: on_err callback for outgoing real-vote endpoints. - * Election-phase branch now counts a post-connect failure through - * vote_reply_dispatch instead of handle_request_vote_reply. */ -static int vote_ep_err_handler(void *ep_raw); - -/* pre_vote_ep_err_handler: on_err callback for outgoing PreVote endpoints. - * Election-phase branch counts a post-connect failure through - * pre_vote_reply_dispatch. */ -static int pre_vote_ep_err_handler(void *ep_raw); - -/* shared helper: identical CTRL_PRIMARY cleanup extracted from both. */ -static void vote_ep_handle_promoted_disconnect(ha_state_t *ha, endpoint_t *ep, - vote_ep_ctx_t *vctx); -``` -- Preconditions: `ep->data` is a populated `vote_ep_ctx_t` for the endpoint kind the handler is registered against. -- Postconditions: exactly one outcome (denied/counted) reaches the applicable tally function per invocation; `CTRL_PRIMARY`-role invocations perform standby cleanup identically to today. -- Invariants: a candidate's own term is never mistaken for an observed peer term through this path (the synthesized outcome carries the candidate's own term, `granted=0` — unchanged from today's behavior). - -**Implementation Notes** -- Integration: registration sites (`ha.c:688-689`, `827-828`) each point at the correspondingly-named handler; no change to endpoint setup beyond the callback pointer. -- Validation: new unit tests per File Structure Plan, using the reused test-construction helper to simulate a post-connect failure without a live peer. -- Risks: touches live election-path code — mitigated by keeping the extracted `CTRL_PRIMARY` branch byte-for-byte identical and requiring the new tests before this task is considered done (see `research.md` Risks & Mitigations). - -## Data Models - -### Data Contracts & Integration - -**`ReplConfigPushReply` (Cap'n Proto, `src/schema/controller.capnp`)** - -| Field | Ordinal | Type | Change | -|---|---|---|---| -| `accepted` | 0 | Bool | unchanged | -| `versionTag` | 1 | UInt64 | unchanged | -| `term` | 2 | UInt64 | **new, additive** | - -Backward/forward compatibility: Cap'n Proto additive fields are safe under mixed-version rolling upgrade — an old-code standby's reply (no `term` field) decodes as `term=0` on a new-code primary. This is safe by construction: `0` can never present as a term higher than the primary's own current term (terms start at 1 and only increase), so no spurious adoption occurs; the primary simply doesn't get early term-discovery from an old-code peer's config-push replies, which is exactly today's (pre-feature) behavior for that peer. No explicit migration phasing is required beyond the existing rolling-upgrade tolerance the cluster already relies on. - -`WalAckReply` is unchanged (already carries `term`, `lsn`, `success`); listed here only to confirm no modification is needed. - -## Error Handling - -### Error Strategy - -This feature's domain is internal cluster wire-protocol reply integrity, not user-facing request handling — the applicable categories are protocol-rejection, transport-failure, and validation-failure, each with a distinct, already-established response pattern that this feature extends consistently. - -### Error Categories and Responses -- **Validation/protocol rejection** (malformed config-push request, SHA-256 mismatch, `conf_save_config` failure): explicit reject reply (`accepted=0`) carrying term, so the primary can distinguish "actively rejected" from "lost" and react (retry, log, or escalate) — this is the category this feature's fixes target. -- **Transport failure** (connection drop, send failure, RDHUP): counted as a failure outcome through the same production tally/reply path used for a delivered rejection wherever the protocol model requires an outcome (vote/PreVote tally); where no reply is architecturally expected (snapshot chunk transfer, by explicit design — see requirements.md Boundary Context), recovery remains via the next heartbeat cycle, unchanged by this feature. -- **Activation-deferred, not an error** (`config_activate_committed` returning `-EAGAIN`): explicitly not classified as a rejection; the existing loud-fence-and-retry pattern (steering `ha-async-io.md`, "controller never self-terminates") already governs it and is preserved unchanged. - -### Monitoring -No new metrics/logging channels introduced; existing `WARN`/`INFO` logging in `handle_repl_config_push` and `vote_ep_err_handler` is preserved and extended with the same log level conventions already used in each function. - -## Testing Strategy - -- **Unit Tests**: - 1. `handle_repl_config_push` sends `ReplConfigPushReply{accepted=0, term}` for each of the four reject conditions (malformed request, missing bytes/sha256, sha256 mismatch, save failure) — via the new `XCE_TESTING` send-hook. (2.2, 2.3, 2.5) - 2. `handle_repl_config_push` still sends `accepted=1` when `config_activate_committed` returns `-EAGAIN` (activation-deferred is not a rejection). (2.5, 7.5) - 3. `ha_send_wal_ack_reply_ex(success=0)` is actually invoked on a WAL catch-up reject (closing the reply-construction-boundary coverage gap identified during design discovery). (2.1) - 4. `vote_ep_err_handler` routes a synthetic post-connect failure through `vote_reply_dispatch`'s tally, using the reused test-construction helper. (4.2, 4.4) - 5. `pre_vote_ep_err_handler` routes the same failure shape through `pre_vote_reply_dispatch`'s tally. (4.3) - 6. A submitted batch with one failing payload propagates that failure to every payload in the batch. (5.3) - 7. Driving the production heartbeat/replication path end-to-end never increments `ha_send_append_entries_stub_call_count`, while `repl_broadcast`/`ha_stream_wal_to_standby` do fire for the same scenario — the Requirement 3.3 non-regression guard. (3.3) -- **Integration Tests**: - 1. A three-node cluster with one standby configured to reject a pushed configuration (bad hash) observes the primary correctly distinguishing the rejection from a timeout and not treating the push as silently lost. - 2. A three-node election round where one peer's vote endpoint fails after connect (simulated) still reaches a quorum decision without waiting for the full election timeout. - 3. Existing `test_term_safety_adapter.c` advisory-vs-authoritative commit tests re-run unmodified as the Requirement 6 regression lock (no new test needed, confirmed sufficient during design discovery). - 4. Existing `test_persist_recovery.c`/`test_config_repl.c` recovery tests re-run unmodified as the Requirement 8 regression lock. -- **Live/Soak Tests** (per project convention for HA changes, `xce-test-ha` skill): - 1. A rolling chaotic-failover soak where the test driver issues a configuration push whose payload deliberately does not match the sha256 it advertises (no special fault-injection hook needed — this exercises `handle_repl_config_push`'s real SHA-256-mismatch reject branch), confirming the primary observes the explicit reject rather than degrading to the timeout path. - 2. A soak round exercising the post-connect vote/PreVote endpoint failure path end-to-end. **Open question for the tasks phase**: unlike item 1, this needs a way to sever an already-established outbound vote connection before a reply arrives (e.g. a firewall-rule toggle or proxy in the soak harness) — no existing `xce-test-ha` mechanism for this was confirmed during design. If no such mechanism exists cheaply, this item may be deferred; the unit tests (Requirement 4.2/4.3, via `vote_ep_ctx_build_for_test`) already give direct, deterministic coverage of the fix itself. - -## Supporting References -- `research.md` — full gap analysis (asset map, options, effort/risk), design-phase discovery log, and the two recorded design decisions (dormant sender resolution, vote-handler split) with full rationale. -- `src/controller/CLAUDE.md` — "TLA+ Action-to-Code Mapping" and "Protocol Verification Checklist," extended by this feature as the durable conformance record (Requirement 1). diff --git a/.kiro/specs/raft-adapter-conformance/requirements.md b/.kiro/specs/raft-adapter-conformance/requirements.md deleted file mode 100644 index 531f0eb1..00000000 --- a/.kiro/specs/raft-adapter-conformance/requirements.md +++ /dev/null @@ -1,120 +0,0 @@ -# Requirements Document - -## Introduction - -This feature performs a systematic, action-by-action conformance audit of the XCE controller's Raft replication **adapter** — the wire-level send/receive/reply code in `src/controller/ha.c`, `repl.c`, and `config_repl.c` — against the actions of the canonical Raft protocol model (`raft/raft.tla`: `AppendEntries` request/response, `RequestVote`/`PreVote` request/response, `ClientRequest`/log append, `AdvanceCommitIndex`, term/`UpdateTerm` handling, and restart/recovery reconcile). It is distinct from and complements `raft-sim-conformance`, whose boundary is the in-process, in-memory library harness (`src/lib/raft/raft_sim.c`) validating the library's own protocol semantics; this feature validates that the controller adapter faithfully carries the library's decisions onto the wire without silently dropping a response, retry, or backoff step the protocol model assumes always happens. - -**Motivating finding.** A fresh live-soak re-verification of bug-1479's fix surfaced that the controller's only `WalAckReply` constructor hardcoded success, so no production code path ever sent a failing reply — a standby's WAL catch-up rejection was silently dropped rather than reaching the primary as a NACK, reproducing bug-1479's permanent-quorum-loss symptom by a different mechanism than the original defect. That gap, plus a second, independently-discovered adapter gap it exposed (a standby's catch-up streaming range was capped at the leader's committed point rather than its real log tail, creating a self-referential deadlock), are already fixed and live-verified as of this spec's seeding (see bug-1479 in `.wolf/buglog.json` and `config-upgrade-snapshot` task 8.6). This feature's job is the broader pass that finding was a symptom of: walk every Raft protocol action against the adapter code that is supposed to realize it, and treat each specific type of gap consistently rather than fixing instances one at a time as they surface via live incidents. - -**Audit results seeded into this requirements pass.** A fresh action-by-action pass performed for this spec, cross-referenced against prior `raft-log-replication`/`raft-commit-safety`/`ha-term-safety`/`raft-sim-conformance` validation notes, confirmed: -- Two areas already conformant and worth protecting with regression coverage: `ClientRequest`/log append integrity, and the durable-majority-authoritative / advisory-raft-index split for `AdvanceCommitIndex`. -- Two live, unfixed gaps of the same class as the motivating finding: a standby's configuration-push rejection (malformed message, missing fields, checksum mismatch, or save failure) never sends an explicit reject reply, so the primary cannot distinguish an actively-rejected push from a lost packet; and a vote/PreVote endpoint failure arriving after connect (not at connect time) is counted through a legacy path instead of the production vote-tally path, reopening — for that one failure window — the same silent-drop defect class bug-1423 already fixed for connect-time failures. -- One architecture question requiring a design decision rather than a behavioral fix: a fully protocol-conformant, per-peer index-driven `AppendEntries` sender exists in the codebase but has zero production callers; the mechanism actually driving replication in production today is a different, coarser substitute (broadcast-plus-catch-up-marking) that converges correctly post-fix but is not the mechanism the design documentation and the protocol model depict. - -Per this feature's scope decision, the confirmed live gaps found during this pass are fixed within this feature rather than deferred to separate specs, following the same precedent as the motivating finding. - -## Boundary Context - -- **In scope**: - - A documented, action-by-action mapping between each `raft.tla` protocol action and the controller adapter code path(s) that realize it, with an explicit conformance verdict per action and citations precise enough to be independently re-verified. - - Reconciling documentation-vs-code drift the audit finds at this boundary (a design document describing a code path that is not the one actually exercised in production) by recording which path is live and which is dormant or superseded. - - Fixing every live gap the audit confirms, where the adapter fails to produce a response, retry, or backoff step that the protocol model or an already-shipped sibling fix assumes exists — including the configuration-push rejection reply and the post-connect vote/PreVote failure routing named above, and any other such gap the systematic pass confirms. - - Regression coverage that locks in both the gaps already fixed before this spec (WAL catch-up rejection replies, catch-up streaming ceiling, vote/PreVote connect-time failure routing, restart/recovery log resync) and the gaps this spec fixes, so a future change cannot silently reopen either. - - A single, unambiguous decision (recorded as a requirement) about which `AppendEntries` send/retry/backoff mechanism is production-authoritative — the existing per-peer index-driven sender, or the broadcast-plus-catch-up substitute — so the codebase does not carry two divergent implementations of the same guarantee. -- **Out of scope**: - - Any change to the Raft protocol logic itself (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`) inside `src/lib/raft/raft.c` — that logic is validated by `raft-log-replication`, `raft-commit-safety`, and `raft-sim-conformance`; this feature validates only that the controller adapter carries the library's decisions onto the wire. - - Configuration-*upgrade* materialized-state consistency (full entity-bearing state snapshot at a configuration change boundary) — remains the independent `config-upgrade-snapshot` spec's scope. - - Changes to the controller voting set (adding/removing a controller peer) — remains `ha-config-bootstrap`'s scope; this feature does not introduce quorum/membership transitions. - - Client-side primary-discovery behavior (bug-854) and catch-up payload handling across multiple WAL segments (bug-847) — tracked separately, not part of the adapter's reply-integrity contract this feature audits. - - Any change to when or how the primary decides to advance `committed_lsn` beyond preserving the existing durable-majority-authoritative behavior — that decision boundary belongs to `ha-commit-engine`/`ha-term-safety`. -- **Adjacent expectations**: - - Relies on the raft library's callback contract (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`, vote/pre-vote handling) exactly as delivered by the Raft TLA+ Conformance series; this feature does not change that contract, only how faithfully the adapter drives it. - - Relies on the out-of-band configuration-push transport (`ha-term-safety`) continuing to carry configuration bytes hash-verified against the committed `(term, lsn)` marker; this feature adds reply integrity to that transport's rejection path without changing what it carries. - - Any gap this feature's audit finds that is genuinely new protocol-model territory (not covered by an existing sibling spec's boundary) is fixed here; any gap that falls within an existing sibling spec's stated scope is cross-referenced to that spec rather than duplicated. - -## Requirements - -### Requirement 1: Systematic Raft adapter conformance audit - -**Objective:** As a maintainer, I want a documented, action-by-action mapping between the Raft protocol model's actions and the controller's adapter code, so that gaps between the model's assumptions and production behavior are found deliberately and once, instead of being rediscovered independently through separate live incidents. - -#### Acceptance Criteria -1. The conformance audit shall produce a written mapping that lists, for each of `AppendEntries` request, `AppendEntries` response, `RequestVote`/`PreVote` request, `RequestVote`/`PreVote` response, `ClientRequest`/log append, `AdvanceCommitIndex`, term/`UpdateTerm` handling, and restart/recovery reconcile, the controller adapter code path(s) that realize it and a conformance verdict. -2. When the audit finds that a sibling spec's design documentation describes an adapter code path that differs from the code path actually exercised in production, the audit shall record which path is live and which is dormant or superseded, rather than leaving the discrepancy implicit. -3. When the audit finds a gap where the protocol model assumes a response, retry, or backoff step that the adapter does not perform, the audit shall record the affected code path, the triggering condition, and the observed effect in enough detail to be independently re-verified without re-deriving the finding from scratch. -4. The audit shall classify each finding as either requiring a code fix (a live path that violates the model's guarantee) or documentation-only drift (the guarantee holds in production, but design documentation describes a different mechanism than what runs). -5. Where a finding is classified as requiring a code fix, the fix shall be delivered under this feature's Requirements 2 through 8, not deferred to a separate spec. -6. Where a finding is classified as documentation-only drift, the audit shall record the correction needed so the affected design documentation can be reconciled without re-investigating the code. - -### Requirement 2: AppendEntries-equivalent rejection reply integrity - -**Objective:** As an operator of an HA controller cluster, I want every replication message a standby rejects to produce an explicit reply back to the primary, so that the primary always learns of a rejection and can retry or re-route rather than the cluster stalling on a silently dropped reject. - -#### Acceptance Criteria -1. When a standby rejects a WAL catch-up entry for any reason (log mismatch, forward gap, or apply failure), the Controller shall send an explicit reply indicating rejection back to the primary. -2. When a standby rejects a configuration push for any reason (malformed message, missing required fields, checksum mismatch, or a local save failure), the Controller shall send an explicit reply indicating rejection back to the primary, mirroring the acceptance reply already sent on a successful push. -3. The rejection reply for a configuration push shall be distinguishable by the primary from a lost or undelivered push, so that the primary's push-and-wait logic never treats an active rejection and a transport loss as the same outcome. -4. While no explicit rejection or acceptance reply has arrived within the applicable transport wait bound, the Controller shall continue to treat the outcome as unknown rather than inferring success. -5. The Controller shall not regress the acceptance-reply behavior for a successful WAL catch-up entry or a successful configuration push. - -### Requirement 3: AppendEntries send retry and backoff guarantee - -**Objective:** As an operator, I want exactly one production-authoritative mechanism driving `AppendEntries`-equivalent replication retries and per-peer backoff, so that a rejected or lost replication attempt is corrected without relying on two divergent, only-one-of-which-is-live implementations of the same guarantee. - -#### Acceptance Criteria -1. When the primary receives an explicit rejection reply for a replicated entry, the Controller shall retry replication to that standby from an earlier point rather than repeating the same rejected attempt. -2. While a standby remains behind after a rejection, the Controller shall continue advancing that standby's catch-up point across successive rounds until it converges, rather than requiring convergence in a single round. -3. The Controller shall have exactly one send/retry/backoff mechanism that is authoritative in production for `AppendEntries`-equivalent replication; where more than one fully-implemented mechanism exists in the codebase, the non-authoritative one shall be resolved by one of: retiring it, making it the production path, or formally documenting it as an intentionally-dormant reference path guarded against silent regression (e.g. an explicit non-production marker plus a test asserting it stays uncalled in production) — so the codebase never carries two divergent, unreconciled implementations of the same guarantee. -4. If a replication send fails at the transport level (not a rejection), the Controller shall not treat the standby as caught up, and shall re-attempt replication to it within a bounded number of subsequent replication cycles. - -### Requirement 4: RequestVote/PreVote reply integrity across all failure modes - -**Objective:** As an operator, I want every vote or pre-vote request outcome — granted, denied, or a failure that prevents a reply — to be counted through the same production tally, regardless of when in the round the failure occurs, so that an election or pre-vote round can reach its fast "all replied" decision instead of always falling back to the full round timeout. - -#### Acceptance Criteria -1. When a vote request cannot be delivered because the connection could not be established, the Controller shall count that outcome through the same production vote-tally path used for a delivered denial. -2. When a vote request connection is established but then fails before a reply arrives (a post-connect failure), the Controller shall count that outcome through the same production vote-tally path used for a connect-time failure and for a delivered denial, rather than through a separate legacy path that does not update the tally. -3. The Controller shall apply the same failure-counting behavior to a pre-vote request as to a real vote request. -4. The Controller shall not regress the existing connect-time and monitor-time failure routing already verified for vote and pre-vote requests. -5. When every peer in an election or pre-vote round has replied or had its outcome counted, the Controller shall be able to reach a quorum decision without waiting for the round's full timeout. - -### Requirement 5: ClientRequest / log append integrity - -**Objective:** As a maintainer, I want a client write to always resolve to either an acknowledged append or an explicit, propagated failure, so that a write is never silently lost between submission and its outcome. - -#### Acceptance Criteria -1. When the primary appends a client-submitted entry, the Controller shall either persist the entry and make it eligible for replication, or return an explicit failure to the submitter. -2. The Controller shall not advance the log length for an entry whose persistence failed. -3. When a batch of submitted entries includes one that fails to persist, the Controller shall propagate that failure to every entry in the same batch rather than silently completing a subset. -4. The Controller shall not regress the existing behavior where a persistence failure is returned to the caller rather than swallowed. - -### Requirement 6: Commit advancement source-of-truth preserved - -**Objective:** As an operator, I want the durable-majority barrier to remain the sole authority for advancing the committed point, so that the advisory Raft commit index can never gate, delay, or override a safe commit advance. - -#### Acceptance Criteria -1. The Controller shall advance the committed point using the durable-majority barrier as the sole authoritative source. -2. The Controller shall treat the Raft library's advisory commit index as informational only; it shall neither block nor be required for the durable-majority barrier to advance the committed point. -3. The Controller shall not reintroduce a hard ceiling that requires both the durable-majority barrier and the advisory Raft commit index to agree before advancing the committed point. -4. When the advisory Raft commit index lags the durable-majority watermark, the Controller shall still advance the committed point according to the durable-majority barrier. - -### Requirement 7: Term discovery completeness - -**Objective:** As an operator, I want every reply channel capable of carrying a peer's observed election term to trigger term adoption when that peer's term is higher, so that a stale-term node cannot remain stuck retrying against a peer that has already moved to a higher term. - -#### Acceptance Criteria -1. When a heartbeat reply carries a term higher than the primary's current term, the Controller shall adopt the higher term and step down from any leadership held at the lower term. -2. When a WAL rejection reply carries a term higher than the sender's current term, the Controller shall adopt the higher term. -3. When a vote or pre-vote reply carries a term higher than the candidate's current term, the Controller shall adopt the higher term. -4. When a configuration-push reply is sent, the Controller shall include the replying member's current term, so a higher term can be discovered through that reply channel as it already is through heartbeat and WAL-rejection replies. -5. The Controller shall not regress the existing heartbeat-reply and WAL-rejection term-adoption behavior already verified for this cluster. - -### Requirement 8: Restart / recovery reconciliation conformance - -**Objective:** As an operator, I want a restarted or rejoining controller to reconcile its applied state, log length, and term from its durable floor every time recovery runs, so that a boot-ordering or partial-failure hazard cannot leave a node permanently out of sync with what it has actually persisted. - -#### Acceptance Criteria -1. When a controller completes startup recovery, the Controller shall raise its applied LSN, log length, and term from the recovered durable floor, never leaving them below what recovery found. -2. If a recovery step fails partway through cluster initialization, the Controller shall retry that step rather than permanently latching a completed state without having finished it. -3. While a term-safe append-entries receive occurs after startup, the Controller shall re-check and, if needed, re-raise its log length from the durable floor, not only at boot time. -4. The Controller shall not regress the existing byte-identical convergence behavior for a single clean failover and restart. diff --git a/.kiro/specs/raft-adapter-conformance/research.md b/.kiro/specs/raft-adapter-conformance/research.md deleted file mode 100644 index 063a4612..00000000 --- a/.kiro/specs/raft-adapter-conformance/research.md +++ /dev/null @@ -1,231 +0,0 @@ -# Gap Analysis: raft-adapter-conformance - -Analysis date: 2026-07-29. Requirements phase: generated (not yet approved). Per skill fallback rule, proceeding — gap analysis findings can also inform requirement refinement before approval. - -## Method - -Codebase investigation used direct `Read`/`grep` against `src/controller/ha.c`, `repl.c`, `config_repl.c`, `cluster.c`, `src/lib/raft/raft.c`/`raft.h`, `src/schema/controller.capnp`, and `src/controller/testsuite/`, plus the two Explore-agent passes already run during requirements generation (their findings are the primary input to Requirement-to-Asset mapping below; this pass adds the concrete integration-point evidence — schema fields, dispatcher code, struct layouts — needed to select an implementation approach). No external dependency research was needed; this is entirely an internal-codebase conformance question. - -## Requirement-to-Asset Map - -### Requirement 1 — Systematic conformance audit deliverable - -| Asset | Status | -|---|---| -| `raft/raft.tla` (canonical action list: `Restart`, `Timeout`, `RequestVote`, `BecomeLeader`, `ClientRequest`, `AdvanceCommitIndex`, `AppendEntries`, `Receive`→`UpdateTerm`/`HandleRequestVoteRequest`/`HandleRequestVoteResponse`/`HandleAppendEntriesRequest`/`HandleAppendEntriesResponse`) | Present, unmodified upstream copy | -| `src/controller/CLAUDE.md` → "TLA+ Action-to-Code Mapping" + "Protocol Verification Checklist" tables | **Present and already the natural home for this deliverable** — but incomplete/partially stale (its own checklist entry admits a 2026-07-18 citation correction; it does not yet list the two already-fixed reply-integrity gaps or the two newly-found ones) | -| `ha-term-safety/spec.json` `known_limitations` (3 entries) + `tasks.md:195` | Present — already contain two of the three documentation-drift notes this audit would otherwise re-derive (dormant `on_send_append_entries`/`raft_send_append_entries` send path; advisory-vs-authoritative commit index) | -| `raft-sim-conformance/spec.json` `known_limitations` (3 entries) | Present — contains the follower-commit-index documentation-drift note | -| A standalone, dedicated conformance-map document | **Missing** (no such file exists yet under `.kiro/specs/raft-adapter-conformance/` or `docs/`) | - -**Gap tag**: Constraint — CLAUDE.md's table is intentionally terse (a living quick-reference, not an evidentiary audit trail); the systematic pass's detailed findings (file:line citations, live-vs-dormant verdicts, reasoning) don't fit there without bloating it. - -### Requirement 2 — AppendEntries-equivalent rejection reply integrity - -| Sub-area | Asset | Status | -|---|---|---| -| WAL catch-up reject reply | `ha_send_wal_ack_reply_ex` (bug-1479, `ha.c` ~6680+), `WalAckReply` schema (`controller.capnp:876-880`, already has `term`, `lsn`, `success`) | **Fixed and live.** Gap tag: Unknown — need to confirm a regression test explicitly asserts a reject reply is sent (not just that the NACK routes once received); `test_ha_core_safety.c` has adjacent pin tests (`test_2_3_termblind_route_payload_guarded`, `test_2_3_unrepairable_catchup_raises_nonpromotable_fence`) but their scope wasn't fully re-verified this pass. | -| Config-push reject reply | `handle_repl_config_push` (`config_repl.c:1261-1337`); dispatch site `cluster.c:652-655` (`rc = handle_repl_config_push(...)`, no reply built on any `rc`); contrast `cluster.c:767-772` `default:` case, which *does* send a generic error reply on unknown opcode | **Missing — confirmed live gap.** Read in full this pass: 4 early returns (`-EINVAL` malformed body ~1266, `-EINVAL` missing bytes/sha256 ~1286, `-EBADMSG` sha mismatch ~1294, `rc` from `conf_save_config` failure ~1309) all `return` before constructing any `repl_config_push_reply_t`. Only the success tail (~1326-1335) builds and sends `{accepted=1, version_tag}`. The outer dispatcher never synthesizes a reply from a nonzero `rc` for this opcode — logged bug-1480. | -| `ReplConfigPushReply` schema | `controller.capnp:805-808`: `struct ReplConfigPushReply { accepted @0: Bool; versionTag @1: UInt64; }` | **Constraint, not itself missing for the reject-reply fix** — adding `accepted=0` needs no schema change (the field already exists); only Requirement 7's term field is a schema addition (see below). | -| Distinguishing reject from activation-deferred | `config_activate_committed` returning `-EAGAIN` (activation gated on committed marker, by design) does **not** early-return in `handle_repl_config_push` — the function still falls through to send `accepted=1` in that case | **Important design nuance for the fix**: `accepted` currently means "bytes saved out-of-band," not "activated." The reject-reply fix must not conflate the two — an `-EAGAIN` activation defer is expected/retriable and must keep sending `accepted=1`; only the four genuine save/validation failures above should send `accepted=0`. | - -### Requirement 3 — AppendEntries send retry/backoff, single authoritative mechanism - -| Asset | Status | -|---|---| -| `raft_send_append_entries` (`raft.c:595`) — protocol-conformant, nextIndex-driven | **Confirmed zero production callers** (only `raft_sim.c` and an `XCE_TESTING` helper `ha_cb_send_append_entries_for_test`) — re-verified this pass via full-repo grep. | -| `ha_cb_send_append_entries_stub` (`ha.c:2533-2574`) — the registered `on_send_append_entries` callback, delegates to `repl_send_append_entries` | Registered (`s_real_cbs`) but **only reachable via `raft_send_append_entries`**, which nothing in the live controller calls — so this callback is wired but dormant, confirmed by its own call graph, not just the sibling-spec note. | -| `repl_broadcast` (`repl.c:295-380`) + `ha_stream_wal_to_standby` (`config_repl.c`, catch-up streamer) | **This is the actual production send/retry mechanism** — unconditional newest-entry broadcast plus heartbeat-driven catch-up range streaming, not nextIndex-indexed. | -| **A prior, already-recorded design decision** — `test_ha_core_safety.c:2474`: *"DECISION: DOCUMENT SAFE-BY-GUARD, do not structurally retire."* Reasoning given: the term-blind branches this dead path would exercise are (a) unreachable with real data today, and (b) even if reached, already guarded elsewhere (`persist_wal_write_replicated`'s (term,lsn,payload-crc) guard + sticky non-promotable fence). | **This materially changes the decision space** for Requirement 3 AC3. A prior investigation already considered and rejected structural retirement, for reasons that still hold. This is not new information the design phase needs to re-derive — it's an existing, reasoned precedent. | - -**Gap tag**: Constraint (design precedent already exists and argues against literal retirement) — see Options below. - -### Requirement 4 — RequestVote/PreVote reply integrity, all failure modes - -| Asset | Status | -|---|---| -| `vote_ep_err_handler` (`ha.c:7564-7608`, read in full this pass) | Confirmed: election-phase branch calls `handle_request_vote_reply(ha, vctx->ctx, vctx->peer_idx, vctx->my_term, 0, 0, 0)` directly — the legacy path, bypassing `vote_reply_dispatch`/`pre_vote_reply_dispatch`. | -| `vote_ep_ctx_t` (`ha.c:46-61`, full struct read this pass) | **Missing a discriminant.** The struct has no field distinguishing a real-vote endpoint from a PreVote endpoint — `peer_idx`, `my_term`, `my_tag`, `my_lsn`, `my_last_log_term/index`, `my_name`, but nothing like `is_prevote`. `vote_ep_err_handler` is registered identically at both call sites (`ha.c:688-689` real-vote, `ha.c:827-828` PreVote) and currently has no way to route correctly even if it tried. | -| `vote_reply_dispatch` (~`ha.c:6333-6394`), `pre_vote_reply_dispatch` (~`ha.c:7982-8000`) | Both exist and are already the correct production tally paths for the connect-time/monitor-time failures bug-1423/1424 fixed. | - -**Gap tag**: Missing (discriminant field + routing branch) — concrete, bounded fix; not an unknown. - -### Requirement 5 — ClientRequest / log append integrity - -| Asset | Status | -|---|---| -| `raft_log_append` (`raft.c:562-593`), `ha_primary_log_append` (`ha.c:4777-4823`), `append_job_wal_entry_local`/`persist_submit_job_payloads_async` (`persist.c:6081-6199`) | Traced and confirmed conformant this pass (persist-before-act, no phantom log-length advance, failures propagated per-payload, first-failure rc propagated across a batch). | - -**Gap tag**: Unknown (test-coverage only) — no code gap found; need to confirm which existing test IDs already assert "failed persist ⇒ no log-length advance ⇒ caller sees the failure" as a locked invariant, or whether this needs a new regression test. Not investigated further this pass (out of gap-analysis scope per the framework's "defer deep research to design phase"). - -### Requirement 6 — AdvanceCommitIndex / commit-source-ownership - -| Asset | Status | -|---|---| -| `raft_advance_commit` (`raft.c:769-821`), `ha_cb_commit_advanced_stub` (`ha.c:2616-2667`) → `commit_set_raft_commit_index` (`commit.c:1082-1095`, advisory-only, monotonic) | Confirmed conformant. | -| `majority_candidate_locked` (`commit.c:121-198`) — the actual authoritative ceiling, capped at `persist_durable_lsn` | Confirmed authoritative; `commit.c:139-155` comment documents *why* a prior `min(durable-majority, raft_commit_index)` coupling was tried and removed (starvation). | -| `XCE_TESTING` assert at `commit.c:222` (`committed_lsn` never exceeds `persist_durable_lsn`) | **Existing regression guard already present** — a strong asset; likely already catches any future accidental re-coupling. | - -**Gap tag**: Low — mostly already regression-guarded by the existing `XCE_TESTING` assert; confirm during design whether an additional explicit test naming `raft_commit_index` as advisory-only is warranted, or whether the existing assert suffices. - -### Requirement 7 — Term discovery completeness - -| Asset | Status | -|---|---| -| Heartbeat-reply (`ha.c:6636-6651`) and WAL-NACK (`ha.c:6746-6784`) term-adoption vectors, both feeding `raft_on_higher_term_observed` (`raft.c:855-874`) | Confirmed fixed and live (bug-1421). | -| `WalAckReply` schema already carries `term @0` | Present — no schema work needed for the WAL-NACK vector. | -| `ReplConfigPushReply` schema (`controller.capnp:805-808`) | **Missing a `term` field** — confirmed by direct read this pass: only `accepted`/`versionTag`. Adding a reply-carried term here is an additive `.capnp` schema change (new field number, e.g. `term @2: UInt64`), not a breaking one. | -| Cap'n Proto codegen build step | **Unknown this pass** — need to confirm which Meson target regenerates the `.capnp.c`/`.capnp.h` from `controller.capnp` (likely `capnpc` invoked from the top-level `meson.build`), and whether the historical `nixSupport/xce.nix` capnpc-pin issue (bug-918, already fixed) has any bearing on a schema *field addition* (as opposed to a `c-capnproto` version bump) — almost certainly not, but worth a one-line confirmation in design rather than assuming. | - -**Gap tag**: Missing (schema field) + Unknown (codegen build-step confirmation, low-risk). - -### Requirement 8 — Restart/recovery reconciliation - -| Asset | Status | -|---|---| -| `ha_sync_applied_lsn_from_recovery`/`ha_sync_raft_log_from_recovery` (`ha.c:5296-5351`), phase2 retry helper (`ha.c:415-501`), promotion call sites (`ha.c:1867-1897`, `2112-2131`), JIT log_len resync (`ha.c:3200-3204`) | Confirmed conformant and already fixed (bug-1042, bug-1477, bug-1475). | - -**Gap tag**: Unknown (test-coverage only, same caveat as Requirement 5/6) — no code gap found. - -## Implementation Approach Options - -### Requirement 1 (audit deliverable) - -- **Option A — Extend `CLAUDE.md`'s existing tables.** Update the "TLA+ Action-to-Code Mapping" and "Protocol Verification Checklist" in place; add rows/checklist items for the reply-integrity guarantees this feature establishes. ✅ single source of truth, no new doc to keep in sync. ❌ CLAUDE.md is meant to stay terse — doesn't have room for full per-finding evidence (file:line trails, live-vs-dormant reasoning) without bloating a document every session reads. -- **Option B — New standalone conformance-map document** (e.g. `.kiro/specs/raft-adapter-conformance/design.md`'s own dedicated section, or a `docs/raft-adapter-conformance.md`). ✅ room for full evidentiary detail. ❌ a second place that can drift from CLAUDE.md's summary table, recreating exactly the "documentation drift" class of problem this feature exists to fix. -- **Option C — Hybrid (recommended direction for design phase).** CLAUDE.md's table stays the terse, always-current pointer (updated as part of this feature's fixes); the full audit trail with citations and verdicts lives in this spec's own `design.md`, referenced *from* CLAUDE.md by name so there is one canonical detailed source and one canonical terse pointer, not two competing summaries. - -### Requirement 2 (rejection reply integrity) - -- **Option A — Extend `handle_repl_config_push` in place**, mirroring the `ha_send_wal_ack_reply_ex` pattern already established for the WAL catch-up fix: build and send a `ReplConfigPushReply{accepted=0}` at each of the four reject early-returns, taking care not to touch the `-EAGAIN` activation-defer path (which correctly stays `accepted=1`). ✅ minimal, uses an established in-codebase pattern, touches one function. ❌ none significant — this is a narrow, well-bounded fix. -- No Option B/C considered necessary — the fix is a direct, small extension of an existing function using a precedent already proven in production (bug-1479). Effort/risk assessment below reflects this. - -### Requirement 3 (single authoritative send/retry mechanism) - -- **Option A — Document as intentionally dormant (recommended, matches existing precedent).** Formalize the `test_ha_core_safety.c:2474` decision into the design doc and CLAUDE.md: `raft_send_append_entries`/`ha_cb_send_append_entries_stub` remain in the codebase as a library-conformant reference implementation exercised only by `raft_sim`/tests, explicitly marked as such; `repl_broadcast` + catch-up streaming remains the sole production mechanism. No behavior change. ✅ lowest risk, respects a reasoned prior decision, zero regression surface. ❌ leaves two implementations in the codebase long-term (a documentation cost, not a correctness one). -- **Option B — Structurally retire the dead path** (delete `raft_send_append_entries`'s production wiring, keep it library-internal/test-only via `#ifdef XCE_TESTING` or move it fully into `raft_sim.c`). ✅ removes the "two implementations" documentation burden entirely. ❌ contradicts the specific reasoning already recorded against this (loses a working, protocol-exact reference implementation `raft_sim` and future specs may want); touches library-public API (`raft.h:378`) that other tests reference directly. -- **Option C — Wire in as the real production path**, replacing `repl_broadcast`/catch-up-streaming with nextIndex-driven sends. ✅ closest to `raft.tla` literally. ❌ highest risk by far — replaces a converging, live-verified (post bug-1479) core replication mechanism with an unexercised-in-production one; would need its own multi-round chaotic-soak verification before trust; effort XL, not justified by any live defect (the current substitute already converges correctly). - -### Requirement 4 (vote/PreVote failure routing) - -- **Option A — Add a discriminant field to `vote_ep_ctx_t`** (e.g. `bool is_prevote`), set at allocation (`ha.c:629` real-vote alloc → `false`, `ha.c:784` PreVote alloc → `true`), then branch inside `vote_ep_err_handler`'s election-phase case to call `vote_reply_dispatch` or `pre_vote_reply_dispatch` accordingly. ✅ single handler retained, minimal struct change, mirrors how `my_term`/`peer_idx` are already threaded per-context. ❌ handler gains a branch, slightly more complex than today. -- **Option B — Split into two handlers** (`vote_ep_err_handler` / `pre_vote_ep_err_handler`), registered at their respective existing call sites (`ha.c:688-689` vs `827-828`, which are already separate registration points today). ✅ no runtime branch, each handler stays single-purpose, matches the existing pattern where connect/monitor failures are *already* handled at separate call sites for the two endpoint kinds. ❌ near-duplicate logic in two functions (the `CTRL_PRIMARY` branch — standby-endpoint cleanup — is identical in both; would need a shared helper to avoid copy-paste). -- Recommendation for design phase: Option B composed with a small shared helper for the `CTRL_PRIMARY` cleanup branch is likely cleanest, but this is a genuine, low-stakes design choice — either is a bounded, low-risk change. - -### Requirements 5, 6, 8 (already-conformant areas) - -- **Option A only, no alternatives needed** — these are regression-lock requirements. The task is to confirm or add test coverage that pins the already-correct behavior (existing `XCE_TESTING` assert at `commit.c:222` for Requirement 6 is a strong existing asset; Requirements 5 and 8 need a design-phase check of which existing test IDs already exercise the traced code paths before deciding whether any new test is needed at all). - -### Requirement 7 (term discovery / config-push term field) - -- **Option A — Add `term @2: UInt64` to `ReplConfigPushReply`, populate it in `handle_repl_config_push`'s reply construction** (both the existing success path and the new Requirement 2 reject paths), and route it through the same `raft_on_higher_term_observed` adoption call already used for the heartbeat and WAL-NACK vectors. ✅ additive schema change (safe, no wire-format break), reuses the established adoption entry point, and — since Requirement 2 already touches every reply-construction site in this function for the reject-reply fix — can be implemented in the same edit pass with no extra integration surface. This pairing should be called out explicitly in the design/tasks phase so it isn't accidentally split into two separate change sets touching the same lines twice. - -## Effort & Risk - -| Requirement | Effort | Risk | Justification | -|---|---|---|---| -| 1 — Audit deliverable | S (1-3 days) | Low | Documentation + citation verification; no code risk. | -| 2 — Rejection reply integrity | S (extends one existing function with a proven pattern) | Low | Bounded to `handle_repl_config_push`; must respect the `-EAGAIN` activation-defer nuance found this pass, otherwise trivial. | -| 3 — Single authoritative send mechanism | S if Option A (document only) / XL if Option C (wire in) | Low (Option A) to High (Option C) | A prior in-codebase decision already argues for Option A; Option C would require full chaotic-soak reverification of a currently-working guarantee for no confirmed live defect. | -| 4 — Vote/PreVote failure routing | S-M | Low-Medium | Small struct/handler change, but touches election-path code; needs a live/soak test that can actually inject a post-connect (not connect-time) endpoint failure — that injection capability's current existence is unconfirmed (see Research Needed). | -| 5 — ClientRequest integrity | S (test-only, likely) | Low | No code gap found; effort is test-coverage confirmation. | -| 6 — Commit source-of-truth | S (test-only, likely) | Low | Existing `XCE_TESTING` assert already guards this. | -| 7 — Term discovery completeness | S-M (schema change + wiring, shares an edit pass with Requirement 2) | Low-Medium | Additive schema field is low-risk; the only real unknown is confirming the capnp codegen build step, not the change itself. | -| 8 — Restart/recovery reconciliation | S (test-only, likely) | Low | No code gap found; already-fixed by bug-1042/1477/1475. | - -## Research Needed (carried forward to design phase) - -1. **Test-coverage confirmation for Requirements 5, 6, 8** — identify the specific existing test IDs (in `test_ha_core_safety.c`, `test_apply_pump.c`, `test_commit_durability.c`, or elsewhere) that already lock each already-conformant guarantee, versus needing a net-new regression test. Not investigated deeply this pass per the gap-analysis framework's "defer deep research to design." -2. **Post-connect endpoint failure injection** — confirm whether the existing test harness (`test_ha_vote_send.c` or similar) can already simulate a failure arriving *after* a successful connect+monitor (as opposed to a connect-time failure, which is already tested per bug-1423/1424's regression coverage) — needed to properly test the Requirement 4 fix. -3. **Cap'n Proto codegen build step** — confirm the exact Meson target/invocation that regenerates `controller.capnp.c/.h` from the schema, for the Requirement 7 field addition. Low risk expected (additive field, not a `c-capnproto` version change like bug-918), but unconfirmed this pass. -4. **`handle_repl_config_push`'s WAL catch-up sibling regression test scope** — re-confirm (rather than assume from prior-session notes) exactly which `test_ha_core_safety.c` test IDs assert the bug-1479 WAL-ACK reject-reply fix, so the new config-push reject-reply test can be modeled on the same pattern. -5. **Requirement 3 wording tension** — Requirement 3 AC3 currently reads "...shall be either retired or made the production path." The asset found this pass (`test_ha_core_safety.c:2474`'s prior "document, do not structurally retire" decision) suggests a third real option — formally document as intentionally dormant — that isn't literally "retired." Flag for the design phase to either treat "retired" broadly enough to cover a documented-dormant-with-guard outcome, or revisit the requirement's wording before design finalizes. - -## Recommendations Summary - -- Requirements 2 and 7 should be planned as a single combined change (same function, same schema, same edit pass) rather than sequenced separately. -- Requirement 3's design decision should start from the existing `test_ha_core_safety.c:2474` precedent (Option A, document-only) rather than re-litigating retire-vs-wire-in from scratch; only revisit if the design phase surfaces a concrete reason the prior reasoning no longer holds. -- Requirements 5, 6, and 8 are low-effort, test-coverage-confirmation items — worth batching into one design-phase task rather than three separate ones. -- Requirement 4's Option A vs B (discriminant field vs split handler) is a genuine, low-stakes design choice; either is acceptable, defer to design phase author preference. - ---- - -# Design-Phase Discovery & Decisions - -This section extends the gap analysis above with the design-phase discovery pass and the decisions it resolved. Discovery type: **Extension** (existing controller subsystem, no new external dependency) — light discovery process applied. - -## Summary -- **Feature**: `raft-adapter-conformance` -- **Discovery Scope**: Extension -- **Key Findings**: - - The Cap'n Proto codegen path for the schema change (Req 2/7) is a pre-existing, unconditional Meson `custom_target` (`src/lib/meson.build:1-10`, `capnp compile -o c:lib`) — an additive field needs no build-system change, closing Research Needed item 3 from the gap analysis with high confidence. - - `ha_send_wal_ack_reply_ex` (`ha.c:3095-3116`) already establishes the exact pattern to mirror for the config-push reject-reply fix: an `XCE_TESTING`-only send-hook (`ha_wal_ack_send_hook`) that lets a unit test observe the reply payload without a live endpoint. This resolves the "how do we test this" question for Req 2 by precedent rather than invention. - - Existing regression coverage was confirmed strong for three of the four "already-conformant" requirements: Req 6 (`test_term_safety_adapter.c:646-780`, explicit bidirectional advisory-vs-authoritative pins) and Req 8 (`test_persist_recovery.c`, `test_config_repl.c`) already lock the guarantees. Req 5 has partial coverage (`test_commit_barrier_rollout.c`, `test_commit_barrier_reply.c`) but no test was found asserting batch-wide failure propagation specifically — flagged as a small net-new test item, not a code gap. - - No existing test exercises `vote_ep_err_handler` at all (`grep` across `test_ha_vote_send.c` for the handler name, `RDHUP`, or `on_err` returned zero matches) — Req 4 needs net-new test infrastructure, not just a net-new test case. This is a real (small) infrastructure gap, not a research unknown anymore. - - No existing test was found asserting that a WAL catch-up reject actually *sends* `WalAckReply{success=0}` on the wire (as opposed to testing what happens downstream once a NACK is routed) — the bug-1479 fix itself is under-tested at the reply-construction boundary. Folded into this feature's Testing Strategy as a regression-lock item for Req 2.1. - -## Research Log - -### Cap'n Proto schema regeneration path -- **Context**: Requirements 2 and 7 both need `ReplConfigPushReply` to gain a `term` field; needed to confirm this doesn't require a Nix/build change given the historical `nixSupport/xce.nix` capnpc-pin issue (bug-918, memory-recorded from a prior session). -- **Sources Consulted**: `src/lib/meson.build` (read directly), `src/schema/controller.capnp` (read directly). -- **Findings**: `src/lib/meson.build:1-10` unconditionally regenerates `controller.capnp.c/.h` from `src/schema/controller.capnp` on every build via `capnp compile -o c:lib` using the `capnpc-c` plugin and the `c-capnproto` dependency. Bug-918 was about the Nix derivation's *own* pin for the `c-capnproto` toolchain source (a mutable upstream `main` ref), not about schema content — an additive field is orthogonal to that concern. -- **Implications**: No meson/Nix change needed for the schema addition; a normal `/xce-build controller` (or `all`) picks it up. No new Technology Stack row needed beyond noting the existing Cap'n Proto toolchain already covers this. - -### Existing reply-hook test pattern -- **Context**: Needed a concrete, low-risk way to unit-test the new config-push reject-reply construction without standing up a live endpoint pair. -- **Sources Consulted**: `ha.c:3085-3131` (`ha_wal_ack_send_hook`, `ha_send_wal_ack_reply_ex`, `ha_send_wal_ack_reply`), read directly. -- **Findings**: The WAL-ACK path already has exactly this shape: an `XCE_TESTING`-guarded global function pointer that, when set, receives the constructed reply payload instead of the function sending over the wire. This is bug-1479's own test-support mechanism. -- **Implications**: The config-push reject-reply fix should add an analogous `XCE_TESTING`-only hook (e.g. `ha_config_push_reply_send_hook`) rather than inventing a different test-injection mechanism — keeps the two reply-integrity fixes symmetric and reviewable by the same pattern. - -### Regression-coverage confirmation for Requirements 5, 6, 8 -- **Context**: Gap analysis flagged these as "Unknown — test-coverage only"; needed to resolve before Testing Strategy could be written concretely. -- **Sources Consulted**: `grep` across `src/controller/testsuite/*.c` for the traced functions/concepts, read matched line ranges directly. -- **Findings**: Requirement 6 has strong existing dedicated coverage (`test_term_safety_adapter.c`, tests explicitly pin both directions: a lagging advisory index does not cap `committed_lsn`, and a well-ahead advisory index does not raise it past the durable-majority floor). Requirement 8 has existing coverage in `test_persist_recovery.c` and `test_config_repl.c`. Requirement 5 has partial coverage; no test specifically named or scoped to "one failed payload in a batch propagates to all" was found. -- **Implications**: Requirements 6 and 8 need no new tests — the design's Testing Strategy references the existing test files as the regression lock. Requirement 5 gets one small net-new test item (batch failure propagation) rather than a full new suite. - -### Vote endpoint failure-path test infrastructure -- **Context**: Needed to confirm whether `vote_ep_err_handler`'s failure path (the Requirement 4 fix target) has any existing test scaffolding. -- **Sources Consulted**: `grep -n "vote_ep_err_handler\|RDHUP\|on_err" src/controller/testsuite/test_ha_vote_send.c` — zero matches. -- **Findings**: No existing test drives `vote_ep_err_handler` at all, for either the already-fixed connect-time paths or the not-yet-fixed post-connect path. This is a genuine test-infrastructure gap (need a way to invoke the handler directly with a synthetic `vote_ep_ctx_t`/endpoint, e.g. a small `XCE_TESTING`-only constructor mirroring `vote_ep_ctx_build_for_test` at `ha.c:7337`, which already exists for a different purpose and can likely be reused/extended). -- **Implications**: Requirement 4's task will need a small test-support addition (reusing `vote_ep_ctx_build_for_test`), not just a new test case in isolation. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Extend in place (chosen for all code fixes) | Add the missing reply/routing logic directly inside the existing adapter functions (`handle_repl_config_push`, `vote_ep_err_handler`) | Matches the bug-1479 precedent exactly; no new abstraction; minimal review surface | None material — these are narrow, well-bounded functions already | Consistent with `ha_send_wal_ack_reply_ex`'s own extension of `ha_send_wal_ack_reply` | -| New shared "reply integrity" abstraction (rejected) | A generic cross-cutting helper/interface for "send an explicit reply on every reject path" spanning WAL-ACK, config-push, and vote replies | Would prevent the class of bug from recurring elsewhere by construction | The three reply shapes (`WalAckReply`, `ReplConfigPushReply`, `RequestVoteReply`) are distinct Cap'n Proto structs with different fields and different dispatch semantics (one is a direct reply, one is a tally-feed); forcing a shared interface over them adds indirection with only three call sites — fails the Simplification lens (no second concrete beneficiary today) | Documented here so a future spec doesn't have to re-derive why this was rejected | - -## Design Decisions - -### Decision: Documentation-only resolution for the dead nextIndex-driven AppendEntries sender (Requirement 3) -- **Context**: `raft_send_append_entries`/`ha_cb_send_append_entries_stub` is fully protocol-conformant but has zero production callers; `repl_broadcast` + catch-up streaming is the actual production mechanism. -- **Alternatives Considered**: - 1. Retire the dead path (remove production wiring, keep library-only/test-only). - 2. Wire it in as the real production path, replacing `repl_broadcast`. - 3. Formally document it as intentionally dormant, matching the reasoning already recorded at `test_ha_core_safety.c:2474`. -- **Selected Approach**: Option 3. No source code changes. Update `src/controller/CLAUDE.md`'s TLA+ Action-to-Code Mapping to note `AppendEntries(i,j)` production behavior is `repl_broadcast`/`ha_stream_wal_to_standby`, with `raft_send_append_entries` explicitly marked as a dormant, protocol-exact reference path exercised only by `raft_sim`/tests. Add a one-line cross-reference note to `ha-term-safety/spec.json`'s existing `known_limitations` entries (index 30) pointing at this spec's resolution, so a future reader lands on the authoritative decision instead of re-discovering the drift. -- **Rationale**: A prior investigation already weighed retire-vs-keep and rejected retirement because the term-blind branches this path would otherwise expose are unreachable with real data today and independently guarded by `persist_wal_write_replicated`'s conflict guard even if reached. Re-litigating that decision without new information would violate the Simplification lens in reverse (removing working, guarded code for documentation tidiness alone). Wiring it in (Option 2) is rejected as disproportionate risk (XL effort, full chaotic-soak reverification) for zero confirmed live defect in the current mechanism. -- **Trade-offs**: Keeps two implementations of the same guarantee in the codebase long-term (a documentation cost); avoids all correctness/regression risk. -- **Follow-up**: Requirement 3's task should be scoped as a documentation task with zero `src/controller/*.c` diff; reviewers should treat any code-touching PR for this requirement as scope creep unless a new live defect is found. - -### Decision: Split `vote_ep_err_handler` into two handlers sharing a helper (Requirement 4) -- **Context**: `vote_ep_ctx_t` has no field distinguishing a real-vote endpoint from a PreVote endpoint, and the shared handler's election-phase branch calls the legacy `handle_request_vote_reply` instead of the production tally path for post-connect failures. -- **Alternatives Considered**: - 1. Add a boolean discriminant field to `vote_ep_ctx_t` and branch inside the one handler. - 2. Split into `vote_ep_err_handler` / `pre_vote_ep_err_handler`, each registered at its existing distinct call site, sharing a small helper for the identical `CTRL_PRIMARY` cleanup branch. -- **Selected Approach**: Option 2. -- **Rationale**: The two endpoint kinds are *already* registered at distinct call sites today (`ha.c:688-689` vs `827-828`) even though they share the `vote_ep_ctx_t` type — the split mirrors an architectural seam that already exists, rather than introducing a new one. It avoids adding a field that only exists to support a runtime branch in one function. -- **Trade-offs**: Two small handler functions instead of one slightly larger one; a shared static helper for the `CTRL_PRIMARY` branch avoids duplicating that logic. -- **Follow-up**: The task should reuse `vote_ep_ctx_build_for_test` (`ha.c:7337`) as the test-construction seam for both new handlers' unit tests, per the test-infrastructure finding above. - -## Risks & Mitigations -- Adding `term` to `ReplConfigPushReply` changes a wire struct consumed during rolling upgrades — mitigated because Cap'n Proto field addition is backward/forward compatible by construction (old readers ignore the new field; an old sender's reply decodes with `term == 0` on a new-code primary, which is naturally safe since `0` can never look like a *higher* term than the primary's own and so triggers no adoption — see Migration Strategy in `design.md`). -- Splitting `vote_ep_err_handler` touches live election-path code — mitigated by keeping the `CTRL_PRIMARY` branch behavior byte-for-byte identical (extracted, not rewritten) and requiring the new dedicated unit tests (via the reused test-construction helper) before this task is considered done. -- No test currently exists for the exact reply-construction boundary of the already-shipped bug-1479 fix — mitigated by adding it as an explicit Requirement 2.1 regression-lock test in this feature rather than assuming the existing catch-up/fence tests already cover it. - -## References -- `src/controller/testsuite/test_ha_core_safety.c:2440-2480` — the prior "DOCUMENT SAFE-BY-GUARD, do not structurally retire" decision informing Requirement 3. -- `src/lib/meson.build:1-10` — Cap'n Proto codegen custom_target. -- `src/controller/ha.c:3085-3131` — `ha_send_wal_ack_reply_ex`/`ha_wal_ack_send_hook`, the precedent pattern for Requirement 2's test hook. diff --git a/.kiro/specs/raft-adapter-conformance/spec.json b/.kiro/specs/raft-adapter-conformance/spec.json deleted file mode 100644 index 84896f92..00000000 --- a/.kiro/specs/raft-adapter-conformance/spec.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "feature_name": "raft-adapter-conformance", - "created_at": "2026-07-29T03:10:00Z", - "updated_at": "2026-08-15T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "related_bugs": ["bug-1479", "bug-1480", "bug-1481", "bug-1466", "bug-1471", "bug-1472", "bug-1474", "bug-1475", "bug-1477", "bug-1478", "bug-1423"], - "related_specs": ["raft-log-replication", "raft-commit-safety", "ha-term-safety", "raft-sim-conformance", "config-upgrade-snapshot", "ha-config-bootstrap", "ha-commit-engine"], - "validate_impl": { - "decision": "GO", - "validated_at": "2026-08-15T00:00:00Z", - "note": "Corrected stale phase metadata (was 'tasks-generated' though every task 1-7 was already [x] complete: config-push reply integrity, vote/pre-vote post-connect failure routing, dormant-sender guard, regression locks, live soaks).", - "known_limitations": [ - "bug-1474 (angle-a only; the repl.c defense-in-depth slot-reuse hardening is already fixed elsewhere): a spurious duplicate/late-joiner reconnect during WAL trim, root-caused to a vote_ep_ctx_t.peer_idx convention mismatch in ha.c's heartbeat-reconnect-sweep (~ha.c:9219-9254). Never converted into a task here despite being tracked in related_bugs -- this spec's tasks 1-7 are entirely about config-push reply integrity and vote/pre-vote failure routing, a different part of ha.c. Natural fit for a new appended task given its scope match.", - "bug-1471 / bug-1472-remainder: listed in related_bugs but never converted into a task -- root mechanism (worker.c/persist.c's WAL_WORKER_CTRL apply path) is outside this spec's ha.c/repl.c-adapter boundary; see config-upgrade-snapshot's own known_limitations for the primary tracking." - ] - } -} diff --git a/.kiro/specs/raft-adapter-conformance/tasks.md b/.kiro/specs/raft-adapter-conformance/tasks.md deleted file mode 100644 index 22b9a75d..00000000 --- a/.kiro/specs/raft-adapter-conformance/tasks.md +++ /dev/null @@ -1,127 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: extend the configuration-push reply schema with a term field - - Add `term @2: UInt64` to `ReplConfigPushReply` in `src/schema/controller.capnp`, alongside the existing `accepted`/`versionTag` fields - - Rebuild the controller so the Cap'n Proto codegen `custom_target` regenerates `controller.capnp.c/.h` with the new field - - Observable: `repl_config_push_reply_t` exposes a `term` member usable from `config_repl.c` and `cluster.c`; existing config-push tests still build and pass unmodified - - _Requirements: 7.4_ - -- [ ] 2. Core: configuration-push reply integrity -- [x] 2.1 (P) Send an explicit accept/reject reply carrying term on every configuration push outcome - - In `handle_repl_config_push`, construct and send `ReplConfigPushReply{accepted=0, term=current_term}` on each of the four reject early-returns (malformed request, missing bytes/sha256, sha256 mismatch, `conf_save_config` failure) instead of returning without a reply - - Populate `term` on the existing success reply path too - - Preserve the existing `-EAGAIN`-from-`config_activate_committed` behavior unchanged: that path still sends `accepted=1` (activation-deferred is not a rejection) - - Add an `XCE_TESTING`-only send-hook (mirroring `ha_wal_ack_send_hook`) so a unit test can observe the constructed reply payload without a live endpoint - - Observable: every code path through `handle_repl_config_push` now sends exactly one `ReplConfigPushReply`, with no path returning silently - - _Requirements: 2.2, 2.3, 2.5, 7.4_ - - _Boundary: config_repl.c reply construction_ - -- [x] 2.2 (P) Wire term-adoption into the primary-side configuration-push reply handler - - In the existing `ReplConfigPushReply` branch of `myreq_handler` (`cluster.c`, the code that already checks `cack->accepted` and calls `repl_on_config_ack` or warns-and-ignores), add a term comparison: when `cack->term` exceeds the primary's current term, call `raft_on_higher_term_observed` — the same adoption entry point the heartbeat-reply and WAL-NACK vectors already use - - Do not alter the existing accept/reject branching logic itself, only add the term check alongside it - - Observable: a config-push reply carrying a higher term causes the primary to adopt that term and step down from leadership held at the lower term - - _Requirements: 7.4, 7.5_ - - _Boundary: cluster.c reply handling_ - -- [x] 2.3 Test configuration-push reply integrity end-to-end - - Unit test: each of the four reject conditions in `handle_repl_config_push` produces `accepted=0` with the current term, via the new send-hook - - Unit test: the `-EAGAIN` activation-defer path still produces `accepted=1` - - Unit test: an inbound config-push reply carrying a higher term triggers adoption via the `cluster.c` wiring from 2.2 - - Observable: new tests in `test_config_repl.c` fail against pre-fix code and pass against the code from 2.1/2.2 - - _Requirements: 2.2, 2.3, 2.4, 2.5, 7.4, 7.5_ - - _Depends: 2.1, 2.2_ - -- [ ] 3. Core: vote and pre-vote endpoint failure routing -- [x] 3.1 (P) Split the shared vote-endpoint error handler and route post-connect failures to the production tally - - Split `vote_ep_err_handler` into a real-vote handler (registered at its existing connection-setup call site) and a `pre_vote_ep_err_handler` (registered at its existing, separate PreVote connection-setup call site) - - Extract the identical `CTRL_PRIMARY`-role cleanup branch (standby-endpoint deregistration, `repl_remove_standby`, quorum-transition logging) into one shared helper called by both, unchanged in behavior - - Change each handler's election-phase branch to route a post-connect failure through the same production tally path already used for connect-time failures (`vote_reply_dispatch` for the real-vote handler, `pre_vote_reply_dispatch` for the pre-vote handler) instead of the legacy `handle_request_vote_reply` - - Observable: a failure arriving after a vote or pre-vote endpoint has already connected now reaches the same tally counter a connect-time failure or a delivered denial would - - _Requirements: 4.2, 4.3, 4.4_ - - _Boundary: ha.c vote endpoint handlers_ - -- [x] 3.2 Test post-connect vote and pre-vote failure routing - - Extend the existing test-construction helper (`vote_ep_ctx_build_for_test`) to synthesize a post-connect failure on a real-vote endpoint and assert it reaches `vote_reply_dispatch`'s tally - - Do the same for a PreVote endpoint against `pre_vote_reply_dispatch` - - Confirm a round where every peer's outcome is now counted (including this failure mode) reaches a quorum decision without waiting for the full round timeout - - Observable: new tests in `test_ha_vote_send.c` fail against pre-fix code (tally never updates) and pass against the code from 3.1 - - _Requirements: 4.2, 4.3, 4.5_ - - _Depends: 3.1_ - -- [ ] 4. Core: document the dormant AppendEntries sender and guard it against silent regression -- [x] 4.1 Record the production-authoritative send mechanism and add a non-regression guard - - Extend the `on_send_append_entries` doc-comment (`ha.c`, the `ha_cb_send_append_entries_stub` callback) to state plainly that this callback is reachable only via `raft_send_append_entries`, which has no production caller, and that `repl_broadcast`/`ha_stream_wal_to_standby` are the actual production `AppendEntries`-equivalent send/retry/backoff mechanism - - Add an `XCE_TESTING`-only call counter to `ha_cb_send_append_entries_stub` - - Observable: the counter exists and is queryable from a test; the doc-comment names both the dormant path and the live substitute explicitly - - _Requirements: 3.3_ - -- [x] 4.2 Test that the dormant sender stays uncalled in production - - Drive the production heartbeat/replication path end-to-end in a test and assert the new call counter stays at zero - - In the same test, confirm `repl_broadcast`/`ha_stream_wal_to_standby` did fire for the same scenario, proving the assertion isn't vacuous - - Observable: the test fails if a future change accidentally gives `raft_send_append_entries` a production caller - - _Requirements: 3.3_ - - _Depends: 4.1_ - - _Boundary: test_ha_core_safety.c_ - -- [ ] 5. Core: regression-lock tests for already-shipped reply and append behavior -- [x] 5.1 (P) Test that a WAL catch-up rejection actually sends the reply on the wire - - Add a test asserting `ha_send_wal_ack_reply_ex(success=0)` is invoked (via the existing `ha_wal_ack_send_hook`) when a WAL catch-up entry is rejected, closing the reply-construction-boundary coverage gap that existed alongside the already-shipped bug-1479 fix - - Observable: new test in `test_ha_core_safety.c` fails if the reject branch is ever changed back to a silent drop - - _Requirements: 2.1_ - - _Boundary: test_ha_core_safety.c_ - -- [x] 5.2 (P) Test that one failing payload in a submitted batch propagates to the whole batch - - Add a test asserting `persist_submit_job_payloads_async` resolves every payload in a batch with the first failure's outcome when one payload fails to persist, rather than completing a subset - - Observable: new test in `test_commit_barrier_reply.c` fails if a future change lets a batch partially succeed - - _Requirements: 5.3_ - - _Boundary: test_commit_barrier_reply.c_ - -- [x] 6. Integration: extend the controller's durable conformance record with this feature's findings - - Update `src/controller/CLAUDE.md`'s "TLA+ Action-to-Code Mapping" table and "Protocol Verification Checklist" with corrected/added rows covering: `AppendEntries` reply integrity (WAL catch-up and configuration push), vote/PreVote failure routing across all failure timings, and the dormant-sender resolution from task 4 - - Append a one-line pointer to the existing `known_limitations` entry (index 30) in `.kiro/specs/ha-term-safety/spec.json`, referencing this spec's resolution of the dormant `on_send_append_entries` send-path drift - - Observable: the CLAUDE.md table cites the current, post-fix code paths for every action this feature touched, with no reference to the pre-fix silent-drop behavior as if it were still current; a reader of `ha-term-safety`'s known_limitations lands on this spec's resolution instead of re-discovering the drift - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_ - - _Depends: 2.1, 2.2, 3.1, 4.1, 4.2_ - -- [ ] 7. Validation: regression confirmation and live verification -- [x] 7.1 (P) Confirm append and commit-source regression coverage is unaffected - - Run `test_commit_barrier_rollout.c`, `test_commit_barrier_reply.c` (including the new test from 5.2), and `test_term_safety_adapter.c` and confirm all pass without modification to the already-conformant code they exercise - - Observable: test run output confirms client-append-integrity and durable-majority-authoritative behavior are unchanged - - _Requirements: 5.1, 5.2, 5.4, 6.1, 6.2, 6.3, 6.4_ - - _Depends: 5.2_ - - _Boundary: test_commit_barrier_rollout.c, test_commit_barrier_reply.c, test_term_safety_adapter.c_ - -- [x] 7.2 (P) Confirm election, replication, and recovery regression coverage is unaffected - - Run `test_ha_core_safety.c` (including the new tests from 5.1 and 4.2), `test_ha_vote_send.c` (including the new tests from 3.2), `test_persist_recovery.c`, and `test_config_repl.c` (including the new tests from 2.3; also confirms 2.4's existing timeout/unknown-outcome handling, unchanged) and confirm all pass - - Run `test_repl_catchup.c`, `test_repl_unit.c`, `test_repl_majority.c`, and `test_ha_repl_push.c` and confirm all pass, closing the coverage gap for the Requirement 3 retry/backoff/catch-up-streaming-ceiling regression lock (`ha_raft_on_repl_nack`, `repl_start_catchup`, `repl_broadcast`) - - Observable: test run output confirms the already-fixed retry/backoff, catch-up streaming ceiling, connect-time vote-failure routing, term-adoption vectors, restart/recovery reconcile, and the new dormant-sender guard all remain correct - - _Requirements: 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.4, 7.1, 7.2, 7.3, 7.5, 8.1, 8.2, 8.3, 8.4_ - - _Depends: 5.1, 2.3, 3.2, 4.2_ - - _Boundary: test_ha_core_safety.c, test_ha_vote_send.c, test_persist_recovery.c, test_config_repl.c, test_repl_catchup.c, test_repl_unit.c, test_repl_majority.c, test_ha_repl_push.c_ - -- [x] 7.3 Live soak: configuration-push rejection end-to-end - - Run a rolling chaotic-failover soak where the test driver issues a configuration push whose payload deliberately does not match its advertised sha256, exercising the real `handle_repl_config_push` SHA-256-mismatch reject branch - - Observable: the soak log shows the primary receiving and logging the explicit reject rather than a timeout - - _Requirements: 2.2, 2.3, 2.5, 7.4, 7.5_ - - _Depends: 2.1, 2.2_ - -- [x]* 7.4 (P) Live soak: vote/PreVote post-connect failure end-to-end (optional, defer if no fault-injection mechanism exists) — NOT deferred, ran successfully - - Attempt a soak round that severs an already-established outbound vote connection before a reply arrives (e.g. a firewall-rule toggle or proxy in the soak harness); if no such mechanism exists cheaply in `xce-test-ha`, defer this task — the unit tests from 3.2 already give deterministic coverage of the fix itself - - Observable, if run: the soak log shows the affected round reaching a quorum decision within the fast-tally path rather than the full election timeout - - _Requirements: 4.2, 4.3, 4.5_ - - _Depends: 3.1_ - - _Boundary: xce-test-ha soak harness_ - -## Implementation Notes - -- Task 1: adding a `$C.mapname`-tagged capnp field requires a companion edit to the hand-written C struct in `src/include/typemap.h` (the codegen writes into that struct by name; it does not synthesize it). `ReplConfigPushReply.term` required `struct repl_config_push_reply_s.term` in typemap.h — design.md's File Structure Plan didn't call this out, but it's the established, required pattern for every existing `$C.mapname` struct in that file, not scope creep. Any future task adding a capnp field should expect the same pairing. -- Authoritative test verification recipe (confirmed working): `rm -rf /tmp/xce_*` then run `ASAN_OPTIONS=detect_leaks=0 src/xce.build/controller/controller_unit_test` directly and read its own stdout `N Tests M Failures` line — meson's aggregate Ok/Fail can show a spurious TIMEOUT/inline FAIL under its own harness even when the direct binary run is clean (1053 Tests 0 Failures confirmed for task 1's baseline). -- Task 2.2 (bug-1482): `myreq_handler`'s first unconditional step is `xce_decode(pkt, &request)`, which itself unconditionally `free_packet(pkt)` + frees the `io_packet_t` struct before returning on every path — `pkt` ownership transfers into `xce_decode`, the caller must never touch/free it afterward. Any new `_for_test` wrapper around `myreq_handler` (or `xce_decode` callers generally) must document this, not the reverse ("caller owns/frees pkt") — that wrong assumption is what caused the original test-only wrapper's doc comment to mislead a reviewer into requesting a double-free. -- Task 2.2: driving a test through `cluster.c`'s real `myreq_handler` (rather than testing extracted logic in isolation) requires adding `cluster.c` to `controller_unit_test_src` in `meson.build` plus link-satisfying fakes for symbols its OTHER unrelated switch-cases reference (task.c/opget.c/sched.c/apiserver.c/myconf.c) — this is pre-existing, established convention in this codebase (`test_worker_join.c`, `test_commit_barrier_reply.c` already do it for the same reason), not scope creep. Any new test driving `myreq_handler` end-to-end should expect this same footprint. -- New test-only wrappers exposing a `static` production dispatch function (the `_for_test` pattern, e.g. `peer_ctrl_msg_handler_for_test`, `myreq_handler_for_test`) MUST be wrapped in `#ifdef XCE_TESTING`/`#endif` — the source file compiles into the production binary too, and an unguarded wrapper ships as a new linked symbol in production with zero production purpose. -- Task 3.2 (bug-1483, not fixed): driving a PreVote round to `RAFT_PRE_VOTE_LOST` in a unit test (`c==NULL`) without first stubbing `ha_prevote_rearm_liveness_hook` falls through to the real `ha_arm_heartbeat_timeout_vtask`, which callocs into the static `s_liveness_ctx` and never frees it when `c==NULL` short-circuits the scheduling guard. `test_ha_prevote.c`'s existing LOST-path tests already stub-and-reset this hook — any new PreVote-round test that reaches LOST must do the same. -- `pre_vote_reply_dispatch` has no `XCE_TESTING` mock-hook seam (unlike `vote_reply_dispatch`'s `xce_vote_reply_hook`) — it calls `raft_on_pre_vote_reply` directly. Tests verifying PreVote routing must assert on `raft_state_t`'s own tally fields (`pre_vote_replies[]`, `pre_votes_granted`, `pre_vote_in_progress`) instead, per `test_ha_prevote.c`'s established technique. -- Task 7.2 (bug-1484, residual gap, NOT fixed — outside this spec's task list): no test anywhere in `testsuite/` drives the real `repl_ack_decode_and_dispatch` path and asserts `raft_on_higher_term_observed` fires for either the heartbeat-reply (ha.c ~6690-6711) or WAL-NACK (ha.c ~6805-6814) term-discovery vector bug-1421 introduced. `test_intake_higher_term_heartbeat_adopts_term` looks like it covers this but actually drives a different AE-request code path. Requirement 7.5's "already verified" premise currently rests on unverified ground for these two vectors specifically (the third, config-push, vector added by this feature IS tested — task 2.2). A future task should add a dedicated test mirroring `test_config_push_reply_higher_term_adopts_and_steps_down`'s pattern for the other two. -- Task 7.3: the real elected-primary's own outbound config-push has no TOCTOU window to inject a sha256 mismatch without new production/relay code (primary hashes and pushes from the same in-memory buffer) — confirmed by reading `config_commit_change`/`push_config_to_standbys`. Live soaks of this specific reject path must act as a direct client/sender against a standby (using `xce_replconfigpush_req_raw`, the real production wire function) rather than corrupting a real primary's push; decode the `ReplConfigPushReply` envelope via `xce_reply()`'s `Request{opcode:reply, body.replymsg}` wrapper (not a bare `RequestReply`) to get real accepted/term content instead of only inferring correctness from the receiver's local log line. -- Task 7.4: found a clean, cheap, reusable live fault-injection technique for "post-connect, pre-reply" endpoint failures — vote/PreVote endpoints are TLS (`ep->ssl = SSL_new(...)`), and `endpoint_connect()`'s return value comes only from the TCP-level `socket_connect()`, never from the TLS handshake (`netio.c:875-903`). Standing up a throwaway TCP listener on a configured peer's port that accepts-then-closes without any TLS bytes makes `endpoint_connect()` succeed (so the candidate genuinely reaches "connected") while deterministically causing an async `SSL_do_handshake` failure milliseconds later — landing squarely in `vote_ep_err_handler`/`pre_vote_ep_err_handler` (task 3.1's fix target), never the already-fixed synchronous connect-refused path (bug-1423). `iptables`/`nft` aren't installed in this sandbox and `tc netem` needs `CAP_NET_ADMIN` (not available); ptrace/gdb mid-flight attach is blocked by `yama ptrace_scope=1`. This accept-then-close-impostor technique generalizes to any "TLS-tier endpoint fails after TCP connect" live-soak need. -- `xce-deploy-ha/SKILL.md` documents `VERBOSE=-vvv scripts/start.sh ctrl-N &`, but `start.sh` never reads `$VERBOSE` — only positional args. Use `-vvv` as a positional arg (`scripts/start.sh ctrl-N -vvv &`), matching `xce-test-ha/SKILL.md`'s documented form, not the env-var form. diff --git a/.kiro/specs/raft-apply-decoupling/brief.md b/.kiro/specs/raft-apply-decoupling/brief.md deleted file mode 100644 index 68627ef5..00000000 --- a/.kiro/specs/raft-apply-decoupling/brief.md +++ /dev/null @@ -1,86 +0,0 @@ -# Brief: raft-apply-decoupling - -## Problem - -The controller's Raft consensus protocol is **embedded inside the notification hook fan-out**, and in-memory state is mutated **before** the entry is committed. This couples three concerns that canonical Raft keeps separate (log → commit → apply → notify): - -- `persist_job_hook_fun` is registered as an `AFTER_EVENT` hook, and *inside* that callback it performs the WAL append (`ha_primary_log_append` → `raft_log_append`) **and** blocks on the synchronous majority-commit barrier (`repl_wait_majority_ack`). The raft protocol step is therefore literally a side effect of running a hook (`persist.c:2532`, `persist.c:2198`, `ha.c:3080`). -- `handle_jobsubmit` mutates the in-memory job table via `create_job()` **before** the WAL/commit step (`job.c:558` runs before the AFTER hook at `job.c:593`). A commit that fails or times out does **not** cleanly revert that optimistic mutation, leaving a window where in-memory state diverges from the committed log. -- Because apply happens before commit, every downstream consumer (scheduler push, client reply) must defensively re-gate on a commit watermark (`sched.c:38` `job_submit_committed`) to avoid acting on not-yet-committed state — scattering commit-awareness across the codebase instead of centralizing it at one apply point. - -This is the structural root behind the bug-795/828/851/1060 divergence family: the controller has no single, ordered "apply committed entry to the state machine" seam. - -## Current State - -- Hooks (`src/lib/hook.c`) are a typed pub/sub fan-out (`xce_run_hooks`) with `HOOK_BEFORE_EVENT` / `HOOK_AFTER_EVENT` situations. Subscribers: `persist_job_hook_fun` (durability — does WAL+commit), `sched_job_hook_fun` (scheduler push), apiserver SSE callbacks. -- Order today: `BEFORE hook → create_job() (state mutation) → AFTER hook { WAL append + majority commit barrier } → client reply → scheduler push (commit-gated)`. -- The raft library (raft-log-replication, raft-commit-safety) already OWNS the replicated log and exposes pure callbacks (`on_persist_entry`, `on_commit_advanced`, etc.). The library half is the right protocol core; the controller half is what is mis-layered. -- Restart replay (`persist_recover` → `replay_wal` → `apply_wal_entry`, `persist.c:1950/1735/1606`) validates only the CRC chain. There is **no** post-replay check that materialized in-memory state equals the committed log. - -## Desired Outcome - -The controller follows canonical Raft ordering with one explicit apply seam: - -1. **pre-hooks** (validation / preparation only — no authoritative state change) -2. **raft step** — append entry → replicate → majority commit (an explicit protocol step, NOT a hook callback) -3. **apply** — a single `apply_committed_entry()` seam mutates the in-memory state machine *after* commit -4. **after-hooks** = pure notification delivery (scheduler push, client reply/SSE, apiserver) — they observe already-committed, already-applied state and no longer carry WAL/commit responsibility - -A commit that never reaches majority therefore never applies, eliminating the optimistic-divergence window. On restart, the same `apply_committed_entry()` seam is reused by replay, and a verification step proves materialized state == committed log. - -## Approach - -Apply-after-commit (canonical), chosen 2026-06-24. Invert the submit path so the raft commit gates the apply: - -- **KEYSTONE PRE-REQUIREMENT (must land first):** convert the protocol drive from blocking to event-driven. `repl_wait_majority_ack()` today blocks on `pthread_cond_timedwait` (commit.c:359) — a parked thread per in-flight commit. Build its async counterpart: append → register `pending(LSN)` → resolve on majority-ACK/timeout/demotion → drive apply + fire reply continuation. B (apply self-deadlock), C (reply lifecycle), and vtask-pool exhaustion are all downstream of this one conversion. See `facts-in-raft.md` § Keystone. The rest of this approach assumes the drive no longer blocks. - -- Extract WAL append + commit barrier out of `persist_job_hook_fun` into an explicit controller-side raft step invoked directly by command handlers (not via the hook system). -- Introduce a single `apply_committed_entry()` function that the commit path calls **after** `on_commit_advanced`, and that WAL replay reuses on restart — guaranteeing online-apply and replay-apply are the same code (no divergence-by-two-implementations). -- Demote the hook fan-out to notification-only: after-hooks fire from the apply step, observing committed state. Remove the now-redundant `job_submit_committed` watermark gating where the new ordering makes it unnecessary. -- **Commands vs facts policy:** make commands (`WAL_MODE_WAIT`: job submit, config) strictly apply-after-commit. Facts (`WAL_MODE_RETAIN`: worker-reported start/finish/fail/rusage) are applied on receipt today because the controller reacts immediately. **Direction set 2026-06-25: route facts through commit-then-apply as well** — candidate plan and 15-step list in `facts-in-raft.md` (this spec dir), formalized as the **Facts-Apply invariant** (criterion 2.9) in `ha-consistency-invariants/requirements.md`. After the change, commands and facts share the one `apply_committed_entry()` seam and differ only in ingress + commit-wait/failure policy (see the comparison table in `facts-in-raft.md`). The one remaining open decision is **D1** (quorum-loss reaction policy: block-but-durable vs local-apply allowlist) — resolve before Phase C. -- **Restart raft-log-applied validation (item #4, in-scope):** add a post-replay verification that the materialized entity set matches the committed log tip (e.g., count/hash of applied entities vs WAL-derived expectation), surfacing a CLEAN/DIVERGENT verdict alongside the existing CRC verdict. - -## Scope - -- **In**: - - Decouple the raft protocol step from the hook fan-out (item #1). - - Canonical ordering: pre-hooks → commit → apply → after-hooks-as-notification (item #2). - - Single shared `apply_committed_entry()` seam used by both online commit and restart replay. - - Post-restart validation that committed raft log was correctly applied to materialized state (item #4). - - Requirement + regression test design for the above (item #5): unit tests for ordering/rollback, plus conformance assertions extended in `raft-sim-conformance`. -- **Out**: - - Job-array single-record collapse — owned by the sibling `job-array-single-wal-record` spec. - - Changes to the raft library protocol core (owned by raft-log-replication / raft-commit-safety). - - Entity-bearing snapshots / config-upgrade snapshot (separate specs). - - Read-consistency / leader-lease optimizations. - -## Boundary Candidates - -- The explicit controller raft step (append + commit barrier) extracted from `persist_job_hook_fun`. -- The single `apply_committed_entry()` apply seam (shared online + replay). -- The hook system's demotion to notification-only (after-hook subscribers re-pointed at the apply step). -- The commands-vs-facts apply policy. -- The post-restart apply-correctness verification. - -## Out of Boundary - -- The raft library callbacks/protocol (consumed, not modified here). -- Array-submission batching. -- Scheduler framework policy changes. - -## Upstream / Downstream - -- **Upstream**: raft-log-replication + raft-commit-safety (library log/commit core providing `on_persist_entry` / `on_commit_advanced`); ha-term-safety (controller-side raft integration umbrella — this spec restructures the same submit→commit path ha-term-safety wires, so it must land in coordination with it); ha-commit-durability (durable-commit semantics, unchanged). -- **Downstream**: `job-array-single-wal-record` (builds the single atomic array record on top of the clean apply seam); worker-rejoin-resync and any future state-machine change benefit from the centralized apply point. - -## Existing Spec Touchpoints - -- **Extends / sequences with**: `ha-term-safety` (controller raft integration umbrella — disjoint-but-adjacent edits to the submit/commit path; coordinate ordering to avoid conflicting refactors). -- **Adjacent**: `raft-sim-conformance` (item #5 regression assertions land here — apply-after-commit ordering + no-apply-on-failed-commit invariants); `ha-commit-barrier-rollout` (commit classification table — the WAIT vs RETAIN distinction this spec must respect); `raft-log-replication` / `raft-commit-safety` (library callbacks). - -## Constraints - -- Must not block the iomgr thread (async-I/O invariant in `ha-async-io.md`): the explicit commit barrier must run off the iomgr thread, as the current `persist_wal_sync_append` already arranges. -- "Controllers never self-terminate" — a divergent restart verdict is a loud, non-promotable fence, never an `exit`/`abort`. -- Preserve persist-before-ACK durability and the synchronous commit barrier as policy (do not canonicalize them away). -- The shared apply seam must be deterministic so online-apply and replay-apply produce identical materialized state. diff --git a/.kiro/specs/raft-apply-decoupling/design.md b/.kiro/specs/raft-apply-decoupling/design.md deleted file mode 100644 index b7d322b7..00000000 --- a/.kiro/specs/raft-apply-decoupling/design.md +++ /dev/null @@ -1,449 +0,0 @@ -# Design Document: raft-apply-decoupling - -## Overview - -**Purpose**: This feature gives the XCE controller one canonical, ordered path from a client/worker request to committed, applied cluster state — append → majority-commit → apply → notify — replacing today's mix of an already-partially-fixed command path (bug-1306) and a still-fully-optimistic fact path (worker-reported job start/finish/fail/rusage, applied and reacted to on receipt, ahead of commit). - -**Users**: Cluster operators (who need failover to never lose or fabricate job state) and maintainers of the controller's submit/commit/replay code (who need one apply seam instead of two divergent ones). - -**Impact**: Changes the commit engine's `COMMIT_MODE_WAIT` path from a blocking wait to an event-driven resolve; introduces a single `apply_committed_entry()` seam invoked by both live commit-advance and restart replay; moves worker-fact state mutation and reactions from receipt-time to post-commit-time; adds a post-replay CLEAN/DIVERGENT verification. - -### Goals -- One explicit raft step (append + majority commit) per state-changing operation, invoked directly by its handler — not incidentally from a hook. -- One apply seam, used identically by live commit-advance and restart replay, applied strictly in commit order. -- Commands and facts converge on that one apply seam; only their ingress and commit-wait policy differ. -- A quorum-loss window degrades safely by construction (retained-but-unapplied), with no special-case "local apply" code path. -- A restart proves its own correctness (CLEAN/DIVERGENT) instead of only checking WAL structural integrity. -- No regression to any HA correctness or throughput property already proven by prior specs. - -### Non-Goals -- Rewriting the raft library's protocol core (`raft_advance_commit`, `on_commit_advanced`, etc.) — consumed as-is. -- Collapsing a job array into a single WAL record (`job-array-single-wal-record`). -- Moving snapshot send / config-push / peer-connect off the iomgr/delaymgr threads (`raft-drive-async`, already done separately). -- Building `worker-rejoin-resync` — this design depends on it as a future backstop, consistent with every sibling spec that already accepted this gap. -- Entity-bearing snapshot content or read-consistency/leader-lease work. - -## Boundary Commitments - -### This Spec Owns -- The explicit raft-commit step for command entries and its call sites (already partially landed for job submit via bug-1306; this spec completes the ordering and generalizes the mechanism). -- The commit engine's `COMMIT_MODE_WAIT` continuation/resolve mechanism (the keystone conversion away from blocking wait). -- The single `apply_committed_entry()` seam and its dispatch table, and the "apply pump" that drives it off commit-advance in strict LSN order. -- The fact-ingestion split (durable-append-only at receipt vs. mutation+reaction at apply). -- The post-replay CLEAN/DIVERGENT verification and its non-promotable fence. -- Conformance/regression assertions for apply-ordering, idempotency, and no-reaction-before-commit. - -### Out of Boundary -- The raft library's callback contracts (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`, `raft_advance_commit`) — consumed unchanged. -- Transport/timer-thread blocking (snapshot send, config-push wait, peer connect) — `raft-drive-async`'s domain; this design only *consumes* its `vtask_enq_classified` / slow-lane primitives. -- Worker join/rejoin protocol changes — `worker-join-refactor` / `worker-rejoin-resync`. -- Entity-bearing snapshot format/content — `ha-entity-snapshots` / `config-upgrade-snapshot`. This design's replay-side changes must remain compatible with `ha-entity-snapshots`' restore-before-advance `load_snapshot` path, which already shares `restore_job`/`restore_queue_state` with WAL replay (see Architecture). -- Removing `job_submit_committed()` (egress.c:47) — becomes a harmless tautology once this design lands, but its removal is a documented follow-up, not in scope (research.md, Design Decisions). - -### Allowed Dependencies -- `ha-commit-engine` / `ha-commit-barrier-rollout`: majority-commit and idempotency primitives, consumed as-is. -- `ha-term-safety` (complete): `on_commit_advanced` / `raft_advance_commit` wiring on the ACK path (advisory `raft_commit_index`, Option A) — this design's apply pump is a **new** consumer of the existing durable-majority commit advance; it does not change how `committed_lsn` itself advances. -- `raft-drive-async` (complete): `vtask_enq_classified` + slow-lane executor + `VTASK_LIVENESS`/`VTASK_LIVE_BLOCK`/`VTASK_REQ_BLOCK` admission classes — reused for dispatching resolved-commit continuations off the ACK-processing thread. -- `raft-sim-conformance`: existing deterministic-simulator harness, extended (not replaced) with new assertions. - -### Revalidation Triggers -- Any change to the commit engine's per-LSN structure (`commit_engine_t`, `ce->...`) or its locking discipline (`p->lock` vs `ce->lock` ordering, persist.c:3347-3350). -- Any change to `WAL_MODE_WAIT`/`WAL_MODE_RETAIN` classification (`wal_classify`, persist.c:3218) — the apply pump's per-type behavior depends on this classification being exhaustive. -- Any new WAL entry type — must be added to the `apply_committed_entry()` dispatch table or it silently fails to apply. -- `worker-rejoin-resync` landing — must re-verify the fact-apply idempotency guards against its replay/resync ordering (shared `mark_job_*` guards). -- `job-array-single-wal-record` landing — builds its array-expand step directly on `apply_committed_entry()`; a signature change here is a breaking change there. - -## Architecture - -### Existing Architecture Analysis - -Current state, verified against the live codebase (not the 9-day-old brief — see `research.md`): - -| Aspect | Current state | Gap vs. this design | -|---|---|---| -| Job-submit raft step | Already extracted from the hook fan-out into `persist_submit_wal_and_commit()`, called directly by `handle_jobsubmit` (job.c:626, bug-1306) | Requirement 1 done for submit. No other command type (config change) has this yet. | -| Job-submit apply ordering | `create_job()` (job.c:583) runs **before** the commit barrier; on commit failure the created job is **not** rolled back | Requirement 2 violated today — a failed/timed-out submit leaves a phantom job in `c->jobmgr` | -| Commit-wait drive | `repl_wait_majority_ack` → `commit_submit(..., COMMIT_MODE_WAIT, ...)` → blocking `pthread_cond_timedwait` (commit.c:409), pooled worker, 500ms default | Requirement 9's keystone: still fully blocking | -| Online vs. replay apply (submit) | Replay uses `restore_job()` (idempotent, LSN-tagged); online uses raw `create_job()` directly | Requirement 3: half-unified — replay already has the reusable primitive, online doesn't use it | -| Online vs. replay apply (facts) | Online uses guarded `mark_job_start/finish/fail/update()` (monotonic-rank checks); replay uses raw `restore_job_state()` (**no** guard) | Requirement 3/7: genuinely divergent — replay can regress state that online would reject | -| Fact mutation timing | `mark_job_*` mutates `job->job.state` synchronously at receipt, **before** the (already-async) WAL append | Requirement 5/6: fully unaddressed — no apply-after-commit exists for facts at all | -| Hook fan-out | `AFTER_EVENT` for job submit is already notification-only (dead WAIT branch, persist.c:3466); facts' `AFTER_EVENT` still triggers the WAL append itself | Requirement 4: partially true for submit, not for facts | -| Restart verification | CRC-chain + entry-CRC only (`replay_wal`, persist.c:2667-2723); no materialized-state-vs-log-tip check | Requirement 8: fully unaddressed | -| Async task submission | `vtask_enq_classified` + slow-lane executor already exist (`raft-drive-async`, complete) | Requirement 9: reusable as-is, no new primitive needed | - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - Client[Client jobsubmit] - Worker[Worker fact report] - Handler[Command handler] - FactRecv[Fact receipt handler] - RaftStep[Explicit raft step] - CommitEngine[Commit engine resolve] - ApplyPump[Apply pump LSN ordered] - ApplySeam[apply_committed_entry] - Replay[Restart replay] - Hooks[After hooks notify only] - Sched[Scheduler dispatch] - Reply[Client reply] - - Client --> Handler - Handler --> RaftStep - RaftStep --> CommitEngine - Worker --> FactRecv - FactRecv -->|durable append only| RaftStep - CommitEngine -->|resolve on majority| ApplyPump - Replay -->|same seam| ApplySeam - ApplyPump --> ApplySeam - ApplySeam --> Hooks - Hooks --> Sched - Hooks --> Reply - CommitEngine -.->|resolve failure or timeout| Reply -``` - -**Architecture Integration**: -- **Selected pattern**: Commit-then-apply pipeline with a single apply seam, mirroring the pattern already used for the raft library's own log/commit split (`raft-log-replication` owns log, `raft-commit-safety` owns commit-index, controller owns apply) — this design is the controller finishing that same separation one layer up, for its own state machine. -- **Domain/feature boundaries**: the raft library and commit engine remain unaware of "jobs"; `apply_committed_entry()` is the only place WAL-entry semantics meet job/queue/worker state. -- **Existing patterns preserved**: the commit engine's per-LSN registry, `wal_classify`'s WAIT/RETAIN table, the hook system's BEFORE/AFTER shape, `restore_job`/`mark_job_*`'s existing idempotency logic — all reused, not replaced. -- **New components rationale**: a continuation-registration API on the commit engine (replaces blocking wait); an LSN-ordered apply pump (didn't exist — nothing today reacts to RETAIN commit); a post-replay verification step. -- **Steering compliance**: preserves persist-before-ACK durability and the synchronous-commit-as-policy invariant (`ha-async-io.md`); the raft step continues to run off the iomgr thread; a divergent-verdict fence reuses the existing "fail loud, stay alive" mechanism (`persist_recover_loud_fence_if_unrecoverable`), never introducing a new terminal path. - -### Implementation Sequencing - -This design has a hard build order, not just a hard conceptual one: - -1. **Commit Engine Resolve** (the keystone) must land first — every other component either registers a continuation with it (Raft Step, Fact Ingestion's future apply trigger) or depends on the apply pump it drives. -2. **Apply Seam** (`apply_committed_entry`) depends on (1)'s apply pump existing to call it live, but its dispatch logic and the `restore_job`/guarded-`mark_job_*` extraction can be built and unit-tested in parallel with (1). -3. **Raft Step (Command)** ordering fix and **Fact Ingestion**'s receipt/apply split both depend on (1) and (2) being in place; they touch disjoint call sites (`handle_jobsubmit` vs. `on_job_start/finish/fail/update`) and can proceed in parallel with each other once (1)-(2) land. -4. **Restart Replay & Verification** depends on (2) (`apply_committed_entry` must exist before replay can be pointed at it) but is otherwise independent of (3). -5. **Conformance & Regression** depends on all of the above being functionally complete — it verifies the assembled behavior, not any single component in isolation. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Controller core | C (gnu17), existing | All changes land in `src/controller/{job.c,persist.c,commit.c,repl.c,ha.c}` | No new language/runtime | -| Async dispatch | `vtaskmgr.c` `vtask_enq_classified` (raft-drive-async) | Dispatches resolved-commit continuations off the ACK-processing path | Reused, not new | -| Serialization | Cap'n Proto (existing `controller.capnp`) | WAL entry payload encode/decode, unchanged | No schema change | -| Test harness | `raft_sim.c` (raft-sim-conformance), `xcelib_test`/`controller_unit_test`, `/xce-test-ha` | Apply-ordering assertions, idempotency assertions, live soak | Extends existing harnesses | - -## File Structure Plan - -### Modified Files -- `src/controller/commit.c` — Replace the `COMMIT_MODE_WAIT` blocking-wait branch with a continuation-registration + `commit_resolve()` mechanism; `commit_advance_locked()` walks newly-decided LSNs and calls `commit_resolve()` for each instead of `cond_broadcast`-ing a single waiter set. `COMMIT_MODE_RETAIN` registration is unchanged (already non-blocking). Also adds the timeout-arming mechanism for registered continuations: removing the blocking wait also removes the only thing that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline), so an explicit timeout must be armed at registration and swept/fired independently. -- `src/controller/repl.c` — **NOT touched (implementation deviation, confirmed 2026-07-07 validate-impl audit).** `repl_wait_majority_ack()`/`repl_submit_majority_commit()` were never built; task 3.2 instead calls `commit_submit_async()` directly on `p->repl.ce`, bypassing `repl.c` entirely (see tasks.md task 3.2's Implementation Note). This narrows the reusable primitive this design originally wanted — a third parallel "append+track a commit" implementation now exists alongside `COMMIT_MODE_WAIT`'s and `COMMIT_MODE_RETAIN`'s. Left as a known follow-up, not fixed retroactively (out of a validation pass's boundary). -- `src/controller/persist.c` — Add `apply_committed_entry()` as the single dispatch seam (extracted from today's `apply_wal_entry()` switch body, persist.c:2452-2562, plus the fact-transition guard fix). `apply_wal_entry()` (replay) and the new apply pump both call it. Add the post-replay CLEAN/DIVERGENT verification at the end of `persist_recover()`/`replay_wal()`. `persist_job_hook_fun`'s dead `WAL_MODE_WAIT` branch is removed (no longer needed once `repl_submit_majority_commit` fully replaces the old call convention). `wal_hook_append_classified()` (fact receipt path) stops mutating job state — it becomes durable-append-only. Also adds a call from the existing WAL-truncation point into the commit engine's `commit_resolve()` so any pending registered continuation for a truncated LSN resolves as failed instead of hanging. -- `src/controller/ha.c` — Adds a call from the existing demotion/role-transition point into the commit engine's `commit_resolve()` so any pending registered continuation invalidated by losing leadership resolves as failed instead of hanging. This is in addition to `ha.c`'s other listed change (wiring the apply pump, below). -- `src/controller/job.c` — `handle_jobsubmit()`: defer `create_job()`/job-object construction to fire from `apply_committed_entry()` after commit succeeds, not before. `mark_job_start/finish/fail/update()`: split into a guard+mutation core (callable from apply) and a receipt-time validation-only entry point; their `AFTER_EVENT` hook-firing relocates to fire from the apply step. `restore_job_state()` is retired in favor of routing replay through the same guarded core. -- **Apply pump wiring — implementation deviation (confirmed 2026-07-07 validate-impl audit).** Task 6.1 wired the pump's trigger into `commit_resolve_pending()` (`commit.c`), not `ha.c`'s `on_commit_advanced` as originally planned above — every one of `commit.c`'s 6 internal `committed_lsn`-advance sites kicks the pump directly, which covers `on_commit_advanced`'s own call path plus every other advance site `on_commit_advanced` alone would have missed. The walk itself (`[last_applied_lsn+1 .. committed_lsn]` in order, invoking `apply_committed_entry()` for each, dispatched via `vtask_enq_classified`) matches this section as designed; only the trigger's wiring point differs. -- `src/controller/egress.c` — No functional change; `job_submit_committed()` (egress.c:47) is left in place as a now-tautological defensive predicate (see research.md). - -### New Test Files -- `src/controller/testsuite/test_apply_seam.c` — unit tests: apply-after-commit ordering, no-mutation-on-failed-commit, idempotent re-apply, strict LSN ordering. -- `src/controller/testsuite/test_commit_resolve.c` — unit tests for the continuation/resolve conversion: exactly-once resolution, resolve on timeout/demotion/truncation, admission behavior under concurrent in-flight commits. -- Extend `src/lib/raft/raft_sim.c` (or its conformance test driver) with apply-ordering and idempotent-re-apply assertions (Requirement 10.1-10.3). -- Extend the `/xce-test-ha` soak scenario set with a kill-mid-fact-stream case (Requirement 10.4). - -## System Flows - -### Command submission (target ordering) - -```mermaid -sequenceDiagram - participant C as Client - participant H as handle_jobsubmit - participant R as Raft step - participant CE as Commit engine - participant AP as Apply pump - participant AS as apply_committed_entry - participant N as After hooks notify - - C->>H: job submit request - H->>H: BEFORE_EVENT validate - H->>R: encode and append WAL_JOB_SUBMIT - R->>CE: register continuation for LSN - CE-->>CE: majority ack or timeout or demotion - CE->>AP: resolve LSN outcome - alt committed - AP->>AS: apply_committed_entry creates job - AS->>N: AFTER_EVENT notify scheduler apiserver - AP->>H: reply success - else failed or timed out - AP->>H: reply failure, no apply - end -``` - -Key decision not obvious from the diagram: the job object itself does not exist in `c->jobmgr` until the committed branch runs — there is no rollback needed because there is nothing to roll back (Requirement 2.3 by construction, not by compensating action). - -### Worker fact ingestion (target ordering) - -```mermaid -sequenceDiagram - participant W as Worker - participant F as Fact receipt handler - participant WAL as Durable append RETAIN - participant CE as Commit engine - participant AP as Apply pump - participant AS as apply_committed_entry - participant N as Reactions slot free dispatch notify - - W->>F: job start or finish report - F->>F: BEFORE_EVENT validate only - F->>WAL: append entry, return immediately - WAL->>CE: register RETAIN LSN - Note over CE: majority may be delayed by quorum loss; entry stays pending, never applied - CE->>AP: resolve on eventual majority commit, in original order - AP->>AS: apply_committed_entry mutates job state guarded - AS->>N: fire reactions post commit -``` - -Key decision: quorum-loss safety (Requirement 6) requires no special-case code — the apply pump only ever walks committed LSNs, so an unreachable majority simply means the pump has nothing new to walk. Retained facts commit and apply in original order automatically once majority is restored, because the commit engine's `committed_lsn` itself only ever advances as a contiguous prefix. - -### Restart replay and verification - -```mermaid -flowchart LR - Start[persist_recover] --> CRC[CRC chain check] - CRC -->|corrupt| Truncated[RECOVER_TRUNCATED] - CRC -->|clean| Replay[replay_wal applies each entry via apply_committed_entry] - Replay --> Verify[Compare materialized state vs committed log tip] - Verify -->|match| Clean[CLEAN verdict] - Verify -->|mismatch| Divergent[DIVERGENT verdict] - Divergent --> Fence[Non promotable fence, stays alive, retries] -``` - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|---|---|---|---| -| 1.1-1.4 | Extract raft step from hook fan-out | Raft Step (job.c, persist.c) | Command submission | -| 2.1-2.3 | Apply-after-commit ordering for commands | Apply Pump, apply_committed_entry | Command submission | -| 3.1-3.4 | Single shared apply seam, online = replay | apply_committed_entry, Restart Replay | Command submission, Restart replay | -| 4.1-4.3 | Hooks demoted to notification-only | Notification (hook.c usage) | Command submission | -| 5.1-5.3 | Facts apply-after-commit | Fact Ingestion, apply_committed_entry | Fact ingestion | -| 6.1-6.3 | Quorum-loss retained-fact behavior | Apply Pump (commit-gated by construction) | Fact ingestion | -| 7.1-7.3 | Idempotent, ordered re-application | apply_committed_entry (guarded transitions) | Command submission, Fact ingestion, Restart replay | -| 8.1-8.4 | Restart apply-correctness verification | Restart Replay & Verification | Restart replay | -| 9.1-9.4 | Bounded resource behavior under concurrent commits | Commit Engine (continuation/resolve) | Command submission | -| 10.1-10.4 | Regression/conformance coverage | Conformance (raft_sim.c), soak suite | (cross-cutting) | -| 11.1-11.4 | Preserve existing behavior and performance | All components (verification gate) | (cross-cutting) | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|---|---|---|---|---|---| -| Commit Engine Resolve | commit.c/repl.c | Convert `COMMIT_MODE_WAIT` from blocking wait to continuation/resolve | 1.3, 1.4, 9.1-9.4 | vtaskmgr `vtask_enq_classified` (P0) | Service | -| Apply Seam | persist.c | Single dispatch point mutating state from a committed entry | 2, 3, 5.2, 7 | restore_job/mark_job_* guards (P0) | Service | -| Raft Step (Command) | job.c/persist.c | Explicit append+commit invoked by command handlers | 1.1, 1.2, 2.1 | Commit Engine Resolve (P0) | Service | -| Fact Ingestion | job.c/persist.c | Durable-append-only at receipt; defers mutation/reaction | 5, 6 | Apply Seam (P0) | Service | -| Restart Replay & Verification | persist.c | Replay via Apply Seam; post-replay CLEAN/DIVERGENT check | 3.1-3.2, 8 | Apply Seam (P0), existing loud-fence mechanism (P0) | Service | -| Notification (hooks) | hook.c usage in job.c/persist.c | AFTER_EVENT fires only post-apply, pure notify | 4 | Apply Seam (P0) | Event | -| Conformance & Regression | raft_sim.c, testsuite, `/xce-test-ha` | Assert ordering/idempotency invariants | 10, 11 | Apply Seam, Commit Engine Resolve (P1) | Batch | - -### Commit Engine - -#### Commit Engine Resolve - -| Field | Detail | -|-------|--------| -| Intent | Replace the blocking majority-commit wait with continuation registration and event-driven resolution | -| Requirements | 1.3, 1.4, 9.1, 9.2, 9.3, 9.4 | - -**Responsibilities & Constraints** -- Owns the per-LSN commit registry's WAIT-mode bookkeeping (already exists; behavior at resolution changes). -- Must resolve every registered LSN's outcome exactly once, across `{majority-ack, timeout, demotion, truncation}` (Req 9.4). -- Must not block the thread that calls `commit_advance_locked()` (the replication-ACK path) — dispatch continuations via `vtask_enq_classified`, never inline. -- Preserves the existing lock-ordering rule: never hold `p->lock` across `ce->lock` (persist.c:3347-3350) — the continuation dispatch happens after `ce->lock` is released. - -**Dependencies** -- Outbound: `vtaskmgr.c` `vtask_enq_classified` — dispatch a resolved continuation (Criticality P0) -- Inbound: `repl_submit_majority_commit()` (repl.c, replaces `repl_wait_majority_ack`) — registers a continuation (Criticality P0) - -**Contracts**: Service [x] - -##### Service Interface -```c -typedef enum { COMMIT_OUTCOME_COMMITTED, COMMIT_OUTCOME_FAILED } commit_outcome_t; - -typedef void (*commit_continuation_fn)(void* ctx, uint64_t lsn, - commit_outcome_t outcome, int rc); - -/* Registers a continuation for lsn under COMMIT_MODE_WAIT. Returns 0 on - * successful registration, negative on admission rejection (mirrors today's - * -XCE_ERR_NOT_READY). Never blocks. */ -int commit_submit_async(commit_engine_t* ce, uint64_t lsn, - commit_continuation_fn fn, void* ctx); - -/* Called from commit_advance_locked() (majority path) and from the existing - * timeout/demotion/truncation handling (failure path). Dispatches the - * registered continuation, if any, via vtask_enq_classified. Idempotent: - * a second resolve() for an already-resolved lsn is a no-op. */ -void commit_resolve(commit_engine_t* ce, uint64_t lsn, - commit_outcome_t outcome, int rc); -``` -- Preconditions: `lsn` was previously appended locally. -- Postconditions: exactly one of `{COMMITTED, FAILED}` is delivered to the registered continuation, exactly once. -- Invariants: resolution order across distinct LSNs is not required to be FIFO by this component alone — LSN application order is enforced by the Apply Seam's pump, not by resolve dispatch order. - -### Apply Seam - -#### apply_committed_entry - -| Field | Detail | -|-------|--------| -| Intent | The single function that mutates in-memory state for one committed WAL entry, used by both live apply and replay | -| Requirements | 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 5.2, 7.1, 7.2, 7.3 | - -**Responsibilities & Constraints** -- Dispatches on WAL entry type: `WAL_JOB_SUBMIT` → job construction via the existing `restore_job()` primitive (already idempotent, already LSN-tagged); `WAL_JOB_{START,FINISH,FAIL,UPDATE,...}` → the guarded transition core factored out of `mark_job_start/finish/fail/update()`; `WAL_QUEUE_CTRL` → `restore_queue_state()`; `WAL_WORKER_CTRL` → `worker_from_capnp()`. -- Fires the type's `AFTER_EVENT` hook after the mutation completes, never before. -- Must be safe to invoke more than once for the same entry (replay re-running an already-applied LSN, or a resync re-delivering an already-applied fact) — this safety comes from the guards already present in `restore_job` (`-EEXIST`-as-success) and the (relocated) `mark_job_*` monotonic-rank checks, not from new dedup logic. -- Must never be invoked out of LSN order by either caller (live pump or replay loop). - -**Dependencies** -- Outbound: `restore_job`, `restore_queue_state`, `worker_from_capnp`, the guarded `mark_job_*` core (all job.c) (Criticality P0) -- Inbound: Apply Pump (live path), `replay_wal` (restart path) (Criticality P0) - -**Contracts**: Service [x] / Batch [x] - -##### Service Interface -```c -/* Applies one committed WAL entry to in-memory state. Called with entries - * strictly in ascending lsn order by both the live apply pump and replay. - * Returns 0 on success (including a no-op idempotent re-apply), negative on - * a genuine apply failure (which is a fenceable condition, not a retry). */ -int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn, - const void* payload, uint32_t len); -``` -- Preconditions: `lsn` is committed (majority-durable); every entry below `lsn` has already been applied. -- Postconditions: in-memory state reflects the entry; the type's `AFTER_EVENT` hook has fired exactly once for this application. -- Invariants: applying the same `(wal_type, lsn, payload)` twice produces the same observable state as applying it once. - -##### Batch / Job Contract -- Trigger: commit-advance (live) walking `[last_applied_lsn+1 .. committed_lsn]`, or replay iterating the recovered log. -- Input / validation: WAL entry header + payload, already CRC-verified by the point apply is reached. -- Output / destination: in-memory job/queue/worker state; `AFTER_EVENT` hook firing. -- Idempotency & recovery: guaranteed by the underlying `restore_job`/guarded-transition primitives; `last_applied_lsn` is the resumable cursor for restart. - -### Raft Step (Command) - -#### Explicit command raft step - -| Field | Detail | -|-------|--------| -| Intent | Append + majority-commit a command entry, invoked directly by the command handler, never from a hook | -| Requirements | 1.1, 1.2, 2.1 | - -**Responsibilities & Constraints** -- For job submit: already implemented as `persist_submit_wal_and_commit()` (job.c:626) — this design changes its caller-side contract from "returns synchronously, caller replies inline" to "registers a continuation via `repl_submit_majority_commit()`, caller's reply logic becomes the continuation." -- The command handler (`handle_jobsubmit`) no longer calls `create_job()` before this step; object construction moves to the continuation's success branch, via `apply_committed_entry()`. -- Must run off the iomgr thread (unchanged from today). - -**Dependencies** -- Outbound: Commit Engine Resolve (`commit_submit_async`) (Criticality P0) -- Outbound: Apply Seam, on the continuation's success branch (Criticality P0) - -**Contracts**: Service [x] - -**Implementation Notes** -- Integration: the existing `persist_submit_wal_and_commit()` encode/append loop is retained; only its commit-wait call changes from blocking to continuation-registering. -- Validation: a config-change command handler follows the identical pattern once introduced (out of scope to build here, but the contract is designed to generalize — see Requirements Traceability 1.1 covering "job submit, configuration change"). -- Risks: converting the one existing `COMMIT_MODE_WAIT` call site is narrow in surface area but touches a durability-critical path — full existing HA soak suite (Req 11.2) must pass before/after. - -### Fact Ingestion - -#### Fact receipt split - -| Field | Detail | -|-------|--------| -| Intent | Split today's receipt-time mutate-then-append into durable-append-only at receipt, mutate+react at apply | -| Requirements | 5.1, 5.2, 5.3, 6.1, 6.2, 6.3 | - -**Responsibilities & Constraints** -- `on_job_start`/`on_job_finish`/`on_job_fail`/`on_job_update` (job.c) validate (existing `BEFORE_EVENT`, unchanged) and then only encode + durably append via `wal_hook_append_classified` (`WAL_MODE_RETAIN`) — no state mutation, no `AFTER_EVENT` firing at this point. -- The guarded mutation core (extracted from today's `mark_job_*`) and the `AFTER_EVENT` hook firing both relocate to run from `apply_committed_entry()`. -- No local-apply allowlist under quorum loss (Req 6.2) — enforced structurally: the apply pump only ever processes committed LSNs, so there is no code path that can apply an uncommitted fact. - -**Dependencies** -- Outbound: durable append (`wal_hook_append_classified`, existing, unchanged) (Criticality P0) -- Outbound: Apply Seam, for the deferred mutation (Criticality P0) - -**Contracts**: Event [x] / Batch [x] - -##### Event Contract -- **Published events — implementation deviation (confirmed 2026-07-07 validate-impl audit).** All nine job-state fact WAL types route through a single guard-only `apply_job_update_transition()` core in `apply_committed_entry()` and fire one generic `HOOK_DETAIL_JOB_UPDATE`, not per-type `HOOK_DETAIL_JOB_START`/`_EXIT`/`_FAIL`/etc. as stated below. Verified safe in practice: `sched_job_update_fun`/`apiserver_job_update_fun` (`src/controller/sched.c:446-465`) already exist and consume exactly the generic `(cluster_t*, job_t*)` form the apply path passes — no subscriber assumed per-type dispatch. Left as a documentation correction, not a functional gap. -- ~~Published events: `AFTER_EVENT` for `HOOK_DETAIL_JOB_START`/`_EXIT`/etc., now fired from the apply step, not the receipt handler.~~ (superseded by the note above) -- Subscribed events: none new. -- Ordering / delivery guarantees: strict commit order via the shared apply pump; at-least-once redelivery for facts lost before append remains `worker-rejoin-resync`'s responsibility (out of boundary). - -**Implementation Notes** -- Integration: `mark_job_start/finish/fail/update`'s guard predicates (`job_state_is_backward`, `job_state_is_terminal`) become the guard logic inside `apply_committed_entry`'s fact-transition case — the same predicates, relocated, not rewritten. -- Validation: audit every `HOOK_DETAIL_JOB_*` subscriber for an assumption that `BEFORE_EVENT` and the state mutation happen together at receipt; only the mutation moves, `BEFORE_EVENT` validation timing is unchanged. -- Risks: this is the widest behavioral change in the design — worker-visible latency for slot-free/dispatch grows by one commit round-trip (accepted cost, documented in `facts-in-raft.md`, Req 11.4 bounds it to "no per-fact synchronous barrier," not "no latency change"). - -### Restart Replay & Verification - -#### Post-replay apply-correctness verification - -| Field | Detail | -|-------|--------| -| Intent | Prove materialized state matches the committed log tip after replay, beyond CRC structural checks | -| Requirements | 3.1, 3.2, 8.1, 8.2, 8.3, 8.4 | - -**Responsibilities & Constraints** -- Runs after `replay_wal()`'s existing CRC-chain verification succeeds (RECOVER_CLEAN), applying every entry via `apply_committed_entry()` (replacing today's direct `restore_job`/`restore_job_state` calls in `apply_wal_entry`). -- Computes a lightweight expectation from the committed log tip (e.g., last-applied LSN reaches the recovered commit watermark, plus a count/hash of materialized entities) and compares it to the actual post-replay state. -- On mismatch: reuses `persist_recover_loud_fence_if_unrecoverable()`'s existing mechanism (persist.c:2990) and the `recovery_unrecoverable` sticky flag / `ha->in_catch_up` promotion gate — a DIVERGENT verdict is a new *input* to that existing fence function, not a new fence mechanism. - -**Dependencies** -- Outbound: Apply Seam (Criticality P0) -- Outbound: existing loud-fence mechanism (`persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable`, `ha->in_catch_up`) (Criticality P0) - -**Contracts**: Batch [x] / State [x] - -##### State Management -- State model: `last_applied_lsn` (new, small addition to `persist_t`) tracks replay/apply progress; `recovery_unrecoverable` (existing) is the sticky fence flag. -- Persistence & consistency: `last_applied_lsn` is derived at replay time (not itself persisted) — restart always recomputes it from the WAL, so it cannot itself drift from disk state. -- Concurrency strategy: single-threaded during replay (unchanged); the live apply pump takes over `last_applied_lsn` advancement once replay completes. - -## Data Models - -No new persistent schema. `WAL_JOB_SUBMIT`/`WAL_JOB_START`/`WAL_JOB_FINISH`/`WAL_JOB_FAIL`/`WAL_JOB_UPDATE`/`WAL_QUEUE_CTRL`/`WAL_WORKER_CTRL` entry formats, `wal_classify`'s WAIT/RETAIN table, and `job_t`'s state bits (`JOB_STAT_PEND/RUN/DONE/EXIT`) are all unchanged. - -**In-memory additions**: -- `persist_t.last_applied_lsn` (`_Atomic uint64_t`) — the apply pump's resumable cursor, analogous to the existing `committed_lsn` / `durable_lsn` watermarks. -- `commit_engine_t`'s per-LSN WAIT entry gains a `commit_continuation_fn` + `void* ctx` field (replacing the `pthread_cond_t` wait registration for that mode; `COMMIT_MODE_RETAIN` entries are unaffected). - -## Error Handling - -### Error Strategy -- **Failed/timed-out command commit**: `commit_resolve(..., COMMIT_OUTCOME_FAILED, rc)` fires the registered continuation with the original error-code vocabulary (`-ETIMEDOUT`, `-XCE_ERR_NOT_READY`, etc., unchanged from today); the continuation replies failure to the client and never calls `apply_committed_entry()` — no rollback needed because no mutation occurred (Req 2.3). -- **Apply-time failure** (e.g., a decode error on an already-committed, already-CRC-valid entry): a genuine programming-invariant violation, not an operational condition — treated per the controller's existing "never self-terminate" rule: log loud, do not apply, and let the DIVERGENT-verdict fence (restart path) or an equivalent live-path fence catch it; never `assert()`/abort on this path in production. -- **DIVERGENT restart verdict**: non-promotable fence via the existing `recovery_unrecoverable` mechanism — stays alive, keeps retrying catch-up, never exits (steering `ha-async-io.md` / controller `CLAUDE.md` process-lifecycle invariant). -- **Quorum loss during a retained fact's commit**: no error surfaces to the worker (facts have no requester awaiting confirmation, unchanged); the fact remains durably appended and unapplied until majority is restored (Req 6). - -### Monitoring -- Log a WARN when a retained fact's backlog age (durably appended, not yet committed) exceeds a threshold, consistent with `facts-in-raft.md`'s D1 refinement (metric on retained-fact backlog age). -- Log the CLEAN/DIVERGENT verdict at INFO/ERROR respectively on every restart, alongside the existing CRC verdict. - -## Testing Strategy - -- **Unit Tests**: - - Apply-after-commit ordering: a failed commit produces zero observable state mutation (Req 2.3). - - Idempotent re-apply: applying the same committed entry twice (simulated replay-of-already-applied) is a no-op (Req 7.1, 7.2). - - Commit-resolve exactly-once: registering a continuation and forcing each of `{majority-ack, timeout, demotion, truncation}` resolves it exactly once (Req 9.4). - - Fact monotonicity under replay: a duplicate/out-of-order FINISH-then-START replay sequence does not regress state (Req 7.2 — directly targets the `restore_job_state` gap found in `research.md`). -- **Integration Tests**: - - End-to-end job submit: commit success → job visible and notified; commit failure → job never visible, client sees failure (Req 2, 11.3). - - End-to-end fact flow: worker reports job start → no slot-free/dispatch/notify until the entry commits (Req 5.3). - - Quorum-loss-then-restore: facts submitted during a majority-loss window apply in original order once quorum returns, with no local-apply escape hatch exercised (Req 6). - - Restart CLEAN verdict on a normal shutdown/restart; forced DIVERGENT verdict (inject a materialization mismatch) confirms the fence engages and the process stays alive (Req 8). -- **Regression / Conformance**: - - Extend `raft_sim.c`'s conformance harness with "no reaction before commit" and "idempotent re-apply" assertions layered on its existing apply-after-`on_commit_advanced` loop (Req 10.1-10.3). - - Extend `/xce-test-ha` with a kill-the-primary-mid-fact-stream scenario asserting no phantom slot-frees and no notification for an uncommitted fact (Req 10.4). - - Full existing unit/integration/soak suite re-run as a regression gate (Req 11.1, 11.2), with the known, intentional exception of tests that asserted the old apply-before-commit ordering (updated, not skipped). - -### Performance & Scalability -- Target: fact-commit throughput under sustained high-volume worker churn must not regress relative to pre-feature behavior (Req 11.4) — the group-commit amortization for `WAL_MODE_RETAIN` entries is unchanged by this design; only the post-commit reaction timing shifts. -- The keystone conversion removes one pooled-worker-thread-per-in-flight-commit cost entirely (today's `COMMIT_WAIT_RESERVED`-gated blocking wait), which should *improve* concurrent-submit headroom, not merely preserve it — validate via Req 9's bounded-resource tests under a concurrent-submit burst. diff --git a/.kiro/specs/raft-apply-decoupling/facts-in-raft.md b/.kiro/specs/raft-apply-decoupling/facts-in-raft.md deleted file mode 100644 index b5e5ff99..00000000 --- a/.kiro/specs/raft-apply-decoupling/facts-in-raft.md +++ /dev/null @@ -1,150 +0,0 @@ -# Design Note: Routing Facts Through Raft (apply-after-commit for RETAIN events) - -**Status:** candidate plan (2026-06-25). Resolves the "commands vs facts apply policy" open question in `brief.md`. **D1 resolved 2026-06-25: block-but-durable (retain-only) under quorum loss.** Design-risk triage complete — see `## Design Risks & Tricky Cases`. - -## Premise - -Today XCE splits state changes into two paths: - -- **Commands** (`WAL_MODE_WAIT`: job submit, config) — client-initiated; synchronous commit before reply. -- **Facts** (`WAL_MODE_RETAIN`: worker-reported job start/finish/fail/rusage) — worker-initiated; **applied + reacted to on receipt**, WAL-appended non-blocking, committed later via a covering standby ACK (`wal_hook_append_classified`, `persist.c:2488`). - -Facts were made apply-on-receipt for liveness: a worker fact (job finished) must promptly free the slot and trigger the next dispatch, and the worker has already moved on (fire-and-forget, zero redelivery — bug-396). - -**"Raft handling facts" means:** a fact becomes a normal committed log entry. Its *state-machine mutation* (job RUN/DONE/FAIL) and its *reactions* (slot free, dispatch trigger, client/apiserver notify) move to the post-commit `apply_committed_entry()` seam. Only the **durable append** stays on-receipt (so the fact is never lost). This makes online-apply ≡ replay-apply for facts too, and eliminates phantom reactions on uncommitted facts. - -## Do we still handle requests and facts differently? (yes — but only at the edges) - -Once both requests (commands) and facts route through commit-then-apply, **the apply step unifies** — both go through the same `apply_committed_entry()` seam. What stays type-specific is the *ingress* and the *commit-wait / failure* policy, not the apply: - -| Aspect | Request / command (was `WAL_MODE_WAIT`) | Fact (`WAL_MODE_RETAIN`) | -|---|---|---| -| Origin / ingress | client; non-leader arrivals redirected via the `libxce-failover` taxonomy | worker; non-leader arrivals rejected/redirected, lost-window recovered by `worker-rejoin-resync` | -| Commit wait | **synchronous** — handler blocks on `repl_wait_majority_ack`; client reply gated on commit (criteria 2.1/2.2) | **async** — append, return immediately; commit via the group-commit batch; never a per-fact blocking barrier (high volume) | -| If commit fails / times out | **report FAILED** to the waiting requester (criterion 2.2) | **nobody to tell** — retain + retry while leader (criterion 2.8) | -| Quorum loss | block/fail (CP) | retained-and-retried; decision **D1** decides any local-react escape hatch | -| State mutation + reaction (apply) | **shared** `apply_committed_entry()`, post-commit | **shared** `apply_committed_entry()`, post-commit | - -**The key shift:** the `WAIT` vs `RETAIN` classification's *meaning narrows*. Today it conflates two things — "apply-after-commit vs apply-on-receipt" **and** "synchronous-confirm vs async". After this change the apply timing is the **same** for both (after commit); the classification governs **only** "does a requester block and get a failure reply" vs "is it async-retained". So we still *route* them differently at the edges, but they *converge* at the one place correctness depends on: the apply seam. - -## Why this is correct, not just possible - -- A fact is an external input to the replicated state machine. Raft already commits external inputs (that's what `ClientRequest` is). Nothing about a fact requires apply-on-receipt at the *log* level. -- Reactions are state-machine **outputs**. Canonical Raft produces outputs at apply time (post-commit), on the leader, idempotently. Today's on-receipt reaction violates this — it can fire on a leader that then loses the entry. -- The latency cost (one commit round-trip before slot-free/dispatch) is the same round-trip commands already pay, and the group-commit flusher (`ha-commit-durability`) already amortizes the high fact volume. The iomgr thread is never blocked: `on_commit_advanced` drives apply asynchronously. - -## Decisions to lock before building - -- **D1 — Quorum-loss fact policy — RESOLVED 2026-06-25: (a) block-but-durable, retain-only.** When the primary cannot reach a majority, a fact is appended durably but NOT applied (no state mutation, no reaction) until its entry commits. No local-apply allowlist. Rationale: option (b) (local-apply allowlist) directly contradicts criterion 2.4 (provisional state must not drive irrevocable exports such as dispatch) and reopens the bug-795/828/851 divergence family. Cost of (a) is bounded and self-healing — only *new* dispatch stalls during the (already-degraded) quorum-loss window; running jobs continue (worker autonomy); retained facts commit+apply in order on quorum restore; primary-death-in-window recovered by `worker-rejoin-resync`. Consistent with CP + "controllers never self-terminate". Refinements: no carve-outs (allowlists rot); accept the documented stale-read deviation; emit a metric/WARN on retained-fact backlog age; facts inherit Requirement 7's quorum-loss posture. - - Rejected: **(b) local-apply allowlist** — buys availability the operator should fix at the quorum level, at the cost of the core safety invariant. -- **D2 — Reaction timing.** Confirm ALL fact reactions (slot free, scheduler dispatch, client/apiserver notify) are safe to defer to post-commit. Expectation: yes — the worker is fire-and-forget; nothing external blocks on the controller's reaction. -- **D3 — Idempotency key.** Fact apply must be idempotent (replay, worker-rejoin-resync re-report, retry). Key on `(jobid, monotonic job-state)` so a re-applied or out-of-order fact is a no-op, never a regression of state. Reuse the `mark_job_*` idempotency guards (shared need with `ha-consistency-invariants`). - -## The list — follow in order - -### Phase A — Preconditions (shared with raft-apply-decoupling core) -1. Land the single `apply_committed_entry()` seam; route BOTH commands and facts through it. -2. Add/confirm `mark_job_*` idempotency guards (D3) so apply is replay/resync/retry safe. -3. Ensure the async group-commit flusher (`ha-commit-durability` `fsync_worker`) wakes a post-commit apply pump via `on_commit_advanced(lsn)` — no iomgr blocking. - -### Phase B — Reclassify facts onto the committed-apply path -4. Change fact ingestion (the `WAL_MODE_RETAIN` path in `wal_hook_append_classified`, `persist.c:2488`) so on receipt it ONLY: validate → `ha_primary_log_append` (append entry) → return. **Remove the on-receipt state mutation and reactions.** -5. Move the fact state mutation (`restore_job_state`-equivalent: RUN/DONE/FAIL/rusage) into `apply_committed_entry()`, keyed on committed LSN. -6. Move the reactions (slot free, scheduler dispatch trigger, client/apiserver notify) to fire from the apply step, after the mutation, leader-only. -7. Implement the apply pump: `on_commit_advanced(lsn)` applies all newly-committed entries (facts + commands) in LSN order exactly once. - -### Phase C — Failover / liveness correctness -8. Reject/redirect facts arriving at a non-leader (workers must target the leader; reuse the `libxce-failover` redirect taxonomy `E_XCE_REDIRECT`/`E_XCE_NOMASTER`/...). -9. Wire `worker-rejoin-resync` as the backstop for facts lost in the appended-but-not-committed or never-arrived window. -10. Implement D1 (RESOLVED: block-but-durable — append durably, no apply/react until commit, no allowlist); gate it, emit the retained-fact backlog metric, and test the quorum-loss-then-restore path. - -### Phase D — Restart / replay parity -11. Restart replay applies fact entries through the SAME `apply_committed_entry()` → online-apply ≡ replay-apply. -12. Extend the brief's post-restart apply-validation (item #4) to cover fact-derived final states, not just submits. - -### Phase E — Verification / regression (item #5) -13. `raft-sim-conformance`: assert no fact reaction (slot-free/dispatch/notify) occurs before that fact's entry commits; assert idempotent re-apply; assert State-Machine-Safety across interleaved fact+command logs. -14. Live soak (`/xce-test-ha`): kill the primary mid-fact-stream; assert no phantom slot-frees, no divergence, no final-state loss beyond what `worker-rejoin-resync` recovers. -15. Performance regression: high worker-churn fact stream must not regress — verify group-commit keeps fact commit amortized (NO per-fact synchronous barrier; facts never use the command `repl_wait_majority_ack` blocking path). - -## Keystone Pre-requirement: async, non-blocking commit drive *(must land FIRST)* - -Everything in this spec is downstream of one conversion. The protocol **drive** is currently **blocking**: `repl_wait_majority_ack()` → `pthread_cond_timedwait(&ce->cond, &ce->lock, &dl)` (commit.c:359, bounded by `DEFAULT_SYNC_TIMEOUT_MS`=500ms, config.h:93) **parks a thread until a majority of WalAcks arrive**. This is *blocking-offloaded* (the wait was moved off the iomgr thread) but NOT *event-driven* — it honors the letter of "never block iomgr" while still parking one worker thread per in-flight commit. - -**Pre-requirement — convert the commit engine from `wait` to `resolve`** (do NOT build a parallel registry; reuse the commit engine's existing per-LSN majority tracking — today it `cond_broadcast`s, replace that with `resolve()`): -- append → register `pending(LSN)` on the existing per-LSN commit structure, return immediately (no park). -- on each WalAck event: if majority reached → `resolve(lsn, COMMITTED)`; on timeout/demotion/truncation → `resolve(lsn, FAILED)`. -- `resolve(lsn, outcome)` → advance commit_index → drive apply (off the ACK thread) → fire the reply continuation (C) and any apply-time proposal callbacks (B). -- the `cond_timedwait` is **deleted, not wrapped**. - -**Why it is the keystone:** B (apply→propose self-deadlock), C (reply continuation), and pool-exhaustion (parked workers starving the bounded vtask pool under a slow/degraded majority) are **all downstream of the blocking drive**. One conversion dissolves all three: with nothing parked, there is no thread to self-deadlock, replies become natural callbacks, and the pool can't be starved by commits. The apply-after-commit work sits on top of this — so the async drive is a **hard prerequisite**, sequenced before Phase A. - -The blocking call is simple and obviously correct; the event-driven version is a state machine across async events (append-complete, each ACK, timeout, demotion, disconnect) carrying the same exactly-once + completeness burden as C, now on the protocol critical path. That cost is why "driving raft" — not the raft algorithm (already library-verified) — is the hardest part of this effort. - -### Two distinct prerequisites — the commit barrier is NOT the only blocking violation - -A broad async-violation audit (2026-06-25) found the commit barrier is actually the *least* severe blocking class (it blocks a pooled vtask worker — bounded, degrades gracefully). **Three CRITICAL violations block the iomgr thread itself** — the single epoll loop driving ALL heartbeats/votes/ACKs/client I/O. Root cause: `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** (NOT handed to the vtask pool like normal cluster messages), so anything it does synchronously runs on the iomgr event loop. When iomgr stalls → heartbeats stop → peers declare the leader dead → election storm. This is a *separate defect class* from the apply refactor: it threatens protocol **liveness today**, independent of apply-after-commit, and is a likely root of the bug-851 chaotic-failover family. - -**CRITICAL (block iomgr / the heartbeat-driver thread):** -1. **Snapshot send on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) → `persist_checkpoint` (fdatasync + dir scan + segment unlinks) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503), all on iomgr. A large snapshot freezes every heartbeat/vote/client request for its whole duration. Worse than the commit barrier. -2. **Config push wait on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172,1177,2672,2677,3973), up to `CONF_PUSH_TIMEOUT_MS`. Code comments (ha.c:75-77) acknowledge the safe-vs-unsafe caller split but unsafe paths exist. -3. **Blocking DNS + connect on the delaymgr (heartbeat) thread** — `endpoint_connect` → `gethostbyname` + blocking `connect()` (netio.c:289,302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unresolvable peer stalls ALL heartbeats/election-timeouts for seconds → false failover. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.) - -**MEDIUM (block a vtask pool worker — same class as the commit barrier):** -- `repl_wait_majority_ack` for `OPCode_upgradeConfigAck` (config_repl.c:1459); 30-second `usleep` checkpoint-wait poll in `do_upgrade_config_with_bytes` (config_repl.c:1661); `ha_persist_term` fsync on a vote grant (ha.c:3407). - -**Correctly event-driven (confirmed):** RequestVote fan-out, heartbeat fan-out, and WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + its fdatasync run on a vtask worker, not iomgr. - -**Verdict: the protocol drive is NOT fully event-driven.** Two separable prerequisites result: -- **P1 — commit-barrier async** (this doc's keystone): unblocks the apply/reply/facts work (B, C, pool-exhaustion). Prerequisite for `raft-apply-decoupling`. -- **P2 — drive hardening** (move snapshot send, config-push wait, and peer DNS/connect OFF the iomgr/delaymgr threads): independently valuable, independently testable, threatens liveness today. **Recommended home: its own spec `raft-drive-async` (or fold into `election-iomgr-refactor`, which already touches this thread model) — NOT buried inside `raft-apply-decoupling`.** - - **Offload mechanism — "trigger, don't execute" (delaymgr/iomgr only schedule; blocking work runs in a pool with admission headroom):** - - The timer/event thread enqueues a task and returns immediately; the blocking work runs on a worker. The danger is relocating the stall: dumping blocking work into the bounded vtask pool can starve request handling. **XCE already solved this for the commit barrier — `COMMIT_WAIT_RESERVED` caps concurrent blocking waits at `max(1, n_vcpus − RESERVED)`, reserving headroom (commit.c:195-203).** Any offloaded blocking task MUST live under the same admission discipline. - - **API gap:** the existing enqueue `vtask_handle_msg` (vtaskmgr.c:356) is endpoint+inbound-packet-bound (`vtype=VTASK_IO`, requires `ep->fsm_in.pkt`). A reconnect has no inbound packet → P2 needs a **general (non-IO) task-submit path** (`vtask_enq` with custom data), or generalize the API. - - **Lane by duration:** short bounded tasks (**reconnect**, ≤200ms connect cap) → vtask pool **under an admission cap** (COMMIT_WAIT_RESERVED-style). Long tasks (**snapshot send, config-push**, seconds) → a **dedicated slow-lane executor**, NOT the request pool (a few concurrent multi-second tasks head-of-line-block request handling even with reservation). - - **Coordination per offloaded task:** (1) idempotent in-flight dedup (`pending[peer]` atomic) so periodic timers don't pile up duplicates; (2) refcount the endpoint at enqueue, release at completion (arg outlives the tick); (3) marshal completion (epoll fd registration) back to the **iomgr** owning thread (bug-646); (4) resolve the outbound-vs-inbound connection race. - - **DNS:** `gethostbyname` is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — **defer unless measured**. The `connect` fix is mostly reordering an existing pattern: non-blocking connect + bounded `poll` timeout already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); the buggy path just sets non-blocking AFTER connect (netio.c:716). - - **Heartbeat-problem completeness (does the offload fix it? — partially):** Heartbeat liveness spans BOTH threads — delaymgr *sends* heartbeats + fires election timeouts; iomgr *processes* incoming ACKs/votes/replies (`peer_ctrl_msg_handler`). Connect-offload protects the delaymgr (sender) side and election-timeout firing — a real win — but is **necessary-not-sufficient**: the snapshot-send-on-iomgr blocker (#1) freezes the *receiver* loop, so the leader stops hearing its peers regardless of how cleanly it sends. **Heartbeat is safe only when BOTH delaymgr AND iomgr never block → P2 must land all three CRITICALs, not just connect.** Residual: delaymgr ticks at 200ms precision (delay.c) — a timer-granularity floor offload doesn't touch; set the failover-detection timeout with margin above 200ms (tuning, not blocking). - - **Missing regression test:** `/xce-test-ha` only *kills* the primary (process-dead → connect fails fast with RST; does NOT reproduce this). The real test is a **partition variant** — make one peer *unreachable* (blackhole packets so connect hangs and snapshot-send stalls) and assert (a) heartbeats to survivors keep flowing and (b) no spurious election. This specifically exercises the *unreachable*-not-*dead* case the blocking findings care about — the bug-851 chaotic-failover family. - -## Design Risks & Tricky Cases - -Triaged 2026-06-25. After triage, only **A** and **B** are genuinely architectural; **C** was reclassified to request-handling correctness; **D–G** are correctness/cost refinements. - -### A — Reactions are not replayable → re-derive from committed state on recovery *(architectural; RESOLVED by existing resync)* -A committed log perfectly records *what state should be*, but is a bad script for *what side effects to re-emit*: replaying a job-finish must rebuild DONE state but must NOT re-notify the client or re-dispatch. So reactions originate from **only two places**: (1) **live-leader apply** of newly-committed entries, and (2) the **full-state resync on consumer reconnect**. Recovery never replays reactions — it rebuilds committed state, and the inevitable reconnect (failover / primary-restart / promotion all look identical to a consumer) re-derives the world. Three apply contexts: leader-live = state + react; standby = state only; recovery/replay = state only, then re-derive via resync. - -**This is solved by XCE's existing reconnect-and-resync** (scheduler/apiserver full-state push on reconnect = the re-derivation step). Conditions for it to hold: -1. Resync must push **committed** state only (post-replay), never provisional (criterion 2.7). -2. **Full reconcile**, not delta (consumer discards old view) — guard against the "lightweight re-attach" optimization eroding this. -3. Resync re-reports flow through **commit-then-apply** too (a rejoining worker's re-reported fact is appended→committed→applied idempotently, never apply-on-receipt — same D1/2.9 gate). -4. Order resync **after** replay (re-reports above the recovered tip → append+commit; already-committed → idempotent no-op). - -**HARD prerequisite:** the worker→controller half depends on `worker-rejoin-resync`, which is **not yet implemented** (workers send only `hostattrs` today, no job state — bug-396). Without it, a fact lost in a failover window has no redelivery path and A reopens for facts. → upgrade the `worker-rejoin-resync` dependency in `brief.md`/roadmap from "designated backstop" to **HARD**. - -### B — Apply → propose reentrancy (deadlock risk) *(architectural; OPEN)* -Apply runs inside the commit-advance callback. A reaction that produces *new cluster state* must **enqueue a fresh proposal**, never mutate inline and never block waiting for its commit (you'd deadlock inside the commit path). Rule: **apply may enqueue proposals; apply may never block on consensus.** The vtask-continuation discipline below (register-don't-block) is the same mechanism and covers B. - -### C — Reply ownership *(RECLASSIFIED: request-handling correctness, NOT architecture; low risk)* -"One request → exactly one reply" is a **pre-existing** request-handling invariant. Apply-decoupling only **defers** the reply to commit; it adds no new architecture. The genuinely architectural sub-concerns are already solved by the existing design: -- **Send race / socket-epoll ownership (bug-646 class):** solved — `xce_reply` is non-blocking (it *enqueues* to the endpoint's output queue; the io worker owns the wire). Reply is generated by the vtask handler-continuation (completion marshaled to the iomgr thread), enqueued, and delivered by the io worker. -- **Endpoint lifetime (bug-1077 class):** solved — the io socket is owned by the endpoint, which is **refcounted**; a continuation holds a ref, so a mid-flight disconnect discards the reply against a dead endpoint rather than a UAF. - -**Residual (request-handling correctness, single-threaded if produced on the iomgr thread):** -1. **Exactly-once deferred-reply resolution** per request across `{commit-success, commit-timeout, leadership-loss, uncommitted-tail truncation}` — an atomic "claimed" flag on the obligation; centralize all exits through one `resolve(lsn, outcome)`. -2. **Bounded in-flight** (the one system-level item, not per-request): the async model loses the blocking model's free thread-pool backpressure, so add admission control on the accept path. - -### D — Which facts need consensus? *(refinement)* -Job *lifecycle* (start/finish/fail) → RETAIN-through-commit (authoritative). `rusage`/telemetry → consider **LOCAL** (soft, re-derivable, highest-volume — don't commit-amplify it). Re-audit the `wal_classify` table. - -### E — Idempotency must be state-monotonic, not LSN-dedupe *(refinement; mandatory)* -Compare job-state rank (PEND<RUN<DONE) and no-op regressions/duplicates — not LSN dedupe. **Resync amplifies this** (it floods re-reported facts), so E is mandatory, not optional. (Implements D3.) - -### F — Strict LSN-ordered apply *(refinement)* -Serial, in-LSN-order apply pump, never reordered/parallelized — this is what makes "fact for a not-yet-created job" races vanish by construction. Lock it as a `raft-sim-conformance` assertion. - -### G — Notify reliability is unchanged *(note, not a regression)* -Moving after-hooks to notify-only does NOT make delivery reliable (still best-effort, bug-396). Resync backstops loss **across a reconnect**, not per-message. Don't let the cleaner architecture imply guaranteed delivery. diff --git a/.kiro/specs/raft-apply-decoupling/requirements.md b/.kiro/specs/raft-apply-decoupling/requirements.md deleted file mode 100644 index c2783131..00000000 --- a/.kiro/specs/raft-apply-decoupling/requirements.md +++ /dev/null @@ -1,160 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE controller's raft protocol step is currently a side effect of the notification hook fan-out rather than an explicit, ordered part of the request path. `persist_job_hook_fun` runs as an `AFTER_EVENT` hook and, inside that callback, performs the WAL append and blocks on the synchronous majority-commit barrier. Meanwhile `handle_jobsubmit` mutates the in-memory job table (`create_job()`) **before** that commit step runs, so a commit that fails or times out does not cleanly revert the optimistic mutation. Because apply happens before commit, downstream consumers (scheduler push, client reply) must defensively re-gate on a commit watermark to avoid acting on not-yet-committed state. This mis-layering is the structural root of the bug-795/828/851/1060 divergence family. - -This feature restructures the controller onto canonical Raft ordering with one explicit apply seam: pre-hooks (validation only) → an explicit raft step (append → replicate → majority commit) → a single `apply_committed_entry()` seam that mutates in-memory state **after** commit → after-hooks that are pure notification delivery observing already-committed, already-applied state. The same `apply_committed_entry()` seam is reused by restart replay, and a post-replay verification proves materialized state matches the committed log. Per `ha-consistency-invariants` criterion 2.9 (the Facts-Apply invariant), this feature also reconciles worker-reported facts (`WAL_MODE_RETAIN`: job start/finish/fail/rusage) onto the same commit-then-apply seam — today they are applied and reacted to on receipt, ahead of commit. - -Everything in this feature is downstream of one conversion: the commit-wait drive is currently blocking (`repl_wait_majority_ack` parks a pooled worker thread per in-flight commit via `pthread_cond_timedwait`). This feature includes converting that drive from blocking-wait to event-driven resolve as its keystone prerequisite, because the apply step, the deferred client reply, and the fact-reaction timing all sit on top of it. The separate transport/timer-thread blocking violations (snapshot send, config-push wait, peer connect on iomgr/delaymgr) are a distinct defect class owned by `raft-drive-async`, not this feature. - -The authoritative subject for the acceptance criteria below is **the XCE controller** — specifically its command/fact submission, commit, and apply path. - -## Boundary Context - -- **In scope**: - - Extracting the raft protocol step (WAL append + majority-commit barrier) out of the `AFTER_EVENT` hook fan-out into an explicit step invoked directly by command handlers. - - Canonical ordering for commands (job submit, configuration change): pre-hook validation → commit → apply → after-hook notification, with no in-memory state mutation before commit. - - A single shared `apply_committed_entry()` seam used by both live commit-advance and restart replay, applying committed entries strictly in commit order. - - Demoting the hook fan-out to notification-only: after-hooks observe already-committed, already-applied state and no longer carry WAL/commit responsibility, and downstream consumers no longer need to defensively re-check a commit watermark before acting on a notification. - - Reconciling worker-reported facts (`WAL_MODE_RETAIN`) onto the same commit-then-apply seam (criterion 2.9): durable append stays on receipt, but the state mutation and reactions (worker slot release, scheduler dispatch trigger, client/apiserver notify) move to after commit. - - The quorum-loss policy for retained facts: block-but-durable (append durably, do not apply or react until commit; no local-apply allowlist) — decision D1. - - Idempotent, state-monotonic, strictly commit-ordered re-application of entries (replay, retry, worker resync). - - Post-restart verification that materialized state matches the committed log tip, surfacing a CLEAN/DIVERGENT verdict. - - The keystone conversion of the commit-wait drive from blocking (`repl_wait_majority_ack` / `pthread_cond_timedwait`) to an event-driven resolve, and the resulting bound on request-handling capacity under concurrent in-flight commits. - - Regression and conformance coverage for the above ordering, apply-idempotency, and no-reaction-before-commit invariants. - - No regression of existing HA behavior, correctness guarantees, or fact-commit throughput as a result of this reordering. -- **Out of scope**: - - Collapsing a job-array submission into a single atomic WAL record — owned by `job-array-single-wal-record`, which builds on top of the apply seam this feature delivers. - - Changes to the raft library's protocol core (`raft_advance_commit`, `on_commit_advanced`, `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`) — owned by `raft-log-replication` / `raft-commit-safety` and consumed as-is. - - Entity-bearing snapshot content and the config-upgrade snapshot format — owned by `ha-entity-snapshots` / `config-upgrade-snapshot`. - - Read-consistency and leader-lease optimizations. - - Moving snapshot send, configuration-push waiting, or peer connect off the iomgr/delaymgr threads — owned by `raft-drive-async` (P2 transport/timer drive hardening); this feature owns only the commit-wait drive conversion (P1). - - Changes to the worker join/rejoin protocol itself — owned by `worker-join-refactor` / `worker-rejoin-resync`. - - Asynchronous DNS resolution for peer connections. - - Extract-from-hook / apply-after-commit ordering for queue-control and worker-control commands: `queue.c`'s `AFTER_EVENT` path and `persist_queue_hook_fun`/`persist_worker_hook_fun` still mutate in-memory state before firing the WAL append, the same double-append-class ordering this feature fixes for job submission and facts — un-fixed here (2026-07-09 validate-impl finding). Tracked as a follow-up, same shape as this feature's Requirement 1/2 but for the queue/worker command family. -- **Adjacent expectations**: - - This feature relies on the majority-commit and idempotency primitives already delivered by `ha-commit-engine` and `ha-commit-barrier-rollout`; it does not re-implement them. - - This feature touches the same submit/commit path that `ha-term-safety` is concurrently wiring into the raft library adapter; the two must be sequenced/coordinated so neither reintroduces a term-blind or apply-before-commit regression. - - Full recovery of a fact that never reached the controller, or that was lost with the controller across a failover window, remains the responsibility of `worker-rejoin-resync` (roadmap backlog, not yet built) — this feature relies on it as the designated backstop but does not own or implement it. Until `worker-rejoin-resync` lands, such facts remain lost, consistent with the umbrella-locked decision already accepted by `ha-consistency-invariants` and `ha-commit-barrier-rollout`. - - `job-array-single-wal-record` depends on the `apply_committed_entry()` seam this feature delivers and must land after it. - - Ordering and apply-correctness assertions introduced here are consumed by `raft-sim-conformance`'s regression harness. - - This feature must preserve the existing "controllers never self-terminate" invariant and the existing persist-before-ACK durability and synchronous majority-commit-barrier policy; it changes ordering and apply timing, not the durability contract. - -## Requirements - -### Requirement 1: Extraction of the Raft Commit Step from the Hook Fan-Out - -**Objective:** As a maintainer, I want the WAL append and majority-commit barrier for a command to run as an explicit step invoked directly by the command handler rather than as a side effect of an `AFTER_EVENT` hook callback, so that the raft protocol step is a first-class, ordered part of the request path instead of incidental hook behavior. - -#### Acceptance Criteria - -1. When a command handler (job submit, configuration change) processes a request, the controller shall perform the WAL append and majority-commit barrier as an explicit step invoked by the handler, not from within a `BEFORE_EVENT` or `AFTER_EVENT` hook callback. -2. The controller shall not register the majority-commit barrier as a hook subscriber for any command entry type. -3. The controller shall perform the explicit raft commit step without blocking the iomgr event-loop thread, preserving the existing async-I/O thread-role contract. -4. The controller shall preserve persist-before-ACK durability and the synchronous majority-commit barrier as policy; extracting the raft step from the hook fan-out shall not weaken or remove either. - -### Requirement 2: Canonical Command Ordering — Apply-After-Commit - -**Objective:** As a cluster operator, I want a job-submit or configuration command's in-memory effect to occur only after its entry is durably committed to a majority, so that a commit that fails or times out never leaves the cluster's observable state showing an operation that did not durably happen. - -#### Acceptance Criteria - -1. When a command entry is accepted for processing, the controller shall not apply its state mutation until the entry has committed to a majority. -2. When a command entry commits to a majority, the controller shall apply its state mutation before producing any success reply or notification for that entry. -3. If a command entry's majority commit fails or times out, then the controller shall report failure to the requester and shall not apply the entry's state mutation, so the controller's observable state is unchanged by the failed command. - -### Requirement 3: Single Shared Apply Seam — Online and Replay Parity - -**Objective:** As a maintainer, I want one shared `apply_committed_entry()` seam used both when a commit advances live and when the controller replays its log after restart, so that a node's materialized state is always the deterministic result of its committed log regardless of how it got there. - -#### Acceptance Criteria - -1. When the controller applies a committed entry during live operation, the controller shall invoke the same `apply_committed_entry()` seam that restart replay invokes for that entry type. -2. When restart replay processes a committed entry, the controller shall apply it through `apply_committed_entry()` rather than through a separate replay-only mutation path. -3. The controller shall apply committed entries strictly in commit order, never applying an entry out of order relative to an entry with a lower commit position. -4. The controller shall produce identical materialized state from a committed log regardless of whether that log is applied via live commit-advance or via restart replay. - -### Requirement 4: Hook Fan-Out Demotion to Notification-Only - -**Objective:** As a maintainer, I want after-hooks to be pure notification delivery observing already-committed, already-applied state, so that scheduler push, client reply, and apiserver updates no longer need to carry or defensively re-check commit responsibility. - -#### Acceptance Criteria - -1. When an `AFTER_EVENT` hook fires for a command or fact entry, the controller shall treat that hook exclusively as a notification of already-committed, already-applied state (client reply, scheduler dispatch trigger, SSE/apiserver update). -2. The controller shall not perform WAL append, commit-barrier waiting, or state-machine mutation from within any `AFTER_EVENT` or `BEFORE_EVENT` hook callback. -3. The controller shall not require a downstream consumer (scheduler, apiserver) to re-check a commit watermark before acting on a notification delivered by an after-hook, since that notification only fires after the entry is committed and applied. - -### Requirement 5: Facts Apply-After-Commit - -**Objective:** As a cluster operator, I want a worker-reported fact's state mutation and its consequential reactions (worker slot release, scheduler dispatch, notification) to occur only after the fact is committed to a majority, so that a leader that loses an uncommitted fact never leaves behind a phantom effect such as a freed slot, a triggered dispatch, or a sent notification. - -#### Acceptance Criteria - -1. When the controller receives a reported fact from a worker, the controller shall durably append the fact to its log without performing the fact's state mutation or reactions at that time. -2. When a reported fact's entry commits to a majority, the controller shall apply its state mutation and trigger its reactions through the same `apply_committed_entry()` seam used for commands. -3. The controller shall not release a worker slot, trigger scheduler dispatch, or send a client/apiserver notification for a fact whose entry has not committed to a majority. - -### Requirement 6: Quorum-Loss Behavior for Retained Facts - -**Objective:** As a cluster operator, I want reported facts to be safely retained — neither dropped nor prematurely applied — when the controller cannot reach a majority, so that the cluster degrades safely and self-heals without operator intervention once quorum is restored. - -#### Acceptance Criteria - -1. While the controller cannot reach a majority of cluster members, the controller shall retain durably-appended, uncommitted facts without applying their state mutations or triggering their reactions. -2. The controller shall not apply a fact's state mutation or reactions through any path other than majority commit, regardless of quorum state; no local-only apply allowlist shall exist. -3. When a majority becomes reachable again after a quorum-loss period, the controller shall commit and apply the retained facts in their original log order. - -### Requirement 7: Idempotent, Ordered Re-Application - -**Objective:** As a maintainer, I want re-applying an already-applied entry — via replay, retry, or worker resync — to be a safe no-op, so that duplicate or out-of-order delivery never regresses or corrupts job state. - -#### Acceptance Criteria - -1. When the controller applies an entry whose effect has already been applied, by prior commit, replay, or resync, the controller shall treat the re-application as a no-op with respect to observable job state. -2. If a duplicate or out-of-order fact is applied for a job, then the controller shall compare it against the job's current lifecycle state and shall not regress that job to an earlier lifecycle stage. -3. The controller shall apply entries in strict commit order and shall never apply an entry ahead of another entry that committed before it. - -### Requirement 8: Restart Apply-Correctness Verification - -**Objective:** As a cluster operator, I want the controller to verify after restart that its materialized state matches its committed log, so that a hidden apply defect is surfaced immediately rather than silently causing future divergence. - -#### Acceptance Criteria - -1. When the controller completes WAL replay after a restart, the controller shall verify whether the materialized state matches the expectation derived from the committed log tip, in addition to the existing CRC chain verification. -2. When the post-replay verification matches, the controller shall report a CLEAN verdict. -3. If the post-replay verification does not match, then the controller shall report a DIVERGENT verdict as a loud, operator-visible condition. -4. If the controller reports a DIVERGENT verdict, then the controller shall enter a non-promotable fenced state and shall continue running and retrying catch-up rather than exiting or aborting, consistent with the "controllers never self-terminate" invariant. - -### Requirement 9: Bounded Resource Behavior Under Concurrent In-Flight Commits - -**Objective:** As a cluster operator, I want the controller to remain responsive to new requests even while many operations are simultaneously awaiting commit, so that a burst of concurrent commands or facts cannot make the controller unresponsive. - -#### Acceptance Criteria - -1. While multiple entries are simultaneously awaiting majority commit, the controller shall continue to accept and begin processing new incoming requests. -2. The controller shall not exhaust its bounded request-handling capacity as a direct result of the number of entries currently awaiting commit. -3. When an entry's commit outcome (success, failure, or timeout) becomes known, the controller shall resolve exactly one reply or notification for that entry's originator. -4. If the controller loses leadership or a pending entry's log tail is truncated while the entry is awaiting commit, then the controller shall resolve that entry's outcome as a failure exactly once, rather than leaving it unresolved. - -### Requirement 10: Apply-Ordering Regression and Conformance Verification - -**Objective:** As a maintainer, I want automated regression coverage asserting the apply-after-commit ordering, so that a future change cannot silently reintroduce apply-before-commit or lost-notification defects. - -#### Acceptance Criteria - -1. The project's conformance test suite shall assert that no entry's state mutation or reactions occur before that entry commits. -2. The project's conformance test suite shall assert that a failed or timed-out commit produces no residual state mutation. -3. The project's conformance test suite shall assert that re-applying an already-committed entry, whether by simulated replay or retry, is idempotent. -4. The project's regression suite shall include a live-cluster scenario that kills the primary mid-fact-stream and asserts no phantom slot-frees, no committed-state divergence, and no notification for a fact that did not commit. - -### Requirement 11: Preservation of Existing Behavior and Performance - -**Objective:** As a cluster operator, I want this refactor to preserve every existing HA behavior, correctness guarantee, and performance characteristic that this feature does not specifically intend to change, so that fixing the apply-ordering defect does not introduce a new regression elsewhere. - -#### Acceptance Criteria - -1. The controller's existing automated unit and integration test suites shall continue to pass after this feature lands, except for tests that specifically assert the pre-existing apply-before-commit ordering, which shall be updated to assert the new apply-after-commit ordering instead. -2. The controller's existing HA failover regression and soak suites shall continue to report zero committed-job loss and zero committed-state divergence across a primary failover, matching the acceptance bar already established by prior HA specs. -3. When a client submits a job, cancels a job, or queries job/queue/worker status, or a worker reports a job's start, finish, failure, or resource usage, the controller shall produce the same externally observable end result as before this feature; this feature shall change only the internal timing of when that result becomes visible, not the result itself. -4. While a sustained high-volume worker-fact stream is in progress, the controller shall not regress fact-commit throughput relative to pre-feature behavior, and shall not introduce a per-fact synchronous commit barrier. diff --git a/.kiro/specs/raft-apply-decoupling/research.md b/.kiro/specs/raft-apply-decoupling/research.md deleted file mode 100644 index 614c70bf..00000000 --- a/.kiro/specs/raft-apply-decoupling/research.md +++ /dev/null @@ -1,103 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `raft-apply-decoupling` -- **Discovery Scope**: Complex Integration (extension of a live, fast-moving HA/Raft subsystem — light discovery upgraded to comprehensive verification because the source brief (`brief.md`, 2026-06-24/25) is 9 days old relative to the code and several sibling specs land in the exact same seam) -- **Key Findings**: - - `bug-1306` (commit `15be5d9`, 2026-06-30) already extracted the WAL-append + majority-commit barrier out of `persist_job_hook_fun` for job submit, ahead of this spec's formal implementation — Requirement 1 is partially pre-landed for the WAIT path. - - The in-memory job-submit mutation (`create_job()`) still runs **before** the commit barrier (`persist_submit_wal_and_commit()`), and a failed/timed-out commit does **not** roll it back — the exact optimistic-divergence window Requirement 2 targets is still live today, just narrowed by the bug-1306 fix (which only fixed the reply-ack race, not the apply-ordering). - - `ha-term-safety` is fully done (20/20 tasks); `on_commit_advanced` / `raft_advance_commit` are wired and live on the replication-ACK path (advisory-only, Option A) — no conflict with this spec's keystone. - - `raft-drive-async` already built `vtask_enq_classified` + a slow-lane executor + admission classes (`VTASK_LIVENESS` / `VTASK_LIVE_BLOCK` / `VTASK_REQ_BLOCK`) — this closes the "no general non-IO task-submit path" gap the brief flagged as missing, so the keystone conversion can reuse it instead of building a new primitive. - - Online-apply and replay-apply are **not** unified today: job submit online uses raw `create_job()`; replay uses `restore_job()` (decode + idempotent `-EEXIST` guard + LSN tag). Fact state transitions online go through guarded `mark_job_start/finish/fail/update()` (monotonic-rank checks); replay applies facts via `restore_job_state()`, a raw `j->job.state = state` overwrite with **no** guard at all. This is concrete, evidenced proof of the divergence Requirement 3/7 exist to close. - - `WAL_MODE_RETAIN` facts already register into the commit engine (`commit_submit(ce, lsn, COMMIT_MODE_RETAIN, 0)`) for durability bookkeeping, but nothing today reacts to a RETAIN entry's commit — the state mutation already happened synchronously at receipt, before the WAL append. The "apply pump" driven off commit-advance (Requirement 5/6) does not exist in any form yet. - - `raft-sim-conformance`'s harness structurally enforces apply-only-after-commit already (it calls `raft_advance_commit`, then applies via `on_commit_advanced`, then checks State Machine Safety) — Requirement 10's conformance assertions mostly compose with existing harness structure rather than requiring new harness machinery. - -## Research Log - -### Current submit/hook path (job.c, persist.c) -- **Context**: Brief claimed `persist_job_hook_fun` does WAL-append + commit-wait inside an `AFTER_EVENT` hook (persist.c:2532/2198, 2026-06-24 citation). -- **Sources Consulted**: `src/controller/job.c:560-662` (`handle_jobsubmit`), `src/controller/persist.c:3392-3478` (`persist_submit_wal_and_commit`, `persist_job_hook_fun`), git log (`15be5d9`). -- **Findings**: - - `handle_jobsubmit` (job.c:560): `BEFORE_EVENT` hook (validation) → `job_submit()` → `create_job()` (in-memory insert into `c->jobmgr`, job.c:583) → **explicit** `persist_submit_wal_and_commit()` (job.c:626, persist.c:3406) reading the commit verdict directly → on failure, reply error and **return** (job.c:627-637) — the already-created job object is **not** removed from `c->jobmgr` → on success, `AFTER_EVENT` hook fires as notification-only (job.c:643, persist.c:3466-3478 confirms the WAIT branch inside `persist_job_hook_fun` is now a no-op) → reply success. - - `persist_job_hook_fun`'s `WAL_MODE_WAIT` branch is dead code by design (returns 0 unconditionally, persist.c:3466-3478) — kept as a documented no-op, not removed, so hook registration order doesn't matter for job submit anymore. -- **Implications**: Requirement 1 (extract raft step from hook) is **done** for the job-submit WAIT path. Requirement 2 (apply-after-commit ordering) is **not** done — `create_job()` still runs before the barrier, and there is no rollback on commit failure. Design must move the job **object construction** itself (not just a secondary mutation) into the post-commit apply step. - -### Commit engine and ha-term-safety completion (commit.c, ha.c) -- **Context**: The keystone prerequisite (`repl_wait_majority_ack` blocking-to-event-driven conversion) is claimed as this spec's first-landed item; needed to confirm nothing else already did it and that `ha-term-safety`'s concurrent work on the same seam doesn't conflict. -- **Sources Consulted**: `src/controller/repl.c:533-558`, `src/controller/commit.c:195-409`, `src/controller/ha.c:1680,1776,1829-1831,3764`, `.kiro/specs/ha-term-safety/tasks.md`. -- **Findings**: - - `repl_wait_majority_ack` (repl.c:557) unconditionally delegates to `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout_ms)` (commit.c:358), which blocks via `pthread_cond_timedwait` (commit.c:409) under the `COMMIT_WAIT_RESERVED` admission cap. Unchanged — still a pooled-worker blocking wait, 500ms default timeout. - - `ha-term-safety`: all 20 tasks done. `on_commit_advanced` (ha.c:1776) is wired and live, firing from `raft_advance_commit` (raft.c call site ha.c:3764) on the replication-ACK path; it feeds `commit_set_raft_commit_index()` as an **advisory** cap only (Option A — the durable-majority barrier remains authoritative; the library commit index does not gate anything today). -- **Implications**: No conflict — `ha-term-safety` supplies the library-side commit-index plumbing this spec's apply pump can key off of (`on_commit_advanced`), but the blocking-wait-to-resolve conversion itself is untouched and squarely this spec's keystone to build. - -### Reusable async primitives from raft-drive-async (vtaskmgr.c) -- **Context**: The brief identified a gap — no general (non-IO) task-submit path exists — as an API prerequisite for the keystone's async continuations. -- **Sources Consulted**: `.kiro/specs/raft-drive-async/tasks.md`, `src/controller/vtaskmgr.c:467,497,528`. -- **Findings**: Task 1.2 (slow-lane executor, non-blocking submit) and Task 1.6 (`vtask_class_t` + `vtask_enq_classified`, admission-gated by class: `VTASK_LIVENESS` / `VTASK_LIVE_BLOCK` / `VTASK_REQ_BLOCK`) are both done and live (default-on since 2026-07-02, bug-1376). `vtask_enq_classified(mgr, task, class)` is a general, non-packet-bound submit path. -- **Implications** (Build vs. Adopt): Adopt `vtask_enq_classified` with a `VTASK_REQ_BLOCK`-equivalent class for the resolved-commit continuation (reply dispatch / apply trigger) instead of building a new submission primitive. This directly resolves the brief's "API gap" concern at zero new-code cost. - -### Restart replay and the apply-seam divergence (persist.c, job.c) -- **Context**: Requirement 3 requires one shared `apply_committed_entry()` seam for online and replay; needed to establish exactly how much divergence exists today. -- **Sources Consulted**: `src/controller/persist.c:2452-2562` (`apply_wal_entry`), `src/controller/persist.c:2667-2723` (`replay_wal` CRC verification), `src/controller/job.c:44-177` (`create_job`, `restore_job`, `restore_job_state`), `src/controller/job.c:664-774` (`mark_job_start`, `mark_job_finish`). -- **Findings**: - - `apply_wal_entry` dispatches `WAL_JOB_SUBMIT` → `restore_job()` (persist.c:2505) and `WAL_JOB_{START,FINISH,FAIL,...}` → `restore_job_state()` (persist.c:2520). - - `restore_job()` (job.c:96) decodes the Cap'n Proto payload, checks for an existing job by jobid (idempotent `-EEXIST` guard, job.c:117-130), and calls the **same** `create_job()` (job.c:44, thin wrapper over `makeJob`) that the online submit path uses. So submit-side apply is **half**-unified: replay already funnels through a shared primitive; online does not (it calls `create_job()` directly, skipping the idempotency guard and LSN tagging that `restore_job()` provides). - - `restore_job_state()` (job.c:164) is a **raw** `j->job.state = (int)state` overwrite — no `job_state_is_backward`/`job_state_is_terminal` check at all. The online path's `mark_job_start`/`mark_job_finish`/`mark_job_update` (job.c:664-850) **do** carry those monotonic-rank guards. This is a genuine, currently-live idempotency gap: a duplicate or out-of-order fact entry replayed today (segment-boundary restart replay, or a future retained-fact retry) can regress job state, because replay's mutation path bypasses the same guard the online path enforces. -- **Implications**: `apply_committed_entry()` should standardize on `restore_job()` for submits (already idempotent, already reusable) and on a **guarded** state-transition function for facts — meaning `mark_job_start/finish/fail/update`'s guard logic must become reachable from both the post-commit online path and replay, and `restore_job_state()` must stop being a second, weaker mutation path. This is a Build-vs-Adopt case: adopt the existing guards, relocate their call site. - -### raft-sim-conformance harness readiness -- **Context**: Requirement 10 needs regression coverage for apply-ordering; checked whether the existing deterministic simulator can carry these assertions without new harness machinery. -- **Sources Consulted**: `.kiro/specs/raft-sim-conformance/{spec.json,requirements.md,design.md}`. -- **Findings**: The harness already asserts Election Safety, Log Matching, Leader Completeness, and State Machine Safety, and its commit/apply loop is structured as `raft_advance_commit` → `on_commit_advanced` → apply → State-Machine-Safety check — i.e., it is already apply-after-commit by construction. -- **Implications**: New assertions (no-reaction-before-commit, idempotent re-apply) are additive checks layered on the existing harness loop, not a new harness. The live-cluster kill-mid-fact-stream scenario (Requirement 10.4) is new and belongs to the `/xce-test-ha`-style soak tooling, not the simulator. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|----------------------|-------| -| Single `apply_committed_entry()` dispatch function, keyed on WAL type | One function, switch on `wal_type`, called from both the post-commit online path and `apply_wal_entry()` | Matches steering's "library-owned, thin-adapter" pattern already used for raft callbacks; smallest change that satisfies Req 3 | Must not become a god-function — keep per-type logic in the existing `restore_*`/`mark_job_*` helpers, `apply_committed_entry()` only dispatches | Selected | -| Separate apply seams per entry type (no unifying dispatcher) | Each WAL type keeps its own online-vs-replay pair, just guard-aligned | Less code to touch per type | Does not satisfy Req 3's "same seam" requirement literally; re-invites future divergence | Rejected | -| Event-sourced full replay-log-as-truth (rebuild all reactions from log on every apply) | Maximal purity | Over-engineered — Design Risk A in `facts-in-raft.md` already concluded reactions must NOT be replayed, only state; resync (existing) is the re-derivation mechanism | Rejected (already settled in brief.md) | - -## Design Decisions - -### Decision: Keystone conversion reuses the existing per-LSN commit registry and `vtask_enq_classified`, not a new registry or submit path -- **Context**: `repl_wait_majority_ack` → `commit_submit(..., COMMIT_MODE_WAIT, ...)` blocks a pooled worker on `pthread_cond_timedwait`. Converting to event-driven `resolve()` needs (a) a way to register a continuation instead of parking a thread, and (b) a way to run that continuation without new infrastructure. -- **Alternatives Considered**: - 1. Build a new pending-request registry alongside the commit engine's existing per-LSN structure. - 2. Reuse the commit engine's existing per-LSN structure (`ce`), replacing `cond_broadcast` with a `resolve(lsn, outcome)` call that invokes a registered continuation. -- **Selected Approach**: (2). `commit_submit` keeps its per-LSN bookkeeping; `COMMIT_MODE_WAIT` callers register a continuation (function pointer + context) instead of blocking; `commit_advance_locked`'s existing `cond_broadcast` call site becomes the single place that walks newly-committed/newly-failed LSNs and invokes `resolve()`, which dispatches the continuation via `vtask_enq_classified` (raft-drive-async) rather than a bespoke thread hop. -- **Rationale**: No new concurrent data structure to get wrong; reuses the already-verified admission-cap discipline (`COMMIT_WAIT_RESERVED` / `VTASK_*` classes) instead of inventing a second one. -- **Trade-offs**: The commit engine's internal contract changes from "callers block on `ce->cond`" to "callers register a callback" — every existing `COMMIT_MODE_WAIT` call site (job submit today; future command types) must be converted together, not incrementally, or the engine would need to support both calling conventions simultaneously. -- **Follow-up**: Verify during implementation that `commit_advance_locked`'s lock-ordering discipline (documented at persist.c:3347-3350: never hold `p->lock` across `ce->lock`) still holds once resolution dispatches into `vtask_enq_classified` from inside the commit-advance path. - -### Decision: `apply_committed_entry()` standardizes on `restore_job()` / a guard-carrying state-transition function, not new mutation primitives -- **Context**: See Research Log finding above — replay already has an idempotent submit primitive (`restore_job`); online submit doesn't use it. Online facts have a guarded transition primitive (`mark_job_*`); replay doesn't use it. -- **Alternatives Considered**: - 1. Write new, from-scratch apply functions for both submit and facts. - 2. Adopt `restore_job()` as-is for submit; extract the guard logic embedded in `mark_job_start/finish/fail/update` into a form callable from both the post-commit path and `apply_wal_entry`, retiring `restore_job_state()`'s unguarded overwrite. -- **Selected Approach**: (2). -- **Rationale**: Both primitives already carry the exact idempotency semantics Requirement 7 asks for; the gap is call-site wiring, not missing logic. Writing new functions would duplicate logic that already exists and is already unit-tested. -- **Trade-offs**: `mark_job_start/finish/fail/update` today fire their own `BEFORE_EVENT`/`AFTER_EVENT` hooks inline (job.c:680-681, 699-700) as part of the online receipt path — that hook-firing must move to occur only when these functions are invoked from the post-commit apply step, not from the (now durable-append-only) receipt handler, or the ordering guarantee (Req 5) would be violated. This is a real refactor, not a pure reuse. -- **Follow-up**: Confirm no other caller of `restore_job_state()` (grep found only `apply_wal_entry`) depends on its current unguarded-overwrite behavior before retiring it. - -### Decision: `egress.c`'s `job_submit_committed()` watermark gate becomes redundant for job submit once Req 2 lands, but is not removed by this spec -- **Context**: Requirement 4.3 says downstream consumers should not need to re-check a commit watermark once notifications only fire post-commit. `job_submit_committed()` (egress.c:47) exists precisely because today a job is visible in `c->jobmgr` before it's committed. -- **Selected Approach**: Design leaves `job_submit_committed()` in place as a defense-in-depth predicate (it becomes a tautology once Req 2 lands — a job is never in `jobmgr` before commit — so it's harmless to keep), and records its removal as a documented candidate cleanup, not an in-scope deletion. -- **Rationale**: Removing it requires confirming every caller of the scheduler-export path relies solely on this gate and not on some other assumption — that audit is a distraction from the core ordering fix and belongs in a follow-up cleanup pass, matching the Simplification lens's guidance not to over-scope "just in case." -- **Follow-up**: Revalidation trigger — if a future spec removes `job_submit_committed()`, it must re-verify no path still relies on it as a correctness gate rather than a no-op. - -## Risks & Mitigations -- **Risk**: Converting every `COMMIT_MODE_WAIT` call site to the continuation-based API in one change is a wide, mechanically risky diff (touches the commit engine's core contract). — **Mitigation**: today there is exactly one `COMMIT_MODE_WAIT` call site (`repl_wait_majority_ack`, used only by job submit); the conversion is narrow in practice even though the contract change is conceptually broad. -- **Risk**: Relocating `mark_job_*`'s hook-firing out of the receipt-time handler could silently change BEFORE_EVENT hook timing/semantics for subscribers that assume BEFORE fires at receipt. — **Mitigation**: audit all `HOOK_DETAIL_JOB_START`/`_EXIT`/etc. subscribers before relocating; BEFORE_EVENT for facts should fire at receipt (validation), only the AFTER_EVENT + state mutation defers to apply. -- **Risk**: `worker-rejoin-resync` (hard dependency for full fact-reaction correctness per `facts-in-raft.md` Design Risk A) does not exist yet. — **Mitigation**: accepted, documented gap, consistent with prior specs (`ha-consistency-invariants`, `ha-commit-barrier-rollout`) that already ship with this same accepted gap. -- **Risk**: The keystone conversion changes lock-ordering-sensitive code in the commit engine that several other specs depend on (`ha-commit-durability`, `ha-term-safety`). — **Mitigation**: preserve the documented lock-ordering rule (never hold `p->lock` across `ce->lock`) as an explicit invariant in design and tests; run the full existing HA soak suite (Req 11) before/after. - -## References -- `.kiro/specs/raft-apply-decoupling/brief.md` — original discovery brief (2026-06-24/25) -- `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` — facts-through-raft candidate plan, D1-D3 decisions, keystone rationale -- `.kiro/specs/ha-consistency-invariants/requirements.md` criterion 2.9 — Facts-Apply invariant, ownership assignment -- `.kiro/specs/ha-term-safety/tasks.md` — confirms full completion, no conflict -- `.kiro/specs/raft-drive-async/tasks.md` — confirms `vtask_enq_classified` / slow-lane availability -- `.kiro/specs/raft-sim-conformance/{requirements.md,design.md}` — existing conformance harness structure -- Commit `15be5d9` — bug-1306 fix, partial pre-landing of Requirement 1 diff --git a/.kiro/specs/raft-apply-decoupling/spec.json b/.kiro/specs/raft-apply-decoupling/spec.json deleted file mode 100644 index d17fd15f..00000000 --- a/.kiro/specs/raft-apply-decoupling/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "raft-apply-decoupling", - "created_at": "2026-07-03T03:32:11Z", - "updated_at": "2026-07-09T21:00:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/raft-apply-decoupling/tasks.md b/.kiro/specs/raft-apply-decoupling/tasks.md deleted file mode 100644 index 7372de84..00000000 --- a/.kiro/specs/raft-apply-decoupling/tasks.md +++ /dev/null @@ -1,221 +0,0 @@ -# Implementation Plan - -- [x] 1. Commit engine event-driven resolve (keystone prerequisite) -- [x] 1.1 (P) Add per-LSN continuation registration to the commit engine for the synchronous (WAIT) commit mode, replacing the blocking wait - - Register a continuation (callback + context) against an appended LSN instead of parking the calling thread - - Preserve the existing per-LSN bookkeeping and admission-cap discipline already used by the commit engine - - Observable: a unit test can register a continuation for an LSN and the calling thread returns immediately, before the LSN commits - - _Requirements: 1.3, 1.4, 9.1, 9.2_ - - _Boundary: Commit Engine Resolve — commit.c per-LSN registry only_ - -- [x] 1.2 Build the generic resolve primitive and wire the majority-ack (committed) outcome - - Implement `commit_resolve()`: invoke a registered continuation with a committed or failed outcome, exactly once, idempotent on re-resolve - - On majority ack, invoke the continuation with a committed outcome - - Dispatch every continuation invocation off the calling thread using the existing non-blocking task-submission primitive, never inline on the replication-ACK or iomgr path - - Observable: forcing majority-ack in a test resolves the registered continuation exactly once with a committed outcome - - _Requirements: 1.3, 9.3, 9.4_ - - _Note: real timeout/demotion/truncation **detection** (as opposed to the generic resolve primitive) is 1.3/1.4 — see Implementation Notes_ - -- [x] 1.3 Arm a timeout for registered async continuations - - `commit_submit_async()` no longer parks a thread that would otherwise notice its own `pthread_cond_timedwait` deadline expire — converting away from the blocking wait removes the ONLY mechanism that ever detected a commit timing out. Add an explicit timeout (e.g., a delaymgr-driven check, or a deadline recorded at registration and swept periodically) that resolves a still-pending registered continuation as FAILED with `-ETIMEDOUT` if majority is not reached within the caller-supplied timeout - - Preserve the existing default timeout value and error code so callers observe identical failure timing/vocabulary to today's blocking path — this is a "no regression" requirement (11.1), not just a new capability - - Observable: an async-registered continuation for an LSN that never reaches majority is resolved as FAILED with `-ETIMEDOUT` within the configured timeout window, with no thread blocked to detect it, verified by a test that lets real time (or a mocked clock) elapse past the deadline without ever calling `commit_resolve()` manually - - _Requirements: 1.3, 1.4, 9.3, 9.4_ - - _Boundary: Commit Engine Resolve — commit.c only_ - -- [x] 1.4 Wire demotion and log-truncation detection to resolve pending continuations as failed - - When this node is demoted (loses leadership) or truncates its log tail (conflict reconcile), any still-pending registered continuation for an LSN affected by that transition must resolve as FAILED — otherwise a demoted/truncated node leaves requesters permanently unresolved - - Find the existing demotion/role-transition point and the existing log-truncation point and add a call into the commit engine's resolve path for every pending LSN they invalidate - - Observable: forcing a demotion or a truncation while a continuation is registered for an affected LSN resolves it as FAILED exactly once, with the existing error-code vocabulary - - _Requirements: 1.3, 1.4, 9.3, 9.4_ - - _Boundary: Commit Engine Resolve integration points — this task is a narrow, explicit exception to the "commit.c only" boundary used by 1.1-1.3: it may touch the demotion/role-transition site and the log-truncation site (wherever they currently live) in addition to commit.c, but must not touch job.c or any command-handler code_ - - _Depends: 1.3_ - -- [x] 1.5 Unit test the commit engine's resolve behavior under concurrent in-flight commits - - Cover exactly-once resolution across majority ack, timeout, demotion, and truncation, exercised through the real detection paths built in 1.3/1.4 (not by calling `commit_resolve()` directly, which 1.2's own tests already cover) - - Cover that a burst of concurrently pending commits does not exhaust the bounded request-handling capacity - - Observable: a test suite run demonstrates bounded worker usage and exactly-once resolution under a concurrent-commit burst - - _Requirements: 9.1, 9.2, 9.3, 9.4_ - - _Depends: 1.4_ - -- [x] 2. Shared apply seam for committed entries -- [x] 2.1 (P) Extract the guarded state-transition logic used by fact handling into a form usable outside the receipt-time call site - - Keep the existing monotonic-rank checks (reject backward transitions, no-op duplicate terminal transitions) intact and unchanged in behavior - - Decouple the transition logic from firing its own hook inline, so hook-firing can be relocated by a later task without touching this logic again - - Observable: the extracted transition logic can be invoked directly by a test without going through the worker-report receipt path, and produces the same guard behavior as before extraction - - _Requirements: 3.3, 7.1, 7.2_ - - _Boundary: Apply Seam — persist.c / job.c apply-dispatch and replay call sites only, no dependency on the commit-engine change in task 1_ - -- [x] 2.2 Build the single dispatch function that applies one committed entry to in-memory state, covering every classified entry type - - Route job submission through the existing idempotent, jobid-addressable construction primitive instead of a raw uncoordinated insert - - Route job state transitions through the extracted guarded logic from 2.1 - - Route queue control and worker control entries through their existing restore primitives - - Fire the entry type's post-mutation notification hook from inside this function, after the mutation completes - - Observable: calling the dispatch function directly with a sample entry of each classified type produces the correct in-memory mutation and fires exactly one post-mutation notification - - _Requirements: 2.2, 3.1, 3.2, 4.1, 4.2, 5.2_ - - _Depends: 2.1_ - -- [x] 2.3 Point restart replay's per-entry application at the new dispatch function - - Replace replay's direct calls to the individual restore primitives with calls to the dispatch function from 2.2 - - Observable: restart replay of a log containing every classified entry type produces identical materialized state to before this change, verified by an existing or updated replay test - - _Requirements: 3.1, 3.2_ - - _Depends: 2.2_ - -- [x] 2.4 Unit test idempotent and ordered re-application of the dispatch function - - Cover applying the same committed entry twice is a no-op with respect to observable state - - Cover that a duplicate or out-of-order fact entry does not regress a job's lifecycle state - - Observable: a test applies a FINISH entry followed by a stale, previously-unapplied START entry for the same job and confirms the job remains in its terminal state - - _Requirements: 3.4, 7.1, 7.2, 7.3_ - - _Depends: 2.2_ - -- [x] 3. Command apply-after-commit ordering -- [x] 3.1 (P) Stop constructing the in-memory job object before the commit barrier - - Change job submission to carry forward only the data needed to build the WAL entry payload, without inserting anything into the in-memory job table until the (still-synchronous, at this point) commit call returns success - - Remove the now-unreachable code path that previously ran the durability step from inside the notification hook for job submission - - Observable: a job submission whose commit fails or times out leaves no job object in the in-memory job table; a successful submission still replies synchronously exactly as before this task - - _Requirements: 1.2, 2.1, 2.3_ - - _Boundary: Raft Step (Command) — job.c handle_jobsubmit and its call chain only, no dependency on task 1_ - -- [x] 3.2 Convert job submission to register its commit outcome through the new continuation mechanism instead of blocking - - Replace the synchronous commit call from 3.1 with a registration against the task 1 continuation API - - On a failed or timed-out outcome, reply failure directly from the continuation — no apply is needed on this path - - On a committed outcome, hand the entry off to the ordered apply pump (task 6) rather than applying or replying from this continuation directly, so this entry's apply and reply happen in strict commit order relative to every other entry - - Observable: the calling thread returns before the commit resolves; a forced failure/timeout outcome still replies failure immediately with no job ever appearing (the committed-outcome path has no independently observable behavior until task 6 completes it) - - _Requirements: 1.1, 1.4, 2.1, 2.3_ - - _Depends: 1.1, 1.2, 1.3, 1.4_ - -- [x] 3.3 Unit test the command failure path leaves no observable trace - - Force repeated commit failures and timeouts for job submission and confirm no job object is ever created - - Observable: the test suite demonstrates zero phantom job objects across repeated forced commit failures - - _Requirements: 2.1, 2.3_ - - _Depends: 3.2_ - -- [x] 4. Fact apply-after-commit split -- [x] 4.1 (P) Change worker-fact receipt handling to validate and durably append only - - Remove the state mutation and the post-mutation notification hook firing from the receipt-time handlers for job start, finish, fail, and update reports - - Keep the existing pre-mutation validation hook firing at receipt time, unchanged - - Observable: after a worker reports a job event, the job's in-memory state is unchanged immediately after the receipt handler returns - - _Requirements: 5.1_ - - _Depends: 2.1_ - - _Boundary: Fact Ingestion — job.c on_job_start/on_job_finish/on_job_fail/on_job_update receipt handlers only, disjoint from handle_jobsubmit_ - -- [x] 4.2 Unit test durable retention of facts submitted while majority is unreachable - - Submit facts while majority is unreachable and confirm they are durably appended and that job state remains unchanged for the duration of the outage - - Confirm no local-only apply code path exists by inspection of the receipt handler from 4.1 (no runtime check needed — the guarantee is structural) - - Observable: the test demonstrates zero state mutation during a simulated quorum-loss window - - _Requirements: 6.1, 6.2_ - - _Depends: 4.1_ - -- [x] 5. Restart apply-correctness verification -- [x] 5.1 (P) Compute a post-replay expectation from the committed log tip and compare it against materialized state - - Derive a lightweight expectation (e.g., last-applied position and a count or checksum of materialized entities) from the recovered log - - Compare the expectation against the actual state produced by replay through the shared apply dispatch function - - Observable: a clean replay produces a CLEAN verdict alongside the existing structural (CRC) verdict - - _Requirements: 8.1, 8.2_ - - _Depends: 2.3_ - - _Boundary: Restart Replay & Verification — persist.c replay/recovery path only_ - -- [x] 5.2 Route a verification mismatch through the existing non-promotable fence mechanism - - Reuse the existing loud-fence function and sticky fence flag rather than introducing a new terminal or fencing path - - Observable: an injected materialization mismatch produces a DIVERGENT verdict, a loud operator-visible log entry, and a non-promotable fenced state - - _Requirements: 8.3, 8.4_ - - _Depends: 5.1_ - -- [x] 5.3 Unit test the DIVERGENT verdict path stays alive and keeps retrying - - Force a DIVERGENT verdict and confirm the process does not exit or abort - - Confirm the fence clears once a subsequent recovery reaches a consistent state - - Observable: the test process remains running and retrying after a forced DIVERGENT verdict, never terminating - - _Requirements: 8.4_ - - _Depends: 5.2_ - - _Boundary: Restart Replay & Verification — retroactive exception (added 2026-07-07 validate-impl audit, mirroring task 1.4's precedent): review found the sticky `recovery_unrecoverable` flag has two pre-existing, trigger-agnostic live-clear call sites in `ha.c` (`promotion_fence_timeout_fn`, `raft_cb_become_leader`'s promotion gate) that could silently release an apply-divergence-triggered fence. The fix is confined to those two existing call sites only — an `apply_divergent` guard ANDed into each — never job.c or any command-handler code._ - -- [x] 6. Ordered apply pump integration -- [x] 6.1 Build the strict-order walk that applies every newly committed entry - - Walk entries from the last-applied position up to the current committed position, in ascending order, with no skipping or reordering - - Invoke the shared apply dispatch function for every entry in that range, regardless of its commit mode - - Observable: a burst of interleaved command and fact entries committing out of submission order still applies in commit order - - _Requirements: 3.3, 7.3_ - - _Depends: 1.2, 2.2_ - -- [x] 6.2 Fire a command's registered continuation immediately after its entry applies - - After the walk in 6.1 applies an entry that has a registered commit continuation (from task 3.2), invoke that continuation now, after the mutation, not before - - Leave fact entries with no further action beyond what the apply dispatch function itself performs - - Observable: a successful job submission's client reply carries the job id only after the job is visible in the in-memory job table; the reply always occurs strictly after the mutation - - _Requirements: 1.1, 2.1, 2.2, 2.3_ - - _Depends: 6.1, 3.2_ - -- [x] 6.3 Integration test command and fact apply/notify timing through the full pump - - Cover a successful job submission becoming visible and notified only after commit, exercised through the full pump (superseding the earlier synchronous-path check from task 3.1) - - Cover that worker slot release, scheduler dispatch triggering, and client/apiserver notification for a fact do not occur until that fact's entry is committed - - Observable: the test suite shows zero fact-reaction side effects during a delayed-commit window, and a submitted job's reply always follows its visibility in the job table - - Confirm a consumer acting on the post-apply notification never needs to separately re-check a commit watermark, since the notification's occurrence already implies the entry is committed and applied - - _Requirements: 2.2, 4.3, 5.2, 5.3_ - - _Depends: 6.2, 4.1_ - -- [x] 6.4 Integration test quorum-loss-then-restore catch-up ordering through the full pump - - Submit a mix of facts and commands while majority is unreachable, then restore majority and confirm every retained entry commits and applies in its original log order with no local-apply escape hatch exercised - - Observable: the test demonstrates full, correctly-ordered catch-up after quorum returns, with zero out-of-order application - - _Requirements: 6.3, 7.3, 9.4_ - - _Depends: 6.1, 4.2_ - -- [x] 7. Conformance, regression, and existing-behavior preservation -- [x] 7.1 Extend the deterministic conformance harness with apply-ordering assertions - - Add an assertion that no entry's state mutation or reactions occur before that entry commits - - Add an assertion that re-applying an already-committed entry is idempotent - - Observable: the conformance harness run reports both new assertions passing alongside its existing safety checks - - _Requirements: 10.1, 10.3_ - - _Depends: 6.3, 6.4_ - -- [x] 7.2 Add a live-cluster regression scenario that kills the primary mid-fact-stream - - Drive a continuous worker-fact stream, kill the primary partway through, and let a new primary take over - - Assert no phantom slot-frees, no committed-state divergence, and no notification for a fact that never committed - - Observable: the scenario completes with zero divergence and zero phantom reactions across repeated runs - - _Requirements: 10.2, 10.4_ - - _Depends: 6.3, 6.4_ - -- [x] 7.3 Re-run the full existing automated test suite and update only the tests tied to the old ordering - - Run the existing unit, integration, and HA soak suites - - Update only the specific tests that asserted the old apply-before-commit ordering to assert the new ordering instead - - Observable: the full suite passes with zero committed-job loss and zero state divergence across the existing failover soak - - _Requirements: 11.1, 11.2, 11.3_ - - _Depends: 7.1, 7.2_ - -- [x] 7.4 Measure sustained fact-stream throughput against the pre-feature baseline - - Record a throughput baseline on the pre-feature code, then drive the same sustained high-volume worker-fact stream against the completed feature - - Confirm no per-fact synchronous commit barrier was introduced and throughput does not regress - - Observable: a recorded throughput comparison shows no regression relative to the pre-feature baseline - - _Requirements: 11.4_ - - _Depends: 7.3_ - -## Implementation Notes - -- **7.1**: correctly identified `src/lib/raft/raft_sim.c` (pre-existing `#ifdef XCE_TESTING` TLA+-conformance simulator, driven by `src/lib/testsuite/test_raft.c` in the `xcelib_test` Unity binary — a different binary/library boundary than every other task in this spec, which use `controller_unit_test`) as the intended "deterministic conformance harness," confirmed verbatim against design.md's Testing Strategy line 442 ("Extend `raft_sim.c`'s conformance harness with 'no reaction before commit' and 'idempotent re-apply' assertions layered on its existing apply-after-`on_commit_advanced` loop"). Added two fault-injectable checker functions (`raft_sim_check_apply_after_commit`, `raft_sim_check_idempotent_reapply`) plus `raft_sim_replay_commit`, gated by two new flags (`RAFT_SIM_BREAK_APPLY_BEFORE_COMMIT`, `RAFT_SIM_BREAK_REAPPLY_GUARD`) and a new `apply_reaction_count[]` field on `raft_sim_node_t`. Wired into `sim_persist_entry`/`sim_conf_on_commit_advanced`, both no-ops (byte-identical to prior behavior, confirmed by reviewer) when the flags are unset — zero regression risk to the 5 pre-existing scenario functions. RED phase independently reproduced by reviewer for both checks (`Expected 39338 Was 0`, `Expected 2 Was 1`); GREEN 287/0/0 confirmed twice (implementer + reviewer), ASan-clean. - **Scoping decision, independently upheld by reviewer**: the two new checks run only via their own 2 dedicated tests, NOT wired into `raft_sim_check_all`/`sim_scen_check` (the umbrella the 5 pre-existing scenario functions use). Reviewer traced this to `raft-sim-conformance` spec's Requirement 3 Criterion 5 (fixed four-invariant, fixed-order, halt-on-first-failure contract) — an already-shipped contract owned by a *different* spec that 7.1 has no charter to modify. Also: these checks operate on the harness's synthetic `applied[]`/`apply_reaction_count[]` model, not real production `apply_committed_entry()`/`persist_apply_pump_run()` — real production coverage of this ordering already exists via tasks 6.1-6.4's `test_apply_pump_integration.c`. Confirmed legitimate scoping, not a corner-cut. - **Residual, non-blocking documentation nitpick**: the `applied[k-1] != 0` "has-been-applied" sentinel is not structurally guaranteed non-zero the way log-entry term 0 is (raft.tla guarantees terms start at 1; arbitrary client payloads have no such guarantee) — the header comment's "same zero-means-absent convention as log-entry terms" analogy slightly overclaims precedent (the pre-existing `raft_sim_check_state_machine_safety` bounds by `commit_index` directly, doesn't use a zero-sentinel at all). Currently harmless because the only two call sites use hardcoded non-zero literals (`0x99AA`, `0x77BB`). **Fast-follow flag for whoever later widens this harness to PRNG-drawn payload values**: `apply_reaction_count[]` cannot substitute as the "ever-applied" indicator for `check_apply_after_commit`, since the `BREAK_APPLY_BEFORE_COMMIT` fault only touches `applied[]` — a real generalization needs a third parallel "ever-written" flag array, not a reuse of either existing array. -- **1.1**: `commit_submit_async()` reuses the exact `wait_admit_locked`/`wait_release_locked`/`inflight_waits` admission-cap discipline as `COMMIT_MODE_WAIT` — task 1.2's `commit_resolve()` MUST call `wait_release_locked()` when resolving an async-registered entry, or the shared admission cap leaks. Continuation registry is `calloc(n_vcpus, ...)`, sized so `n_continuations >= wait_cap(ce)` always holds; `lsn == 0` is the free-slot sentinel. -- **1.1 review**: bare `controller_unit_test` (no `--run` filter) segfaults deterministically in this environment (SSL_CTX_free/iomgr_fini crash, `test_ha_vep_teardown.c`) — reproduced identically on the pre-task base commit, confirmed pre-existing and unrelated. Logged as bug-1406. Also `test_commit_advance_retry_clears_push_before_commit_fence` (`test_config_repl.c`) fails independently of this feature, also reproduced on base — logged as bug-1407, flagged because it's adjacent to `commit_advance_locked`, which task 1.2 modifies. **Always validate with a filtered `--run <group>` invocation or the tightly-coupled commit-engine cluster, not the bare unfiltered binary, until 1406/1407 are separately fixed.** -- **1.2**: implemented `commit_resolve()` + wired it into all 6 commit.c-internal paths that can advance `committed_lsn`/`writable_lsn` (the COMMITTED/majority-ack outcome). Chose `VTASK_LIVE_BLOCK` for dispatch (never admission-gated, unlike `VTASK_REQ_BLOCK` which can fast-reject — unacceptable for "exactly-once, never unresolved"). **Reviewer caught a real task-graph gap, not a defect in 1.2 itself**: converting `COMMIT_MODE_WAIT` away from a blocking wait also removes the *only* mechanism that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline) — nothing yet arms an equivalent timeout for async-registered continuations, and nothing yet wires demotion/log-truncation detection to call `commit_resolve(FAILED, ...)`. Without this, a pending async command during quorum loss would hang forever instead of failing — worse than today's behavior. Split out as new tasks **1.3** (timeout arming) and **1.4** (demotion/truncation wiring), inserted before the renumbered **1.5** (was 1.3, concurrency test — now exercises the real detection paths from 1.3/1.4 instead of calling `commit_resolve()` directly). Task 3.2 now also depends on 1.3/1.4, since its own "failure/timeout outcome replies" observable is meaningless without them. -- **1.3**: added a dedicated per-commit-engine sweep thread (not the shared `delaymgr` — would violate boundary + delaymgr's 200ms poll granularity vs. `DEFAULT_SYNC_TIMEOUT_MS`=500ms would add timing jitter). Race rule: majority-satisfied always wins over an already-expired deadline when both are true under the same lock hold. **Round 1 REJECTED**: `commit_sweep_once()` computed the next-deadline snapshot under `ce->lock`, unlocked to dispatch, then slept on the *stale* snapshot — a `commit_submit_async()` registration/broadcast landing during that unlocked dispatch window was lost (no thread parked on the condvar to receive it), so the sweep thread could sleep on an absent/late deadline indefinitely (worst case: unbounded delay in a low-traffic quorum-loss scenario, violating the "identical failure timing" no-regression bar). **Fixed** by holding `ce->lock` continuously across scan→wait decisions (releasing only for the unlocked dispatch step) and always re-scanning under the lock immediately before ever calling `cond_wait`/`cond_timedwait` — the standard recheck-then-wait idiom. Round 2 reviewer independently reproduced the round-1 bug shape to confirm the fix actually closes it, not just narrows it. **Precedent for 1.4**: any future code that needs to interrupt the sweep thread's wait (e.g., a demotion/truncation handler resolving a pending LSN out-of-band) must go through the same lock+broadcast discipline `commit_submit_async()` uses, or it risks the identical lost-wakeup class of bug. -- **1.4**: added `commit_resolve_pending_above(ce, lsn_floor, rc)` (snapshot-then-`commit_resolve()`-per-entry, no parallel bookkeeping). Truncation: a single new `on_truncate`/`truncate_cb_ctx` callback slot on `persist_t` (same inversion-of-control shape as the existing `on_durable_advance` hook) fires from `persist_truncate_wal_above` after releasing `p->lock`, only when a truncation actually removed entries — this ONE wrapper covers all three real truncation call sites (`raft_cb_become_leader`'s post-election truncate, the library's `on_truncate_from` conflict-reconcile, `ha_demotion_convergence_fn`'s post-demotion truncate), verified by the reviewer reading all three. Demotion: `raft_cb_become_follower` calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` (lsn_floor=0 → fails everything — a demoted node can never authoritatively confirm anything pending; verified against the CP posture). **Known residual, non-blocking**: a fourth internal truncation path (`persist.c` inside `persist_wal_write_replicated`'s tip-conflict-repair branch, STANDBY replicated-apply only) calls `truncate_wal_above_locked()` directly, bypassing the new wrapper — safe today only because demotion resolves ALL pending continuations before role flips to STANDBY (so nothing can still be registered by the time that path runs), but this is an *implicit* ordering invariant, not an asserted one. Flag for whoever next touches the STANDBY replicated-apply path: add a defensive comment/assertion there, or route it through the wrapper too, before relying on it further. Also: bug-1407 was believed closed at this point (70/70 across 3 runs) — **see 1.5's note below, this was a false reading, not a real fix**. -- **7.4**: closes out task-group 7 and the spec. Measured sustained `jfinish` fact-stream throughput at the pre-feature baseline commit vs. the feature-complete commit on a single-node HA cluster, distinguishing two signals since facts are fire-and-forget: commit throughput (durable-append + majority-registration only) and apply/observable throughput (commit + the ordered apply pump's mutation, the new hop this spec introduces). Commit throughput: unregressed (~3111/s baseline vs ~3033/s feature, ~2.5% delta, within noise) — Req 11.4's "no per-fact synchronous commit barrier" claim independently confirmed both by this measurement and by code inspection (`commit_submit(COMMIT_MODE_RETAIN, ...)` in `commit.c` returns immediately, byte-identical mechanism to pre-feature code). Apply/observable throughput: **initially found regressed 56%** (2831/s baseline vs 1237/s feature) with clearly superlinear scaling (2x fact volume → 5.2x elapsed time) — a genuine, in-scope defect this task's own benchmark surfaced, not a false alarm, correctly reported BLOCKED rather than silently accepted or patched outside boundary. Root-caused (fresh debug investigation) to `apply_pump_read_entry` (`persist.c`, introduced this spec's task 6.1): the function has zero memory across calls — every single LSN lookup re-runs a full `opendir`/`readdir`/segment-sort and rescans its segment linearly from byte offset 0 — and its only caller, `persist_apply_pump_run`, calls it once per LSN in strict ascending order with no batching, so applying N contiguous entries does O(N²) total work (entry k's read rescans ~k prior headers). Fixed with a minimal segment-read cache on `persist_t` (`apply_pump_cache_fd`/`_seg_lsn`/`_off`/`_next_lsn`, guarded entirely by the existing `apply_pump_lock`): a fast path `pread`s the next expected LSN directly at a remembered offset on an already-open fd (O(1) hit, same CRC verification as the slow path), falling back unchanged to the original full-rediscovery logic on ANY miss (gap, first call, post-truncation, post-recovery) — correctness never depends on the cache, only performance does. Invalidated at all four lifecycle points a stale cursor could otherwise leak or misread from: `persist_init` (starts invalid), `persist_fini` (closes if open), `persist_recover` (resets before any pump kick can trust a pre-recovery cursor), and `persist_truncate_wal_above` (closes inside the existing real-truncation gate, since a truncation can `ftruncate`/`unlink` segment bytes a stale cached fd/offset might otherwise still successfully, and wrongly, read). **Round-1 review REJECTED** on two findings, both remediated in round 2: (1) `persist_recover`'s cache invalidation initially mutated the new fields without holding `apply_pump_lock`, contradicting the fields' own documented invariant (fixed: wrapped in lock/unlock matching `persist_truncate_wal_above`'s existing pattern; independently re-verified deadlock-free by tracing every call site of both `persist_recover` and `apply_pump_lock` itself — the lock is touched in exactly 3 places, all private to `persist.c`, no re-entrant path exists); (2) the initial fix's "no worse than before" characterization of the cache/trim interaction understated a real, bounded risk — the OLD code failed loudly and immediately (`-ENOENT`) if a needed segment was already trimmed, while the NEW cache can (POSIX unlink-keeps-fd-alive) silently advance `last_applied_lsn` past already-deleted data if the pump is lagging when `trim_wal_segments` fires, a reachable scenario since apply-lag-behind-commit is this feature's entire premise — bounded by the pre-existing `APPLY_DIVERGENT` boot fence (tasks 5.1/5.2) catching it at next restart (deferred loud fencing, not silent corruption, but a real behavior change, not literally risk-free) — corrected via an accurate doc comment, no redesign of `trim_wal_segments` attempted or needed. **Verified end-to-end** (implementer + two independent review rounds): post-fix apply-throughput scales sublinearly (1.46x elapsed for 2x volume, vs. the pre-fix 7.4x reproduced independently on a stashed pre-fix binary in the same environment) and exceeds both the regressed 1237/s and the original 2831/s pre-feature baseline; `test_apply_pump` 5/5, `test_apply_pump_integration` 4/4, `test_apply_correctness_verification` 11/11, `test_persist_recovery` 7/7, `test_persist_truncate` 8/8, plus a broader sweep (`test_demotion_convergence`, `test_ha_core_safety`, `test_commit_resolve`, `test_repl_apply`, `test_commit_barrier_reply`) all pass with zero new failures, independently re-run by both reviewers, not just trusted from the implementer's report. **Two items explicitly flagged as out-of-scope, not fixed here**: (1) an intermittent (~2 of 9 attempts) client hang in the synchronous `jobsub` (`COMMIT_MODE_WAIT`) path shortly after a fresh single-node election, reproduced identically on both pre-fix and post-fix binaries (confirmed unrelated to this fix), resembling but not confirmed identical to the closed bug-1416 boot-race class — worked around via retry for benchmarking only, needs separate root-cause investigation; (2) `trim_wal_segments`'s retention floor has no awareness of the local apply pump's own lag (a distinct, pre-existing architectural gap, same family as an already-tracked standby-cursor variant) — out of scope for this targeted perf fix, flagged for a future spec/task rather than redesigned here. -- **7.3**: closes out task-group 7's regression gate. Full re-run confirmed exactly 2 of 33 tests in `test_commit_barrier_reply.c` failing — `test_jobsubmit_returns_before_commit_and_registers_pending_apply` and `test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements` — both asserting task 3.2's own explicitly-documented interim contract ("no independently observable behavior until task 6 completes it") that tasks 6.1/6.2 (already committed, ordered apply pump) closed by construction. Updated both to assert the new deterministic settled state (poll for reply, then assert success/job-exists), mirroring the idiom already used by the file's other, already-updated passing tests — no production code touched. One judgment call flagged and independently confirmed by reviewer: in the mixed-outcome array test, a committed sibling now applies (job visible) independently of a differently-failing sibling, while the batch's single client-visible reply still reflects the failure — this is the documented, intended interim behavior (collapsing array submission into one atomic WAL record is explicitly out-of-scope, owned by the separate `job-array-single-wal-record` spec), not a regression or scope violation. Verified (implementer + independent reviewer, both fresh runs): `test_commit_barrier_reply` 33/33 pass (reviewer ran 3x to rule out flakiness); full 79-group/926-test suite 0 failures including both bare-binary runs (reviewer ran the full unfiltered `controller_unit_test` binary directly and it passed clean, even through the two tests flagged elsewhere in this file's notes as environment-flaky, bug-1406/1407 — no regression signal, strengthens confidence); `/xce-test-ha --sequential 5` HA soak PASS twice independently (implementer's own run + reviewer's fresh re-run in a separate scratch dir), zero job loss, zero divergence, all 3 nodes led both times. -- **7.2**: added `--factstream` mode to `run-ha-soak.sh` (kill-mid-fact-stream against a live 3-node cluster, real `jstart`/`jfinish`/`jfail` wire traffic via `xce_test`'s `worker` REPL context, extended in this task since libxce already had the calls but the REPL didn't expose them) and a phantom-fact checker that reads `waldump`'s decoded `state_raw` bits (extended in this task to decode every worker-fact WAL type, not just `WAL_JOB_SUBMIT`) rather than inferring "did START apply" from the pretty-printed job state string — `job_state_str()`'s RUN-checked-before-EXIT priority ordering hides co-set EXIT/PERR bits on a fail-without-finish job, a genuine phantom-check false-positive caught and fixed here (pre-existing, minor, display-only quirk in `job_state_str` itself is out of boundary, not touched). **Round 2 found a real production bug this task's own scenario was needed to surface**: bug-1416, a boot-time TOCTOU in `persist_verify_apply_correctness()` — a returning node's `persist_recover()` replay+verify pass can race the live apply pump and the standby live-replication apply path (both write the same `last_applied_lsn`/job-table state `persist_verify_apply_correctness` re-reads live), producing a false `APPLY_DIVERGENT` that permanently, incorrectly fences a healthy node (task 5.3's sticky fence never self-clears on a false positive) — reproduced 10/10 across ASAN/release builds, 8-20 job counts, only on the RESTART/catch-up path (a clean 3-node cold boot with no kill never hit it). Fixed with a new `_Atomic bool persist_t.recovery_complete` gate (`persist.h`/`persist.c`): `persist_apply_pump_kick()` no-ops while closed; `ha_apply_single_entry`/`ha_apply_term_entry_via_library` (`ha.c`) skip only the in-memory `apply_wal_entry()` call while closed — the durability write, replication ACK, and `applied_lsn` advance are never gated (mirrors the bug-1043 durability/materialization-decoupling precedent, confirmed in `.wolf/cerebrum.md` since bug-1043 predates the current buglog retention window). `persist_recover()` closes the gate at entry, reopens it plus issues one deliberate catch-up kick right before returning, unconditionally of verdict — a genuine divergence still fences exactly as before, this flag only prevents the *comparison itself* from racing. **A second, harder-to-find gap surfaced only via live verification, not code review**: a node stuck retrying its own losing/timing-out election as a candidate can receive and durably apply live-replicated entries through the *ungated* (`persist_init` default=`true`) path for tens of seconds before `cluster_init_phase2()` ever calls `persist_recover()` for the first time — reproducing the identical false-DIVERGENT symptom through an entry point `persist_recover()`'s own bracket can't see. Closed with `persist_mark_recovery_pending(&c.persist)` in `myapp_run()` (`myapp.c`), called immediately after `persist_init()`/`persist_start()` succeed and before `ha_start_election_async()`/`cluster_run()` can process any network traffic. `persist_init()`'s default stays `true` so none of this spec's ~20 direct-`persist_t`-construction unit fixtures are affected — `persist_mark_recovery_pending` is called only from `myapp.c`. **Round-2 review independently verified, not just re-reported**: ran `test_apply_correctness_verification` itself (11/11, including all 3 real-divergence fence tests, confirming detection wasn't accidentally suppressed), confirmed the ACK/durability paths are genuinely ungated by reading the `ha.c` call sites directly, confirmed the `myapp.c` fix is load-bearing (not noise) by reverting only that hunk and reproducing the original failure live (2 of 3 nodes fenced), confirmed bug-1043 precedent against `.wolf/cerebrum.md` (not in `.wolf/buglog.json` — predates current retention). Verified: live 3-node `--factstream` soak (2 rounds, kill+restart both) PASS, zero `APPLY_DIVERGENT` across 5 `persist_recover()` calls; broader `controller_unit_test` sweep (23 groups) zero new failures (only the 2 pre-existing task-6.1/6.2-documented `test_commit_barrier_reply.c` failures reserved for task 7.3 remain). **Do-not-repeat lesson**: when gating a live path behind a "recovery in progress" flag, the flag's own bracket around the recovery function itself is insufficient if OTHER code paths (election retry, live replication) can mutate the same state before that function is ever first called — the gate must close at the earliest point in the real process lifecycle traffic could possibly begin, not just around the specific function whose output it protects. -- **6.4**: closes out task-group 6. Approved first round, zero production changes — group ends 2-for-4 on real bugs found (6.1, 6.2), 2-for-4 clean confirmations (6.3, 6.4). Extended `test_apply_pump_integration.c` (task 6.3) with 2 tests covering the MIXED facts+commands scenario 6.3 explicitly deferred here. **Test 3**: genuinely interleaved COMMAND/FACT/COMMAND/FACT/COMMAND sequence through a real `repl_remove_standby` outage, polls for zero commit/apply/reply throughout, restores quorum, and asserts STRICT original-log-order catch-up via a real append-only sequence spy (not a set-membership check) — an adversarial probe (corrupting the expected slot-0 jobid) was independently reproduced by the reviewer to confirm the assertion genuinely discriminates reordering, not just typos. **Test 4 — the highest-value test in this task**: wires a REAL 4-vcpu `vtaskmgr_t` (mirroring task 6.2's bug-1415 regression harness) to drive 4 commands + 2 facts through ONE covering ACK, independently traced by the reviewer to confirm this genuinely creates 5 vtasks (4 continuation dispatches + 1 apply-pump walk) competing for 4 real worker threads — true N-way concurrent racing of task 6.2's self-healing fix, not just one isolated forced pair like 6.2's own test. The "exactly once" check is per-jobid (not an aggregate total), so a skewed double-fire-on-one/hang-on-another distribution would be caught even though the aggregate count would look fine. Adversarial probe (corrupting expected reply count) independently reproduced. **Two honestly-scoped, explicitly non-blocking caveats, not gaps requiring fixes**: (1) task 6.2's own reviewer flagged a narrow, nanosecond-scale TOCTOU residual (unlocked self-heal check vs. locked list-link in `persist_register_pending_apply()`) as a non-blocking fast-follow — Test 4 does NOT deliberately stress this specific window (no injected synchronization primitive forces it, unlike 6.2's own barrier-based test); 50 clean runs across two independent builds are consistent with "didn't hit it by chance," not proof it's closed — stated explicitly here so it isn't assumed covered by this task's "several concurrent races" framing. (2) Test 4 doesn't directly re-assert per-command reply-after-apply ordering the way Test 1 (task 6.3) does via `reply_already_enqueued` — this property holds BY CONSTRUCTION (both the walk's take-call and the self-heal's `lsn <= last_applied_lsn` check only ever fire after `last_applied_lsn` already reflects that entry, verified in the actual code) and is independently covered by task 6.2's own dedicated test, so this is non-blocking, not a real gap. **Task-group 6 final summary**: the ordered apply pump is now live, activated (6.1), correctly hands off command replies (6.2), proven correct for both isolated (6.3) and mixed/bursty (6.4) command+fact scenarios under both normal and quorum-loss conditions — with 2 real, independently-verified, production-severity bugs found and fixed along the way (6.1's job-state corruption from stale receipt-time computation; 6.2's client-hang race from unordered async dispatch), both closed with dedicated regression coverage, the first ever in this spec to wire a real multi-worker `vtaskmgr_t`. **Optional fast-follow noted, not blocking**: a dedicated stress test that deliberately widens the `persist_register_pending_apply()` check-to-lock window (e.g. via a test-only injection point) to reliably exercise 6.2's disclosed residual TOCTOU gap, since neither 6.2's nor 6.4's tests reliably reach it today. -- **6.3**: pure integration-test close-out, approved first round, zero production changes — breaks task-group 6's 2-for-2 real-bug streak (6.1, 6.2 each found genuine production-severity bugs; 6.3 confirmed the pump's existing behavior is correct rather than finding a new one). New `test_apply_pump_integration.c` (2 tests) exercises the FULL live pump (real commit engine, real apply dispatch, real hook subscribers — `pool=NULL` like every other fixture, does not overclaim bug-1415/6.2's dispatch-race coverage, which is that test's own job) for both a job submission and a fact. **Key design decision, independently verified true by the reviewer, not just asserted**: `apply_committed_entry()` fires ALL job-state facts via the generic `HOOK_DETAIL_JOB_UPDATE` only (task 2.2's documented deviation, confirmed UNCHANGED by 6.1's `_pad` fix — that fix changed which state-computation core runs, not which hook detail fires) — so `sched_job_start_fun`/`_exit_fun`/`_fail_fun`/apiserver.c's equivalents are structurally unreachable for any live-applied fact, meaning ONE reaction spy wired to `HOOK_DETAIL_JOB_NEW`+`HOOK_DETAIL_JOB_UPDATE` legitimately covers all three named reactions (worker slot release, scheduler dispatch trigger, client/apiserver notify) since they're all downstream of the identical single firing. **Strong independent confirmation this is load-bearing, not incidental**: the type-specific handlers are unconditionally `va_arg`-typed for `job_pidinfo_t*`/`xce_jobfinish_t*`/`job_failinfo_t*` — types `apply_committed_entry()` never passes (always `job_t*`) — so if a type-specific hook ever DID fire from the live pump, it would be an immediate type-confusion crash, not just a coverage gap. Repo-wide re-audit (not just job.c/persist.c) confirms exactly 2 live call sites ever fire these hook details: `apply_committed_entry()` (guarded) and `job_ctrl`'s `mark_job_update` (legitimate SIGCONT/SIGSTOP) — no orphaned third site. Both tests prove REAL ordering, not coincidental timing: reply-after-visibility is sampled synchronously inside the reaction-spy callback (not two separately-timed checks); Req 4.3's "no separate watermark re-check needed" is proven by construction (the walk only ever invokes `apply_committed_entry()` for LSNs already `<= committed_lsn`, so `committed_lsn_at_fire >= entry_lsn` holds structurally, not by scheduling luck) rather than merely asserted in prose. Delayed-commit and genuine quorum-loss windows are both polled repeatedly across real time (5×20ms), not checked once. RED phase independently reproduced by both implementer and reviewer via re-injecting the pre-4.1 eager-mutation bug class. **Task-group 6 running tally**: 3 tasks in, 2 real production-severity bugs found and fixed (6.1 data corruption, 6.2 client-hang race), 1 clean confirmation (6.3) — the pump's core mechanics and hook-routing design are now thoroughly adversarially tested from multiple independent angles. Remaining: 6.4 (quorum-loss catch-up ordering, mixing facts AND commands — 6.3's fact-only quorum-loss scenario is explicitly non-duplicative defense-in-depth per the reviewer, not a preemption of 6.4's charter). -- **6.2**: closes the standing operational gap flagged since task 3.2 — successful job submissions finally get a client reply again. Wired `persist_apply_pump_run()` (task 6.1) to call `persist_take_pending_apply(p, lsn, &out_fn, &out_ctx)` immediately after each entry's `apply_committed_entry()` call, invoking the continuation when found — strictly after the mutation, per the task's named observable. Facts need no special-casing: `persist_take_pending_apply()` naturally returns 0 for them since the fact-receipt path (task 4.1) never calls `persist_register_pending_apply()`. Added `jobsubmit_reply_success_once()` (job.c, sibling of the pre-existing `jobsubmit_reply_failure_once()`) — `jobsubmit_apply_complete()` now calls it once every array element resolves, sending `RequestReply_data_jobid` to the client. Took 2 review rounds. **Round 1 REJECTED — a second real, production-severity bug found this task-group, on top of task 6.1's data-corruption bug**: self-disclosed as bug-1415, confirmed by the reviewer via direct code trace: `commit_resolve_pending()` (commit.c) enqueues the WAIT-mode continuation's dispatch and `persist_apply_pump_kick()`'s walk as TWO INDEPENDENT `VTASK_LIVE_BLOCK` vtasks with NO completion-order guarantee — `vdispatcher_entry()` binds each ready task to a different idle vcpu and does not wait for a prior dispatch to complete. Confirmed reachable on EVERY default production deployment (`cluster_init_phase2()` always wires a real `vtaskmgr_t`, default `n_vcpus=8`, never NULL) — and confirmed completely untested by every fixture in this entire spec, all of which use `pool=NULL` (which trivially serializes the two dispatches in enqueue order by falling back to synchronous inline execution, and so can never exercise the race). **Consequence**: if the apply-pump task wins, it applies the entry and advances `last_applied_lsn` past the LSN before the continuation task ever calls `persist_register_pending_apply()` — when that registration finally arrives for an already-passed LSN, nothing will ever come back to consume it. **The job is correctly constructed and visible, but the client hangs forever, plus a leaked `jobsubmit_async_ctx_t`/endpoint ref** — a coin-flip-or-worse failure mode on the feature's own primary success path, for every default multi-vcpu production controller. **Round 2 fixed it** with a self-healing check at the top of `persist_register_pending_apply()`: `if (lsn <= persist_last_applied_lsn(p)) { on_applied(ctx, lsn); return; }` — using the existing `persist_last_applied_lsn()` accessor, no signature change. Verified race-safe in both interleavings: non-racing case is a no-op (confirmed unchanged behavior for every existing test, all of which use `pool=NULL` and so never trigger the check); racing case fires the callback exactly once, immediately, on the registering thread — and can never double-fire because a self-healed registration is never linked onto `pending_applies`, so the walk's own `persist_take_pending_apply()` call for that LSN (which already ran and found nothing, which is precisely how we know the registration is late) has nothing left to find. **First and only test in this entire spec to wire a REAL multi-vcpu `vtaskmgr_t`** (`n_vcpus=4`, via `commit_set_pool()`) — every other fixture in the spec deliberately uses `pool=NULL` and structurally cannot reproduce this class of bug; the new test's poll-barrier ties directly to the real race predicate (`persist_last_applied_lsn(p) < lsn`), not a timing guess, and self-checks its own precondition so it fails loudly rather than silently degrading if the forced interleaving doesn't occur. RED→GREEN independently reproduced by both rounds' reviewers. **One narrow, non-blocking residual finding for a fast-follow, not required to close bug-1415**: the self-heal check (an unlocked atomic read) and the subsequent list-link (under `pending_applies_lock`) are not one atomic unit — a sufficiently long preemption between the check and the lock could theoretically still reproduce a vastly narrower version of the original race. Recommended fix: perform the self-heal check while already holding `pending_applies_lock` (the same lock `persist_take_pending_apply()` uses), making check-and-link atomic with respect to the walk's take. Not blocking since the reported, reproduced, production-relevant (dispatch-level) race is genuinely and completely closed. **Two pre-existing tests in test_commit_barrier_reply.c still fail as EXPECTED** (unchanged from task 6.1 — same root cause, task 7.3's job). **Task-group-6 running tally: 2 of 2 tasks landed so far have each required a full REJECTED→fixed round for a genuine, independently-verified production-severity bug** (6.1: job-state corruption from stale receipt-time fact computation; 6.2: a client-hang race from unordered async dispatch) — this task group is where deferred-apply's real concurrency implications are surfacing, exactly as expected for the task that finally activates the live path. Remaining: 6.3/6.4 (integration tests through the full pump) should be reviewed with the same level of scrutiny for further concurrency surprises. -- **6.1**: the most consequential task in the entire spec — this is what finally makes `apply_committed_entry()` live (previously only restart replay, task 2.3, ever called it). Built `persist_apply_pump_run()` (walks `[last_applied_lsn+1..committed_lsn]` strictly ascending, calls `apply_committed_entry()` for every entry regardless of commit mode) and `persist_apply_pump_kick()` (off-thread trigger via `vtask_enq_classified(VTASK_LIVE_BLOCK)`, inline fallback for pool-less fixtures, CAS-coalesced so at most one caller runs the walk at a time — the loser hands off asynchronously, by design, to whichever thread wins). **Design deviation, verified correct and a strict superset**: wired the trigger into `commit_resolve_pending()` (commit.c) rather than only ha.c's `on_commit_advanced` — task 1.2 already discovered `commit_resolve_pending` is the single choke point all 6 internal `committed_lsn`-advancing paths funnel through (verified: exactly 6 call sites, commit.c:353/558/1042/1060/1099/1116, all firing after `ce->lock` releases), while `on_commit_advanced` covers only 1 of them; wiring only there would silently starve the pump for single-node/self-quorum clusters. design.md should be updated to reflect this as the actual integration point. **Fixed Hazard 1 (double-append, flagged since task 4.1)**: a naive `detail==HOOK_DETAIL_JOB_UPDATE` filter would have broken `job_ctrl`'s legitimate live SIGCONT/SIGSTOP append (same hook detail, still-legitimate command path) — fixed instead with a `_Thread_local bool t_apply_seam_notifying` call-provenance guard set only around `apply_committed_entry()`'s own hook-firing calls; `persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun` early-return when set. Also caught and fixed an UNDOCUMENTED instance of the identical hazard class in the queue/worker hook functions (neither had ANY mode-based early-return before this, unlike job's pre-existing dead `WAL_MODE_WAIT` branch) — would have double-appended+re-triggered a nested majority-commit wait for every live-applied `WAL_QUEUE_CTRL`/`WAL_WORKER_CTRL` entry. Took 2 review rounds. **Round 1 REJECTED — the most severe finding of the entire session, a real data-corruption bug, not a doc/log issue**: Hazard 2 (merge-staleness, also flagged since 4.1) was investigated and ESCALATED as requiring wire-format changes beyond scope, while still landing the pump that activates it. Round-1 review proved with field-level precision this was wrong to ship: job state DONE, correctly baked by a FINISH fact — followed shortly after (an entirely ordinary HPC lifecycle event, not adversarial) by a POST_EXEC-stage FAIL fact whose receipt-time snapshot is STILL stale (RUN, since FINISH hadn't applied yet, apply now being deferred) — bakes `RUN|EXIT|PERR` into its WAL entry. Both apply in correct LSN order (FINISH then FAIL), but FAIL's stale absolute overwrite wins over the now-correct DONE, because the terminal/non-terminal guard doesn't catch it (both values happen to carry a terminal-ish bit). **Final result: a job's correct DONE completion silently replaced by RUN|EXIT|PERR — a Req 11.3 regression, and a real, operator-and-billing-visible corruption newly introduced by deferring apply (this feature's whole purpose), reachable under completely ordinary traffic.** The reviewer also found a scoped fix contradicting the "needs wire-format enrichment" escalation: `wal_jobstate_t`'s existing unused `_pad` field had room. **Round 2 fixed it properly**: `_pad` now carries a discriminator (`WAL_JOB_FAIL_STAGE_FLAG`/`WAL_JOB_UPDATE_POSTDONE_MARK`) plus the minimal data needed (FAIL's `stage`) to RECOMPUTE against the job's ACTUAL CURRENT state AT APPLY TIME via the real `apply_job_fail_transition()`/new `apply_job_postdone_transition()` guard cores — no more baking a stale absolute value at receipt time. `WAL_JOB_START`/`WAL_JOB_FINISH` are unchanged (confirmed safe on their own — deterministic, state-independent payloads that were never the problem, only the thing getting clobbered). Backward-compatible: `_pad==0` (all pre-existing/legacy data) deterministically routes to the same absolute-overwrite behavior that was universal before this fix — no crash, no new corruption, correctly out of scope for further hardening since this feature has never been deployed. **Bonus finding during round-2 review — a real test-flakiness hazard, root-caused and fixed on the spot, not just reported**: a rare (initially estimated ~2%, actually verified ~7.5%) flake in the new regression test was root-caused to a genuine SECOND, independent kicker of the apply pump (`commit_on_durable_advance`, fired from the real `fsync_worker` background thread on every `durable_lsn` advance, entirely separate from `repl_on_ack`) that the test's own header comment didn't account for — this is by-design eventually-consistent behavior (task 6.2's continuation-firing will correctly rely on it), not a production bug, but the TEST needed a proper poll-for-applied-state helper instead of asserting immediately. Fixed (`tap_wait_applied()` polling helper added to 2 tests) and a SECOND, related flake found during the investigation in a third test, also fixed — 120/120 clean runs afterward. Also fixed a stale orphaned forward-declaration left over from the round-2 rename. **Also fixed a genuine, unrelated bug found via regression sweep, logged as bug-1409**: an early version of `apply_pump_drain`'s retry loop didn't check `persist_apply_pump_run`'s return code, causing an unbounded busy-loop (18.5M+ log lines) in `test_demotion_convergence` whenever `committed_lsn` advances without a corresponding real on-disk WAL entry (a legitimate scenario for commit-engine-focused unit fixtures) — fixed by returning immediately on a negative rc. **Hand-off built for task 6.2**: `int persist_take_pending_apply(persist_t* p, uint64_t lsn, pending_apply_notify_fn* out_fn, void** out_ctx)` — call AFTER `apply_committed_entry()` applies `lsn`; returns 1+populates once (atomically unlinking) when a registration exists (the command path, task 3.2), 0 otherwise (the common case, facts never register); a second call for the same LSN is guaranteed to return 0; safe to call unconditionally for every applied LSN. Task 6.2 should call this right after each apply inside/after the walk and invoke the returned continuation — deliberately NOT wired into the walk itself, per boundary. **Two pre-existing tests in test_commit_barrier_reply.c now fail as EXPECTED (not a regression)**: their own assertion text literally says "no job may be constructed until task 6's apply pump exists" — now correctly invalidated by this task landing; explicitly task 7.3's job to update, per task 3.2's own Implementation Notes anticipating exactly this. **Minor, non-blocking coverage gap for a fast-follow**: no test explicitly replays a `WAL_JOB_FAIL`/`WAL_JOB_UPDATE` entry with a real `_pad` marker through the REPLAY path specifically (only through live apply) — low risk given `apply_committed_entry()` is the literal shared function both paths call, but untested for that specific combination. -- **5.3**: closes task-group 5. Took 3 review rounds — the heaviest task in the entire series, and the only one that surfaced a genuine CRITICAL production safety bug (not a doc/log issue like 5.1/5.2's rounds). **Round 1 (test-only) REJECTED**: while writing tests for "the fence stays alive" and "the fence clears on a subsequent consistent recovery," review discovered the sticky `recovery_unrecoverable` flag is trigger-agnostic — it has ZERO provenance about WHY it was set. Task 5.2 correctly reused this flag for the new `apply_divergent` trigger, but TWO PRE-EXISTING, live, running-process call sites in ha.c already clear this SAME flag based purely on STRUCTURAL conditions (never checking the apply verdict): `promotion_fence_timeout_fn` (a self-re-arming periodic timer) and the promotion-eligibility gate in `raft_cb_become_leader`. **Concrete, reachable attack path**: a node's local apply dispatch has a defect (materialized state diverges from the committed log) while its WAL bytes are otherwise fine (structurally healthy — orthogonal to the apply defect by construction). The apply-divergent fence sets correctly at boot. But WHILE THE PROCESS KEEPS RUNNING, the periodic timer sees the structural conditions were always fine and silently calls `persist_clear_recovery_unrecoverable()` — releasing a fence that was never actually resolved. If this node then wins a later election, the promotion gate now sees `unrecoverable==false` and promotes it to primary — **a genuinely divergent node serving authoritative state**, a direct Req 8.4 / CLAUDE.md "never serve divergent state" violation. **Round 2 fixed the code**: added `apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT` as an additional guard ANDed into BOTH clear-call-site conditions — `promotion_fence_timeout_fn` (line ~876) and `raft_cb_become_leader`'s gate (line ~1130) — so an apply-divergence-triggered fence can now ONLY be cleared by a fresh process restart's clean `persist_recover()` (since `apply_verdict` has exactly two production write sites: `persist_init()`'s reset-to-`APPLY_CLEAN` and `persist_recover()`'s boot-time computation — no live path resets it). Verified via full trace + RED→GREEN for site 1. **Round 2 REJECTED anyway**: the fix was logically sound, but review PROVED site 2's guard had ZERO test coverage — by deleting only that guard and re-running the ENTIRE 298-test regression sweep across 13 groups, every single test still passed. An unverified defense-in-depth guard on promotion/election-safety code doesn't meet the bar. **Round 3 (test-only) closed it**: added a test that structurally isolates site 2 from site 1 — proven not to be exercising the already-covered sticky-flag path, since `ha_promotion_gate_decision()` never populates `apply_divergent` in its own inputs, so ONLY the round-2-added standalone guard at the call site can catch this scenario. RED→GREEN independently reproduced by two separate reviewers reverting the guard and confirming the exact predicted test failure. **Final closed picture, verified end-to-end across all 3 rounds**: (a) apply-divergence sets the fence correctly (5.2); (b) survives the structural timer's periodic re-check (round 2); (c) survives the promotion gate's re-check even in the hypothetical where the sticky flag itself somehow reads false (round 3); (d) only clears via a fresh process restart replaying cleanly — confirmed by round 1's own two-separate-`persist_t`-instances test. **Forward note for task 6, already flagged in-repo (test_apply_correctness_verification.c comments), not a defect today**: this entire safety picture rests on `apply_verdict` only ever being recomputed once, at boot. If task 6's live apply pump ever adds a live re-verification of apply-correctness (not currently planned, but worth checking), the "restart-only" assumption these 3 rounds of guards depend on would need re-examination at that time. **Lesson for the whole series**: reusing an existing mechanism correctly at the logic/decision level (5.2's `apply_divergent` short-circuit) does not automatically make every OTHER consumer of that same mechanism's output (ha.c's two independent, pre-existing live-clear call sites, written before this feature existed and with no reason to have considered a future trigger) still behave safely — a shared, trigger-agnostic sticky flag is exactly the kind of implicit coupling that needs an explicit audit of ALL its readers/writers whenever a new trigger is added, not just the one you're actively wiring. -- **5.2**: routed `APPLY_DIVERGENT` (task 5.1) into the EXISTING loud-fence mechanism as a new INPUT, not a new fencing path — added `bool apply_divergent` to the existing `recover_policy_inputs_t`; `persist_recover_decision()` now unconditionally short-circuits to the pre-existing `RECOVER_DECISION_FENCE_UNRECOVERABLE` enum value when it's true (correct and design-mandated: a divergent local materialization can't be trusted for a peer/catch-up comparison either, so it must always fence regardless of peer state); reuses the same sticky `recovery_unrecoverable` flag and the same `persist_recover_loud_fence_if_unrecoverable()` function — confirmed by both review rounds to be the identical pre-existing symbols, zero new fence mechanism. Confirmed the existing promotion-gate consultation in ha.c (~line 1078, `persist_recovery_unrecoverable(&c->persist)`) already independently reads this flag — zero ha.c changes needed. Zero new self-termination paths (verified by grep both rounds) — stays alive, keeps retrying, per the CLAUDE.md process-lifecycle invariant. Took 2 review rounds. **Round 1 REJECTED — real, test-reproduced defect, not documentation**: the operator-facing ERROR() log's fixed boilerplate clause ("no healthy peer reachable to catch up; durable_floor_lsn=X < cluster committed high-water hwm=Y") fired UNCONDITIONALLY for every fence decision, including the new apply_divergent trigger — but for that trigger the fields are either hardcoded placeholders or explicitly-set values that made the clause LITERALLY FALSE in test output ("100 < 100", "2 < 0" — numerically impossible claims presented as fact to an operator during a real incident). Before this task, `RECOVER_DECISION_FENCE_UNRECOVERABLE` was only reachable via the structural "no healthy peer" branch, so the clause was truthful by construction; the new unconditional short-circuit broke that invariant. This directly undermines Req 8.3's "loud, operator-visible" requirement — a log an operator can't trust during an incident is worse than no log. **Round 2 fixed** by branching the ERROR() call itself (not just the inner reason string) on `apply_divergent` — that branch now states only the real, verified reason with no peer/HWM claim; the structural-fence branch is byte-identical to before. New regression test uses a genuine (non-mocked) stdout-capture mechanism relying on `controller_unit_test` never calling `tlog_init()` (verified true across all ~85 files in the test binary — confirmed the vendored tinylog fallback path used is a real raw `write(STDOUT_FILENO,...)`, not a stub) — both review rounds independently reverted the fix and reproduced the exact false-clause failure before re-confirming GREEN. **Lesson, consistent with task 2.2's earlier pattern**: reusing an existing decision path correctly at the LOGIC level doesn't automatically make every SIDE EFFECT of that path (like a log message assuming context from the old, narrower set of triggers) still valid for a newly-added trigger — check accompanying log/error text for assumptions that were true only because of the old, narrower reachability, not just the returned enum/decision value itself. -- **5.1**: added a new `apply_verdict_t` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) computed alongside the EXISTING, untouched structural `recover_verdict_t` (CRC) verdict — a dual-check design: (1) an entity-count check comparing `WAL_JOB_SUBMIT` entries scanned during replay against the live `c->jobmgr` table's actual growth (`count_jobmgr_entries()`), genuinely independent since it queries live state rather than trusting `apply_committed_entry()`'s return code — verified real by both review rounds via a forced job-count-mismatch test; (2) a position check comparing an "expected" LSN against `p->last_applied_lsn` (new `_Atomic uint64_t` field, per design.md, forward-looking infrastructure for task 6's live pump). Took 2 review rounds. **Round 1 REJECTED — real, live-today bug, not documentation**: the position check's "expected" value was computed as `recovered_tip = current_lsn - 1` (the PHYSICAL WAL-scan tip, which `replay_wal()` advances for every entry including `WAL_CHECKPOINT` markers and unknown types) instead of the LOGICAL apply cursor. `persist_checkpoint()` durably fsyncs the checkpoint marker in one step, then persists the advanced `checkpoint_lsn` to the on-disk snapshot in a LATER step — a crash between those two steps (a real, unavoidable crash-safety window this codebase already engineers hard for, cf. bug-661/ha-commit-durability) leaves a trailing checkpoint marker as the highest-LSN entry on next boot, causing `recovered_tip` to exceed the true `last_applied_lsn` and falsely report `APPLY_DIVERGENT` on a perfectly clean recovery — a direct Req 8.2 violation, reachable TODAY on any node that crashes at exactly the wrong moment during a checkpoint. **Round 2 fixed** by reading `p->last_applied_lsn` directly (the value `replay_wal()` already correctly advances only for entries actually dispatched through `apply_committed_entry()`, skipping checkpoint-marker/unknown-type entries) instead of `recovered_tip`; `recovered_tip` itself is untouched and still correctly feeds `durable_lsn`, which does need the physical tip. New regression test genuinely reproduces the trigger condition (real `persist_checkpoint()`, then `unlink()`s the exact snapshot file it wrote, landing on-disk state in the identical place a real crash would) — both review rounds independently reverted the fix and reproduced the false-DIVERGENT failure before re-confirming GREEN. **Known, disclosed, accepted scope gap (not a defect, flagged for whoever extends this check)**: the entity-count check covers jobs only. Queues are correctly excluded (config-derived, `WAL_QUEUE_CTRL` never creates a new one, verified by both review rounds). **Workers are NOT covered but CAN diverge**: `worker_from_capnp()`/`restore_worker()` genuinely creates a new worker on a `WAL_WORKER_CTRL`/`HOOK_DETAIL_WORKER_JOIN` entry when absent from `workermgr` — a worker-creation divergence is currently invisible to this verification. Extend `persist_verify_apply_correctness()` with a worker-count check (mirroring `count_jobmgr_entries()`) if/when this needs to be closed. **Self-disclosed, accepted limitation**: because `replay_wal()`'s single-threaded caller in `persist_recover()` derives both the "expected" and "actual" position values from the same atomic within moments of each other on the same thread, the position check is diagnostically tautological FOR THIS SPECIFIC CALL SITE today (though the entity-count half remains genuinely independent) — this will change once task 6's live apply pump gives expectation and actual state a real chance to diverge across threads/time; task 5.1's boundary does not require solving that now. -- **4.2**: pure test-only close-out of task-group 4 — one new test `test_fact_state_unchanged_throughout_quorum_loss_outage` in `test_commit_barrier_reply.c` (30→31), approved first round, zero production changes. Drives a genuine `repl_remove_standby`-based quorum-loss window through all 4 real fact-receipt handlers, polls live job state + `committed_lsn` at 5 time-spread points (real 20ms sleeps, ~100ms total) rather than once, asserts a real hook-spy never sees `AFTER_EVENT` fire, and confirms durable WAL presence via direct on-disk segment read (`cbrp_count_jobstate_entries`) — combining and strengthening two pre-existing tests that each covered only half (4.1's own single-point-in-time check under normal conditions; an older pre-4.1 test that proved WAL durability but never inspected live state). **Static/inspection half of Req 6.2 confirmed independently by the reviewer, re-reading 4.1's landed diff**: none of the four `record_*_fact` functions call `apply_committed_entry`/`restore_job`/mutate the real job object — `record_job_fail_fact` specifically operates on a throwaway `job_t scratch = *job` shallow copy. **Architecturally significant finding for task 6 planning**: facts have NO commit-timeout/failure concept at the receipt layer, unlike commands (task 3.2's `commit_submit_async`). `persist_submit_job_fact()` → `wal_hook_append_classified(WAL_MODE_RETAIN)` → `commit_submit(..., COMMIT_MODE_RETAIN, 0)`, and `COMMIT_MODE_RETAIN`'s branch in commit.c is an immediate, unconditional return with NO continuation registered (explicit comment: "RETAIN never honors the timeout") — `commit_resolve()` never fires for a retained fact. A fact's outage is therefore open-ended by construction; its eventual commit is discovered only when `commit_advance_locked()`/`commit_on_ack` notices `committed_lsn` can advance past it via the ACK-registry watermarks, purely. **Task 6's apply pump must rely entirely on `committed_lsn` advancement to discover newly-eligible retained facts — there is no per-fact signal to wait on.** Confirmed neither of 4.1's two flagged hazards (double-append, merge-staleness) is triggered by this test, since it never drives apply. **Task-group 4 (fact apply-after-commit split) is now complete**: 4.1 (receipt converted to durable-append-only, two critical hazards discovered and documented for task 6) → 4.2 (retention proven under sustained quorum loss, fact-vs-command architectural asymmetry documented). **Both task-groups 1-4 are now done; task 6 (ordered apply pump) is the critical next milestone that must resolve, before going live: the double-append hazard, the merge-staleness hazard, AND design around facts' lack of a per-entry commit signal (commit_advance/committed_lsn-driven discovery only).** Groups 5 (restart verification) and 6 (apply pump) remain; 5.1 depends only on 2.3 (already done) so could run in parallel with 6, per tasks.md's dependency graph — continuing in numeric order (5 next) unless reprioritized. -- **4.1**: converted the four worker-fact receipt handlers (`on_job_start`, `on_job_finish`, `on_job_fail`, `on_job_postdone` — NOT literally named `on_job_update`, task text's naming was approximate) to validate-and-durably-append-only: BEFORE_EVENT validation hook unchanged, AFTER_EVENT (post-mutation) hook no longer fires at receipt, live state mutation removed. **Key discovery**: the durable WAL append was NOT a separate call — it lived INSIDE the very `persist_job_hook_fun` AFTER_EVENT subscriber this task needed to silence, so a new `persist_submit_job_fact()` (persist.c) was added that calls the same unmodified `wal_hook_append_classified(WAL_MODE_RETAIN, wal_jobstate_t{jobid,state}, ...)` directly — verified wire-format-identical to before, `apply_committed_entry()`'s existing switch (task 2.2) routes it unchanged. `mark_job_start`/`mark_job_finish` (single caller each, verified via diff against 41d3620) renamed in place to `record_job_start_fact`/`record_job_finish_fact`; `mark_job_fail`/`mark_job_update` (shared with `job_ctrl`'s OUT-OF-BOUNDARY client-command path, 2 and 3 additional call sites) left untouched, with new durable-append-only siblings `record_job_fail_fact`/`record_job_update_fact` added instead — `job_ctrl`/`do_jobcontrol_op` confirmed byte-identical to pre-task by the reviewer. Approved first round. **Real, permanent regression accepted, not fixed (Req 5.1 required the removal)**: `on_job_postdone`'s `finish_time = time(NULL)` anomaly backstop is gone with no wire-format-level equivalent — concretely, a job hitting this rare no-prior-exit-status path now has `finish_time` stuck at 0 forever, which `job_should_retain()` (persist.c) evaluates as immediately GC-eligible instead of retained for the configured window. Narrow, rare-path, but real. **TWO CRITICAL HAZARDS DISCOVERED BY REVIEW, MUST BE RESOLVED BEFORE TASK 6 WIRES apply_committed_entry() INTO A LIVE PATH — currently inert only because replay runs before persist_start() registers hooks (cluster.c: persist_recover() at line ~1451 vs persist_start() at ~1694)**: - 1. **Latent double-append**: `persist_job_hook_fun` is STILL registered on `(XCTL_HOOK_JOB, HOOK_AFTER_EVENT)` with no detail filter for `HOOK_DETAIL_JOB_UPDATE`, and does NOT early-return for that detail (it only early-returns for the WAIT-mode submit path). The moment task 6 wires `apply_committed_entry()` into live commit-advance, EVERY fact `apply_committed_entry()` applies will fire `HOOK_DETAIL_JOB_UPDATE`, which `persist_job_hook_fun` will catch and append AGAIN — a duplicate, mis-typed WAL entry per live-applied fact. **Task 6.1/6.2 must retire or detail-filter `persist_job_hook_fun`'s fact-handling branch before going live**, since no live receipt path fires AFTER_EVENT for facts anymore (this task removed the only other source). - 2. **Merge-staleness on deferred apply**: `record_job_fail_fact`/`record_job_update_fact` precompute a FINAL target `state` at receipt time from a snapshot of current job state and bake it verbatim into the WAL entry (their own doc comments claim this "can never drift from what apply time will eventually compute" — that claim was TRUE only when receipt and apply were the same event; it becomes FALSE the moment they're separated by a commit round-trip, which is this entire feature's purpose). `apply_committed_entry()` routes all job-state facts through the generic `apply_job_update_transition`, which does a blind `job->job.state = state` overwrite — not the type-specific `|=` merge `apply_job_fail_transition` uses. Once two facts for the same job are in flight simultaneously (now newly possible, since apply is deferred), the second one to apply can silently clobber bits the first one set. Root cause is task 2.2's wire-format/generic-routing reality, not this task's diff — but no task in the current plan (4.2, 5.x, 6.x) tests or fixes this. **Task 6 planning must resolve this before wiring live apply** — likely requires either wire-format enrichment (baking the merge operation itself into the WAL entry, not just the precomputed result) or an apply-time re-derivation strategy. -- **3.3**: pure test-only close-out of task-group 3 — 4 new tests added to `test_commit_barrier_reply.c` (25→29), approved first round, zero production changes. Closes two gaps 3.2 flagged: (1) an explicit-FAILED test via `persist_truncate_wal_above`/`commit_on_truncate`/`commit_resolve_pending_above(-ENOENT)`, distinct from the existing timeout (`-ETIMEDOUT`) path; (2) the `array_size > 1` mixed-outcome gap — one element ACKed to real majority while two time out, proving the committed sibling's `persist_register_pending_apply` hand-off isn't blocked by failing siblings, zero phantom jobs for the failures, and exactly one batch-level failure reply (via `jobsubmit_reply_failure_once`'s ref-counted first-failure-wins gate). Also added two repeated-failure loop tests (5 independent cycles each, fresh jobid/LSN per iteration) proving the no-phantom-job guarantee isn't a one-shot fluke. **Confirmed correct, non-obvious harness detail**: this file's fixture wires job-submit continuations through a standalone `commit_engine_t g_hce`, NOT `cluster_t`'s embedded `c->commit` field that demotion (`raft_cb_become_follower`) resolves against — so truncation, not demotion, is the only deterministic way to force an explicit FAILED (non-timeout) outcome in this specific test file. Demotion-forces-FAILED already has its own dedicated coverage in `test_demotion_convergence.c` (tasks 1.4/1.5) via a different fixture — not a gap, a file-boundary distinction. **Non-blocking observation carried forward, not fixed here** (pre-existing since 3.2, correctly out of scope for a test-only task): the single batch-level failure reply carries no per-element granularity — a client can't tell "1 of 3 committed" from "all 3 failed" from the reply alone. **Task-group 3 (command apply-after-commit ordering) is now complete**: 3.1 (job construction deferred past commit, bug-1408 caught+fixed) → 3.2 (converted to real async continuation-based commit, majority-ack safety independently verified intact, several documented design deviations) → 3.3 (failure-path guarantee proven under repetition and array mixed-outcomes). **Standing operational blocker carried from 3.2, still true**: no live path applies a successfully-committed job submission until task 6 exists — this branch must not be soak-tested or deployed against live traffic yet. -- **3.2**: converted `handle_jobsubmit()` from 3.1's still-synchronous commit to a real async registration against task 1's continuation API. New `persist_submit_job_payloads_async()` (persist.c) appends the WAL entries via the same `ha_primary_log_append`/`persist_wal_async` primitive the pre-existing RETAIN-fact path (`wal_hook_append_classified`) already uses, then calls `commit_submit_async()` directly on `p->repl.ce` — bypassing `repl_wait_majority_ack`/`repl_submit_majority_commit` in repl.c, which is what design.md's text actually specifies as the integration site. **Independently verified safe** (reviewer traced the full ACK→`commit_resolve()` path in commit.c): majority-ack detection is driven entirely by `(ce, lsn)` events inside the commit engine, agnostic to which wrapper registered the continuation — no false-commit risk. But it's a real, permanent design deviation, not just an implementation detail: this creates a THIRD parallel "append+track a commit" implementation (alongside the old sync one, now alive only in 3 test files, and the RETAIN-fact one), narrowing the reusable primitive design.md wanted — a future command handler (e.g. a config-change command) gets no shared plumbing from this work and would need its own bespoke path. On FAILED/timeout, replies failure immediately via `jobsubmit_reply_failure_once()`, no job ever constructed. On COMMITTED, calls new `persist_register_pending_apply(p, lsn, on_applied, ctx)` — a hand-off linked list on `persist_t` for task 6.1 to consume — and does NOT apply or reply, per the task's explicit instruction. **Deleted (not preserved-dead) task 3.1's bug-1408 create_job-loop/notify block**: verified correct — `apply_committed_entry()`'s existing `WAL_JOB_SUBMIT` case (task 2.2) already fires `HOOK_DETAIL_JOB_NEW` per-entry independently, structurally immune to bug-1408's batch-suppression class; resurrecting the old loop in task 6 would double-construct/double-notify. Original code fully recoverable from `git show 6f1fa97:src/controller/job.c` if ever needed. **CRITICAL OPERATIONAL CALLOUT — this branch must NOT be soak-tested or deployed against live traffic until task 6 lands**: there is currently NO live code path that applies a successfully-committed job submission (only restart replay, task 2.3, does) — every successful job submission produces a durably-committed WAL entry but ZERO reply to the client, forever, until task 6.1/6.2 exist to consume the pending-apply hand-off and complete the reply. This is exactly what the task text sanctions ("no independently observable behavior until task 6"), not a bug, but it means this commit range is only safe in isolated unit-test form right now. **Hand-off mechanism for task 6.1's briefing**: `persist_register_pending_apply()`/`persist_has_pending_apply()` exist; there is NO "take and remove" accessor yet — task 6.1 must build the consumer side itself. The per-batch `jobsubmit_async_ctx_t` (ref-counted across array elements, frees only once every element has either failed-and-replied or been applied via `jobsubmit_apply_complete`) will not free for any batch containing a committed element until task 6 exists — an accepted, bounded-by-test-scope leak today, but would be unbounded growth under any live/soak run before task 6 lands. Outcome encoding uses a plain `int rc` (not `commit_outcome_t`+rc) to avoid a persist.h→commit.h circular dependency — verified unambiguous today (every real FAILED call site in commit.c passes non-zero rc) but not compiler-enforced; fragile by convention, not by construction. **New regression found by review, not yet fixed, non-blocking**: `array_size == 0` now hangs the client forever and leaks one `jobsubmit_async_ctx_t` per request (3.1's version replied synchronously even for this input; 3.2's async loop never runs so the ctx never releases) — confirmed NOT reachable via any current legitimate client (Go apiserver always sets `arraySize >= 1`; `xce_test`'s `cmd_jobsub` clamps `arsize <= 0` to 1), so flagged as a defense-in-depth hardening gap for a future task rather than fixed here. Also no `array_size > 1` mixed-outcome test exists yet (per-element independent resolution verified correct by code inspection, not by test) — fast-follow recommended. -- **3.1**: reordered `handle_jobsubmit()` (job.c) so job objects are only constructed via `create_job()` AFTER the (still-synchronous) WAL commit succeeds — extracted `jobid_range_exists()` (fast-fail pre-check, no WAL work), `build_job_submit_payloads()` (WAL-payload construction with no jobmgr insertion), and a new persist.c sibling `persist_submit_job_payloads_and_commit()` (operates on raw `xce_job_t*` instead of pre-existing `job_t*`, single production call site in job.c — legitimately within `handle_jobsubmit`'s call chain, not a boundary spillover; no task-1 continuation API dependency, genuinely still synchronous). Took 3 review rounds. **Round 1 REJECTED — real defect, not documentation**: the post-commit array-construction loop suppressed the `AFTER_EVENT`/`HOOK_DETAIL_JOB_NEW` notification for an ENTIRE array batch if even one sibling element failed to construct (e.g. a jobid collision slipping past `jobid_range_exists()`'s inherent TOCTOU window, or a plain allocation failure — no race needed) — stranding OTHER elements that were durably committed AND successfully constructed, invisible to the purely event-driven scheduler/apiserver until the next restart's replay. Logged as bug-1408. **Round 2 fixed the defect** (moved the `construct_rc < 0` early-return to AFTER the jlist-build + hook-fire block, so every successfully-constructed sibling is always notified regardless of a sibling's failure; client reply still separately reports the batch's `construct_rc`) but was REJECTED again on a narrower, documentation-only ground: bug-1408's RED-phase writeup claimed a clean assertion FAIL, but under the standard full-group `--run test_commit_barrier_reply` invocation the failing assertion actually longjmps past the test's own `cbrp_teardown()` call (Unity `TEST_ABORT` semantics) and HANGS the next test — the same longjmp-past-teardown pathology already known from task 1.3's era. **Round 3 (parent-handled directly, no code regression risk)**: corrected bug-1408's buglog.json description to accurately describe the hang mode, and hardened the new test itself — every assertion-relevant observable is now captured into a local, `cbrp_teardown()` is called unconditionally, and assertions run only after teardown, so a future regression in this exact test can no longer skip cleanup and deadlock sibling tests. No production code changed in round 3; job.c's fix from round 2 was independently verified correct by the round-2 reviewer (traced jlist-build/hook-fire at job.c:717-766 running unconditionally before the `construct_rc` check at line 775) and needed no further changes. **Known low-severity residual, not blocking, flagged for a future task**: `apiserver_push_joblist` (apiserver.c) lacks the `nfiltered > 0` guard `sched.c`'s equivalent has, so a total-batch construction failure (every array element fails, jlist ends up empty) now sends an empty/no-op jobs push to every connected apiserver — informationless, can't corrupt state, but a minor behavioral quirk introduced by firing the hook unconditionally. -- **2.4**: pure test-only close-out of task-group 2 — new `test_apply_reapply_idempotency.c` (3 tests), no production changes, approved first round. Covers three distinct cases beyond 2.2's existing coverage: (1) triple-reapplication of an identical WAL_JOB_SUBMIT through `apply_committed_entry()` with hook-fire-count AND pointer-identity assertions across all 3 calls (not just final state); (2) the `JOB_TRANSITION_NOOP_DUPLICATE` guard branch (state == current state) — a genuinely untested path, distinct from 2.2's forward-once and backward-rejected cases; (3) the task's named scenario, driven entirely through the real dispatch seam (not seeded/shortcut): real SUBMIT → real FINISH → stale out-of-order START (lower LSN, arriving after), asserting the job stays in its FINISH/terminal state and no third hook fires for the rejected START. Test assertions key on hook-fire COUNT and job STATE, never a specific `HOOK_DETAIL_*` value — so these tests stay valid even if 2.2's documented generic-hook-routing deviation is later corrected. **Task-group 2 (shared apply seam) is now complete**: 2.1 (guard cores extracted) → 2.2 (dispatch function built, generic job-state-family routing is a confirmed permanent deviation from design.md, tracked) → 2.3 (replay wired to the seam, duplicate-notification-on-restart hazard confirmed inert today but must be re-verified before 4.1/6.1/6.2/6.3) → 2.4 (idempotency proven through the real dispatch path). Next: task-group 3 (command apply-after-commit ordering), which is the first consumer of both this seam and task-group 1's commit-engine continuation API. -- **2.3**: rewired `replay_wal()` (persist.c) to call `apply_committed_entry()` (task 2.2's seam) instead of `apply_wal_entry()` for restart replay's per-entry apply. `apply_wal_entry`/`apply_committed_entry` are siblings, not wrapper/wrappee — `apply_wal_entry` is NOT retired, it remains ha.c's live standby catch-up/replicated-apply dispatch (untouched, different call sites, out of this task's boundary). **Confirmed hazard, investigated to resolution by the reviewer (not left open)**: `replay_wal(p, p->checkpoint_lsn, ...)` replays the FULL WAL range since the last snapshot on every restart (`last_applied_lsn` is deliberately not persisted — recomputed at replay time per design.md), not just a narrow crash-gap tail. Since `apply_committed_entry` fires `AFTER_EVENT` hooks (unlike the old `apply_wal_entry`, which never did), this means every restart re-fires notification hooks for entries already applied+notified in a prior process life — a duplicate-notification hazard in principle. **Independently traced and confirmed inert today** for both existing hook consumers: scheduler hooks aren't registered until `on_scheduler_join()` (sched.c), which happens post-boot via network JOIN handshake, strictly after `persist_recover()` runs; apiserver hooks ARE registered before `persist_recover()` (`apiserver_init()` in `cluster_init_phase2()`), but every apiserver hook body gates its actual network notify on `apiservers[i].ep != NULL`, and the apiserver-join table starts empty each process life — a no-op except via a narrow reconnect race. **MUST be re-verified before tasks 4.1/6.1/6.2/6.3** make this path unambiguously live and broad (e.g. if a future task registers apiserver connections earlier, or scheduler hooks get registered pre-replay) — at that point this either needs a replay-mode flag suppressing hook-firing, or an explicit accepted-idempotent-duplicate-notification decision recorded in design.md. A loud in-line comment cross-referencing this note now sits at the `apply_committed_entry()` call site in `replay_wal()` (persist.c). Also confirmed: job-state facts now replay through 2.1's guarded `apply_job_update_transition()` instead of the previously-unguarded `restore_job_state()` — intentional (Req 3/7 online-replay divergence fix), verified no existing test depended on the old unguarded semantics. `apply_committed_entry`'s missing oversized-payload guard (vs. `apply_wal_entry`'s `MAX_WAL_PAYLOAD_BYTES`/`-EMSGSIZE` check) confirmed inert — `persist_wal_write_replicated()` already rejects oversized payloads at write time, before anything reaches disk. -- **2.2**: implemented `apply_committed_entry(persist_t*, uint32_t wal_type, uint64_t lsn, const void* payload, uint32_t len)` in persist.c/persist.h, matching design.md's signature. Routes `WAL_JOB_SUBMIT` through `restore_job()` (idempotent on duplicate jobid — no-op, Req 7.1), `WAL_QUEUE_CTRL`/`WAL_WORKER_CTRL` through their existing restore primitives, and fires each type's post-mutation hook exactly once, after mutation. **Confirmed permanent, non-cosmetic deviation from design.md's Apply Seam wording**: ALL nine job-state fact WAL types (`WAL_JOB_{START,FINISH,FAIL,STOP,RESUME,UPDATE,RUSAGE,CKPT,PREASON}`) route through the single guard-only `apply_job_update_transition()` core from 2.1 — NOT the type-specific `apply_job_{start,finish,fail}_transition` cores — and fire one generic `HOOK_DETAIL_JOB_UPDATE` hook, not the type-specific `HOOK_DETAIL_JOB_START`/`_EXIT`/`_FAIL`. Root cause, independently verified by two review rounds: the on-disk wire format for every one of these types (`wal_jobstate_t {jobid, state}`) genuinely only carries the resulting state, not the richer `job_pidinfo_t`/`xce_jobfinish_t`/`job_failinfo_t` payload the type-specific cores/hooks require — this is a pre-existing wire-format reality (`apply_wal_entry` already collapsed these the same way before this task), not a shortcut invented here. **design.md's Apply Seam section and Fact Ingestion → Event Contract section currently describe per-type hook dispatch that this wire format cannot support as written** — this is a real, standing design/reality conflict, not yet reconciled in design.md itself. No task in the current plan (2.3, 4.1, 4.2, 6.1, 6.2, 6.3) enriches the wire format to close this gap — confirmed by reading all of them; task 4.1 is exclusively about removing receipt-time mutation/hook-firing, not payload enrichment. **Explicit blocking-style call-out for whoever implements 4.1/6.2/6.3**: wiring this dispatch function into a live fact-apply path will permanently change every worker-reported job start/finish/fail notification to a generic UPDATE detail instead of the type-specific one, unless the wire format is enriched first or design.md is corrected to formally accept this substitution — verify this is still the accepted behavior before wiring it live, don't assume it'll be fixed by then. Took 3 review rounds to land: round 1 rejected two doc-comment defects (a citation blaming the wrong crash site, and a dangling pointer to task 4.1 as the fix location); round 2's fix corrected the crash site (real one is `sched.c`'s `sched_job_start_fun`/`_exit_fun`/`_fail_fun`, which unconditionally dereference `info->jobid` before any NULL check) but fabricated a nonexistent function name (`encode_JobPIDInfo_ptr`) as supporting evidence; round 3 replaced it with a narrower, fully source-verified claim (apiserver.c's equivalent functions store the raw pointer into a notify union without dereferencing it at that call site — real, checked line-by-line). **Lesson**: when a reviewer catches a citation error, the safest fix is the narrowest claim you can personally verify by reading the actual code, not a broader-sounding replacement — round 2's attempt to explain "why the alternative is fine" reached for an unverified downstream mechanism and got caught fabricating it. -- **2.1**: split `mark_job_start`/`mark_job_finish`/`mark_job_fail`/`mark_job_update` (job.c) into four hook-free cores (`apply_job_{start,finish,fail,update}_transition`, new `job_transition_result_t` enum in cluster.h) containing only the monotonic-rank guard + mutation, verified an unmodified lift line-by-line against pre-extraction code. Wrappers peek the same guard predicate to decide hook-firing, then delegate to the core — no unlock/relock gap between peek and delegated call, so no new race from the double-evaluation. `mark_job_fail`'s pre-existing lack of a guard is preserved, not added or removed. New `test_apply_seam.c` (8 tests) calls the cores directly, bypassing `on_job_*` receipt handlers entirely — this is the form task 2.2 (dispatch function) and task 6.1 (apply pump) will call into. **Non-blocking follow-up flagged by reviewer**: the new doc-comments on `apply_job_fail_transition`/`apply_job_update_transition` overclaim a universal "caller holds RTTI_LOCK_WRITE" precondition — `job_ctrl` (job.c:396, pre-existing, untouched) calls `mark_job_update`/`mark_job_fail` under only RTTI_LOCK_READ. No new race (extraction added no unlock/relock), but whoever writes task 2.2/6.1 should not trust that comment's blanket claim — verify the actual lock held at each call site instead. -- **1.5**: pure test-only close-out of task-group 1 — 3 new genuine-multi-thread tests added across `test_commit_resolve.c`/`test_demotion_convergence.c`/`test_persist_truncate.c` (real pthreads, not sequential calls dressed as concurrent): an admission-cap burst with a concurrent monitor thread, a mixed majority-ack/timeout-sweep burst proving bounded dispatch-thread usage, and a genuine two-real-thread ack-vs-sweep-thread race (closing the "single-thread white-box only" gap both task-1.3 review rounds had flagged as a nice-to-have). No production code touched. **bug-1407 saga, definitively resolved**: it was NEVER a source bug. 1.4's "closed, 70/70" and 1.5's initial "reopened, 10/10 failing" were BOTH real, correct, *contradictory* measurements of the exact same source commit (9b4b805) — the difference was a stale/corrupted incremental-build artifact in the shared `build/` directory, which had been `ninja -C build`-rebuilt incrementally across all 5 tasks without ever a fresh `meson setup`. Proven by isolating each measurement in its own throwaway `git worktree add` + fresh `meson setup`/`ninja`: pre-feature commit and the exact failing commit both pass cleanly (0 failures) on a fresh build; only the main worktree's stale incremental `build/` produced the failure. Fixed with `rm -rf build && meson setup build --buildtype=debug -Db_sanitize=address && ninja -C build`; bug-1407 is now genuinely CLOSED with high confidence. **Lesson for every remaining task in this series**: distrust a surprising/contradictory `ninja -C build` incremental result before concluding a source regression — rebuild clean (or isolate via a throwaway worktree) first. Never use `git checkout <sha> -- .` to inspect an old commit in place — it silently reverts working-tree files against current HEAD; use `git worktree add <path> <sha>` instead. Both lessons are now in `.wolf/cerebrum.md` Do-Not-Repeat and `.wolf/buglog.json` bug-1407. diff --git a/.kiro/specs/raft-commit-safety/brief.md b/.kiro/specs/raft-commit-safety/brief.md deleted file mode 100644 index 6894d9f8..00000000 --- a/.kiro/specs/raft-commit-safety/brief.md +++ /dev/null @@ -1,31 +0,0 @@ -# Brief: raft-commit-safety - -## Problem -After `raft-log-replication` gives the library a real `(term, index)` log, two `raft.tla` commit/election-safety rules are still missing in XCE (gap analysis: `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`): -- **G4** — commit-index advance is **term-blind**. `raft.tla` `AdvanceCommitIndex` (line 259) only advances `commitIndex` to `Max(agreeIndexes)` **if `log[Max].term == currentTerm`** (the Figure-8 safety rule: a leader never commits a previous-term entry by replica-count alone). XCE's commit barrier + `ha_compute_safe_committed_lsn` advanced committed "from 1 granted voter" with no current-term check (because the WAL term was dead); the own-term `WAL_NOOP` gate is only a partial mitigation. -- **G6** — vote freshness is **LSN-only**. `raft.tla` `HandleRequestVoteRequest` `logOk` (line 285) compares `(lastLogTerm, lastLogIndex)` lexicographically; the library's `on_vote_eligible` gate uses `config_tag ≥` + `applied LSN ≥` only — it never compares `lastLogTerm`. - -## Approach -Add the commit-safety and freshness rules to the library, strictly per `raft.tla`, on top of the `raft-log-replication` log model. - -### TLA+ actions implemented -- `AdvanceCommitIndex(i)` (raft.tla:259) → `raft_advance_commit(...)`: compute `agreeIndexes` from per-peer `matchIndex` (a quorum agrees), and advance `commitIndex` to `Max(agreeIndexes)` **only when `on_get_entry_term(Max) == currentTerm`**. Otherwise hold. -- `HandleRequestVoteRequest` `logOk` (raft.tla:285) → upgrade the freshness gate to `(lastLogTerm, lastLogIndex)`: grant only if `cand.lastLogTerm > local.lastLogTerm ∨ (== ∧ cand.lastLogIndex ≥ local.lastLogIndex)`, applied to both `raft_on_vote_request` and the PreVote eligibility path. - -## Authoritative shared contract -Uses the `raft-log-replication` contract verbatim (1-based index = WAL LSN; per-entry `uint64_t term`; `on_get_entry_term`, per-peer `matchIndex`). New callback: -- `void on_commit_advanced(void *ctx, uint64_t commit_index)` — fired AFTER `commitIndex` advances; the controller adapter feeds this into the **existing synchronous commit barrier / durability** (persist-before-ACK preserved — this callback does not replace the barrier, it triggers commit-eval). -Vote RPC freshness gains `last_log_term` alongside the existing `last_log_index`/`config_tag` carried to `on_vote_eligible` (extend the eligibility signature or the vote-request payload accordingly). - -## Scope -- **In**: `AdvanceCommitIndex` with the current-term rule; `(lastLogTerm, lastLogIndex)` vote/PreVote freshness; `on_commit_advanced` callback wired to the controller commit barrier; simulator assertion of **Leader Completeness** (a committed entry is present in every future leader's log). -- **Out**: the log model + AppendEntries (owned by `raft-log-replication`); the conformance harness (`raft-sim-conformance`); config markers/snapshot (`ha-term-safety`); the barrier/durability mechanism itself (controller — this spec only triggers it via `on_commit_advanced`). - -## Preserve as policy -The synchronous commit barrier + persist-before-ACK durability is STRONGER than canonical Raft async apply and MUST remain: `on_commit_advanced` triggers the existing barrier; it does not weaken it. PreVote and priority leadership unchanged. - -## Verification -Extend `raft_sim.c`: construct Figure-8 scenarios (a previous-term entry replicated to a quorum but not yet committed by the current leader) and assert the library refuses to commit it until a current-term entry is committed; assert **Leader Completeness** across forced leader changes. Replace `ha_compute_safe_committed_lsn`'s term-blind advance with the library's current-term-gated `commitIndex`. - -## Dependencies -`raft-log-replication` (HARD — needs the `(term,index)` log + `matchIndex`). Feeds `raft-sim-conformance` and `ha-term-safety`. Closes bug-851 G4/G6. diff --git a/.kiro/specs/raft-commit-safety/design.md b/.kiro/specs/raft-commit-safety/design.md deleted file mode 100644 index 6a5a53b4..00000000 --- a/.kiro/specs/raft-commit-safety/design.md +++ /dev/null @@ -1,392 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: This feature adds the **commit-safety and election-freshness** rules of `raft/raft.tla` (Ongaro canonical) to the pure Raft library (`src/lib/raft/raft.{c,h}`), on top of the log half delivered by `raft-log-replication`. It closes the two remaining safety gaps of bug-851: **G4** (term-blind commit advance) and **G6** (LSN-only vote freshness). - -**Users**: The XCE controller (as a Raft node, which will route its commit barrier through the new entry point in the downstream `ha-term-safety` integration), the deterministic simulator (`raft_sim.c`), and the downstream `raft-sim-conformance` spec (Leader Completeness oracle). - -**Impact**: Today the commit position advances **term-blind** in the controller (`ha_compute_safe_committed_lsn` + the commit barrier) — it can mark a previous-term entry committed by replica-count alone (G4) — and vote freshness compares LSNs only, never the last-log term (G6). After this feature, the library owns `commit_index` and advances it exactly as `raft.tla` `AdvanceCommitIndex` does (quorum agreement **and** current-term entry), notifying the controller via `on_commit_advanced` to **trigger** the existing synchronous commit barrier (the barrier itself is unchanged and remains the controller's). Vote and PreVote freshness become the `raft.tla` `(lastLogTerm, lastLogIndex)` lexicographic comparison. The change is a pure, synchronous, deterministic addition to the core; all durability/I/O stays in adapters. - -### Goals -- Add a library-owned `commit_index` and `raft_advance_commit(...)` implementing `raft.tla` `AdvanceCommitIndex` (line 259) **with** the current-term rule `log[Max(agreeIndexes)].term == currentTerm` (line 270) — the Figure-8 safety rule (G4). -- Add the required callback `on_commit_advanced(ctx, commit_index)` that **triggers** the existing synchronous commit barrier; the library never performs the barrier itself. -- Upgrade vote freshness to the `raft.tla` `HandleRequestVoteRequest` `logOk` `(lastLogTerm, lastLogIndex)` comparison (line 285) on both the binding vote path and the PreVote eligibility path (G6). -- Extend `raft_sim.c` to assert the **Figure-8** refusal and **Leader Completeness** across forced leader changes. - -### Non-Goals -- **No log model / AppendEntries / per-peer progress** — owned by `raft-log-replication` and reused verbatim (`log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`, `on_persist_entry`, `on_send_append_entries`, `on_truncate_from`, `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`). -- **No commit barrier / durability mechanism** — the synchronous barrier, fsync/group-commit, persist-before-ACK, reply gate, and snapshot transport stay in the controller (`ha-term-safety` adapter). This spec only triggers the barrier. -- **No config-change markers / upgrade snapshot** — owned by `ha-term-safety`. -- **No conformance / TLC trace-check harness** — owned by `raft-sim-conformance`. -- **No controller wiring** — replacing `ha_compute_safe_committed_lsn`'s term-blind advance with the library's `commit_index` is `ha-term-safety`; this spec defines only the library entry point + callback it will route through. - -## Boundary Commitments - -### This Spec Owns -- The library `commit_index` field in `raft_state_t` and its restore field in `raft_config_t` (`initial_commit_index`, default 0). -- The new entry point `raft_advance_commit(raft_state_t *s)` implementing `AdvanceCommitIndex` with the current-term rule. -- The new required callback `on_commit_advanced` in `raft_callbacks_t` (fired only on strict advance) + its `raft_init` validation. -- The `(lastLogTerm, lastLogIndex)` freshness upgrade on the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path (`raft_on_pre_vote_request`), including carrying `last_log_term` alongside `last_log_index`. -- Simulator extensions in `raft_sim.c` + tests in `src/lib/testsuite/test_raft.c` asserting Figure-8 refusal and Leader Completeness. - -### Out of Boundary -- The log model, AppendEntries, and per-peer `match_index[]` themselves (`raft-log-replication`) — consumed verbatim, never redefined. -- The synchronous commit barrier, fsync/group-commit, persist-before-ACK durability, reply gate, snapshot transport (controller / `ha-term-safety`). -- Config markers in the unified log + the change-boundary snapshot (`ha-term-safety`). -- The conformance / TLC trace-check harness (`raft-sim-conformance`). -- The controller's replacement of `ha_compute_safe_committed_lsn` with `commit_index` and the controller's freshness-floor plumbing of `last_log_term` onto the wire (`ha-term-safety`). This spec specifies only **what the library entry point/callback route**, not the controller mechanics. - -### Allowed Dependencies -- The implemented election core (`raft-library`) and the log half (`raft-log-replication`): `raft_state_t`, `raft_callbacks_t`, `raft_init`, the vote/PreVote path, `log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`. All reused; the vote path is extended additively. -- `errno.h`, `stdint.h`, `string.h` only — the core stays free-standing for the simulator. -- The simulator harness (`raft_sim.c`) for verification. -- Must NOT depend on any controller header, WAL type, capnp type, or transport — the core stays pure. - -### Revalidation Triggers -Dependent specs (`raft-sim-conformance`, `ha-term-safety`) must re-check integration if any of the following change: -- The signature or semantics of `raft_advance_commit` or the `on_commit_advanced` callback (especially the strict-advance / fire-once rule and "trigger only, never performs the barrier"). -- The placement of the current-term rule (`on_get_entry_term(Max(agreeIndexes)) == currentTerm`) or the `agreeIndexes`/quorum definition. -- The freshness signature change — the addition of `last_log_term` to `raft_on_vote_request`, the PreVote eligibility path, and any vote-request payload / `on_vote_eligible` extension. -- The `commit_index` restore convention (`initial_commit_index`, default 0) or its monotonicity guarantee. - -## Architecture - -### Existing Architecture Analysis - -After `raft-log-replication`, the library owns the log as pure, synchronous, deterministic state: per-entry term via `on_get_entry_term`, per-peer `next_index[]` / `match_index[]`, and `leader_commit` carried **opaquely** (never acted on). The election core delegates freshness to `on_vote_eligible`, which today receives only `(from_idx, term)` — the caller checks `config_tag` / applied-LSN floor but has no last-log **term**. This feature follows the same discipline: - -- **Pure / synchronous / deterministic core** — `raft_advance_commit` reads entry terms via `on_get_entry_term` and emits a single `on_commit_advanced` notification; it owns no entry bytes and performs no I/O. The freshness upgrade is a predicate computed from values already in the library log model plus the candidate's advertised `(last_log_term, last_log_index)`. -- **Persist-before-act ordering** — unchanged on the vote path: a vote that passes the new `(lastLogTerm, lastLogIndex)` predicate is still persisted via `on_persist` before being granted. `commit_index` advance is not itself persisted by the library — durability of the committed position remains the controller barrier's job, triggered by `on_commit_advanced`. -- **Status returns, never self-terminate** — `raft_advance_commit` returns a status; no new path aborts. -- **Separate actions** — mirroring `raft.tla`, `AdvanceCommitIndex` is a **distinct** action from `HandleAppendEntriesResponse`. `raft_on_append_entries_reply` (owned upstream) only updates `match_index[]`; commit advance is the separate `raft_advance_commit` call. This keeps atomic regions small and lets a single-node leader commit (the `{self}` term in `agreeIndexes`). - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TB - subgraph PureCore["src/lib/raft/raft.c — PURE CORE (this spec)"] - ADV["raft_advance_commit()<br/>AdvanceCommitIndex (tla:259)<br/>+ current-term rule (tla:270)"] - VOTE["raft_on_vote_request()<br/>logOk (lastLogTerm,lastLogIndex) (tla:285) G6"] - PVOTE["raft_on_pre_vote_request()<br/>same (lastLogTerm,lastLogIndex) freshness"] - end - subgraph Upstream["raft-log-replication — REUSED VERBATIM (not this spec)"] - MI["match_index[] / log_len"] - GET["on_get_entry_term(index) -> term|0"] - REPLY["raft_on_append_entries_reply()<br/>updates match_index[] only"] - end - subgraph CBs["raft_callbacks_t — NEW seam (this spec)"] - C5["on_commit_advanced(ctx, commit_index)<br/>fired ONLY on strict advance"] - end - subgraph Consumers["ADAPTERS — implement callback (NOT this spec's heavy wiring)"] - CTRL["controller adapter (ha-term-safety):<br/>on_commit_advanced -> EXISTING sync commit barrier<br/>(persist-before-ACK preserved, NOT replaced)"] - SIM["raft_sim.c: drive raft_advance_commit,<br/>assert Figure-8 + Leader Completeness (this spec)"] - end - REPLY -.updates.-> MI - ADV --> MI & GET - ADV --> C5 - VOTE --> GET - PVOTE --> GET - C5 --> CTRL - C5 --> SIM - ADV --> SIM -``` - -**Architecture Integration**: -- **Selected pattern**: Pure state machine + callback ports (Hexagonal/Ports-and-Adapters), already established by the election and log cores. Commit advance is a new pure action; `on_commit_advanced` is a new port adapters implement. -- **Domain boundaries**: the commit-safety **decision** (which index is safe to commit, given quorum + current-term) stays in the library; the durable **act** (the synchronous commit barrier, fsync, ACK) stays in the controller adapter, triggered by the port. -- **Existing patterns preserved**: `raft_config_t` extensibility, persist-before-act on the vote path, NULL-checked required callbacks, status-code returns, `XCE_TESTING`-gated simulator. -- **Steering compliance**: matches `src/lib/CLAUDE.md` (free-standing C, gnu17), `ha-async-io.md` (no blocking in the core; the barrier/async lives in the adapter), `testing.md` (TDD: simulator/test first), `ha-async-io.md` durability policy ("persist-before-ACK + synchronous commit barrier" preserved — the callback triggers it), "controller never self-terminates" (library returns statuses only). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Library core | C (gnu17), `errno.h`/`stdint.h`/`string.h` only | Pure commit-safety + freshness | Free-standing; no controller/capnp deps | -| Callbacks | `raft_callbacks_t` function pointers | Commit-advance trigger seam | 1 new required callback (`on_commit_advanced`) | -| Verification | `raft_sim.c` (XCE_TESTING) + `test_raft.c` | Figure-8 + Leader Completeness | Reuses the in-memory log backing from `raft-log-replication` | -| Adapter (downstream) | controller commit barrier (`ha-term-safety`) | Durable commit / ACK | Not built here; only triggered | - -## File Structure Plan - -### Modified Files -- `src/lib/raft/raft.h` — Add `uint64_t commit_index;` to `raft_state_t`; add `uint64_t initial_commit_index;` to `raft_config_t`; add the `on_commit_advanced` callback to `raft_callbacks_t`; declare `raft_advance_commit`; extend the vote-path signatures to carry `last_log_term` (alongside the existing `last_log_index`) per the chosen freshness-plumbing option below; add a Figure-8 / Leader-Completeness driver + assertion helper to the simulator section. -- `src/lib/raft/raft.c` — Implement `raft_advance_commit` (agreeIndexes + current-term rule + monotonic strict-advance firing `on_commit_advanced`); add the `on_commit_advanced` NULL check + `initial_commit_index` restore in `raft_init`; upgrade the freshness predicate in `raft_on_vote_request` and `raft_on_pre_vote_request` to the `(lastLogTerm, lastLogIndex)` comparison. -- `src/lib/raft/raft_sim.c` — Drive `raft_advance_commit` after delivery; the Figure-8 construction; the Leader-Completeness assertion across forced leader changes; the freshness-gate exercise. -- `src/lib/testsuite/test_raft.c` — New tests for `raft_advance_commit` (current-term hold/advance, monotonicity, non-leader), `on_commit_advanced` fire-once/never semantics, the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness, and the simulator invariants (TDD-first). - -### Directory Structure (unchanged) -``` -src/lib/raft/ -├── raft.h # + commit_index, on_commit_advanced, raft_advance_commit, last_log_term on vote path -├── raft.c # + AdvanceCommitIndex(current-term rule), (lastLogTerm,lastLogIndex) freshness -└── raft_sim.c # + commit-advance driver, Figure-8, Leader Completeness assert -src/lib/testsuite/ -└── test_raft.c # + commit-safety + freshness + Figure-8/Leader-Completeness tests -``` - -> Each file keeps one responsibility: `raft.c` = pure protocol, `raft_sim.c` = deterministic harness, `test_raft.c` = assertions. No new file is introduced; the existing module pattern is extended. - -## System Flows - -### Commit advance: separate action after replication (raft.tla AdvanceCommitIndex) - -```mermaid -sequenceDiagram - participant L as Leader (raft.c) - participant LC as Leader adapter (cb) - Note over L: replication already updated match_index[]<br/>via raft_on_append_entries_reply (raft-log-replication) - Note over L: raft_advance_commit() — separate action (tla:259) - L->>L: agreeIndexes = { i in 1..log_len :<br/>{self} ∪ {p: match_index[p]>=i} ∈ Quorum } - alt agreeIndexes empty - Note over L: hold commit_index (else-branch) - else Max(agreeIndexes) from a PREVIOUS term - L->>LC: on_get_entry_term(Max) != currentTerm - Note over L: hold commit_index (Figure-8 refusal, G4) - else Max(agreeIndexes) from CURRENT term - L->>LC: on_get_entry_term(Max) == currentTerm - Note over L: commit_index = Max(agreeIndexes) (strict advance) - L->>LC: on_commit_advanced(ctx, commit_index) - Note over LC: trigger EXISTING sync commit barrier<br/>(persist-before-ACK, NOT replaced) - end -``` - -### Vote freshness: (lastLogTerm, lastLogIndex) comparison (raft.tla HandleRequestVoteRequest logOk) - -```mermaid -sequenceDiagram - participant V as Voter (raft.c) - participant VC as Voter adapter (cb) - Note over V: raft_on_vote_request(..., cand_last_log_term, cand_last_log_index) - Note over V: UpdateTerm / stale-term / one-vote-per-term (unchanged) - V->>VC: on_get_entry_term(log_len) -> local_last_log_term (0 if empty) - Note over V: logOk = cand_last_log_term > local_last_log_term<br/>∨ (== ∧ cand_last_log_index >= log_len) - alt !logOk - Note over V: DENY (even if all other gates would grant) — G6 - else logOk AND application gate AND invariants pass - V->>VC: on_persist(term, from_idx) %% persist-before-grant unchanged - Note over V: GRANT - end -``` - -Flow-level decisions: -- **Commit is a separate action** — `match_index[]` is updated by `raft_on_append_entries_reply` (upstream); the commit index advances only when `raft_advance_commit` is driven, exactly as `raft.tla` separates the two actions (R2.6). This lets a single-node leader commit (`{self}` ∈ `agreeIndexes`). -- **Current-term rule is the Figure-8 gate** — `commit_index` advances to `Max(agreeIndexes)` only when that entry is from `currentTerm`; a previous-term entry replicated to a quorum is **held** until a current-term entry above it commits, then committed indirectly (R1.3/R1.4, closes G4). -- **`on_commit_advanced` fires only on strict advance** — never on a held commit index, never from `raft_on_append_entries_reply` (R2.2/R2.3/R2.6). -- **Freshness denies before any grant** — a stale `(lastLogTerm, lastLogIndex)` denies even when the term/vote invariants and the application gate would grant (R3.4, closes G6); persist-before-grant is unchanged (R3.6). - -## Freshness Plumbing — design decision - -`raft.tla` `logOk` needs the candidate's `(lastLogTerm, lastLogIndex)` and the voter's own. Two options were considered for carrying the candidate values into the library: - -- **Option A — extend the `on_vote_eligible` callback signature** to `on_vote_eligible(ctx, from_idx, term, last_log_term, last_log_index)` and have the library compute `logOk` against `on_get_entry_term(log_len)`/`log_len` before calling the gate. **Rejected as the sole mechanism**: it pushes the protocol-defining `logOk` either into the adapter (violating "library owns protocol decisions") or splits it. -- **Option B (chosen) — the library owns `logOk`.** The candidate's `last_log_term` and `last_log_index` are accepted as explicit arguments on `raft_on_vote_request` (and on the PreVote eligibility path), the library computes `logOk` against its own `on_get_entry_term(log_len)` / `log_len`, and only then consults the unchanged application gate `on_vote_eligible` for XCE-specific policy (primary-deny / heartbeat stickiness / catch-up). This keeps the `raft.tla` `logOk` predicate **inside** the library (R3.1, R5.5) while leaving `on_vote_eligible` for application policy that is genuinely outside Raft. - -The candidate advertises its `(last_log_term, last_log_index)` in the `RequestVote` / PreVote wire RPC; the controller adapter decodes them and passes them to the library entry point. The on-wire encoding of `last_log_term` (and the controller's freshness-floor plumbing) is the `ha-term-safety` adapter's job — out of this spec's scope; this spec fixes only the library signatures and the predicate. - -> **Note — freshness-decision locus moves from caller/dispatcher into the library (both paths).** Today the freshness decision lives **outside** the library: `raft.h` (lines 246–248) documents that PreVote freshness "is applied by the dispatcher before this call", and the binding vote path delegates the decision to `on_vote_eligible`. Option B **moves that decision locus into the library** for **both** the binding (`raft_on_vote_request`) and the PreVote (`raft_on_pre_vote_request`) paths: `raft.c` now computes the `(lastLogTerm, lastLogIndex)` `logOk` itself rather than relying on the caller/dispatcher. This **supersedes the existing `raft.h` lines 246–248 doc contract**, and updating those `raft.h` doc comments (so the header no longer states the dispatcher owns PreVote freshness) is part of the header-contract task. - -## Requirements Traceability - -| Requirement | Summary | Design Elements | raft.tla action | -|-------------|---------|-----------------|-----------------| -| 1.1–1.7 | Library `commit_index`, agreeIndexes, current-term rule, monotonic, leader-only | `raft_state_t.commit_index`; `raft_advance_commit()`; `on_get_entry_term`; `match_index[]` (upstream) | `AdvanceCommitIndex` (259), current-term rule (270) | -| 2.1–2.6 | `on_commit_advanced` trigger, fire-once, separate from reply | `raft_callbacks_t.on_commit_advanced`; strict-advance firing; `raft_init` validation | (seam; `AdvanceCommitIndex` distinct from `HandleAppendEntriesResponse`) | -| 3.1–3.6 | `(lastLogTerm,lastLogIndex)` binding-vote freshness | `raft_on_vote_request` logOk; `on_get_entry_term(log_len)`/`log_len`; persist-before-grant | `HandleRequestVoteRequest` logOk (285) | -| 4.1–4.5 | Same freshness on PreVote, side-effect-free | `raft_on_pre_vote_request` logOk; no persist/term/role mutation | `HandleRequestVoteRequest` logOk (285), PreVote extension | -| 5.1–5.4 | Barrier/durability preserved as policy; PreVote/priority unchanged; no abort | `on_commit_advanced` as trigger only; additive-only changes | (policy; not a TLA+ action) | -| 6.1–6.6 | Simulator Figure-8 + Leader Completeness + freshness gate | `raft_sim.c` commit driver, Figure-8 construction, Leader-Completeness assert, freshness exercise | Figure-8 / Leader Completeness invariant | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| `commit_index` (`raft_state_t`) | Library core | Library-owned commit position | 1 | log core (P0) | State | -| `raft_advance_commit` | Library core | AdvanceCommitIndex + current-term rule | 1, 2 | `on_get_entry_term`, `match_index[]`, `on_commit_advanced` (P0) | Service | -| `on_commit_advanced` callback | Seam | Commit-advance trigger | 2, 5 | controller barrier / sim (P0) | Service | -| `raft_on_vote_request` freshness | Library core | `(lastLogTerm,lastLogIndex)` logOk | 3, 5 | `on_get_entry_term`, `log_len`, `on_persist`, `on_vote_eligible` (P0) | Service | -| `raft_on_pre_vote_request` freshness | Library core | Same logOk, side-effect-free | 4, 5 | `on_get_entry_term`, `log_len`, `on_vote_eligible` (P0) | Service | -| Simulator | Verification | Figure-8 + Leader Completeness | 6 | all of the above (P0) | State | - -### Library Core - -#### `commit_index` addition to `raft_state_t` - -| Field | Detail | -|-------|--------| -| Intent | Hold the library's view of the safe commit position (TLA+ `commitIndex`) | -| Requirements | 1.1, 1.5 | - -**Responsibilities & Constraints** -- `uint64_t commit_index;` — the highest 1-based index (== WAL LSN) the library considers safely committed. Monotonic non-decreasing. -- Restored at init from the adapter (`raft_config_t.initial_commit_index`, default 0; the library never reads storage — consistent with `initial_term` / `initial_log_len`). -- The library never advances `commit_index` except through `raft_advance_commit`; `raft_on_append_entries_reply` (upstream) leaves it untouched (R2.6). - -#### `raft_advance_commit` — AdvanceCommitIndex (raft.tla:259) with the current-term rule (line 270) - -| Field | Detail | -|-------|--------| -| Intent | Advance `commit_index` to `Max(agreeIndexes)` only when that entry is from `currentTerm` | -| Requirements | 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3 | - -##### Service Interface -```c -/* - * raft_advance_commit: leader recomputes its commit index (raft.tla - * AdvanceCommitIndex, 259). A SEPARATE action from append-entries reply - * handling (mirrors raft.tla, so a single-node leader can commit and to - * keep atomic regions small). - * - * Computes agreeIndexes = { index in 1..log_len : - * ({self} ∪ {peer : match_index[peer] >= index}) is a quorum }. - * If agreeIndexes is non-empty AND on_get_entry_term(Max(agreeIndexes)) - * == s->term, sets commit_index = Max(agreeIndexes) (Figure-8 current-term - * rule) and, on a STRICT advance, fires on_commit_advanced(ctx, commit_index) - * exactly once. - * Otherwise leaves commit_index unchanged and does NOT fire on_commit_advanced. - * - * The library performs NO durability work here — on_commit_advanced only - * TRIGGERS the controller's existing synchronous commit barrier. - * - * Returns RAFT_OK on success (whether or not the index advanced); - * -EPERM if role != RAFT_LEADER (commit_index left unchanged). - */ -int raft_advance_commit(raft_state_t *s); -``` -- Preconditions: `s->role == RAFT_LEADER` (else `-EPERM`, no change — R1.6). -- Postconditions: `commit_index` either unchanged or advanced to `Max(agreeIndexes)` when that entry is current-term; `on_commit_advanced` fired iff `commit_index` strictly increased. -- Invariants: `commit_index` never decreases (R1.5); the library reads entry terms only via `on_get_entry_term` (R1.7); `commit_index ≤ log_len`. - -> **Quorum**: `(n_peers / 2) + 1`, identical to the election-core quorum; `match_index[self]` is implicitly `log_len` (self always agrees up to its own log length), realized by seeding the agree-count with self for each candidate index. `agreeIndexes` is computed by scanning candidate indexes from `log_len` downward and taking the first that reaches quorum (equivalent to `Max(agreeIndexes)` without materializing the set). - -#### Freshness upgrade — `raft_on_vote_request` (raft.tla:285, closes G6) - -| Field | Detail | -|-------|--------| -| Intent | Add the `(lastLogTerm, lastLogIndex)` lexicographic logOk before the application gate | -| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 | - -##### Service Interface -```c -/* - * raft_on_vote_request: process an incoming RequestVote RPC (extended). - * Adds the raft.tla HandleRequestVoteRequest logOk (285): the library - * computes - * logOk = (cand_last_log_term > LastTerm(local)) - * || (cand_last_log_term == LastTerm(local) - * && cand_last_log_index >= log_len) - * where LastTerm(local) = (log_len > 0) ? on_get_entry_term(log_len) : 0. - * Step order (unchanged except the new logOk gate): - * 1. stale-term deny / UpdateTerm (unchanged) - * 2. role gate, one-vote-per-term (unchanged) - * 3. logOk gate (NEW, G6): deny if !logOk, before on_vote_eligible - * 4. application gate on_vote_eligible (unchanged) - * 5. on_persist before grant (unchanged) - * Returns RAFT_GRANT / RAFT_DENY; sets *reply_term. - */ -int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term, - uint64_t cand_last_log_term, - uint64_t cand_last_log_index, - uint64_t *reply_term); -``` -- Preconditions: none beyond a valid `s`. -- Postconditions: a vote is granted only when the existing invariants, the new `(lastLogTerm, lastLogIndex)` logOk, and the application gate all pass; persisted before grant. -- Invariants: `logOk` is evaluated **before** `on_vote_eligible`; a `!logOk` denies regardless of application policy (R3.4); no weakening of stale-term / one-vote-per-term (R3.5). - -> **Signature change is a Revalidation Trigger.** `raft_on_vote_request` gains `cand_last_log_term` and `cand_last_log_index`. The controller and the simulator (the two callers) must pass the candidate's advertised last-log `(term, index)`. The PreVote path mirrors this. - -#### Freshness upgrade — `raft_on_pre_vote_request` (side-effect-free) - -| Field | Detail | -|-------|--------| -| Intent | Apply the identical `(lastLogTerm, lastLogIndex)` logOk on the straw poll, mutating nothing | -| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5 | - -##### Service Interface -```c -/* - * raft_on_pre_vote_request: answer a PreVote probe (extended). - * Applies the SAME (lastLogTerm, lastLogIndex) logOk as the binding vote - * (Requirement 3.1), in addition to the existing stale-term deny and the - * on_vote_eligible gate. Mutates NOTHING: no term/voted_for/role/commit_index - * change, no on_persist (PreVote invariant preserved). - * Returns RAFT_GRANT / RAFT_DENY. - */ -int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term, - uint64_t cand_last_log_term, - uint64_t cand_last_log_index); -``` -- Preconditions: none beyond a valid `s`. -- Postconditions: grant only when stale-term, logOk, and the application gate all pass; no persistent state touched. -- Invariants: identical freshness rule to the binding path (R4.1, R4.2); side-effect-free (R4.4). - -### Seam — `raft_callbacks_t` addition (verbatim Authoritative shared contract) - -Appended to `raft_callbacks_t` and validated **required** in `raft_init`. Reproduced verbatim from the brief so downstream specs depend on the exact shape: - -```c -/* REQUIRED: fired AFTER commit_index strictly advances. The controller adapter - * feeds this into the EXISTING synchronous commit barrier / persist-before-ACK - * durability — this callback TRIGGERS commit-eval; it does NOT replace the - * barrier. */ -void on_commit_advanced(void *ctx, uint64_t commit_index); -``` - -**Contract rules (R2.4, R5.1)**: the library decides *whether* the commit index advanced; the callback only triggers the adapter's commit work. It is fired exactly once per strict advance (R2.2), never on a held commit index (R2.3), and never from `raft_on_append_entries_reply` (R2.6). It is NULL-checked and required in `raft_init` (`-EINVAL` if NULL — R2.5), matching how the other required callbacks are validated. - -### Controller adapter (seam only — heavy wiring is `ha-term-safety`) - -This spec describes the adapter **only at the seam**; the full controller integration is the downstream `ha-term-safety` consumer. - -- `on_commit_advanced` → the controller routes the new safe `commit_index` into its **existing** synchronous commit barrier (commit-eval sweep, persist-before-ACK, reply gate) — replacing the term-blind `ha_compute_safe_committed_lsn` advance with the library's current-term-gated value. The barrier mechanism is unchanged; only its **input** (the safe commit position) now comes from the library. -- The candidate's advertised `(last_log_term, last_log_index)` is decoded from the `RequestVote` / PreVote wire RPC and passed to `raft_on_vote_request` / `raft_on_pre_vote_request`; `last_log_term` is read back from the WAL term (`on_get_entry_term`-backed index) on recovery. The on-wire encoding and the freshness-floor plumbing are `ha-term-safety`. -- On recovery, the controller restores `commit_index` (= recovered committed LSN) into `raft_config_t.initial_commit_index`. - -## Data Models - -### Domain Model -- **Commit position (logical)**: `commit_index : uint64` — the highest 1-based index (== WAL LSN) safely committed, owned by the library, monotonic. - - **Convention**: callers pass `s->commit_index` as the `leader_commit` argument to `raft_send_append_entries` — the leader's committed value rides AppendEntries from `commit_index`, which this spec owns (upstream `raft-log-replication` carries `leader_commit` opaquely; the value it carries is the `commit_index` defined here). -- **Freshness tuple**: `(last_log_term : uint64, last_log_index : uint64)` per candidate, compared lexicographically against the voter's own `(on_get_entry_term(log_len), log_len)`. - -### Logical Data Model -- `raft_state_t` gains `commit_index`. -- `raft_config_t` gains `initial_commit_index` (restored committed LSN at boot; 0 on first boot — the library never reads storage, consistent with `initial_term` / `initial_log_len`). -- No new persisted on-disk format: `commit_index` durability is the controller barrier's existing responsibility; the library imposes no new persisted layout. `last_log_term` reuses the WAL `hdr.term` already written by `raft-log-replication`'s adapter. - -### Data Contracts & Integration -- **Freshness tuple**: `raft_on_vote_request` and `raft_on_pre_vote_request` accept `(cand_last_log_term, cand_last_log_index)` as explicit arguments. The voter's own values come from `on_get_entry_term(log_len)` (term `0` when `log_len == 0`, consistent with `raft.tla` `LastTerm`) and `log_len`. -- **Commit trigger**: `on_commit_advanced(ctx, commit_index)` carries only the new committed index; no entry bytes, no term — the adapter looks up whatever it needs through its own state. - -## Error Handling - -### Error Strategy -The library returns status codes only and **never aborts** (preserves "controller never self-terminates"). Callbacks for commit-advance are trigger-only and return void; the freshness path keeps the existing persist-failure handling. - -### Error Categories and Responses -- **Caller misuse**: `raft_advance_commit` while not leader → `-EPERM`, `commit_index` unchanged, no `on_commit_advanced`. NULL `on_commit_advanced` at `raft_init` → `-EINVAL`. -- **Held commit (Figure-8)**: when `Max(agreeIndexes)` is a previous-term entry, `commit_index` is held — this is the correct safety behavior, not an error; no callback fires. -- **Empty agreeIndexes**: no quorum agrees on any index → hold, no callback. -- **Stale freshness**: `!logOk` → vote/PreVote DENY (not an error; the standard freshness rejection). -- **Persist failure on grant**: unchanged from the existing vote path — a failed `on_persist` denies the vote without recording it. - -### Monitoring -Reuse the optional `on_trace` callback for new events: `"commit_advance"`, `"commit_hold_prevterm"`, `"commit_hold_noquorum"`, `"vote_denied_log_term"` (stale `lastLogTerm`), `"vote_denied_log_index"` (tie term, stale index), `"pre_vote_denied_freshness"`. NULL-checked exactly like existing trace calls. - -## Testing Strategy - -TDD-mandatory (`testing.md`): tests and simulator drivers are written **before** the `raft.c` implementation (RED → GREEN). All tests are `XCE_TESTING`-gated and live in `src/lib/testsuite/test_raft.c` driving `raft_sim.c`. They reuse the in-memory per-node log backing added by `raft-log-replication`. - -### Unit Tests (entry points, against the in-memory sim log) -1. `raft_advance_commit` current-term advance: a leader whose `match_index[]` shows a quorum agreeing through index `k`, with the entry at `k` from `currentTerm`, advances `commit_index` to `k` and fires `on_commit_advanced(k)` once (R1.2, R1.3, R2.2). -2. `raft_advance_commit` Figure-8 hold: a leader with a quorum agreeing through a **previous-term** entry at `k` (no current-term entry above it) holds `commit_index` and fires nothing (R1.4, R2.3 — G4). -3. `raft_advance_commit` monotonicity + non-leader: never decreases `commit_index`; returns `-EPERM` with no change and no callback when not leader (R1.5, R1.6). -4. `on_commit_advanced` separation: `raft_on_append_entries_reply` updating `match_index[]` does **not** by itself advance `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does (R2.6). -5. `raft_init` requires `on_commit_advanced`: `-EINVAL` when NULL; restores `initial_commit_index` (R2.5, R1.1). -6. `(lastLogTerm, lastLogIndex)` vote freshness: a candidate with a stale `last_log_term` is denied even with a greater `last_log_index`; a candidate with a fresher `last_log_term` is granted with a smaller `last_log_index`; a tie on term grants only when `last_log_index >= log_len` (R3.1, R3.4). PreVote applies the identical rule with no persist/term/role mutation (R4.1, R4.4). - -### Integration / Simulator Tests (Figure-8, Leader Completeness) -1. **Figure-8 scenario**: construct a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does **not** move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over **both** (indirect commit) (R6.2, R6.3 — G4). -2. **Leader Completeness**: across forced leader changes (timeout / partition-driven), assert that any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader (R6.4) — the `(lastLogTerm, lastLogIndex)` freshness gate (R6.5) is what makes this hold, so the two tests are complementary. -3. **Freshness gate exercise**: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome (stale-term-but-higher-index denied; fresher-term-but-lower-index granted) (R6.5). -4. **Trigger semantics**: assert `on_commit_advanced` fires exactly on strict advance and not while the current-term rule holds the commit index (R6.6). diff --git a/.kiro/specs/raft-commit-safety/requirements.md b/.kiro/specs/raft-commit-safety/requirements.md deleted file mode 100644 index e16b9fa7..00000000 --- a/.kiro/specs/raft-commit-safety/requirements.md +++ /dev/null @@ -1,112 +0,0 @@ -# Requirements Document - -## Introduction - -`src/lib/raft/raft.{c,h}` is the pure, callback-driven Raft library for XCE. The `raft-library` spec gave it the **election half** of `raft/raft.tla` (Ongaro's canonical Raft TLA⁺), and the upstream `raft-log-replication` spec gave it the **log half**: a library-owned log where every entry carries a `uint64_t` term (1-based index = WAL LSN), `ClientRequest` append (`raft_log_append`), `AppendEntries` emission/handling, per-peer `next_index[]` / `match_index[]`, the `on_get_entry_term` / `on_persist_entry` / `on_send_append_entries` / `on_truncate_from` callbacks, and `UpdateTerm` on every RPC. That upstream spec deliberately carries `leader_commit` **opaquely** through `on_send_append_entries` and `raft_on_append_entries_request` **without acting on it** — it never advances a commit index. - -This feature closes the remaining two `raft.tla` **safety** gaps from the gap analysis (`.kiro/specs/ha-term-safety/TLA-gap-analysis.md`): - -- **G4 — term-blind commit advance.** `raft.tla` `AdvanceCommitIndex` (line 259) advances `commitIndex` to `Max(agreeIndexes)` **only if `log[Max(agreeIndexes)].term == currentTerm`** — the Figure-8 safety rule: a leader must never mark a previous-term entry committed by replica-count alone; it commits the older entry only indirectly, once an entry from its **own** current term reaches a quorum. XCE's commit barrier + `ha_compute_safe_committed_lsn` advanced `committed` term-blind ("from 1 granted voter") because the WAL term was dead (G1); the own-term `WAL_NOOP` gate is only a partial mitigation. This spec adds `raft_advance_commit(...)` to the library, computing `agreeIndexes` from per-peer `match_index[]` and advancing `commit_index` **only when the entry at the max agreed index is from the current term**, with a new `on_commit_advanced(ctx, commit_index)` callback that **triggers** the existing synchronous commit barrier. - -- **G6 — LSN-only vote freshness.** `raft.tla` `HandleRequestVoteRequest` `logOk` (line 285) compares the candidate's `(lastLogTerm, lastLogIndex)` against the voter's **lexicographically**: grant only if the candidate's last log term is strictly greater, or the terms tie and the candidate's last log index is at least the voter's. XCE's `on_vote_eligible` gate uses `config_tag ≥` plus an applied/durable-LSN floor **only** — it never compares `lastLogTerm`. With G6 a node whose log has a higher term but a lower index can be denied (correct) or, worse, a stale-term-but-higher-index candidate can be wrongly granted. This spec upgrades the freshness gate on **both** the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path to carry and compare `last_log_term` alongside the existing `last_log_index`. - -The library core stays **pure / synchronous / deterministic** (no I/O, no threads, no allocation in the hot path): `raft_advance_commit` reads entry terms only through the existing `on_get_entry_term` callback and reports the new commit index only through `on_commit_advanced`; the controller adapter performs the durable commit work. The feature is verified end-to-end in the deterministic simulator (`raft_sim.c`) against **Leader Completeness** and the Figure-8 scenario, without a live cluster. - -## Boundary Context - -- **In scope**: - - `raft_advance_commit(...)`: compute `agreeIndexes` from the library's per-peer `match_index[]` (a quorum agrees, including self), and advance `commit_index` to `Max(agreeIndexes)` **only when `on_get_entry_term(Max(agreeIndexes)) == currentTerm`** (the Figure-8 current-term rule, `raft.tla` `AdvanceCommitIndex` line 259, closes G4). Otherwise hold `commit_index` unchanged. - - A new `commit_index` field in the library state and the new required callback `void on_commit_advanced(void *ctx, uint64_t commit_index)` fired **after** `commit_index` strictly advances, so the controller adapter feeds it into the **existing synchronous commit barrier / persist-before-ACK durability** (the callback triggers commit-evaluation; it does not perform or replace the barrier). - - `(lastLogTerm, lastLogIndex)` vote freshness (`raft.tla` `HandleRequestVoteRequest` `logOk` line 285, closes G6): carry `last_log_term` alongside `last_log_index` into the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path, and grant only on the lexicographic `(lastLogTerm, lastLogIndex)` comparison. - - Simulator coverage in `raft_sim.c`: a **Figure-8** scenario (a previous-term entry replicated to a quorum but not yet committed by the current leader) asserting the library refuses to commit it until a current-term entry is committed; and a **Leader Completeness** assertion (a committed entry is present in every future leader's log) across forced leader changes. - -- **Out of scope** (owned elsewhere): - - The library log model + `ClientRequest` / `AppendEntries` / `HandleAppendEntries*` / `UpdateTerm`-on-any-RPC, the per-entry term, and the per-peer `next_index[]` / `match_index[]` themselves → `raft-log-replication` (consumed here verbatim; not redefined). - - The synchronous commit-barrier mechanism, fsync/group-commit, persist-before-ACK durability, the reply gate, and snapshot transport → controller (`ha-term-safety` adapter). This spec only **triggers** the barrier via `on_commit_advanced`; it neither implements nor weakens it. - - Configuration-change markers in the unified log and the upgrade snapshot → `ha-term-safety`. - - The full conformance / TLC trace-check harness → `raft-sim-conformance`. - - The replacement of `ha_compute_safe_committed_lsn`'s term-blind advance in the controller with the library's current-term-gated `commit_index` → that controller wiring is `ha-term-safety`; this spec defines only the library entry point and callback the controller will route through. - -- **Adjacent expectations**: - - Builds on `raft-log-replication` (HARD dependency): needs the `(term, index)` library log, `on_get_entry_term`, and per-peer `match_index[]`. All of that contract is reused verbatim and not redefined here. - - Preserves as policy (must NOT be canonicalized away): the synchronous commit barrier + persist-before-ACK durability are **stronger** than canonical Raft's async apply and remain — `on_commit_advanced` triggers the existing barrier, it does not replace it. PreVote and priority leadership are unchanged in mechanism; only the freshness predicate they consult gains `last_log_term`. - - "Controller never self-terminates": the library returns status codes only; it never aborts a process. The controller owns liveness/fencing. - - Feeds `raft-sim-conformance` (Leader Completeness oracle) and `ha-term-safety` (the controller integration that routes the WAL/commit barrier through `raft_advance_commit` + `on_commit_advanced`). Closes bug-851 G4/G6. - -## Requirements - -### Requirement 1: Library-owned commit index with the current-term advance rule (closes G4) - -**Objective:** As the leader of a Raft cluster, I want my commit index to advance only over an index that a quorum holds **and** whose entry is from my current term, so that a previous-term entry is never marked committed by replica-count alone (Figure-8 safety) and Leader Completeness holds. - -#### Acceptance Criteria - -1. The Raft Library shall maintain, per node, a `commit_index` (the `raft.tla` `commitIndex`) initialized to the value restored from the adapter at boot (default 0), so the library — not term-blind controller code — owns the commit position. -2. When `raft_advance_commit(...)` is called while the node's role is `RAFT_LEADER`, the Raft Library shall compute `agreeIndexes` as the set of indexes `index` in `1..log_len` for which the set `{self} ∪ {peer : match_index[peer] ≥ index}` is a quorum, mirroring `raft.tla` `AdvanceCommitIndex` `agreeIndexes`. -3. When `agreeIndexes` is non-empty and `on_get_entry_term(Max(agreeIndexes)) == currentTerm`, the Raft Library shall set `commit_index` to `Max(agreeIndexes)`, mirroring the `raft.tla` `AdvanceCommitIndex` `newCommitIndex` then-branch (the Figure-8 current-term rule). -4. If `agreeIndexes` is empty, or `on_get_entry_term(Max(agreeIndexes)) != currentTerm`, then the Raft Library shall leave `commit_index` unchanged, mirroring the `raft.tla` `AdvanceCommitIndex` else-branch (refusing to commit a previous-term entry by replica-count alone). -5. The Raft Library shall never decrease `commit_index`; `raft_advance_commit` shall be monotonic, only ever moving `commit_index` forward. -6. If `raft_advance_commit(...)` is called while the node's role is not `RAFT_LEADER`, then the Raft Library shall leave `commit_index` unchanged and report a non-success status, mirroring the `raft.tla` `AdvanceCommitIndex` `state[i] = Leader` precondition. -7. The Raft Library shall obtain the term of the entry at `Max(agreeIndexes)` exclusively through the existing `on_get_entry_term` callback and shall perform no storage I/O itself. - -### Requirement 2: on_commit_advanced callback triggering the existing commit barrier - -**Objective:** As the XCE controller adapter, I want the library to notify me exactly when the committed position moves forward, so that I can drive the existing synchronous commit barrier / persist-before-ACK durability off the library's safe commit index without the library performing or weakening that durability work. - -#### Acceptance Criteria - -1. The Raft Library shall define the required callback `void on_commit_advanced(void *ctx, uint64_t commit_index)`, fired with the new `commit_index` after it strictly advances. -2. When `raft_advance_commit(...)` advances `commit_index` to a strictly greater value, the Raft Library shall invoke `on_commit_advanced(ctx, commit_index)` exactly once with the new value. -3. If `raft_advance_commit(...)` does not change `commit_index` (held by the current-term rule, empty `agreeIndexes`, or non-leader), then the Raft Library shall not invoke `on_commit_advanced`. -4. The Raft Library shall treat `on_commit_advanced` as a trigger only: it shall not itself fsync, persist entries, send the client ACK, or otherwise implement the commit barrier — those remain the controller adapter's responsibility (persist-before-ACK durability preserved as policy). -5. The Raft Library shall validate `on_commit_advanced` as a required, non-NULL callback at `raft_init`, returning `-EINVAL` when it is NULL, consistent with how existing required callbacks are validated. -6. The Raft Library shall not advance `commit_index` or fire `on_commit_advanced` as a side effect of `raft_on_append_entries_reply`; updating per-peer `match_index[]` and advancing the commit index remain separate steps (mirroring `raft.tla`, where `AdvanceCommitIndex` is a distinct action from `HandleAppendEntriesResponse`). - -### Requirement 3: (lastLogTerm, lastLogIndex) vote freshness on the binding vote path (closes G6) - -**Objective:** As any cluster member receiving a `RequestVote`, I want to grant my vote only to a candidate whose log is at least as up-to-date as mine by the `(lastLogTerm, lastLogIndex)` comparison, so that an election can never elect a leader missing a committed entry (Leader Completeness). - -#### Acceptance Criteria - -1. When `raft_on_vote_request(...)` evaluates freshness, the Raft Library shall compute `logOk = (cand_last_log_term > local_last_log_term) ∨ (cand_last_log_term == local_last_log_term ∧ cand_last_log_index ≥ local_last_log_index)`, mirroring the `raft.tla` `HandleRequestVoteRequest` `logOk` predicate (closes G6). -2. The Raft Library shall accept the candidate's `last_log_term` alongside the existing `last_log_index` on the vote-request path so the `(lastLogTerm, lastLogIndex)` comparison can be made (extending the eligibility signature or the vote-request payload accordingly). -3. The Raft Library shall obtain the voter's own `local_last_log_term` and `local_last_log_index` from its library log model (`on_get_entry_term(log_len)` and `log_len`, with term `0` when the log is empty), consistent with `raft.tla` `LastTerm(log)`. -4. If `logOk` is false, then the Raft Library shall deny the vote, even when all other Raft invariants (correct term, no prior vote this term) and application gates would otherwise grant it. -5. The Raft Library shall apply the `(lastLogTerm, lastLogIndex)` freshness check without weakening the existing election-safety invariants (stale-term deny, one-vote-per-term, persist-before-grant) or the existing application policy gate (primary-deny / heartbeat stickiness / catch-up), which remain in force. -6. The Raft Library shall preserve the existing persist-before-grant ordering: a vote that passes the `(lastLogTerm, lastLogIndex)` freshness check is persisted via `on_persist` before being granted. - -### Requirement 4: (lastLogTerm, lastLogIndex) freshness on the PreVote eligibility path (closes G6 for PreVote) - -**Objective:** As a node answering a PreVote straw poll, I want to apply the same `(lastLogTerm, lastLogIndex)` freshness rule as the binding vote, so that PreVote never green-lights a candidacy that a real election would reject for log staleness — without mutating any persistent state. - -#### Acceptance Criteria - -1. When `raft_on_pre_vote_request(...)` evaluates eligibility, the Raft Library shall apply the same `(lastLogTerm, lastLogIndex)` freshness predicate as the binding vote path (Requirement 3.1), using the candidate's advertised `last_log_term` and `last_log_index`. -2. The Raft Library shall carry `last_log_term` alongside `last_log_index` on the PreVote eligibility path consistently with the binding vote path, so the two paths apply an identical freshness rule. -3. If the `(lastLogTerm, lastLogIndex)` freshness predicate is false, then the Raft Library shall deny the PreVote. -4. While answering a PreVote, the Raft Library shall not mutate `currentTerm`, `voted_for`, `voted_for_term`, `role`, or `commit_index`, and shall not invoke `on_persist` — preserving the existing PreVote side-effect-free invariant. -5. The Raft Library shall keep the existing PreVote stale-term deny and application eligibility gate (`on_vote_eligible`) in force; the `(lastLogTerm, lastLogIndex)` freshness is applied in addition to them, not in place of them. - -### Requirement 5: Preserve the synchronous commit barrier and durability as policy - -**Objective:** As the operator of an XCE HA cluster, I want the commit-safety upgrade to keep XCE's stronger-than-canonical durability guarantees intact, so that acknowledged work remains durable on a majority and the term-safety fix does not regress availability or durability behavior. - -#### Acceptance Criteria - -1. The Raft Library shall expose the safe commit index only as a trigger (`on_commit_advanced`); the synchronous commit barrier, persist-before-ACK durability, fsync/group-commit, and the reply gate shall remain the controller adapter's responsibility and shall not be moved into or weakened by the library. -2. The Raft Library shall not change the existing PreVote mechanism or priority-leadership behavior; the freshness predicate these paths consult gains `last_log_term`, and the locus of that freshness decision moves from the caller/dispatcher into the library (see Requirement 5.4). -3. The Raft Library shall return status codes only and shall never abort a process on any commit-safety or freshness condition, preserving the "controller never self-terminates" invariant. -4. The Raft Library shall leave all existing election and log-replication entry points and callbacks (from `raft-library` and `raft-log-replication`) behaviorally unchanged, **except** for: (a) the additive freshness arguments on the vote/PreVote paths, (b) the additive `commit_index` / `on_commit_advanced` state and callback, and (c) the **freshness-decision locus move** — the vote/PreVote entry points gain the `last_log_term` / `last_log_index` arguments **and** the library now owns the `logOk` decision that was previously the caller's/dispatcher's (the binding path no longer delegates freshness to `on_vote_eligible`, and the PreVote path no longer relies on the dispatcher applying freshness before the call). - -### Requirement 6: Deterministic Figure-8 and Leader Completeness verification in the simulator - -**Objective:** As a maintainer, I want the deterministic simulator to drive commit advance and forced leader changes and assert the Figure-8 refusal and Leader Completeness, so that the commit-safety and freshness rules are verifiable without a live cluster. - -#### Acceptance Criteria - -1. The Raft Simulator shall drive `raft_advance_commit` on the leader after replication so the commit index advances deterministically over the in-memory per-node log backing the existing callbacks. -2. The Raft Simulator shall construct a **Figure-8** scenario — a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry — and shall assert that the library does **not** advance `commit_index` over that previous-term entry until an entry from the current term is committed. -3. When the current-term entry that covers the previous-term entry is replicated to a quorum and `raft_advance_commit` is driven, the Raft Simulator shall assert that `commit_index` then advances over both entries (indirect commit of the previous-term entry). -4. The Raft Simulator shall assert the **Leader Completeness** invariant across forced leader changes: any entry that was committed (present at or below `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader. -5. The Raft Simulator shall exercise the `(lastLogTerm, lastLogIndex)` freshness gate: a candidate with a stale last-log-term shall be denied the vote even when its last-log-index is greater, and a candidate with a fresher last-log-term shall be granted regardless of a smaller last-log-index, asserting the lexicographic comparison. -6. The Raft Simulator shall assert that `on_commit_advanced` fires exactly when `commit_index` strictly advances and does not fire when the current-term rule holds the commit index, so the trigger semantics consumed by the controller barrier are verified. diff --git a/.kiro/specs/raft-commit-safety/research.md b/.kiro/specs/raft-commit-safety/research.md deleted file mode 100644 index 6423983b..00000000 --- a/.kiro/specs/raft-commit-safety/research.md +++ /dev/null @@ -1,53 +0,0 @@ -# Gap Analysis — raft-commit-safety (2026-06-24) - -> Scope: the **commit-safety (G4 Figure-8 commit advance) + election-freshness (G6 `(lastLogTerm,lastLogIndex)` vote/PreVote)** half of `raft/raft.tla`, in `src/lib/raft` only. Boundary owns the library entry points + callbacks; the controller barrier/durability, config markers/snapshot, and conformance harness are owned elsewhere (`ha-term-safety`, `raft-sim-conformance`). HARD dep: `raft-log-replication`. - -## Executive Summary - -This spec is **DONE**. All tasks 1.1–8.1 are `[x]` in `tasks.md`, and the actual code confirms it: the library half (G4 commit-advance + G6 vote/PreVote freshness) is implemented in `src/lib/raft/{raft.h,raft.c,raft_sim.c}`, fully tested in `src/lib/testsuite/test_raft.c`, and **already consumed by the controller** via the `ha-term-safety` adapter. The historical "(0,0) stopgap disables G6 / build broken" state recorded in project memory is **stale** — it has been superseded: the controller build is green (later commits `bug-1060`/`bug-1067` build on top of it), the wire schema carries `last_log_term`/`last_log_index`, and `cluster.c` feeds the real candidate tuple into `raft_on_vote_request`. **No unsafe stopgap remains in this spec's surface.** - -## Requirement-to-Asset Map - -| Req | Asset (file:line) | Status | -|-----|-------------------|--------| -| **R1** Library-owned `commit_index` + current-term advance (G4) | `raft.h:62` (`commit_index`), `raft.h:199` (`initial_commit_index`), `raft.c:733` `raft_advance_commit` — leader-only `-EPERM` (`raft.c:738`), `Max(agreeIndexes)` scan (`raft.c:740`), current-term gate before advance, monotonic | **DONE** | -| **R2** `on_commit_advanced` callback triggers barrier (no durability in lib) | `raft.h:169` decl (REQUIRED), `raft.c:58` NULL-check `-EINVAL` at init, `raft.c:780` fired once on strict advance; AE-reply does not advance (`test_ae_reply_does_not_advance_commit`) | **DONE** | -| **R3** `(lastLogTerm,lastLogIndex)` binding-vote freshness (G6) | `raft.c:157` `raft_log_ok`, applied in `raft_on_vote_request` (`raft.c:234`) BEFORE `on_vote_eligible`; extended sig `raft.h:267` | **DONE** | -| **R4** Same freshness on PreVote, side-effect-free | `raft_on_pre_vote_request` (`raft.c:291` `raft_log_ok`), no state mutation / no `on_persist`; sig `raft.h:329` | **DONE** | -| **R5** Preserve sync commit barrier + durability as policy | Library returns status codes only; barrier remains in controller `commit.c`/`ha.c`; `on_commit_advanced` is trigger-only | **DONE** | -| **R6** Deterministic Figure-8 + Leader Completeness + freshness sim | `test_raft.c`: `test_advance_commit_figure8_hold_previous_term` (2765), `test_sim_7_1_figure8_commit_refusal` (4095), `test_sim_7_2_leader_completeness_across_changes` (4161), `test_sim_7_3_freshness_gate_lexicographic` (4245), `test_sim_7_4_on_commit_advanced_trigger_semantics` (4345); all `RUN_TEST`-registered | **DONE** | - -No gaps tagged Missing/Unknown within the spec boundary. - -## Downstream Consumption (out-of-scope here, but verified present) - -The controller integration that routes the WAL/commit barrier through the library is owned by **`ha-term-safety`**, and it is wired: - -- **Wire schema**: `src/schema/controller.capnp:833-834` adds `lastLogTerm @4` / `lastLogIndex @5` to the RequestVote message. -- **Vote path**: `src/controller/cluster.c:395-396,435-436` reads the candidate's `last_log_term`/`last_log_index` from the wire and passes the real tuple into `raft_on_vote_request(...)`. **No `(0,0)` stopgap** — the application freshness gate (`config_tag`/`durable_floor`) is layered IN ADDITION to the library `logOk`, not in place of it. -- **PreVote path**: `cluster.c:489,533` plumbs the same tuple into `raft_on_pre_vote_request`. -- **Commit source**: `src/controller/ha.c:1686` `ha_cb_commit_advanced_stub` (name is legacy — it is a real, wired callback) calls `commit_set_raft_commit_index(...)`, making the library's current-term-gated `commit_index` the steady-state commit SOURCE; the term-blind `ha_compute_safe_committed_lsn` is no longer the steady-state driver. Callback table at `ha.c:1741`. -- **Build state**: green. Later commits `61e1391 fix bug-1060 bug-1067` and the `ha-term-safety` series build on top of the extended vote signature, confirming the earlier "vote-signature extension BROKE cluster.c" condition is resolved. - -## Options (retrospective — for any follow-up only) - -- **Option A — Extend existing (CHOSEN, complete).** The library log/election core already existed (`raft-library` + `raft-log-replication`); this spec extended `raft.{c,h}` with one new action, one callback, and two freshness args. Trade-off accepted: keeps everything in the pure library, no new TU. ✅ minimal surface, leverages existing callback contract. -- **Option B — Library owns `logOk` (CHOSEN for freshness plumbing).** Vote/PreVote signatures gained `cand_last_log_term`/`cand_last_log_index` and the library now owns the `logOk` decision (formerly the caller/dispatcher's). This is the "freshness-decision locus move" called out in R5.4. ✅ single authoritative predicate (`raft_log_ok`) shared by both paths. -- **Option C — Hybrid / new component.** Not needed; rejected implicitly. A separate commit module would have split the `commit_index` invariant away from the log model it depends on. - -## Effort & Risk - -- **Effort: S (already delivered).** Remaining work for *this spec* = none. Verification re-run only. -- **Risk: Low.** Pure, synchronous, deterministic library code with full simulator coverage (Figure-8 hold/release, Leader Completeness across forced leader changes, lexicographic freshness, trigger-fire-count). No I/O, no threads; controller integration validated downstream by `ha-term-safety` live soak. - -## Recommendations for design phase - -- **No further design needed for raft-commit-safety.** Treat as closed; G4/G6 of bug-851 are addressed at the library layer. -- **Carry-forward (owned elsewhere, not this spec):** - - `ha-term-safety` owns the controller wiring already in place — keep the bug-828/854/991/1060/1067 follow-ups there. - - `raft-sim-conformance` owns the full TLC trace-check harness (Leader Completeness oracle) — this spec only seeds the deterministic sim assertions it will consume. -- **Stale-memory correction (research item closed):** the project-memory note "vote-signature extension BROKE the controller build (cluster.c); (0,0) stopgap unsafe—disables G6" is **no longer current**. Verified 2026-06-24: build green, real tuple wired, no stopgap. Update the memory entry accordingly. - -## Research-Needed - -- None within this spec's boundary. (Open HA items — bug-828 residual / bug-854 client discovery — are tracked under `ha-term-safety`, not here.) diff --git a/.kiro/specs/raft-commit-safety/spec.json b/.kiro/specs/raft-commit-safety/spec.json deleted file mode 100644 index e4c903fd..00000000 --- a/.kiro/specs/raft-commit-safety/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "raft-commit-safety", - "created_at": "2026-06-21T12:00:00Z", - "updated_at": "2026-06-25T00:00:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation_status": "implemented — commit-safety + vote-freshness half landed + validated (18/18 tasks, raft_sim GO per memory project_raft_commit_safety_done). Phase reconciled 2026-06-25 (was mislabeled tasks-generated).", - "note": "Auto-approved (-y mode). Adds the commit-safety + election-freshness half of raft.tla on top of raft-log-replication's library log. raft_advance_commit (AdvanceCommitIndex, raft.tla:259) advances commit_index to Max(agreeIndexes from match_index[]) ONLY when on_get_entry_term(Max)==currentTerm (Figure-8 current-term rule, G4); new callback on_commit_advanced(ctx, commit_index) TRIGGERS the existing synchronous commit barrier / persist-before-ACK durability (does NOT replace it). Vote/PreVote freshness upgraded from LSN-only to (lastLogTerm,lastLogIndex) lexicographic compare (HandleRequestVoteRequest logOk, raft.tla:285, G6) carried via last_log_term in raft_on_vote_request + on_vote_eligible/vote-request payload. Simulator asserts Figure-8 refusal + Leader Completeness. Boundary: owns commit-index advance + vote freshness in src/lib/raft ONLY; NOT the log model/AppendEntries (raft-log-replication), the barrier/durability mechanism (controller), config markers/snapshot (ha-term-safety), or the conformance harness (raft-sim-conformance). Depends: raft-log-replication (HARD). Closes bug-851 G4/G6." -} diff --git a/.kiro/specs/raft-commit-safety/tasks.md b/.kiro/specs/raft-commit-safety/tasks.md deleted file mode 100644 index 44649992..00000000 --- a/.kiro/specs/raft-commit-safety/tasks.md +++ /dev/null @@ -1,165 +0,0 @@ -# Implementation Plan - -> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/lib/testsuite/test_raft.c` (and the simulator backing in `raft_sim.c`) that fails before the `raft.c` change and passes after. Build/run library tests via `/xce-build lib`. -> -> **Spec boundary** (applies to ALL tasks): this spec owns the commit-index advance (`raft.tla` `AdvanceCommitIndex` + the current-term Figure-8 rule, closes G4) and the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness (`raft.tla` `HandleRequestVoteRequest` `logOk`, closes G6) **in `src/lib/raft` only**. It does **NOT** own the log model / AppendEntries / per-peer `match_index[]` (those come from `raft-log-replication` and are reused verbatim), the synchronous commit-barrier / durability mechanism (controller — only TRIGGERED via `on_commit_advanced`), config markers / upgrade snapshot (`ha-term-safety`), or the conformance harness (`raft-sim-conformance`). No task here edits any controller file; the controller adapter is described, not built. -> -> **Depends (HARD): `raft-log-replication`** — needs `log_len`, `match_index[]`, and `on_get_entry_term` already present in `raft.{c,h}` and the in-memory per-node log + AppendEntries routing already in `raft_sim.c`. - -## 1. Header contract: commit_index, on_commit_advanced, raft_advance_commit, freshness signatures - -- [x] 1.1 Extend `raft.h` with the commit-index state and restore field - - Add `uint64_t commit_index;` to `raft_state_t` - - Add `uint64_t initial_commit_index;` to `raft_config_t` (restored committed LSN at boot, default 0; library never reads storage — consistent with `initial_term`/`initial_log_len`) - - Observable completion: `raft.h` compiles; `sizeof(raft_state_t)` grows by the new field; no entry-byte storage added - - _Requirements: 1.1, 1.5_ - - _Boundary: raft.h_ - -- [x] 1.2 Add the `on_commit_advanced` REQUIRED callback to `raft_callbacks_t` (verbatim contract) - - Declare `void on_commit_advanced(void *ctx, uint64_t commit_index)` with the exact signature from design.md / brief - - Document it as REQUIRED, fired only on strict advance, and as a TRIGGER for the existing commit barrier (never performs durability itself) - - Observable completion: the function-pointer member exists in `raft_callbacks_t` with the verbatim signature; header doc comment matches the Authoritative shared contract - - _Requirements: 2.1, 2.4, 5.1_ - - _Boundary: raft.h_ - - _Depends: 1.1_ - -- [x] 1.3 Declare `raft_advance_commit` and extend the vote/PreVote signatures with last_log_term/last_log_index - - Declare `int raft_advance_commit(raft_state_t *s)` - - Extend `raft_on_vote_request` and `raft_on_pre_vote_request` to accept `cand_last_log_term` and `cand_last_log_index` (the chosen "library owns logOk" plumbing — Option B in design.md) - - Update the header doc comments to describe the new logOk gate and the commit-advance action - - Observable completion: header declares `raft_advance_commit` and the extended vote/PreVote signatures; a stub TU including `raft.h` compiles - - _Requirements: 1.2, 2.2, 3.2, 4.2_ - - _Boundary: raft.h_ - - _Depends: 1.2_ - -## 2. Init validation and commit-index restore - -- [x] 2.1 RED+GREEN: `raft_init` requires `on_commit_advanced` and restores `initial_commit_index` - - Test: `raft_init` returns `-EINVAL` when `on_commit_advanced` is NULL; returns 0 when present; `s->commit_index == cfg->initial_commit_index` after init - - Implement the NULL check in `raft_init` (mirroring the existing required-callback checks); restore `s->commit_index = cfg->initial_commit_index` - - Observable completion: new `-EINVAL` test case passes; restore assertion passes; existing election + log-replication init tests still green - - _Requirements: 2.5, 1.1_ - - _Boundary: raft.c, test_raft.c_ - - _Depends: 1.3_ - -## 3. AdvanceCommitIndex with the current-term Figure-8 rule (raft.tla:259/270, closes G4) - -- [x] 3.1 RED: commit-advance tests (current-term advance, Figure-8 hold, monotonic, non-leader) - - Test (advance): leader with `match_index[]` quorum-agreeing through index `k`, entry at `k` from `currentTerm` → `commit_index` advances to `k`, `on_commit_advanced(k)` fired once - - Test (Figure-8 hold): quorum agrees through a previous-term entry at `k` with no current-term entry above → `commit_index` unchanged, `on_commit_advanced` NOT fired - - Test (empty agreeIndexes): no index reaches quorum → unchanged, no callback - - Test (monotonic): a later call computing a smaller `Max(agreeIndexes)` never lowers `commit_index` - - Test (non-leader): `raft_advance_commit` returns `-EPERM`, no change, no callback - - Observable completion: tests compile and FAIL (no implementation yet) - - _Requirements: 1.2, 1.3, 1.4, 1.5, 1.6, 2.2, 2.3_ - - _Boundary: test_raft.c_ - - _Depends: 2.1_ - -- [x] 3.2 GREEN: implement `raft_advance_commit` - - Leader-only guard (`-EPERM` otherwise); compute `Max(agreeIndexes)` by scanning candidate indexes from `log_len` downward, seeding the agree-count with self, until a quorum `((n_peers/2)+1)` is reached - - Apply the current-term rule: advance to `Max(agreeIndexes)` only when `on_get_entry_term(Max) == s->term`; else hold - - Monotonic: only ever move `commit_index` forward; fire `on_commit_advanced(commit_index)` exactly once on a strict advance; read entry terms only via `on_get_entry_term` - - Observable completion: 3.1 tests pass; the current-term check is unmistakably present before the advance in `raft.c`; `commit_index <= log_len` holds - - _Requirements: 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3_ - - _Boundary: raft.c_ - - _Depends: 3.1_ - -- [x] 3.3 RED+GREEN: commit-advance is a separate action from append-entries reply - - Test: a `raft_on_append_entries_reply` that updates `match_index[]` to a committable quorum does NOT by itself change `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does - - Verify (no implementation change expected in `raft_on_append_entries_reply` — it is owned upstream and must stay untouched); if a regression is found, the fix belongs upstream, not here - - Observable completion: test passes; `raft_on_append_entries_reply` remains free of any commit-advance side effect - - _Requirements: 2.6_ - - _Boundary: test_raft.c_ - - _Depends: 3.2_ - -## 4. (lastLogTerm, lastLogIndex) vote freshness (raft.tla:285, closes G6) - -- [x] 4.1 RED: binding-vote freshness tests - - Test: candidate with stale `last_log_term` denied even when `last_log_index` is greater; candidate with fresher `last_log_term` granted with a smaller `last_log_index`; tie on term grants only when `cand_last_log_index >= log_len`; the voter's own last-log term/index come from `on_get_entry_term(log_len)`/`log_len` (term 0 when empty) - - Test: `!logOk` denies even when `on_vote_eligible` would grant (logOk applied BEFORE the application gate); persist-before-grant unchanged on the grant path - - Observable completion: tests compile and FAIL - - _Requirements: 3.1, 3.3, 3.4, 3.5, 3.6_ - - _Boundary: test_raft.c_ - - _Depends: 1.3_ - -- [x] 4.2 GREEN: implement the logOk gate in `raft_on_vote_request` - - Compute `local_last_log_term = (log_len > 0) ? on_get_entry_term(log_len) : 0`; `logOk = (cand_last_log_term > local_last_log_term) || (cand_last_log_term == local_last_log_term && cand_last_log_index >= log_len)` - - Insert the logOk deny BEFORE `on_vote_eligible`, after the existing stale-term/UpdateTerm/one-vote-per-term steps; leave persist-before-grant and all existing invariants in place - - Observable completion: 4.1 tests pass; no regression in existing vote tests; the logOk block is clearly before `on_vote_eligible` in `raft.c` - - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_ - - _Boundary: raft.c_ - - _Depends: 4.1_ - -## 5. (lastLogTerm, lastLogIndex) PreVote freshness (raft.tla:285, side-effect-free) - -- [x] 5.1 RED: PreVote freshness tests (identical rule, no side effects) - - Test: `raft_on_pre_vote_request` applies the SAME `(lastLogTerm, lastLogIndex)` logOk as the binding vote; a stale tuple is denied; the existing stale-term deny and `on_vote_eligible` gate remain in force - - Test: across grant and deny, no mutation of `term`/`voted_for`/`voted_for_term`/`role`/`commit_index` and no `on_persist` call (PreVote invariant) - - Observable completion: tests compile and FAIL - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_ - - _Boundary: test_raft.c_ - - _Depends: 1.3_ - -- [x] 5.2 GREEN: implement the logOk gate in `raft_on_pre_vote_request` - - Apply the identical logOk predicate (factored to match 4.2) before/with the existing eligibility gate; mutate nothing; no `on_persist` - - Observable completion: 5.1 tests pass; no regression in existing PreVote tests; the predicate matches the binding-vote path exactly - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_ - - _Boundary: raft.c_ - - _Depends: 5.1_ - -## 6. Simulator backing: commit-advance driver + freshness plumbing - -- [x] 6.1 Wire `on_commit_advanced` and a commit-advance driver in `raft_sim.c` - - Add a per-`raft_sim_node_t` `commit_index` mirror updated from `on_commit_advanced`; implement the callback (record the value + count fires for assertions) - - Add a sim helper to drive `raft_advance_commit` on a given node (or all leaders) after delivery - - Pass each node's restored `initial_commit_index` (default 0) in `raft_sim_init`; wire `on_commit_advanced` for every node - - Observable completion: after replication + delivery, driving the helper advances the leader's mirrored `commit_index`; the callback fire-count is observable - - _Requirements: 6.1, 6.6, 2.5_ - - _Boundary: raft_sim.c_ - - _Depends: 3.2_ - -- [x] 6.2 Plumb `(last_log_term, last_log_index)` through the sim vote/PreVote routing - - Update the sim's vote and PreVote delivery to read the candidate's last-log `(term, index)` from its in-memory log (`term at log_len` / `log_len`) and pass them to the extended `raft_on_vote_request` / `raft_on_pre_vote_request` - - Observable completion: existing election sim tests still pass with the extended signatures; a node with a stale last-log term is denied a vote it would previously have won on LSN alone - - _Requirements: 6.5, 3.2, 4.2_ - - _Boundary: raft_sim.c_ - - _Depends: 4.2, 5.2_ - -## 7. Simulator invariant tests (Figure-8, Leader Completeness, freshness, trigger semantics) - -- [x] 7.1 Figure-8 commit-refusal test (G4) - - Test: construct a previous-term entry replicated to a quorum but not covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does NOT move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over BOTH (indirect commit) - - Observable completion: test passes end-to-end via the sim; the held phase and the released phase are both asserted - - _Requirements: 6.2, 6.3_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 6.1_ - -- [x] 7.2 Leader Completeness across forced leader changes - - Test: drive replication to a commit, then force leader changes (timeout / partition); assert any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader - - Observable completion: test passes; no future leader is missing a committed entry across the churn - - _Requirements: 6.4_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 6.2, 7.1_ - -- [x] 7.3 Freshness-gate lexicographic exercise - - Test: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome — stale-term-but-higher-index denied; fresher-term-but-lower-index granted; tie-term decided by index - - Observable completion: test passes for both vote and PreVote paths - - _Requirements: 6.5_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 6.2_ - -- [x] 7.4 Trigger-semantics test for `on_commit_advanced` - - Test: `on_commit_advanced` fires exactly once on a strict advance and does NOT fire while the current-term rule holds the commit index (Figure-8 held phase) or when not leader - - Observable completion: test passes; the callback fire-count matches the number of strict advances - - _Requirements: 6.6, 2.2, 2.3_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 6.1, 7.1_ - -## 8. Verification - -- [x] 8.1 Full library suite green under ASAN - - Run `/xce-build lib`; all existing election + log-replication tests plus the new commit-safety and freshness tests pass under ASAN with zero leaks - - Observable completion: test binary exits 0; ASAN clean; no regression in the election-core or log-replication tests - - _Requirements: 1.2, 1.3, 2.2, 3.1, 4.1, 6.2, 6.4_ - - _Boundary: src/lib/raft, src/lib/testsuite_ - - _Depends: 7.1, 7.2, 7.3, 7.4_ diff --git a/.kiro/specs/raft-drive-async/brief.md b/.kiro/specs/raft-drive-async/brief.md deleted file mode 100644 index 5afd9cb9..00000000 --- a/.kiro/specs/raft-drive-async/brief.md +++ /dev/null @@ -1,92 +0,0 @@ -# Brief: raft-drive-async - -## Problem - -The Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals (iomgr and delaymgr). When either stalls, heartbeats/votes/ACKs stop flowing → peers declare the leader dead → spurious election. This is a likely root of the **bug-851 chaotic-failover family**. - -A blocking audit (2026-06-25) found three CRITICAL violations of the (newly corrected) thread-role contract in `ha-async-io.md`: - -1. **Snapshot send + `persist_checkpoint` inline on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) runs `persist_checkpoint` (fdatasync + WAL-segment trim) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503) on the iomgr event loop. A large snapshot freezes ALL heartbeat/vote/client processing for its whole duration. -2. **Config-push wait inline on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172/1177/2672/2677/3973), up to `CONF_PUSH_TIMEOUT_MS`. -3. **Blocking `gethostbyname` + `connect` on the delaymgr heartbeat thread** — `endpoint_connect` (netio.c:289/302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unreachable peer stalls ALL heartbeats + election-timeouts for seconds. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.) - -**Root cause:** `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** on peer/vote endpoints — it runs inline on iomgr instead of dispatching to the vtask pool like normal cluster messages (`handle_msg` → `vtask_handle_msg`). So any work it does, including the heavy I/O above, executes on the event loop. - -Plus MEDIUM vtask-pool blockers (pool-exhaustion, not drive-thread): config-ack wait (config_repl.c:1459), a 30-second `usleep` checkpoint poll (config_repl.c:1661), `ha_persist_term` fsync on a vote grant (ha.c:3407 — adds election latency). - -## Current State - -- The corrected **thread-role contract** is now steering (`ha-async-io.md`, 2026-06-25): iomgr = non-blocking I/O + dispatch only; delaymgr = trigger-only, never blocks; vtask pool = bounded blocking only under admission reservation; long blocking I/O → dedicated slow lane. This spec brings the **code into compliance** with that contract. (The contract SUPERSEDED the old "delaymgr for blocking ops = Permitted" rule, which was the design flaw behind these findings — delaymgr is both the heartbeat driver and was the documented blocking sink.) -- Existing primitives to reuse: `COMMIT_WAIT_RESERVED` admission cap (commit.c:195-203) already keeps vtask-pool headroom against blocking waits; the non-blocking-connect + bounded-`poll` pattern already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); `vtask_handle_msg` (vtaskmgr.c:356) is the dispatch path but is endpoint+inbound-packet-bound (`vtype=VTASK_IO`). -- Confirmed already-correct: RequestVote fan-out, heartbeat fan-out, WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + fdatasync run on a vtask worker, not iomgr. - -## Desired Outcome - -The protocol drive is non-blocking on both liveness threads, by construction: - -- **iomgr** does only non-blocking socket I/O + dispatch — `peer_ctrl_msg_handler`'s real work is dispatched to the vtask pool (conforming to the existing `handle_msg` pattern), so no future handler addition can reintroduce inline blocking. -- Inside the vtask, **fast protocol work runs inline** (decode ack → update match_index → advance commit → send next heartbeat — all bounded); **long I/O is triggered to a slow lane** (snapshot transfer, config-push), never run in the request-pool worker. -- **delaymgr** only fires timers and triggers work; the peer connect is **bounded** (non-blocking + `poll` timeout) and **offloaded** off the timer thread. -- Heartbeat liveness is provably protected: a partitioned (unreachable, not merely dead) peer cannot stall heartbeats to healthy peers or trigger a spurious election. - -## Approach - -Bring the code into compliance with the steering thread-role contract, structurally rather than per-op: - -1. **Dispatch `peer_ctrl_msg_handler` to the vtask pool** (the structural fix). iomgr becomes epoll + enqueue only. Preserve per-connection message ordering (VTASK_IO per-endpoint serialization is a feature here). Marshal any iomgr-owned state mutations (epoll/endpoint lifecycle) back to the iomgr thread (bug-646 discipline). -2. **Two-tier execution inside the vtask:** fast bounded protocol work inline; **trigger** long I/O (snapshot send + `persist_checkpoint`, config-push) to a **dedicated slow-lane / admission-capped executor** so multi-second tasks cannot starve the request pool. Per-peer dedup (one snapshot in flight per standby); coordinate with ongoing WAL-catchup stream and the primary's appends/WAL-trim. -3. **delaymgr connect → bounded + offloaded:** reuse the `xce_probe_config_tag` non-blocking-connect + bounded-`poll` pattern; trigger the reconnect as an idempotent, refcounted, deduped task off the delaymgr thread; marshal the connected-fd epoll registration back to iomgr. DNS (`gethostbyname`) is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — defer async DNS unless measured. -4. **General (non-IO) task-submit path:** add the submit primitive needed for #2/#3 (current `vtask_handle_msg` requires an inbound packet); or generalize it. -5. **Admission discipline:** any offloaded blocking task lives under a reservation (extend the `COMMIT_WAIT_RESERVED` pattern) so liveness-critical messages always have a free worker. - -## Scope - -- **In**: - - Dispatch `peer_ctrl_msg_handler` real work to the vtask pool (root-cause structural fix). - - Snapshot send + `persist_checkpoint` off iomgr → slow lane, with per-peer dedup + WAL coordination. - - Config-push wait off iomgr → slow lane / async ACK. - - delaymgr peer connect → bounded non-blocking + offloaded; reconnect dedup/refcount/marshal-back. - - General non-IO vtask submit path + admission reservation for offloaded blocking tasks. - - Partition (unreachable-peer) regression test; verify no spurious election + heartbeats to survivors continue. -- **Out**: - - The commit-barrier async conversion (P1 / `repl_wait_majority_ack` → `resolve`) — owned by `raft-apply-decoupling` as its keystone (different concern: that's the *commit* drive, this is the *transport/timer* drive). - - Apply-after-commit / hook decoupling / facts-through-raft (raft-apply-decoupling). - - Async DNS (deferred unless measured). - - The MEDIUM vtask-pool blockers beyond what the admission discipline naturally covers (note them; fix opportunistically). - - Entity-bearing snapshots / config-upgrade snapshot (separate specs). - -## Boundary Candidates - -- The `peer_ctrl_msg_handler` → vtask dispatch seam. -- The dedicated slow-lane executor + admission reservation. -- The general non-IO vtask-submit primitive. -- The bounded-connect + offloaded-reconnect path. -- The iomgr-owned-state marshal-back mechanism (epoll/endpoint lifecycle). - -## Out of Boundary - -- Commit-engine internals (P1 owns the commit-drive async). -- Snapshot *content* format (entity-bearing snapshots is a separate spec) — this spec only moves the *transfer* off the drive thread. - -## Upstream / Downstream - -- **Upstream**: the `ha-async-io.md` thread-role contract (authority); existing `COMMIT_WAIT_RESERVED` (commit.c) and `xce_probe_config_tag` bounded-connect pattern (reuse); `election-iomgr-refactor` (COMPLETED upstream — provides reusable primitives `delay_task_ms`, `on_connected`, `ha_start_election_async`/non-blocking vote fan-out). -- **Downstream**: `raft-apply-decoupling` (its async commit drive + apply pump reuse the slow-lane + admission infrastructure this spec establishes; and the drive must be non-blocking before apply is restructured on top of it); safe failover for the whole bug-851 family depends on this landing. - -## Existing Spec Touchpoints - -- **Adjacent (RESOLVED standalone 2026-06-25)**: `election-iomgr-refactor` is ADJACENT-SAME-AREA — it fixed the *startup* blocking path and is fully implemented; this spec fixes the *runtime* drive-thread paths it never touched. Keep separate; consume it as upstream. (Scope-check verdict in research.md.) -- **Coordinate**: `ha-term-safety` (touches the same `peer_ctrl_msg_handler` / repl dispatch region); `ha-wal-replication` (snapshot/catch-up transport); `controller-status` (heartbeat/committed propagation on the same drive). -- **Verification home**: `raft-sim-conformance` (add the partition/liveness assertion) and `/xce-test-ha` (partition variant). - -## Constraints - -- The corrected thread-role contract (`ha-async-io.md`) is authoritative — iomgr and delaymgr never block inline; long I/O on the slow lane; bounded blocking in vtask only under admission reservation. -- "Controllers never self-terminate" — an unreachable peer is a loud retry, never an abort. -- Preserve per-connection message ordering when dispatching `peer_ctrl_msg_handler` to vtask. -- Must not regress the heartbeat fast path measurably at cluster scale (3–5 peers); per-message vtask overhead accepted in exchange for uniformity + liveness safety. -- Verified by a **partition** (unreachable, not killed) soak — the case `/xce-test-ha`'s current kill-only soak does not reproduce. - -## Provenance - -Surfaced 2026-06-25 during the `raft-apply-decoupling` discovery as prerequisite **P2** (drive hardening); promoted to its own spec because it threatens liveness independently of the apply refactor. Full analysis: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § "Two distinct prerequisites" + P2 offload mechanism. Thread-role contract: `.kiro/steering/ha-async-io.md` (corrected 2026-06-25). diff --git a/.kiro/specs/raft-drive-async/design.md b/.kiro/specs/raft-drive-async/design.md deleted file mode 100644 index 1c5f9451..00000000 --- a/.kiro/specs/raft-drive-async/design.md +++ /dev/null @@ -1,385 +0,0 @@ -# Design Document — raft-drive-async - -## Overview - -**Purpose**: Make the XCE controller's Raft protocol **drive** fully event-driven by removing every blocking operation from the two threads that carry cluster liveness — the iomgr epoll loop and the delaymgr timer — so that a slow/large transfer or an unreachable peer can no longer stall heartbeats and trigger a spurious election. - -**Users**: Operators of multi-node controller clusters, who today see false failover under partition or large-snapshot conditions (the likely root of the bug-851 chaotic-failover family). - -**Impact**: Changes *how* drive-thread work is scheduled, not *what* the protocol does. `peer_ctrl_msg_handler`'s real work moves from inline-on-iomgr to the vtask pool; long blocking I/O (snapshot transfer, config-push, peer connect) moves to a new dedicated **slow-lane executor**; the delaymgr peer connect becomes bounded and offloaded. All existing consensus semantics, durability (persist-before-ACK), and the synchronous commit barrier are preserved unchanged. - -### Goals -- No blocking operation executes inline on the iomgr loop or the delaymgr timer (R1, R2, R5). -- Long blocking I/O runs on a bounded, admission-controlled lane isolated from the request vtask pool (R3). -- Per-connection processing order and consensus correctness preserved when peer control moves to the pool (R4). -- A network-partition (unreachable-peer) regression test proves heartbeat liveness and no spurious election (R7). -- No regression of existing HA features; steady-state heartbeat path unchanged (R6, R8). - -### Non-Goals -- Commit-barrier async (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` (P1). -- Apply-after-commit / hook decoupling / facts-through-raft. -- Asynchronous DNS resolution — deferred (HPC peers use IP/`/etc/hosts`); only `connect` is bounded here. -- Snapshot *content*/format (entity-bearing snapshots) — this spec moves the *transfer* off the drive threads only. -- The MEDIUM vtask-pool blockers (config-ack wait, 30 s checkpoint poll, term fsync on vote) beyond what the lane and existing `COMMIT_WAIT_RESERVED` already cover. -- The admin `OP_upgradeConfig` push path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459) — it has a synchronous requester and stays a vtask-blocking operation under `COMMIT_WAIT_RESERVED`; only the requester-less iomgr transport push is offloaded. - -## Boundary Commitments - -### This Spec Owns -- The scheduling discipline of drive-thread work: dispatching `peer_ctrl_msg_handler` to the vtask pool; the two-tier rule (fast protocol inline in the vtask, long I/O triggered to the slow lane). -- A new **slow-lane executor** component: a bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight deduplication. -- The bounded, offloaded peer-connect path (reorder of `socket_connect` + offloaded reconnect). -- The network-partition regression scenario in the soak harness. - -### Out of Boundary -- The commit engine and `repl_wait_majority_ack` (untouched; P1 owns its async conversion). -- WAL/replication protocol content, snapshot format, config-upgrade materialization. -- Election/vote protocol logic (consumed as-is; `election-iomgr-refactor` already made vote fan-out async). -- The durability guarantee and synchronous commit barrier (preserved, not modified). - -### Allowed Dependencies -- `ha-async-io.md` thread-role contract (authority). -- Existing primitives reused, not modified in contract: `endpoint_block_tasks`/`endpoint_unblock_tasks` (per-endpoint serialization), lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (thread-safe epoll registration), `vtask_handle_msg`/VTASK_IO dispatch, `delay_task_ms` (non-blocking trigger only), `COMMIT_WAIT_RESERVED` (admission pattern), the `xce_probe_config_tag` bounded-connect pattern, the `fsync_worker`/`push_thread` background-thread precedent. -- `election-iomgr-refactor` (completed upstream): `delay_task_ms`, `on_connected`, non-blocking vote fan-out. - -### Revalidation Triggers -- Any change to the slow-lane submit contract or its admission cap. -- Any change that makes a drive thread (iomgr/delaymgr) execute work synchronously again. -- Changes to per-connection ordering semantics of peer-control dispatch (affects consensus correctness assumers: `ha-term-safety`, `raft-apply-decoupling`). -- A change to `socket_connect`'s blocking/timeout behavior (affects all peer connection callers). - -## Architecture - -### Existing Architecture Analysis - -Three drive threads exist (`ha-async-io.md`): **iomgr** (single epoll loop), **vtaskmgr** (N-worker pool, default 8), **delaymgr** (single timer). The corrected thread-role contract (2026-06-25) requires iomgr and delaymgr to never block inline. Current violations: -- `peer_ctrl_msg_handler` is registered directly as `ep->msg_handler` (ha.c:4265/4307/6349) → runs inline on iomgr; it triggers snapshot send (ha.c:2702 → config_repl.c:2446) and config-push (config_repl.c:1332) synchronously on iomgr. -- `socket_connect` (netio.c:289/302) blocks (sets `O_NONBLOCK` only *after* connect, netio.c:716); reached from `primary_heartbeat_fn` on the delaymgr thread → an unreachable peer stalls all heartbeats. -- The codebase's only deferral idiom, `delay_task_ms`, runs the deferred callback *on the delaymgr thread* — a liveness thread — so it cannot host blocking work under the new contract. - -### Architecture Pattern & Boundary Map - -Selected pattern: **dispatch-and-offload** — iomgr dispatches, the vtask pool does bounded protocol work, a new dedicated slow lane absorbs long blocking I/O; completions re-enter iomgr only through the existing lock-guarded registration call. - -```mermaid -graph TB - subgraph drive[Drive threads - never block] - iomgr[iomgr epoll loop] - delaymgr[delaymgr timer] - end - subgraph pool[vtask pool - bounded work] - vtask[peer_ctrl vtask entry] - end - subgraph lane[slow-lane executor - dedicated bounded threads] - slow[blocking IO worker] - end - iomgr -->|dispatch VTASK_IO packet| vtask - vtask -->|fast protocol inline| vtask - vtask -->|trigger long IO| slow - delaymgr -->|trigger reconnect| slow - slow -->|blocking connect snapshot configpush| slow - slow -->|register connected fd via lock-guarded iomgr_monitor| iomgr -``` - -**Architecture Integration**: -- Domain boundaries: iomgr = I/O + dispatch; vtask = bounded protocol handling (serialized per endpoint by `endpoint_block_tasks`); slow lane = long blocking I/O only; delaymgr = trigger-only. -- Existing patterns preserved: VTASK_IO per-endpoint serialization, lock-guarded epoll registration, `delay_task_ms` as a non-blocking trigger, `COMMIT_WAIT_RESERVED` admission style. -- New component rationale: the slow lane is the missing *non-liveness execution lane* — the one genuine gap; using dedicated threads (not the vtask pool) keeps long ops from starving request handling (R3 by construction). -- Dependency direction: `netio` (lib) → `slowlane` → `ha`/`config_repl` (controller). Drive threads depend on the lane; the lane depends only on netio + libc. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Backend / Services | C (gnu17), pthreads | slow-lane executor; drive-thread dispatch changes | matches controller stack | -| Messaging / Events | existing iomgr/vtaskmgr/delaymgr + new slow lane | thread-role-compliant scheduling | no new external dep | -| Infrastructure / Runtime | epoll, `poll(2)`, pthread mutex/cond/sem | bounded connect, lane workers, admission | reuse existing patterns | -| Test | soak harness + firewall DROP (iptables/nft) | partition (unreachable-peer) scenario | capability-gated; see Testing | - -## File Structure Plan - -### New Files -``` -src/controller/ -├── slowlane.c # Bounded slow-lane executor: submit, worker loop, admission cap, per-key dedup -└── include/slowlane.h # slow-lane public API (submit, init, fini) -``` - -### Modified Files -- `src/lib/vtaskmgr.c` / `src/include/vtaskmgr.h` — **LivenessAdmission**: add `vtask_class_t` + a classified submit (`vtask_enq_classified`); dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus, fast-rejecting `VTASK_REQ_BLOCK` at the cap. Shared lib primitive → Revalidation Trigger. -- `src/controller/commit.c` — fold `inflight_waits`/`COMMIT_WAIT_RESERVED` into the pool reservation: `commit_submit` submits a `VTASK_REQ_BLOCK` instead of self-counting. -- `src/lib/netio.c` — `socket_connect`: set `O_NONBLOCK` before `connect`, bound with `poll(timeout)` (reuse `xce_probe_config_tag` pattern). Add a connect-timeout parameter/config. -- `src/controller/ha.c` — register peer endpoints to dispatch peer-control through the vtask pool (revise `ha.c:4265/4307/6349`); convert `peer_ctrl_msg_handler` into a vtask entry that does fast protocol work inline and `slowlane_submit`s long ops; offload snapshot send (ha.c:2702) and reconnect (`primary_heartbeat_fn`/`ha_connect_to_standby`) to the slow lane with per-peer dedup. -- `src/controller/config_repl.c` — **only the iomgr/promotion path**: run `push_config_transport_only` on the slow lane and replace the `wait_all_with_timeout` drive-thread join (reached from `peer_ctrl`/promotion at ha.c:1172/1177/2672/2677/3973) with a slow-lane-internal wait + non-blocking return. **In scope = the fire-and-forget transport push with NO waiting requester.** The admin `OP_upgradeConfig` path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459, "VTASK-CONTEXT ONLY") is **OUT of scope** — it has a synchronous admin requester awaiting the version/ACK reply and is a MEDIUM vtask-pool blocker bounded by `COMMIT_WAIT_RESERVED`; it must NOT be offloaded (doing so would break the synchronous admin reply). -- `src/controller/cluster.c` (or `myapp.c`) — init/fini the slow-lane executor in the controller lifecycle; pass connect-timeout config. -- `src/include/config.h` / `ha.h` — connect-timeout + slow-lane size/cap constants. -- `.claude/skills/xce-test-ha/run-ha-soak.sh` (+ `xce_test`) — partition (DROP) primitive and the partition scenario. - -> Each file keeps one responsibility; `slowlane.c` is the only new logic module. peer-control logic stays in `ha.c` (moved to a vtask entry, not relocated to a new file) to preserve reviewability against existing tests. - -## System Flows - -### Reconnect to an unreachable peer (R2) -```mermaid -sequenceDiagram - participant DM as delaymgr timer - participant SL as slow lane - participant IO as iomgr - DM->>DM: heartbeat tick - peer disconnected - DM->>SL: slowlane_submit reconnect peer if not in-flight - DM-->>DM: return immediately - other peers still heartbeat - SL->>SL: nonblocking connect + poll bounded timeout - alt connect succeeds - SL->>SL: prepare endpoint set msg_handler - SL->>IO: iomgr_monitor lock-guarded register fd - else timeout or refused - SL->>SL: clear in-flight - retry next tick - end -``` -Key decisions: delaymgr only triggers (idempotent per-peer flag); the connect is bounded so it never approaches the heartbeat timeout; registration reuses the lock-guarded `iomgr_monitor` (no new primitive). - -### Snapshot send triggered by a catch-up request (R1) -```mermaid -sequenceDiagram - participant IO as iomgr - participant VT as peer_ctrl vtask - participant SL as slow lane - IO->>VT: dispatch repl-ack packet VTASK_IO - VT->>VT: fast - decode, update match_index - VT->>SL: slowlane_submit snapshot if not in-flight per standby - VT-->>IO: handler returns - endpoint unblocked, peer messages flow - SL->>SL: persist_checkpoint + fread chunk loop + send -``` -Key decisions: the vtask must **not** hold `ep->busy` across the transfer (R4.3) — it triggers and returns; per-standby dedup prevents concurrent transfers (R1.5). - -## Requirements Traceability - -| Requirement | Summary | Components | Flows | -|-------------|---------|------------|-------| -| 1.1–1.4 | liveness during snapshot/config I/O | SlowLane, PeerCtrlDispatch | Snapshot flow | -| 1.5 | no concurrent snapshot per standby | SlowLane (dedup) | Snapshot flow | -| 2.1–2.2 | unreachable peer doesn't stall heartbeats | BoundedConnect, ReconnectOffload | Reconnect flow | -| 2.3 | bounded connect timeout | BoundedConnect | — | -| 2.4–2.5 | retry without self-terminate; dedup | ReconnectOffload, SlowLane | Reconnect flow | -| 2.6 | connect success registers without loss/reorder | ReconnectOffload (iomgr_monitor) | Reconnect flow | -| 3.1–3.4 | admission protection; long IO off pool/drive | SlowLane (cap), LivenessAdmission (pool reservation), PeerCtrlDispatch | — | -| 4.1–4.2 | per-connection processing order | PeerCtrlDispatch (endpoint_block_tasks) | — | -| 4.3 | don't hold block across long IO | PeerCtrlDispatch | Snapshot flow | -| 4.4 | iomgr-state mutation via lock-guarded iface | ReconnectOffload (iomgr_monitor) | Reconnect flow | -| 4.5–4.6 | preserve vote/ack/term + persist-before-vote | PeerCtrlDispatch, LivenessAdmission (ack-progress reservation) | — | -| 5.1–5.4 | thread-role contract compliance | all | — | -| 6.1–6.7 | no regression of existing HA features | all (regression suite) | — | -| 7.1–7.4 | partition verification | PartitionTest | — | -| 8.1–8.2 | heartbeat fast-path unchanged | PeerCtrlDispatch | — | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| SlowLane | controller/infra | bounded executor for long blocking I/O | 1, 3 | pthreads, sem (P0) | Service, Batch | -| LivenessAdmission | lib/vtaskmgr | pool-level task-class reservation: liveness work always has a worker | 3, 4.5 | vtaskmgr dispatcher (P0) | Service | -| PeerCtrlDispatch | controller/ha | move peer-control to vtask; two-tier | 1,4,5,8 | vtaskmgr, LivenessAdmission, endpoint_block_tasks (P0) | Service | -| BoundedConnect | lib/netio | non-blocking + poll-bounded connect | 2 | poll (P0) | Service | -| ReconnectOffload | controller/ha | delaymgr triggers offloaded reconnect | 2 | SlowLane, iomgr_monitor (P0) | Service | -| PartitionTest | test | unreachable-peer regression | 7 | soak harness (P1) | Batch | - -### Controller / Infrastructure - -#### SlowLane - -| Field | Detail | -|-------|--------| -| Intent | Bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight dedup | -| Requirements | 1.1, 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4 | - -**Responsibilities & Constraints** -- Owns a fixed set of dedicated worker threads (sized to peer count; small constant) — isolated from the vtask request pool, so long ops cannot starve request handling. -- `slowlane_submit` is non-blocking (enqueue + return); callers on drive threads never wait. -- Enforces an admission cap on concurrent in-flight tasks; excess is queued, never run on a drive thread (R3.3/3.4). -- Per-key dedup: a submit with an in-flight key is a no-op (R1.5 snapshot-per-standby; R2.5 reconnect-per-peer). -- Does not own the work content (connect, snapshot, config-push bodies stay in their modules); it owns scheduling/isolation. - -**Dependencies** -- Inbound: PeerCtrlDispatch — submits long ops (P0); ReconnectOffload — submits reconnect (P0). -- Outbound: `iomgr_monitor` (via the submitted task) — register connected fd (P0). -- External: pthreads (threads, mutex, cond, sem) (P0). - -**Contracts**: Service [x] / Batch [x] - -##### Service Interface -```c -typedef void (*slowlane_fn_t)(void *arg); - -int slowlane_init(slowlane_t *sl, int n_workers, int max_inflight); -/* submit returns 0 (enqueued), -EEXIST (key in flight, deduped), -EAGAIN (cap full, queued/deferred); - never blocks the caller. key==0 disables dedup. */ -int slowlane_submit(slowlane_t *sl, uint64_t key, slowlane_fn_t fn, void *arg); -void slowlane_fini(slowlane_t *sl); /* drain + join on shutdown only */ -``` -- Preconditions: `fn` performs the blocking work and is responsible for clearing any caller-side in-flight state on completion. -- Postconditions: `fn` runs exactly once on a lane worker; never on iomgr/delaymgr. -- Invariants: concurrent running tasks ≤ `max_inflight`; one in-flight task per non-zero key. - -**Snapshot/checkpoint concurrency invariant (Phase 3, R1.1/1.3/1.5 — MUST hold before snapshot offload lands)** -Moving `persist_checkpoint` (which runs `trim_wal_segments` — *deletes* WAL segments) + the snapshot `fread` loop onto a lane worker runs them concurrently with live WAL appends, the `fsync_worker` flusher, and the WAL-catchup stream. The lane snapshot path must obey: -- **Frozen boundary**: the snapshot is taken at a fixed `(term, lsn)` checkpoint boundary captured at submit time; the `fread`/send loop reads only data at or below that boundary, never live tail bytes. -- **Trim safety (MECHANISM, not just goal — see bug-1147)**: `trim_wal_segments` (`persist.c:1374`) TODAY bounds deletion ONLY by `active_seg_lsn` (the open segment) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** snapshot boundary, and there is **no** working snapshot backfill (`write_snapshot` emits only a 16-byte P0 sentinel, not materialized state). So a standby lagging by more than ~one open segment already hits a permanent committed-prefix gap (the bug-795/828 family). Moving checkpoint/trim onto a concurrent lane worker makes this latent race *easier* to hit. Therefore this spec MUST add the cursor-bound guard as a stopgap: `trim_wal_segments` computes `floor = min(all standby acked_lsn, all in-flight snapshot boundaries)` under `p->lock` and refuses to delete any segment whose range reaches `floor` or above — never holding `p->lock` across the `fread`/send I/O. This is strategy-A retention discipline that *should* exist regardless of this spec; the full fix for empty/far-behind join (cursor=0) requires entity-bearing snapshots (separate spec) and is explicitly NOT solved here. The design must not assume "a snapshot will backfill the gap" — that path is not built. -- **Single in-flight checkpoint**: at most one checkpoint/trim in flight (lane dedup key = checkpoint), and at most one snapshot per standby (dedup key = standby id, R1.5). -- **No correctness change**: the lane only relocates execution; the checkpoint/trim/snapshot *content* is byte-identical to the current inline path. - -**Implementation Notes** -- Integration: init in controller lifecycle (cluster.c/myapp.c); size from peer count. -- Validation: assert submit never called from a context expecting completion; metric on queue depth/age (fail-loud on backlog); a test asserting concurrent append + checkpoint/trim produces no segment-below-boundary deletion and a consistent snapshot. -- Risks: a lane worker blocked indefinitely (e.g., snapshot to a half-open peer) — **bound every in-lane blocking op**: the connect uses the bounded `socket_connect`; the snapshot send must use a bounded socket write (write timeout, mirroring the bounded-connect approach) so a standby going unreachable mid-transfer cannot pin a lane worker. Otherwise stuck transfers exhaust the bounded lane and delay reconnects (which share it). Phase-3 task constraint; size/segment the lane so reconnect and transfer do not starve each other. - -#### LivenessAdmission (lib/vtaskmgr) - -| Field | Detail | -|-------|--------| -| Intent | Make "keep K workers free for liveness" a property of the vtask **pool**, not of the commit engine — so it covers every blocking op the pool runs once peer-control moves onto it, not just commit-waits | -| Requirements | 3.1, 3.2, 3.3, 4.5 | - -**Why this exists (the gap it closes):** Today `vtaskmgr` (`include/vtaskmgr.h`) has a single FIFO `taskq`, N `vcpu` workers, and **no priority/reservation**; a blocking task holds its `vcpu` in `VCPU_BUSY` for the whole block (`vtaskmgr.c:248-275`). The only admission anywhere is `commit.c`'s `inflight_waits` counter (`COMMIT_WAIT_RESERVED=2`, `commit.c:200`) — external to vtaskmgr and counting **only** commit-waits. Once `peer_ctrl` moves onto the pool (PeerCtrlDispatch), the pool also runs the vote-grant term fsync (R4.6) and config-ack waits — blocking ops **invisible** to the commit counter. So "≥2 workers free of commit-waits" no longer means "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. The reservation must therefore become pool-level. - -**Responsibilities & Constraints** -- Classify every vtask at enqueue into `VTASK_LIVENESS` (never parks — WAL-ack decode, heartbeat send, vote *receipt/decode*; `commit_on_ack` is documented never-block, `commit.c:16`), `VTASK_LIVE_BLOCK` (bounded mandatory park — vote-grant term fsync, R4.6), or `VTASK_REQ_BLOCK` (may park seconds — commit-wait, config-ack wait). Class is knowable from the message type at the dispatch shim. -- The dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus: a `VTASK_REQ_BLOCK` task is **not admitted** when `count(vcpus running *_BLOCK) >= n_vcpus - VTASK_LIVENESS_RESERVED` — it **fast-rejects with `-XCE_ERR_NOT_READY`** (preserving today's client backpressure, Req 7.1/7.2), it does **not** silently queue or park a worker. -- `VTASK_LIVENESS` tasks are **never** admission-gated → always find a reserved vcpu → acks always advance commit → parked `*_BLOCK` tasks always wake. **Deadlock closed by construction.** -- `VTASK_LIVE_BLOCK` (vote fsync) runs in the reserved headroom and is bounded, so a vote response never queues behind a seconds-long commit-wait (removes a second spurious-election vector). -- `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` is **folded into** this — `commit_submit` becomes "submit a `VTASK_REQ_BLOCK`"; the pool owns the single reservation (`commit_engine_init` no longer needs `n_vcpus`, `cluster.c:1332`). - -**Dependencies**: Inbound — PeerCtrlDispatch, commit (P0). Outbound — none (lib-internal). External — pthreads. - -**Contracts**: Service [x] -```c -typedef enum { VTASK_LIVENESS=0, VTASK_LIVE_BLOCK, VTASK_REQ_BLOCK } vtask_class_t; -/* submit a classified task; VTASK_REQ_BLOCK returns -XCE_ERR_NOT_READY when the - reservation would be violated (retriable, never parks a worker). */ -int vtask_enq_classified(vtaskmgr_t*, vtask_t*, vtask_class_t); -``` -- Invariant: `count(vcpus in *_BLOCK) <= n_vcpus - VTASK_LIVENESS_RESERVED`; ≥`VTASK_LIVENESS_RESERVED` vcpus always available to `VTASK_LIVENESS`. - -**Implementation Notes** -- **Sizing (peer-scaled):** `R = VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)`; commit/`REQ_BLOCK` cap = `n_vcpus − R`. Rationale: per-endpoint serialization (`endpoint_block_tasks`) caps in-flight tasks at one per peer connection, so worst-case concurrent liveness work is **N−1** (one ack/vote per other peer) — reserving N−1 means no peer's liveness message ever queues behind a busy reserved worker. **Correctness needs only the floor of 1** (one guaranteed-free worker + liveness-never-gated + sub-ms cycling = deadlock-free regardless of peer count); the peer-scaled middle is a *latency* optimization. **Ceiling `n_vcpus − 1`** keeps ≥1 `REQ_BLOCK` slot so commits can still run (cap never 0). Both `n_peers` and `n_vcpus` are final at init (`cluster.c:1332` already passes both). Config-overridable. On a degenerate small-core box (`n_vcpus < n_peers`) R clamps to `n_vcpus − 1` (full per-peer reservation impossible) — still correct via fast cycling; emit a loud WARN at init so the under-provisioning is operator-visible. Validate the chosen `R` at supported sizes (R8). -- Validation: misclassification is the one failure mode — add a debug assert/loud-WARN if a `VTASK_LIVENESS` task parks beyond a threshold; task 3.4 (acks advance commit under saturation) is the regression oracle. -- Risk: a future blocking path tagged `VTASK_LIVENESS` silently breaks the guarantee — review discipline + the assert + the saturation test. -- **Revalidation Trigger**: any change to the class set, the reservation size, or the dispatch admission rule (consensus correctness + the slow-lane admission contract depend on it). - -#### PeerCtrlDispatch - -| Field | Detail | -|-------|--------| -| Intent | Dispatch peer-control message handling to the vtask pool; fast work inline, long work to the lane | -| Requirements | 1.4, 4.1, 4.2, 4.3, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2 | - -**Responsibilities & Constraints** -- Peer endpoints dispatch through the existing VTASK_IO path (the message is packet-triggered → reuses `vtask_handle_msg`), inheriting `endpoint_block_tasks` per-endpoint serialization (R4.1/4.2). -- The vtask entry performs only bounded protocol work inline (decode, match_index update, commit eval, send next heartbeat) and `slowlane_submit`s any long op; it must not hold `ep->busy` across long I/O (R4.3). -- Preserves existing vote/ack/term semantics and persist-before-vote ordering (R4.5/4.6) — the logic is unchanged, only its execution context moves. -- **Commit-advance preservation (R4.5):** WAL-ack handling — which wakes the commit barrier (`repl_wait_majority_ack` / `commit_set_raft_commit_index`) — moves off iomgr to the vtask pool. The design preserves that a quorum of acks still advances `committed_lsn` promptly: ack handling is bounded fast work (never offloaded to the slow lane), per-endpoint serialization does not delay *cross-endpoint* acks (each peer's acks process independently), and the commit-advance path itself is untouched. This is the only interaction with P1 (`raft-apply-decoupling`, which owns the barrier's async conversion); this spec must not change ack→commit-advance semantics, only the thread the ack is decoded on. - -- **Deadlock-freedom argument (R4.5/R3.1 — REQUIRED reasoning, reverses a documented invariant):** Today `commit_on_ack` is explicitly "called from the iomgr read path and never blocks" (`commit.c:16`, `repl.c:220`) — the ack that advances `committed_lsn` and the jobsub vtasks blocked in `commit_submit(COMMIT_MODE_WAIT)` (the synchronous barrier this spec PRESERVES) run on **different** threads. Moving peer-control to the vtask pool **co-locates the wakers (acks) and the waiters (blocked commit-submits) on one pool** — exactly the configuration the iomgr placement was chosen to avoid. Safety rests on a worker reservation — but the existing `commit.c` cap is **insufficient as-is**: `COMMIT_WAIT_RESERVED` (`commit.c:200`) counts **only** commit-waits, while moving `peer_ctrl` onto the pool also puts the vote-grant term fsync (R4.6, `VTASK_LIVE_BLOCK`) and config-ack waits on it — blocking ops the commit counter can't see. "≥2 workers free of commit-waits" therefore does **not** mean "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. **Closing the deadlock requires the LivenessAdmission component** (above) — the reservation must become a pool-level property covering every blocking class, with `VTASK_LIVENESS` (acks) never admission-gated. Required invariants this spec must hold (and test): (1) WAL-ack/commit-advance decode is tagged `VTASK_LIVENESS`, never routed to the slow lane, never gated — admissible independently of any blocking count; (2) the reserved headroom is sized for all concurrent liveness work (N-peer acks + vote responses + heartbeat sends), so the reservation — not luck — guarantees an ack always finds a worker. This co-location is **transient by design**: `raft-apply-decoupling` P1 converts `repl_wait_majority_ack` to event-driven (no parked thread per commit), removing the `VTASK_REQ_BLOCK` waiters from the pool entirely — but until P1 lands, correctness depends on LivenessAdmission holding. **Dependency: PeerCtrlDispatch (3.1) requires LivenessAdmission to land first (Phase 1).** Gate: task 3.4 (quorum-of-acks advances `committed_lsn` with the pool saturated by `*_BLOCK` tasks) is the **admission gate for 3.1** — 3.1 is not "done" until 3.4 is green. - -**Dependencies** -- Inbound: iomgr — dispatches the packet (P0). -- Outbound: vtaskmgr (P0); SlowLane — long ops (P0); existing commit/repl/persist logic (P0). - -**Contracts**: Service [x] - -**Implementation Notes** -- Integration: change peer endpoint `msg_handler` registration (ha.c:4265/4307/6349) to the vtask-dispatching shim; convert `peer_ctrl_msg_handler` body into the vtask entry. -- Validation: existing election/heartbeat/repl tests must pass unchanged (R6); assert no long op runs inline in the entry. -- Risks: a heavy op accidentally left inline → per-peer head-of-line blocking (guard by review + a test asserting the handler returns promptly). - -#### BoundedConnect (lib/netio) - -| Field | Detail | -|-------|--------| -| Intent | Make `socket_connect` non-blocking + `poll`-bounded | -| Requirements | 2.3 | - -**Responsibilities & Constraints**: set `O_NONBLOCK` before `connect`; on `EINPROGRESS`, `poll` up to a configured timeout; return success/timeout/error. Reuses the `xce_probe_config_tag` pattern. -**Contracts**: Service [x] -```c -/* connect_timeout_ms: bound the attempt; 0 = use default (config). */ -int socket_connect(const char *host, int port, int connect_timeout_ms); -``` -**Implementation Notes**: callers updated to pass the timeout; default in config.h. This is a behavior change to a shared helper → Revalidation Trigger for all connect callers. - -#### ReconnectOffload (controller/ha) - -| Field | Detail | -|-------|--------| -| Intent | delaymgr triggers an offloaded, deduped, bounded reconnect | -| Requirements | 2.1, 2.2, 2.4, 2.5, 2.6 | - -**Responsibilities & Constraints**: `primary_heartbeat_fn` (delaymgr) detects a disconnected peer and `slowlane_submit`s the reconnect (per-peer key for dedup); the lane worker does the bounded connect, prepares the endpoint, registers it via lock-guarded `iomgr_monitor`; on failure clears the in-flight flag and lets the next tick retry (never self-terminates). -**Contracts**: Service [x] -**Implementation Notes**: refcount the endpoint across the in-flight window; resolve the inbound-vs-outbound connection race per the existing connection-ownership rules (`ha-async-io.md`). - -### Test - -#### PartitionTest - -| Field | Detail | -|-------|--------| -| Intent | Make one peer unreachable (blackhole), assert liveness + no spurious election + heal | -| Requirements | 7.1, 7.2, 7.3, 7.4 | - -**Contracts**: Batch [x] -**Implementation Notes**: the partition primitive must render one peer **unreachable** (traffic dropped, connect hangs), distinct from `kill -9`. Two mechanisms, **primary = in-process** so R7 actually runs in this project's environment (the sibling specs `ha-term-safety` 5.2/5.4 are already perpetually ENV-BLOCKED on the unprivileged/overlayfs sandbox; a firewall-only R7 would be a permanent skip and the spec's headline guarantee would never be exercised): -- **Primary (portable, no privilege): in-process socket-drop hook in `xce_test`** — a test-only seam (`XCE_TESTING`) that makes one node silently drop/blackhole inbound+outbound peer traffic at the endpoint layer (e.g. stop reading/writing the peer fd, or a drop flag in the I/O path), simulating "unreachable" without any kernel/firewall change. Runs anywhere the unit/soak suite runs. -- **Secondary (infra-level, optional): firewall `iptables`/`nft` DROP** on the node's port for real-network fidelity; capability-gated with a **loud skip** when unprivileged. Complements, does not replace, the in-process path. - -Assert (both): heartbeats to survivors continue within the interval, no new-leader election while the leader holds majority, peer rejoins on heal, zero committed-job loss. - -## Error Handling - -### Error Strategy -- **Connect timeout/refused** → bounded `poll` returns; lane clears in-flight; delaymgr retries next tick; loud WARN with peer + elapsed (fail-loud, never abort — controllers never self-terminate). -- **Lane backlog** → submit returns `-EAGAIN` when the cap is reached; the work is queued, never run on a drive thread; emit a metric/WARN on queue depth + oldest-age so a saturated lane is operator-visible. -- **Snapshot/config-push failure on the lane** → reported through the existing repl/config error paths unchanged; lane only relocates execution, not error semantics. -- **Dedup race** → submit with an in-flight key is a no-op (`-EEXIST`); idempotent. - -### Monitoring -- Metrics: lane queue depth, oldest-task age, in-flight count, per-peer reconnect attempts. WARN thresholds make a degraded/partitioned window visible (consistent with fail-loud). - -## Testing Strategy - -### Unit Tests -- SlowLane: `slowlane_submit` is non-blocking; admission cap bounds concurrent tasks; per-key dedup returns `-EEXIST`; `fn` runs exactly once (3.1–3.4, 1.5). -- BoundedConnect: `socket_connect` returns within the timeout against an unreachable host; succeeds against a reachable one (2.3). -- PeerCtrlDispatch: the vtask entry returns promptly (no long op inline); per-endpoint serialization holds via `endpoint_block_tasks` (4.1–4.3). - -### Integration Tests -- Reconnect offload: with one peer unreachable, delaymgr ticks continue and heartbeats to reachable peers are sent within the interval; reconnect is deduped per peer (2.1, 2.2, 2.5). -- Snapshot offload: a catch-up triggering a snapshot does not block iomgr; other peers' heartbeats/votes processed during the transfer; no concurrent transfer per standby (1.1, 1.3, 1.4, 1.5). -- Commit-advance under pool load (R4.5): with the vtask pool near saturation, a quorum of WAL-acks still advances `committed_lsn` promptly; cross-endpoint acks are not serialized behind one another. -- Snapshot/checkpoint concurrency (R1.1/1.3/1.5): concurrent live appends + a lane checkpoint/trim never delete a segment below an in-flight snapshot boundary or a standby catch-up cursor; the produced snapshot is consistent. -- Regression sweep (R6): run the existing suites unchanged — `test_ha_election`, `test_ha_prevote`, `test_ha_liveness`, `test_ha_failover`, `test_ha_rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability`; all pass without behavior changes (6.1–6.7). - -### E2E / Soak Tests -- Partition soak (R7): blackhole one peer → assert heartbeats to survivors continue, no spurious election while leader holds majority, peer rejoins on heal, zero committed-job loss (7.1–7.4). -- Failover soak (existing, kill-based) continues to pass (6.3). - -### Performance -- Steady-state heartbeat cadence at 3–5 controllers shows no added latency attributable to the new dispatch path; the heartbeat/vote send paths remain non-blocking enqueues (8.1, 8.2). - -## Migration Strategy - -Phased, each phase independently testable (matches research recommendation): - -**Ordering rule:** the slow lane is foundational — every offload (reconnect, snapshot, config-push) submits to it, so it must exist first. And the peer_ctrl→vtask move must land **together with** its long-op offload in the same phase: moving peer_ctrl to a vtask while still running snapshot/config-push inline would hold `endpoint_block_tasks` (`ep->busy`) across multi-second I/O (transient R4.3 violation) and consume a pool worker. The two are therefore coupled. - -```mermaid -flowchart TD - P1[Phase 1: SlowLane executor + BoundedConnect] --> P2[Phase 2: ReconnectOffload sender-side] - P2 --> P3[Phase 3: PeerCtrlDispatch to vtask PLUS snapshot and configpush offload coupled] - P3 --> P4[Phase 4: PartitionTest + regression sweep] -``` - -- **Phase 1** (foundational, lowest risk): build the SlowLane executor (bounded pool + admission + dedup — new component, no behavior change yet, unit-testable in isolation); reorder `socket_connect` to bounded non-blocking + `poll` (independent; already bounds the delaymgr stall even before reconnect is offloaded); **and add LivenessAdmission to vtaskmgr** (task-class reservation; fold in `COMMIT_WAIT_RESERVED`). LivenessAdmission is a **hard prerequisite of Phase 3** — peer_ctrl cannot move onto the pool deadlock-safely without it — so it must land in Phase 1 as dormant infra (no behavior change until peer_ctrl dispatches through it). -- **Phase 2**: offload the delaymgr peer reconnect to the lane (deduped per peer, registered via lock-guarded `iomgr_monitor`) → closes sender-side heartbeat starvation. Shippable. -- **Phase 3** (coupled — the R4.3-safe step): dispatch peer-control to the vtask pool **and** offload the long ops it triggers (snapshot send, config-push transport-only) to the lane in the same change, so no phase ever runs long I/O inline in a blocked vtask. Carries the snapshot/checkpoint concurrency invariant (frozen `(term,lsn)` boundary, trim-vs-catch-up-cursor safety, single in-flight checkpoint) and the bounded in-lane snapshot write. -- **Phase 4**: partition test + full regression sweep. -- Rollback: each phase is behavior-preserving; a phase can be reverted independently since the contract (non-blocking drive) is the only invariant added. P1 adds dormant infra; the first observable change is P2. diff --git a/.kiro/specs/raft-drive-async/requirements.md b/.kiro/specs/raft-drive-async/requirements.md deleted file mode 100644 index 0d1d2b8f..00000000 --- a/.kiro/specs/raft-drive-async/requirements.md +++ /dev/null @@ -1,111 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE controller's Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals — the iomgr epoll loop and the delaymgr timer thread. A blocking audit (2026-06-25) found three CRITICAL violations of the corrected `ha-async-io.md` thread-role contract: snapshot send + checkpoint inline on iomgr (`ha.c:2702`, `config_repl.c:2446`), config-push busy-poll + join inline on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr heartbeat thread (`netio.c:289/302`). Root cause: `peer_ctrl_msg_handler` is registered directly as the iomgr message handler (`ha.c:4265/4307/6349`) instead of dispatching to the vtask pool as normal cluster messages do (`cluster.c:711`). When a drive thread stalls, heartbeats/votes/ACKs stop flowing, peers declare the leader dead, and a spurious election follows — the likely root of the bug-851 chaotic-failover family. - -This feature brings the controller into compliance with the steering thread-role contract (`ha-async-io.md`, corrected 2026-06-25) so that no liveness-carrying thread blocks inline, while preserving every existing HA behavior and the persist-before-ACK / synchronous-commit policies. It is verified by a new network-partition (peer unreachable, not merely killed) regression scenario that the current process-kill-only soak cannot reproduce. - -The authoritative subject for the acceptance criteria below is **the XCE controller** and its named drive threads (iomgr loop, delaymgr timer, vtask pool). - -## Boundary Context - -- **In scope**: keeping the iomgr loop and delaymgr timer non-blocking; moving long replication/config I/O (snapshot transfer, config-push) off the drive threads to an admission-controlled execution path; bounding and offloading the peer connect; preserving per-connection message ordering and processing correctness; admission protection so liveness-critical message handling is never starved; a network-partition regression test; no regression of existing HA features. -- **Out of scope**: the commit-barrier async conversion (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` as prerequisite P1; the apply-after-commit / hook-decoupling refactor; asynchronous DNS resolution (deferred unless measured — HPC peers use IP/`/etc/hosts`); the snapshot content/format (entity-bearing snapshots is a separate spec) — this feature only moves the snapshot *transfer* off the drive threads; the MEDIUM vtask-pool blockers beyond what admission control covers. -- **Adjacent expectations**: relies on the corrected `ha-async-io.md` thread-role contract as the authority; reuses the existing `COMMIT_WAIT_RESERVED` admission cap and the `xce_probe_config_tag` bounded-connect pattern; consumes `election-iomgr-refactor` (completed) as upstream for its primitives (`delay_task_ms`, `on_connected`, non-blocking vote fan-out) and stays a standalone spec (scope-check 2026-06-25: ADJACENT-SAME-AREA — that spec fixed startup blocking, this fixes runtime drive-thread blocking); touches the same `peer_ctrl_msg_handler`/repl-dispatch region as `ha-term-safety`; preserves the durability and synchronous-commit guarantees owned by `ha-commit-durability` and `ha-commit-engine`. - -## Requirements - -### Requirement 1: Liveness During Long-Running Replication and Configuration I/O - -**Objective:** As a cluster operator, I want the controller to keep heartbeating and processing protocol messages while it transfers a snapshot or pushes a configuration to a standby, so that a large or slow transfer to one peer does not trigger a false failover of the cluster. - -#### Acceptance Criteria - -1. While the controller is transferring a snapshot to one standby, the controller shall continue to send heartbeats to every other peer within the configured heartbeat interval (default 3000 ms). -2. While the controller is pushing a configuration to standbys, the controller shall continue to send heartbeats to every peer within the configured heartbeat interval and continue to process inbound votes and acknowledgments. -3. When the controller initiates a snapshot transfer, checkpoint creation, or configuration push, the iomgr loop shall remain available to dispatch other inbound messages without waiting for that operation to complete. -4. While a long-running replication or configuration operation is executing, the controller shall continue to process inbound RequestVote messages and respond within the configured election timeout (default 5000 ms). -5. If a snapshot transfer to one standby is already in progress, then the controller shall not start a second concurrent transfer to that same standby. - -### Requirement 2: Liveness and Bounded Behavior When a Peer Is Unreachable - -**Objective:** As a cluster operator, I want an unreachable peer to never stall heartbeats to the healthy peers, so that one partitioned node cannot cause the surviving majority to elect a new leader unnecessarily. - -#### Acceptance Criteria - -1. While one peer is unreachable (its host accepts no connection or silently drops traffic), the controller shall continue to send heartbeats to all reachable peers within the configured heartbeat interval. -2. While one peer is unreachable, the delaymgr timer thread shall continue to fire periodic and election-timeout callbacks without being delayed by the connection attempt to the unreachable peer. -3. When the controller attempts to connect to a peer, the controller shall bound the connection attempt to a configured timeout and report failure on expiry rather than waiting for the operating-system connect timeout. -4. If a connection attempt to a peer fails or times out, then the controller shall retry the connection later without terminating, consistent with the "controllers never self-terminate" invariant. -5. When the controller is already attempting to (re)connect to a given peer, the controller shall not enqueue a duplicate concurrent connection attempt for that same peer. -6. When a connection attempt to a peer succeeds, the controller shall register and begin using the new connection without losing or reordering messages already queued for that peer. - -### Requirement 3: Worker-Pool Admission Protection - -**Objective:** As a cluster operator, I want offloaded blocking work to never consume all worker capacity, so that liveness-critical message handling always has a worker available even under replication or reconnection load. - -#### Acceptance Criteria - -1. The controller shall reserve worker capacity such that liveness-critical message handling (heartbeats, votes, acknowledgments) always has at least one available worker even when blocking offloaded tasks are in flight. -2. While the maximum number of admitted blocking offloaded tasks is in flight, the controller shall continue to admit and process non-blocking liveness-critical messages. -3. When the number of concurrently requested blocking offloaded tasks exceeds the admission limit, the controller shall queue or defer the excess rather than block a liveness-carrying thread. -4. The controller shall route long/unbounded blocking I/O (snapshot transfer, configuration push, peer reconnect) to an execution path that is subject to the admission limit, not to the iomgr loop or the delaymgr timer thread. - -### Requirement 4: Message Ordering and Processing Correctness Preserved - -**Objective:** As a cluster operator, I want peer control messages to be processed with the same ordering and correctness as today after they are moved off the iomgr loop, so that consensus correctness is unaffected by the threading change. - -#### Acceptance Criteria - -1. When peer control messages arrive on a single connection, the controller shall handle (process/apply) them in arrival order, even though handling is dispatched to the worker pool rather than run inline on the iomgr loop. (This concerns inbound *processing* order, distinct from the already-guaranteed outbound *send* order of the endpoint output queue.) -2. While handling a peer control message for a given connection, the controller shall not begin handling another message from that same connection until the first completes, so per-connection processing remains strictly sequential. -3. While handling a peer control message, the controller shall not hold its connection's sequential-processing exclusivity across a long-running offloaded operation (snapshot transfer, configuration push); such operations shall be triggered and the handler shall return, so the connection's subsequent messages (including its heartbeat acknowledgments) are not queued behind the long operation. -4. When peer control message processing mutates iomgr-owned connection or epoll state, the controller shall do so only through the thread-safe (lock-guarded) registration interface, without data races against the iomgr event loop. -5. The controller shall preserve the existing semantics of vote handling, acknowledgment handling, and term updates after relocating peer control message processing off the iomgr loop. -6. The controller shall continue to persist term and vote state before responding to a vote, preserving the existing durability ordering on the vote path. - -### Requirement 5: Thread-Role Contract Compliance - -**Objective:** As a maintainer, I want the controller's drive threads to obey the steering thread-role contract by construction, so that future changes cannot reintroduce inline blocking on a liveness thread. - -#### Acceptance Criteria - -1. The iomgr loop shall perform only non-blocking I/O and message dispatch; it shall not execute blocking calls or inline message processing. -2. The delaymgr timer thread shall only fire timers and trigger work; it shall not execute blocking calls (connect, file I/O, or busy-wait sleeps). -3. The controller shall execute bounded blocking work in the worker pool only under the admission reservation, and long/unbounded blocking I/O only on the dedicated offload path. -4. The controller shall conform to the thread-role contract defined in `ha-async-io.md` for all drive-thread code paths brought into scope by this feature. - -### Requirement 6: No Regression of Existing HA Features - -**Objective:** As a cluster operator, I want all existing HA behaviors to keep working unchanged after the drive is made non-blocking, so that the hardening introduces no functional regression. - -#### Acceptance Criteria - -1. When the existing controller test suites for election/voting, heartbeat/liveness, failover, snapshot recovery, configuration push/upgrade, WAL replication/catch-up, and the commit barrier are run, the controller shall pass them without modification to their asserted behavior. -2. The controller shall continue to elect a single primary on quorum, increment terms, and reject stale-term votes exactly as the existing election tests assert. -3. The controller shall continue to detect a dead primary and fail over within the configured heartbeat timeout (default 10000 ms) exactly as the existing liveness/failover tests assert. -4. The controller shall continue to deliver snapshots and replicated WAL entries to standbys and recover them on restart exactly as the existing replication and recovery tests assert. -5. The controller shall continue to push and apply configuration changes through majority acknowledgment exactly as the existing config-replication tests assert. -6. The controller shall preserve the persist-before-ACK durability guarantee and the synchronous commit-barrier behavior, which this feature does not modify. -7. If any existing HA regression test fails after the change, then the feature shall be considered incomplete until the behavior is restored. - -### Requirement 7: Network-Partition Verification - -**Objective:** As a cluster operator, I want a regression test that makes a peer unreachable (not merely killed), so that the heartbeat-starvation failure mode this feature fixes is provably closed and stays closed. - -#### Acceptance Criteria - -1. The verification suite shall include a scenario that renders one peer unreachable by dropping its traffic (network partition / blackhole), distinct from process termination. -2. When one peer is made unreachable during the scenario, the verification shall assert that heartbeats to the surviving peers continue and that no spurious election of a new leader occurs while the existing leader remains alive and reachable to the majority. -3. When the partition is healed, the verification shall assert that the previously unreachable peer rejoins and catches up without operator intervention. -4. While the partition scenario runs, the verification shall assert that no committed job is lost, consistent with the existing failover soak guarantees. - -### Requirement 8: Heartbeat Fast-Path Performance - -**Objective:** As a cluster operator, I want the steady-state heartbeat and vote path to remain responsive after the threading change, so that making the drive non-blocking does not degrade normal operation at supported cluster sizes. - -#### Acceptance Criteria - -1. While the cluster is healthy and all peers are connected, the controller shall send heartbeats at the configured interval without added latency attributable to the new dispatch path at supported cluster sizes (3–5 controllers). -2. The controller shall not introduce additional blocking or unbounded queuing on the steady-state heartbeat and vote send paths, which are already non-blocking enqueues. diff --git a/.kiro/specs/raft-drive-async/research.md b/.kiro/specs/raft-drive-async/research.md deleted file mode 100644 index c1a7ef11..00000000 --- a/.kiro/specs/raft-drive-async/research.md +++ /dev/null @@ -1,100 +0,0 @@ -# Gap Analysis — raft-drive-async (2026-06-25) - -Analyzes the gap between the 8 requirements and the existing controller code, to inform the design phase. Evidence from this session's blocking audits + targeted greps. - -## Current State Investigation - -### The existing deferral idiom and why it's insufficient -XCE already practices "don't block iomgr": the established pattern is `delay_task_ms(&c->delaymgr, ...)`, which `ordq_put`s a callback and returns immediately. Its own comments confirm the intent — "run off the iomgr walAck thread … delay_task_ms only enqueues" (`config_repl.c:2369`, `ha_convergence.c:207`, `config_repl.c:428`). **But the deferred callback then runs on the delaymgr thread — the heartbeat/election-timeout driver.** So the only existing "offload" target is itself a liveness thread. This is the conceptual heart of the gap: the codebase can *defer*, but its single deferral lane is the thread the corrected `ha-async-io.md` contract must keep free. **The missing capability is a non-liveness execution lane (a "slow lane").** - -### Inline-on-iomgr root cause -`peer_ctrl_msg_handler` is assigned directly as `ep->msg_handler` at `ha.c:4265` (on_vote_connected), `ha.c:4307` (on_pre_vote_connected), `ha.c:6349` (standby accept) — so it runs on the iomgr loop. Normal inbound messages instead go `handle_msg` → `vtask_handle_msg` → vtask pool (`cluster.c:711`, registered `cluster.c:772`). The peer-control path is the anomaly that bypasses the pool. - -### Reusable building blocks that already exist -- **vtask submit primitives**: `vtask_alloc` / `vtask_init` (vtaskmgr.c:21) / `vtask_enq` (vtaskmgr.c:342) exist and compose; the only public entry `vtask_handle_msg` (vtaskmgr.c:356) hardcodes `vtype=VTASK_IO` and *requires* `ep->fsm_in.pkt`. A general non-IO submit is a **small composition** of the existing primitives, not new infrastructure. -- **Per-endpoint serialization (explicit primitive)**: `endpoint_block_tasks()` / `endpoint_unblock_tasks()` (netio.c:896-911) set a per-endpoint `ep->busy` flag; the VTASK_IO path auto-blocks on task start (vtaskmgr.c:249), unblocks on completion (vtaskmgr.c:66), and the handler path uses it (xcectx.c:440/449), while `vtask_eligible` (vtaskmgr.c:203) skips a busy endpoint's other tasks. This **directly and explicitly satisfies R4.1** (per-connection processing order) when peer control reuses the IO path — a reuse, not new logic. Cross-peer isolation is preserved (block is per-`ep`, so peer X busy ≠ peer Y blocked). **Design trap:** the block is held for the whole task duration — so a long op (snapshot send) run *inline* in a blocked vtask would queue that peer's own subsequent heartbeat-ACKs behind it. This makes the two-tier rule (fast work in vtask, **trigger** long I/O to the slow lane) mandatory for R4.1 + R1 to both hold. -- **Admission cap pattern**: `COMMIT_WAIT_RESERVED=2` of `nvcpus` (default 8) caps blocking commit-waits at `max(1, n-2)` (commit.c:200, commit.h:17) — a reusable *pattern* for R3, but it is commit-engine-specific today. -- **Dedicated-background-thread precedent**: `fsync_worker` (persist.c:624) is a long-lived background thread; `push_config_transport_only` **already spawns per-standby `push_thread`s** (config_repl.c:1412) — then blocks by `pthread_join`/`wait_all_with_timeout` on the drive thread (config_repl.c:1332). So the *threads* exist; the defect is the **blocking join on the caller**, not absent concurrency. -- **Bounded-connect pattern**: `xce_probe_config_tag` already does `O_NONBLOCK`-before-connect + `poll(500ms)` (config_repl.c:1753-1761). The buggy path (`socket_connect`, netio.c:289/302) sets non-blocking *after* connect (netio.c:716) — so R2.3 is largely a **reorder + reuse**, not new code. - -### Confirmed absent -- ~~No iomgr cross-thread wake primitive~~ — **CORRECTED 2026-06-25 (was overstated).** epoll registration is ALREADY thread-safe: `iomgr_monitor` (EPOLL_CTL_ADD), `iomgr_unmonitor` (DEL), `iomgr_update_poll` (MOD) all wrap `epoll_ctl` in `mylock_lock(&iomgr->lock)` (netio.c:1046-1082), and `epoll_ctl` is kernel-thread-safe regardless. The connect→register flow already runs from delaymgr/main threads today (e.g. `on_vote_connected`). So a worker can do the blocking connect off-thread, fully prepare the endpoint (set `ep->msg_handler` etc.) BEFORE it is monitored (an unmonitored fd yields no events → no race), then call the lock-guarded `iomgr_monitor`. **No new eventfd/iomgr-post primitive is needed; the marshal-back is "reuse `iomgr_monitor`."** Residual (design-verify, not a primitive): ensure worker-side handling does not concurrently touch an endpoint's read-FSM buffer that iomgr owns — already mitigated by `endpoint_block_tasks` per-endpoint serialization + lock-guarded poll changes. -- **No network-partition test primitive**: the soak (`run-ha-soak.sh`) is process-kill only (`kill -9`, lines 74/440); unit "unreachable peer" cases use closed ports, not traffic drops. R7 has **no existing harness**. - -## Requirement → Asset Map - -| Req | Existing asset | Gap tag | -|-----|----------------|---------| -| R1 liveness during snapshot/config I/O | snapshot send `ha.c:2702`+`config_repl.c:2446`; config-push threads `config_repl.c:1412` | **Missing** off-load lane for snapshot; **Constraint** config-push blocks on join | -| R2 unreachable-peer / bounded connect | `endpoint_connect`/`socket_connect` netio.c:289/302 (blocking); `xce_probe_config_tag` pattern (reuse); reconnect via delaymgr `primary_heartbeat_fn` | **Constraint** (reorder connect) + **Missing** (offload reconnect off delaymgr) | -| R3 admission protection | `COMMIT_WAIT_RESERVED` pattern (commit-specific) | **Missing** (generalize to the slow lane) | -| R4 ordering/correctness | `endpoint_block_tasks`/`endpoint_unblock_tasks` per-endpoint sequential primitive (netio.c:896-911, auto on VTASK_IO path); lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (netio.c:1046-1082) for thread-safe epoll registration | **Reuse** (both primitives exist; no new marshal-back primitive needed) | -| R5 contract compliance | `ha-async-io.md` contract (steering authority) | verified by R1–R4 (no standalone asset) | -| R6 no regression | existing suites: `test_ha_election/prevote/liveness/failover/rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability` | **Reuse** as regression gate | -| R7 partition verification | `run-ha-soak.sh` (kill-only) | **Missing** (no traffic-drop primitive) | -| R8 heartbeat perf | heartbeat fan-out already non-blocking enqueue | **Constraint** (don't regress fast path) | - -## Implementation Approach Options - -### Option A — Extend in place (minimal) -Dispatch `peer_ctrl_msg_handler` through the existing `handle_msg`/`vtask_handle_msg` path; add a general non-IO `vtask_enq` wrapper; reorder `socket_connect` to non-blocking + `poll`; replace config-push `wait_all_with_timeout` join with a non-blocking completion check; reuse `delay_task_ms` only for *non-blocking triggers*. Touches ha.c, config_repl.c, netio.c, vtaskmgr.c. -- ✅ Least new surface; reuses every existing pattern; fastest path to closing the delaymgr+iomgr inline blocks. -- ❌ Long I/O (snapshot, config-push) still lands in the **request** vtask pool → relocates the stall to pool exhaustion (R3 only partially met). - -### Option B — New dedicated slow-lane executor -Build a dedicated bounded **slow-lane executor** (generalize `fsync_worker`) for all long blocking I/O. Peer control still dispatched to vtask, but heavy work re-offloaded to the slow lane. (No iomgr-post primitive needed — epoll registration reuses the lock-guarded `iomgr_monitor`.) -- ✅ Clean separation; R3/R4 fully met by construction; isolates seconds-long I/O from request handling and from delaymgr. -- ❌ One genuinely new component; more interface design than extending in place. - -### Option C — Hybrid (recommended) -- **Extend** (reuse): peer_ctrl → vtask via the existing dispatch pattern; general non-IO vtask submit (compose `vtask_alloc/init/enq`); bounded connect (reorder + reuse `xce_probe_config_tag` pattern); preserve ordering via `VTASK_IO` serialization. -- **New** (the one real gap): a small bounded **slow-lane executor** for snapshot-transfer + config-push (generalize the `fsync_worker`/`push_thread` precedent; per-peer dedup + admission cap parallel to `COMMIT_WAIT_RESERVED`). Epoll registration of a connected fd reuses the lock-guarded `iomgr_monitor` (no new primitive). -- **Phased**: - - *Phase 1 (S–M, highest value/lowest risk):* bounded connect + offload reconnect off delaymgr → closes the heartbeat *sender*-side starvation (R2). Independent, shippable. - - *Phase 2 (M):* dispatch peer_ctrl to vtask (epoll registration reuses lock-guarded `iomgr_monitor`) → closes the iomgr *receiver*-side inline processing (R4, R5). - - *Phase 3 (L):* slow-lane executor + move snapshot-send/config-push to it + admission generalization (R1, R3). - - *Phase 4 (M):* partition test harness + regression sweep (R6, R7, R8). -- ✅ Each phase is independently testable and de-risks the next; matches what already exists vs. what's truly missing. -- ❌ Requires phase coordination; must keep `ha-async-io.md` compliance consistent across phases. - -## Effort & Risk - -| Work item | Effort | Risk | Justification | -|-----------|--------|------|---------------| -| Bounded connect + reconnect offload (Ph1) | S–M | Medium | reorder existing pattern; offload target needs to be non-delaymgr | -| General non-IO vtask submit | S | Low | compose existing `vtask_alloc/init/enq` | -| peer_ctrl → vtask dispatch | M | Medium | correctness-sensitive: ordering (mitigated by `endpoint_block_tasks`), term/vote semantics; epoll register reuses lock-guarded `iomgr_monitor` | -| Slow-lane executor + snapshot-send offload | L | Medium-High | per-peer dedup + coordination with ongoing append/WAL-trim/catch-up stream | -| Admission generalization (R3) | M | Medium | extend `COMMIT_WAIT_RESERVED` pattern to slow lane | -| Partition test harness (R7) | M | Medium | no traffic-drop primitive today; soak infra is kill-only | -| **Overall** | **L (1–2 wks)** | **Medium** | riskiest: snapshot-offload coordination + moving peer_ctrl off iomgr (no new epoll-core primitive needed) | - -## Recommendations for Design Phase - -- **Preferred approach:** Option C (Hybrid), phased. Ship Phase 1 (bounded connect) early — it independently closes the sender-side heartbeat starvation and is low-risk. -- **Key decisions to settle in design:** - 1. **Connected-fd registration:** reuse the lock-guarded `iomgr_monitor` (netio.c:1058) — a worker prepares the endpoint fully, then registers it; no new primitive. Design-verify only that worker handling does not touch an endpoint's read-FSM buffer concurrently with iomgr (serialized by `endpoint_block_tasks`). - 2. **Slow-lane shape:** a shared bounded pool vs. reuse the existing per-standby `push_thread` model for config-push and a small pool for snapshot. Size to peer count. - 3. **Fold-in question — RESOLVED 2026-06-25: keep `raft-drive-async` STANDALONE.** Scope-check verdict ADJACENT-SAME-AREA: `election-iomgr-refactor` is fully implemented and fixed the *startup* blocking path (pthread election before `cluster_run`); it touched none of raft-drive-async's runtime targets (`peer_ctrl_msg_handler` dispatch, delaymgr `connect`, snapshot/config-push on iomgr) and explicitly deferred async DNS as a non-goal. It is consumed as completed *upstream*, providing reusable primitives (`delay_task_ms`, `on_connected` fire-once callback, the `ha_start_election_async`/non-blocking vote-fan-out contract). Folding in would reopen a done spec for net-new infra it never held. - -## Research Needed (carry to design) - -- ~~election-iomgr-refactor scope overlap~~ — **RESOLVED 2026-06-25** (scope-check): ADJACENT-SAME-AREA, keep standalone; reuse its primitives (`delay_task_ms`, `on_connected`, `ha_start_election_async`). See Recommendations decision #3. -- **Endpoint read-FSM ownership**: confirm worker-side peer-control handling never touches an endpoint's read-FSM buffer concurrently with the iomgr read path (expected safe via `endpoint_block_tasks` + handoff of `fsm_in.pkt`). (Epoll registration itself is already settled — lock-guarded `iomgr_monitor`; no new primitive.) -- **Partition primitive choice** for R7: `iptables`/`tc` (root, infra-level) vs. an in-process socket-drop hook in `xce_test`. Determines test portability. -- **Snapshot-send coordination** with the concurrent WAL-catch-up stream and the primary's append/`trim_wal_segments` while the transfer runs on the slow lane. -- **Admission interaction** between the existing `COMMIT_WAIT_RESERVED` (commit waits) and the new slow-lane cap — shared budget vs. independent, so the two together can't starve the 8-worker pool. - ---- - -## Design Synthesis (2026-06-25, /kiro-spec-design) - -Key simplification chosen at design time: **the offload target for long blocking I/O is a dedicated slow-lane executor (separate threads), NOT the vtask request pool.** Consequences: - -1. **R3 (pool protection) becomes structural** — long ops never run on the vtask pool, so they cannot starve liveness-critical message handling. The existing `COMMIT_WAIT_RESERVED` continues to cover commit-waits. -2. **The "general non-IO vtask submit" research item is SUPERSEDED** — not needed. Peer-control handling is packet-triggered, so it reuses the existing VTASK_IO `vtask_handle_msg` path; the only packet-less trigger (delaymgr reconnect) goes to the slow lane, not the vtask pool. No new vtask submit primitive required. -3. Combined with the already-refuted iomgr-post primitive (epoll registration reuses lock-guarded `iomgr_monitor`), **the only genuinely-new component is the slow-lane executor** (`slowlane.{c,h}`). Everything else is reuse + integration. - -Build-vs-adopt: slow lane generalizes the existing `fsync_worker`/`push_thread` background-thread precedent (adopt pattern, build the small bounded executor). Bounded connect reuses the `xce_probe_config_tag` pattern (reorder, not new). Per-endpoint ordering reuses `endpoint_block_tasks`. - -Phased migration (P1 bounded-connect → P2 peer_ctrl→vtask → P3 slow-lane+offload → P4 partition test) keeps each step behavior-preserving and independently revertible. diff --git a/.kiro/specs/raft-drive-async/spec.json b/.kiro/specs/raft-drive-async/spec.json deleted file mode 100644 index ad9cc761..00000000 --- a/.kiro/specs/raft-drive-async/spec.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "feature_name": "raft-drive-async", - "created_at": "2026-06-25T11:37:13Z", - "updated_at": "2026-07-18T14:45:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-18T14:45:00Z", - "note": "Re-validated after bug-1447 (the prior blocking finding) was root-caused and FIXED. \n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh lib, then controller) with the fix applied. TBD/TODO grep on this spec's owned files (slowlane.c/.h, vtaskmgr.c/.h, commit.c, netio.c/.h, ha.c, config_repl.c, cluster.c, config.h, ha.h, run-ha-soak.sh) found only the one pre-existing, already-flagged item (ha.c:1844, ha-term-safety-owned stub) -- zero new TBD/TODO/FIXME introduced by the fix. Secrets grep clean. Full test suite: libxce xcelib_test 294/0/0 (ASAN), controller_unit_test 972/0/0 (ASAN), zero AddressSanitizer/heap-use-after-free/heap-buffer-overflow/double-free/SEGV hits anywhere in the full controller test log.\n\nRUNTIME LIVENESS / SMOKE (the check that previously blocked): re-ran the canonical /xce-test-ha SEQUENTIAL 10-round soak TWICE on fresh ASAN binaries built with the fix, on real ext4 (not overlayfs, bug-924). Both fully clean: run 1 (/home/claude/tmp/xce-bug1447-verify-1) 10/10 rounds, 33 jobs, zero job loss, all 3 nodes led; run 2 (/home/claude/tmp/xce-bug1447-verify-2) 10/10 rounds, 36 jobs, zero job loss, all 3 nodes led including a forced-election leadership-coverage check on ctrl-3. An explicit grep of every node's log in both runs for bug-1447's exact signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake) found zero matches. This is a concurrency/UAF bug class, so two independent clean 10-round soaks (66 rounds of kill/elect/reconnect churn combined across both verification runs) plus a deterministic unit regression test (below) are treated as the evidence bar, not a single pass.\n\nROOT CAUSE (bug-1447, now closed): NOT the originally-suspected 'two io_worker threads racing the same SSL*' -- there is exactly ONE io_worker thread per iomgr_t (iomgr_start creates one pthread), so that hypothesis was retracted. The real defect: io_worker's epoll dispatch loop processed every ready fd from ONE epoll_wait() batch in a plain for-loop with no reference held on any of them. ha.c's on_standby_connected (raft-drive-async Task 2.1/2.2's own reconnect-offload completion callback) evicts a stale duplicate outbound peer channel via iomgr_unmonitor()+endpoint_unref() when a fresher reconnect completes -- but epoll_ctl(EPOLL_CTL_DEL) only stops a FUTURE epoll_wait() from returning events for that fd; it does not invalidate an event for that SAME stale endpoint already sitting in the CURRENT batch's events[] array (very plausible during the reconnect burst this spec's own faster cadence produces right after a new-primary election). endpoint_unref() reaching refcnt 0 immediately freed the endpoint (and its SSL object); the loop's next iteration then read that freed memory in io_handler -- a heap-use-after-free that (when ep->ssl still looked like a plausible pointer) manifested as a deep SEGV inside OpenSSL's ssl3_read_bytes, exactly matching the original crash.\n\nFIX: src/lib/netio.c -- io_worker's per-batch dispatch was extracted into io_worker_dispatch_batch() (~line 639), which now takes an endpoint_ref() on every endpoint in the batch BEFORE dispatching any of them and endpoint_unref()s each one only AFTER the whole batch is processed -- the same endpoint_ref/endpoint_unref bracket idiom this codebase already uses for the vtask dispatch path (bug-1298) and INVOKE_MSG_HANDLER, applied here to the one dispatcher that didn't yet have it. Lock-free atomics only, no I/O under the bracket -- satisfies the async-io steering no-blocking-in-the-hot-path rule. New regression test src/lib/testsuite/test_netio_batch_dispatch_uaf.c drives the real batch-dispatch loop with a synthetic 2-event epoll batch reproducing the exact same-batch eviction; TDD-verified RED (ASAN heap-use-after-free at netio.c:587) before the fix, GREEN after. Full detail in .wolf/buglog.json bug-1447 (status FIXED).\n\nINTEGRATION / COVERAGE / DESIGN: netio.c is an explicitly-owned Modified File in this spec's File Structure Plan; the fix sits entirely inside io_worker's own dispatch loop (\"the scheduling discipline of drive-thread work\", squarely in this spec's Boundary Commitments -- This Spec Owns). No Revalidation Trigger fired (the fix does not change the slow-lane submit contract, does not make iomgr/delaymgr execute work synchronously, does not change per-connection ordering semantics, and does not touch socket_connect). No new files, no dependency-direction change (netio lib -> ha controller unchanged). Requirements coverage unchanged from the prior GO-track pass (this was a runtime defect found BY this spec's own verification task 4.2/4.1 soak gate, not a requirements gap). No tasks are _Blocked:_; all 17 tasks (1.1-4.3) remain [x].\n\nDistinguish from the separate, already-accepted chaotic-mode finding (task 4.2's notes): chaotic-mode's round-4 committed-prefix divergence remains the documented pre-existing bug-851/795 family, diff-confirmed against pre-Phase-3 binaries, correctly out of scope for this spec. bug-1447 was different (a hard crash in the fully-supported SEQUENTIAL lifecycle) and is now closed.", - "known_limitations": [], - "ownership": "LOCAL", - "upstream_spec": "N/A", - "blocking_bug": null, - "remediation": [] - } -} diff --git a/.kiro/specs/raft-drive-async/tasks.md b/.kiro/specs/raft-drive-async/tasks.md deleted file mode 100644 index b7b98054..00000000 --- a/.kiro/specs/raft-drive-async/tasks.md +++ /dev/null @@ -1,163 +0,0 @@ -# Implementation Plan - -Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask. - -- [x] 1. Phase 1 — SlowLane executor and bounded connect (foundation) -- [x] 1.1 Add drive-async configuration constants - - Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count). - - Define the liveness reservation as a peer-scaled derivation, not a flat literal: `VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)` (floor 1 = deadlock-freedom; ceiling `n_vcpus − 1` keeps ≥1 commit slot); WARN at init when clamped by `n_vcpus`. - - Wire the constants into the controller config so they are overridable. - - Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init. - - _Requirements: 2.3, 3.1_ - -- [x] 1.2 Implement the SlowLane executor - - Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately. - - Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle. - - Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op. - - _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_ - - _Boundary: SlowLane_ - -- [x] 1.3 Bound the peer connect path - - Reorder `socket_connect` to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error. - - Update connect callers to pass the configured timeout. - - Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds. - - _Requirements: 2.3_ - - _Boundary: BoundedConnect_ - - _Depends: 1.1_ - -- [x] 1.4 (P) SlowLane unit tests - - Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once. - - Observable: tests pass and fail if the cap or dedup is removed. - - _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_ - - _Boundary: SlowLane_ - - _Depends: 1.2_ - -- [x] 1.5 (P) Bounded-connect unit test - - Assert `socket_connect` returns within the timeout against an unreachable host and succeeds against a reachable one. - - Observable: test passes; reverting the reorder makes it hang/fail. - - _Requirements: 2.3_ - - _Boundary: BoundedConnect_ - - _Depends: 1.3_ - -- [x] 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr - - Add `vtask_class_t` (`VTASK_LIVENESS` / `VTASK_LIVE_BLOCK` / `VTASK_REQ_BLOCK`) + a classified submit; the dispatcher reserves the `VTASK_LIVENESS_RESERVED` value **resolved in 1.1** (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a `VTASK_REQ_BLOCK` with `-XCE_ERR_NOT_READY` when admitting it would breach the reservation (never parks a worker, never silently queues). - - Fold `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` into this — `commit_submit` submits a `VTASK_REQ_BLOCK`; the pool owns the single reservation. - - Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). **Hard prerequisite of 3.1** — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters). - - Observable: a `VTASK_LIVENESS` task always finds a worker even when `*_BLOCK` tasks saturate `n_vcpus − VTASK_LIVENESS_RESERVED`; commit-wait still fast-rejects `-XCE_ERR_NOT_READY` at the cap (backpressure unchanged). - - _Requirements: 3.1, 3.2, 3.3, 4.5_ - - _Boundary: LivenessAdmission_ - - _Depends: 1.1_ - -- [x] 1.7 (P) LivenessAdmission unit test - - Assert: a `VTASK_LIVENESS` task is dispatched while `*_BLOCK` tasks hold `n_vcpus − VTASK_LIVENESS_RESERVED` vcpus; `VTASK_REQ_BLOCK` over the cap returns `-XCE_ERR_NOT_READY` without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers. - - _Requirements: 3.1, 3.2, 3.3, 4.5_ - - _Boundary: LivenessAdmission_ - - _Depends: 1.6_ - -- [x] 2. Phase 2 — Reconnect offload (sender-side liveness) -- [x] 2.1 Offload the delaymgr peer reconnect to the slow lane - - The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline. - - Refcount the endpoint across the in-flight window. - - Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time. - - _Requirements: 2.1, 2.2, 2.5_ - - _Boundary: ReconnectOffload, SlowLane_ - - _Depends: 1.2, 1.3_ - -- [x] 2.2 Complete the reconnect on the lane worker - - The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules. - - Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying. - - _Requirements: 2.4, 2.6, 4.4_ - - _Boundary: ReconnectOffload_ - - _Depends: 2.1_ - -- [x] 2.3 Reconnect-offload integration test - - With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer. - - Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall. - - _Requirements: 2.1, 2.2, 2.5_ - - _Boundary: ReconnectOffload_ - - _Depends: 2.2_ - -- [x] 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side) -- [x] 3.1 Dispatch peer-control message handling to the vtask pool - - Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline. - - Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues. - - Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order. - - **Deadlock-freedom gate (bug-invariant reversal):** moving ack decode to the pool co-locates the commit-barrier *wakers* (acks) and *waiters* (`commit_submit(COMMIT_MODE_WAIT)`) on one pool — today acks run on iomgr precisely to avoid this (`commit.c:16`). The old commit-only `COMMIT_WAIT_RESERVED` is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on **LivenessAdmission (task 1.6)**: ack/vote-receipt dispatched as `VTASK_LIVENESS` (never gated, never offloaded), `commit_submit` as `VTASK_REQ_BLOCK` under the pool reservation. **Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.** - - _Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2_ - - _Boundary: PeerCtrlDispatch_ - - _Depends: 1.2, 1.6_ - - _2026-07-01/07-02 session (production flip — DONE, both flags now default ON): with 4.1+4.2 green, executed the "full fix, defer prod flip" decision's flip. (1) Closed the snapshot-lane borrowed-endpoint hazard (task 3.2's Part A) properly: `ha_snapshot_send_dispatch` takes `endpoint_ref(ep)` before `slowlane_submit`, released on every non-lane terminal path and by `snapshot_lane_fn` on completion — reuses the general bug-1298 refcount system (standby-ep teardown already routes through `endpoint_unref` via `cluster.c`'s `endpoint_killer`, confirmed by reading it — this infra landed in raft-ha-core-safety task 5.8, AFTER this spec's 3.2/3.4 notes were written, so the hazard they describe is now closed for real). New unit test `test_snapshot_offload_holds_endpoint_ref_across_transfer` (RED→GREEN). (2) Flipped `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch_enabled` to default ON; full controller suite green; a 6-round `--chaotic` soak passed clean (zero divergence, zero job loss). (3) A `--force-snapshot` soak (genuinely empty node cold-start rejoin — the ONE scenario chaotic mode never exercises, since it restarts nodes with WAL intact) then surfaced a SEPARATE pre-existing bug, unmasked (not caused) by the flip's timing shift: `ha_stream_wal_to_standby` had no filter for `WAL_CHECKPOINT` markers (local-only, never a logical Raft entry — consumes a real LSN slot via `persist_write_checkpoint_marker`'s own `p->current_lsn++`) and shipped one to a catching-up standby, which correctly refused it and got stuck retrying the same LSN forever. FIXED (`config_repl.c`, skip `hdr.type == WAL_CHECKPOINT` entirely, mirroring the existing `replay_wal`/term-index precedent of treating checkpoint-marker LSNs as invisible gaps) with a RED→GREEN unit test (`test_wal_stream_skips_checkpoint_marker`, differential-verified). (4) Re-ran `--force-snapshot` with BOTH flags ON and the checkpoint-skip fix in place — it STILL failed, this time with a THIRD, different issue logged as **bug-1376**: the freshly-wiped empty node's own self-election races `cluster_init_phase2`'s vtask-pool wiring, then the process hangs silently. Live-repro'd deterministically via a `GDB_WRAP` debug seam added to `run-ha-soak.sh` (launches a named node directly under `gdb --batch`, so it becomes ptrace-legal as gdb's direct child under yama scope 1 — bypasses a live-attach restriction) and a bounded `SIGINT`-to-gdb interrupt for a full ASan-free thread backtrace. **Root-caused and FIXED**: `peer_ctrl_msg_handler` held `ep->dispatch_inflight` incremented across the ENTIRE `peer_ctrl_dispatch_to_pool` call, including its inline fallback (the common cold-start shape when `ha->pool` isn't wired yet). That inline body can reach `raft_cb_become_follower`, which — now that `cluster_init_phase2` ran synchronously earlier in the SAME call chain and wired the pool — tears down its OWN loser vote-ep (the very ep whose reply triggered this call) via the deferred path (`vote_ep_teardown_async`), whose `endpoint_mark_tearing_down` spins waiting for `dispatch_inflight==0` on that SAME ep — a count only this exact call, on this exact thread, holds — a permanent self-deadlock. Fixed by moving the `dispatch_inflight` release into `peer_ctrl_dispatch_to_pool` itself, at the exact right point per exit path: released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure) — safe because `io_worker` is single-threaded and the inline path never enqueues anything, so there is no pool-worker race left to protect against once inline execution is committed to — and only AFTER the (non-blocking) enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test `test_vep_self_teardown_does_not_deadlock` (`test_ha_vep_teardown.c`, bounded-timeout `pthread_timedjoin_np` join so a regression fails loud instead of hanging the suite; RED confirmed via a temporary manual revert — `Expected 0 Was 110` (ETIMEDOUT) — then GREEN restored). **Live-verified with both flags ON**: `--force-snapshot` soak now completes without hanging (2/3 runs confirmed a real snapshot install + `committed_divergence=0 committed_job_loss=0`; 1/3 hit the separate known-flaky "checkpoint didn't fire before the wipe this round" — a timing non-issue the harness itself already calls out, NOT a hang); a 6-round `--chaotic` soak with both flags ON also passed clean; full controller suite green throughout. **Both `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch_enabled` now default ON in production.** `g_drive_async_configpush_offload` (task 3.3) stays OFF — it has its own DISTINCT, still-open hazard (see task 3.3's note) unrelated to bug-1376._ - -- [x] 3.2 Offload snapshot send to the slow lane <!-- Part B (trim guard) LIVE; Part A (offload) now ON by default (bug-1197 endpoint_ref/unref fix, bug-1376 fix) — see Implementation Notes --> - - Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby. - - Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker. - - **Add the trim cursor-bound guard (bug-1147, stopgap):** `trim_wal_segments` (`persist.c:1374`) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec). - - Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby `acked_lsn` or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap). - - _Requirements: 1.1, 1.3, 1.5, 4.3_ - - _Boundary: PeerCtrlDispatch, SlowLane_ - - _Depends: 3.1_ - -- [x] 3.3 (P) Offload the transport-only config-push to the slow lane <!-- LANDED, flag g_drive_async_configpush_offload DEFAULT ON as of 2026-07-02. bug-1381 (endpoint UAF) and bug-1386 (silent chunk-drop) and bug-1389 (vtask-pool chunk reordering) all CLOSED; flip soak-validated: 6-round --chaotic PASS (zero divergence/loss) + 3/3 --force-snapshot PASS (genuine snapshot installs confirmed each time), see Implementation Notes --> - - Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged. - - Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged. - - _Requirements: 1.2_ - - _Boundary: PeerCtrlDispatch, SlowLane_ - - _Depends: 3.1_ - -- [x] 3.4 Receiver-side integration tests <!-- 3.4a UAF fix + 3.4b integration tests DONE (committed); production flag-flip deferred to after 4.1/4.2 per the "full fix, defer prod flip" decision --> - - **3.4a (DONE, committed 6d510d8)**: bug-1197 endpoint-quiescence fix landed so flag-ON peer-control dispatch is UAF-free (deferred teardown-vtask + RCU dispatch_inflight drain). - - **3.4b (DONE)**: receiver-side liveness integration tests (`test_ha_dispatch_integration.c`) with the dispatch flags flipped ON in-test — P1 a parked snapshot/config-push lane transfer does not block another endpoint's peer-control on the pool; P2 a VTASK_LIVENESS ack runs under REQ_BLOCK pool saturation (reservation never starves it; one-over fast-rejects to prove the boundary); P3 cross-endpoint peer-control runs concurrently (per-ep FIFO does not serialize across eps). Fixed a flaky teardown-race hang (bug-1224: poll end-of-body completion counters before the cancel-based vtaskmgr_fini). 5/5 deterministic, full controller suite EXIT=0, ASan-clean. - - **Production flags STILL DEFAULT OFF** (peerctrl/snapshot/configpush). Per the "full fix, defer prod flip" decision, the production flip happens only after 4.1 partition soak + 4.2 regression sweep pass; the snapshot/configpush lane borrowed-ep refcount (distinct from 3.4a) is also required before flipping those two ON. - - Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another. - - Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers. - - _Requirements: 1.1, 1.3, 1.4, 4.3, 4.5_ - - _Boundary: PeerCtrlDispatch_ - - _Depends: 3.2, 3.3_ - -- [x] 4. Phase 4 — Verification, partition test, and regression -- [x] 4.1 Network-partition (unreachable-peer) soak scenario - - Add a partition primitive that renders one node unreachable (distinct from process kill). **Primary = in-process socket-drop hook in `xce_test` (`XCE_TESTING`)** — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of `ha-term-safety` 5.2/5.4). **Secondary (optional) = firewall `iptables`/`nft` DROP** for real-network fidelity, capability-gated with a loud skip when unprivileged. - - Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost. - - Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election). - - _Requirements: 7.1, 7.2, 7.3, 7.4_ - - _Boundary: PartitionTest_ - - _Depends: 2.3, 3.4_ - -- [x] 4.2 Existing-feature regression sweep <!-- NO new regression: unit suites green + live failover soak baseline-compared (bug-1225) --> - - **DONE — verdict: raft-drive-async introduces NO new regression.** Unit-level: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites — election/prevote/liveness/failover/rdhup/repl/config-repl/persist-recovery/commit-barrier/durability — green, dormant flags OFF). Live kill-based failover soak (real fs ~/tmp, release+ASAN builds): **zero committed-JOB loss** across every completed round (jobs 3→12 monotonic). The 6-round CHAOTIC soak does not run clean to completion — it fails around round 4 — but a **baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4 IDENTICALLY** (election stall / reconverge), and the round-4 committed-prefix divergence at LSN 9 is the documented pre-existing **bug-851/795 family** (chaotic-failover silent divergence), present before Phase 3 and out of scope for this spec (needs a separate HA-core effort). See bug-1225. - - Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak. - - Observable: all listed suites pass with no change to their asserted behavior. - - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7_ - - _Depends: 3.4_ - -- [x] 4.3 Thread-role compliance and heartbeat performance check - - Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane. - - Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path. - - Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline. - - _Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2_ - - _Depends: 2.3, 3.4_ - -## Implementation Notes - -- **Build/test env**: builds MUST run in the nix dev shell (`nix develop --command meson compile -C src/xce.build <target>`). The IDE clang reports false errors (missing openssl/unity headers, unknown `-fvar-tracking`) because it isn't in the nix shell — ignore those. Controller tests: `rm -rf /tmp/xce_*` first (bug-924 overlayfs), run `./src/xce.build/controller/controller_unit_test`, set `ASAN_OPTIONS=detect_leaks=0`. -- **Pre-existing leak (not ours)**: the default-options full controller suite exits non-zero due to a LeakSanitizer leak in `test_commit_barrier_reply.c` / `test_worker_join.c` / `test_egress_filter.c` (~174KB). Unrelated to this spec. Run with `detect_leaks=0` (or a `-t` filter) so it doesn't mask raft-drive-async regressions. -- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader. -- **2.1**: reconnect offload is gated by a default-ON kill-switch `g_drive_async_reconnect_offload` (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via `ha_state_t.reconnect_inflight[]` CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = `reconnect_lane_fn` → `ha_connect_finish_peer` (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). **FOLLOW-UP (test hygiene, LOW)**: the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient. -- **1.6**: pool reservation lives in vtaskmgr (`vtask_class_t` + `blocking_inflight`/`liveness_reserved` + `vtaskmgr_blocking_acquire/release`); commit folds in via opt-in `commit_set_pool` (kept `commit_engine_init(n_vcpus)` signature — the `n_vcpus` local cap is the genuine pool==NULL fallback for unit fixtures). **Production wiring is in `cluster.c` phase2** (once-guarded, right after `commit_engine_init`): `vtaskmgr_set_liveness_reserved(R)` + `commit_set_pool` — DO NOT remove; without it the reservation is dead. Lock order `p->lock → ce->lock → blk_lock` (cycle-free). New-cap = `n_vcpus−R` (tighter than old `n_vcpus−2` for 5/7-node — intended). LOW deferred: `vtaskmgr_blocking_cap()` reads `liveness_reserved` without `blk_lock` (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in `run_task` — ensure no double-count vs the commit path. -- **3.1 (DONE — flag now DEFAULT ON, bug-1376 fixed 2026-07-02)**: `peer_ctrl_msg_handler` is now a shim gated by `g_drive_async_peerctrl_dispatch_enabled` (**DEFAULT 1 / ON**; toggle via `ha_set_peerctrl_dispatch_for_test` for a regression comparison). Flag OFF → `peer_ctrl_msg_handler_inline` runs on io_worker, byte-for-byte prior path. Flag ON → `peer_ctrl_dispatch_to_pool` builds a `VTASK_IO` task and `vtask_enq_classified`s it (per-endpoint order auto-enforced by `bind_vtask`→`endpoint_block_tasks`). The 3 routers (`vote_reply`/`pre_vote_reply`/`repl_ack`_decode_and_dispatch) take `pkt` as an explicit param now (no cross-thread `ep->fsm_in.pkt` re-read). **Packet ownership**: shim clears `ep->fsm_in.pkt` immediately (sole-owner transfer); `peer_ctrl_vtask_entry` NULLs `iod->pkt` before the body frees it once; new `vtask_t.owns_io_pkt` (default 0; controller sets 1) lets `taskq_item_killer` free an un-executed queued task's packet without double-freeing the libxce endpoint-owned path (bug-1199). `ha_state_t.pool` wired in `cluster.c` to `&c->vtaskmgr`. - - **TWO FLAG-ON GATE ITEMS — MUST be resolved before 3.4 flips the flag ON** (documented with loud code comments; harmless while flag OFF): - 1. **bug-1197 (CRITICAL UAF)**: `raft_cb_become_follower` `endpoint_fini()`s peer vote-eps from a pool worker → frees `ep->data` while the single io_worker can concurrently read `ep_j->data` in `peer_ctrl_msg_handler_inline` for a same-epoll-batch queued event. FIX (part of 3.4 enablement): `iomgr_unmonitor(ep)` BEFORE `endpoint_fini(ep)` in every pool-reachable teardown. - 2. **bug-1198 (HIGH classification)**: `peer_ctrl_classify` returns `VTASK_LIVENESS` for all, but the CTRL_PRIMARY repl-ack path can run a blocking `HA_REPL_SEND_SNAPSHOT`→`persist_checkpoint` inline under the never-gated liveness reservation → breaks deadlock-freedom. The 3.2 SlowLane offload mechanism now EXISTS but is DORMANT (flag OFF, below), so bug-1198 stays OPEN until 3.4 flips it ON together with the peerctrl flag. - -- **3.2 — Part B (trim guard, bug-1147) is LIVE; Part A (snapshot offload) is now DEFAULT ON (bug-1197 + bug-1376 fixed 2026-07-02)**: - - **Part B (production-live, default-ON, no flag)**: `trim_wal_segments` (`persist.c`) now bounds deletion by `floor = repl_trim_lsn(&p->repl, active_seg_lsn, snap_interval_s)` = `min(active_seg_lsn, every active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn)`, after a `repl_expire_disconnect_slots` call to age out stale slot guards. `floor ≤ active_seg_lsn` structurally (repl_trim_lsn caps at its `checkpoint_lsn` arg), so the new `seg_start_lsn < floor` check subsumes the old `< active_seg_lsn` guard and can only PRESERVE more segments — never delete one a live standby still needs (closes the bug-795/828 trim-vs-cursor race for connected/recently-disconnected standbys; does NOT fix empty/cursor=0 join — needs entity-bearing snapshots, separate spec). Lock-safe: `trim_wal_segments` runs with `p->lock` released; `repl_trim_lsn`/`repl_expire_disconnect_slots` take only `r->lock` — no inversion. Tests: `test_trim_guard_preserves_standby_segments`, `test_trim_without_standbys_proceeds_normally` (test_e2e_checkpoint.c). - - **Part A (DORMANT, `g_drive_async_snapshot_offload` DEFAULT 0/OFF)**: `ha_snapshot_send_dispatch` routes `HA_REPL_SEND_SNAPSHOT` inline (flag OFF — byte-for-byte legacy, send on io_worker) or to `slowlane_submit`+`snapshot_lane_fn` (flag ON). Send body factored into `ha_snapshot_do_send` (shared by both paths, holds the XCE_TESTING hook). Per-peer single-flight via `ha_state_t.snapshot_inflight[MAX_HA_CANDIDATES]` CAS + slowlane key `peer_idx+1`; flag cleared on all 4 terminal paths (alloc-fail, -EEXIST, hard-error, lane-fn completion) exactly like reconnect (2.1). Tests: `test_snapshot_offload_runs_on_lane`, `test_snapshot_inline_when_flag_off`, `test_snapshot_offload_dedups_per_peer` (test_ha_liveness.c). - - **WHY IT WAS DORMANT, AND HOW IT CLOSED**: `snapshot_lane_fn` captures a BORROWED standby endpoint (unlike reconnect, which owns its ep). With the flag ON the send runs on a lane thread while the io_worker may RDHUP-`endpoint_fini` that ep → cross-thread UAF (bug-1197 family). Flag OFF kept send+teardown serialized on the io_worker. Caught + fixed pre-commit early on: the offload flag was initially authored DEFAULT-ON (would have shipped the UAF) → corrected to OFF in both ha.c and the ha.h accessor doc (bug-1210). **Closed 2026-07-02**: `ha_snapshot_send_dispatch`/`snapshot_lane_fn` now bracket the lane dispatch with `endpoint_ref`/`endpoint_unref` (reusing the general bug-1298 refcount system), so the borrowed-endpoint hazard cannot recur regardless of thread timing — no lane-serialized-teardown redesign needed. `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch` both now default ON together (bug-1376, a distinct cold-start self-deadlock unmasked by flipping both, is also fixed). - -- **3.3 (LANDED — checkbox `[x]`, flag DEFAULT ON)**: requester-less config-push transport offload, flag `g_drive_async_configpush_offload` **DEFAULT 0/OFF** (toggle `ha_set_configpush_offload_for_test`). `ha_config_push_offloadable(c, bytes, blen, sha256, new_tag)` routes inline (flag OFF — verbatim `push_config_transport_only`, caller still owns+frees `bytes`) or to `slowlane_submit`+`configpush_lane_fn` (flag ON — DEEP-COPIES `bytes`+sha into a ctx the lane frees). Single cluster-wide single-flight via scalar `ha_state_t.configpush_inflight` CAS + lane key=1 (config push is a cluster-wide fan-out, not per-peer); flag cleared on all terminal paths (alloc-fail×2, -EEXIST, hard-error, lane completion). Send body factored into `ha_config_push_do_send` (production → `push_config_transport_only`; XCE_TESTING → `ha_configpush_probe` stub since cluster.c/transport isn't linked in test). THREE requester-less drive-thread call sites converted: `raft_cb_become_leader` (promotion, both bytes/NULL branches), the `HA_REPL_PUSH_CONFIG` heartbeat case, and the direct-CTRL_PRIMARY FSM promotion branch (~ha.c:4476). UNCHANGED: `push_config_to_standbys` admin-upgrade barrier (has a waiting requester) and `join.c`. Tests: `test_configpush_offload_runs_on_lane`, `test_configpush_inline_when_flag_off`, `test_configpush_offload_dedups` (test_ha_liveness.c). **bug-1381 CLOSED 2026-07-02** (see task 3.4a's note below for the fix) — `push_config_transport_only` itself (not the lane fn) no longer races raw standby endpoint pointers. cpp-review: no CRITICAL/HIGH/MEDIUM; controller suite EXIT=0. **Flip attempted same day, REVERTED**: with bug-1381 closed, flipped `g_drive_async_configpush_offload` to default ON, rebuilt `xce.build` (confirmed binary mtime > source mtime — see the Do-Not-Repeat cerebrum entry this session added after almost repeating the earlier stale-binary mistake), then ran `--chaotic 6` (PASS, 21 jobs, zero divergence/loss) and `--force-snapshot` x3 (0/3 PASS — "no snapshot install" / "did not reconverge to 3-of-3"). A flag-OFF control on the identical rebuilt binary went 1/2 on the same two failure signatures, proving the defect is PRE-EXISTING and NOT caused by this flag: `ha_send_snapshot_to_standby` (config_repl.c) never checks `xce_snapshot_chunk_req_raw`'s return in its chunk-send loop, so a chunk that fails to enqueue (e.g. `-ECONNRESET` during the empty-rejoin reconnect window) is silently dropped while the function still logs "transfer complete" and returns 0 — logged as **bug-1386** (open). Flag-ON's 0/3 vs flag-OFF's 1/2 is too small a sample to rule out a secondary timing interaction (configpush lane traffic narrowing the snapshot lane's completion window before reconnect), so the flag was reverted to OFF rather than accepted on a "probably fine" basis. Full controller suite re-confirmed green (230/230) at OFF. **bug-1386 FIXED 2026-07-02**: `ha_send_snapshot_to_standby` (config_repl.c) now captures the return value of `xce_snapshot_chunk_req_raw`/`xce_snapshot_chunk_hook` in its chunk loop, breaks on a non-zero `send_rc`, and propagates it as `rc` instead of silently continuing and reporting "complete"; the success log is gated on `rc==0`, a new WARN logs the failure (checkpoint_lsn/size/sent-offset/chunk_index/rc) otherwise. No receive-side timeout/re-request was added — unnecessary: `snapshot_inflight[peer_idx]` already clears on every send-thread exit (`snapshot_lane_fn`), so the next heartbeat's SEND_SNAPSHOT decision naturally retries a peer whose `applied_lsn` never advanced. New RED->GREEN test `test_snapshot_transfer_stops_and_propagates_send_failure` (test_config_repl.c) injects a mid-loop `-ECONNRESET` via the existing `xce_snapshot_chunk_hook` seam; RED confirmed via temporary revert ("Expected -104 Was 0"), GREEN restored, full controller suite clean (0 failures). **Re-soak to verify bug-1386's fix (2026-07-02, same session) surfaced a THIRD distinct pre-existing bug, STILL BLOCKING**: ran `--force-snapshot` (flag still OFF, just verifying bug-1386 in isolation before re-attempting the flip) — FAIL again, but a NEW signature: ctrl-1 (sender) now correctly logs "transfer complete" for the full 735546-byte/12-chunk transfer both times (confirming bug-1386's fix works — no more false-complete), but ctrl-2 (receiver) fails with `ha_handle_snapshot_chunk: persist_load_snapshot failed rc=-5` after receiving all 12 chunks. Byte-comparing the sender's source checkpoint against the receiver's assembled file (two independent repro runs, checkpoint LSNs 0x75 and 0x76) showed IDENTICAL total size but DIFFERENT CONTENT — the received file diverges from the source at an exact `SNAPSHOT_CHUNK_SIZE` (65536-byte) chunk boundary, then contains the correct data byte-SHIFTED (same job-frame strings, same occurrence count, wrong offset), re-synchronizing later. Shift direction/magnitude differed between the two attempts (-912 bytes / +4684 bytes) — nondeterministic, not fixed-offset. This is a chunk-content-CORRUPTION bug, not a drop (bug-1386) — logged as **bug-1389** (open, NOT fixed). Prime suspect: `xce_snapshot_chunk_encode_raw`'s capnp `Data` encode (`request.c:630-671`, `capn_new_list8`/`capn_setv8` pattern) — same family as bug-847 (WAL catch-up streaming payload corruption at a capnp segment boundary). Receive-side `persist_load_snapshot` is confirmed NOT the bug — it correctly rejects the corrupted file. **bug-1389 ROOT-CAUSED AND FIXED 2026-07-02**: added live per-chunk crc32 TRACE instrumentation at both `ha_send_snapshot_to_standby` (send) and `ha_handle_snapshot_chunk` (receive), correlated by `(snapshot_lsn, chunk_index)`, then re-ran `--force-snapshot` — the crc32 log proved every one of 12 chunks' CONTENT crossed the wire byte-for-byte intact (all 12 crc32s matched exactly), so capnp/transport was NEVER the bug (the earlier "capnp encode suspected" hypothesis, and an exhaustive prior trace of the wire framing and far-pointer resolution, are both correctly refuted). The real defect: the RECEIVE log showed chunks arriving out of `chunk_index` order — `0,1,3,4,2,5,6,7,8,9,10,11` — because the vtask pool dispatches one transfer's chunks across multiple workers (`cpu#0-3` in the log) with no ordering guarantee, but `ha_handle_snapshot_chunk` (ha.c) appended every chunk's bytes to `snap.part` in ARRIVAL order (`fopen("ab")`/`fwrite`) instead of at the chunk's own `chunk_offset` (a field already carried on the wire but never used). Reordered chunks landed at the wrong file position — same total byte count, wrong content from the first reordered chunk onward, exactly the "shifted, resyncing later, nondeterministic magnitude" signature originally observed. **Fix**: (1) `ha_handle_snapshot_chunk` now writes each chunk via `pwrite(fd, payload, len, chunk->chunk_offset)` — order-independent and safe for concurrent writers (positioned writes to disjoint byte ranges of the same fd don't race). (2) New `ha_state_t.snapshot_recv_count`/`snapshot_recv_total` fields (ha.h, guarded by the existing `ha->lock`) track how many distinct chunks have actually been written; finalize (verify/rename/`persist_load_snapshot`) now fires only once `count >= total` (known once `is_last`'s `chunk_index+1` is seen), not merely because THIS call happens to carry `is_last` — under reordering, `is_last` can process before an earlier chunk. `snapshot_recv_total` is claimed (reset to 0) by whichever thread crosses the completion threshold, preventing a late straggler from double-finalizing. `chunk_index==0` still resets both the stale-file unlink and the new counters (Req 4.4 unchanged). New RED->GREEN test `test_snapshot_recv_out_of_order_chunks_assemble_correctly` (test_config_repl.c) replays the exact non-sequential delivery order and asserts byte-identical assembly; RED confirmed via temporary revert (`Expected 0 Was -22`, `job_count` field corrupted by the misordered chunk boundary), GREEN restored, full controller suite clean. Also added 2 new full-size (65536-byte) capnp encode/decode round-trip tests (`test_wal_wire.c`) — permanent regression coverage that definitively rules out the capnp layer (the prior test only covered 300 bytes). **Live-validated**: 3/3 `--force-snapshot` rounds PASS (genuine `HA: snapshot loaded ... full job/queue entity restoration` installs confirmed each time, zero divergence/loss) + 6-round `--chaotic` PASS. **Flag re-flipped to ON 2026-07-02, LANDED**: with bug-1389 closed, `g_drive_async_configpush_offload` flipped to default 1/ON, full build clean (all components, mtime-verified), full controller suite 0 failures. Soak-validated on the flag-ON binary: `--chaotic 6` PASS (21 jobs, zero divergence/loss); `--force-snapshot` 3/3 genuine PASS (each confirmed a real `HA: snapshot loaded` install, zero divergence/loss) — one intervening run hit the unrelated pre-existing job-submission flakiness ("could not submit 110 jobs to trigger a checkpoint", never reached the snapshot-transfer phase at all) and was retried clean. Task 3.3 is now fully landed — all three requester-less config-push call sites route through the slow lane by default. - -- **3.4a (bug-1197 endpoint-quiescence fix — committed; flags STILL default OFF)**: makes the flag-ON peer-control dispatch path UAF-free so `raft_cb_become_follower` (running as a pool vtask) can free peer vote-eps while other pool workers process peer-control on the same eps. Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO + busy serialization): - - **Deferred teardown-as-vtask**: the loser loop, when `ha_peerctrl_dispatch_enabled() && ha->pool`, does per vote-ep `endpoint_mark_tearing_down(ep)` → `iomgr_unmonitor(ep)` → enqueue a `VTASK_IO`/`VTASK_LIVENESS` teardown vtask bound to that ep (`vote_ep_teardown_vtask_entry`). Per-ep busy FIFO makes it run strictly AFTER any in-flight/queued same-ep peer-control vtask; it NULLs `iod->ep` before `endpoint_fini`+`free(ep)` to disarm `run_task`'s post-body `endpoint_unblock_tasks` (which would otherwise touch the freed `ep->lock`). OFF / pool-unwired path is byte-for-byte the legacy `endpoint_fini` (no free). - - **RCU-like drain closes the TOCTOU window**: a new `_Atomic int endpoint_t.dispatch_inflight` (+ `tearing_down:1` bit). The ON dispatch brackets its flag-check+enqueue: `fetch_add(seq_cst)` → if `tearing_down` drop pkt + `fetch_sub` → else enqueue → `fetch_sub`. `endpoint_mark_tearing_down` sets the flag under `ep->lock`, releases, then spins `while(load(dispatch_inflight,seq_cst)>0) sched_yield()`. become_follower marks (drains) BEFORE enqueuing the teardown vtask, so any dispatch that read the flag false is FIFO-before teardown; any later dispatch drops. Could NOT hold `ep->lock` across the enqueue — it inverts with the taskq lock (dispatcher takes taskq→ep in `vtask_eligible`); the counter is the lock-free handshake. Drain is bounded (the held window is just calloc+non-blocking enqueue on the single io_worker). - - Files: `src/include/netio.h` (`tearing_down:1`, `dispatch_inflight`, mark/is helpers), `src/lib/netio.c` (helpers + drain + `do_endpoint_init` zero-init), `src/controller/ha.c` (`vote_ep_teardown_*`, loser-loop async branch, dispatch bracket + late-drop, test seams), `src/controller/ha.h`. Tests: `test_ha_vep_teardown.c` (5 ASan tests: ordering, late-drop, disarm, OFF-path, deterministic drain-blocks). xcelib_test 282/0; controller EXIT=0; ASan-clean. Adversarial cpp-review: no CRITICAL/HIGH; handshake formally sound (SC total order on the counter + monotone `tearing_down` under `ep->lock`); flag-ON path proven race-free. - - **CLOSED for `configpush_offload`'s prod flip**: `snapshot_lane_fn`'s borrowed-ep hazard closed 2026-07-02 (endpoint_ref/unref, see task 3.2's note). `configpush_lane_fn`'s hazard was a DIFFERENT race — logged as **bug-1381**: `push_config_transport_only` (not `configpush_lane_fn` itself) grabbed raw standby endpoint pointers and spawned unprotected `pthread_create` workers for the fan-out, a race that existed independent of the offload flag. **Fixed 2026-07-02**: each standby ep is now `endpoint_ref()`'d under `c->ha.lock` at the same point its pointer is snapshotted into `push_ctx_t`, with exactly one matching `endpoint_unref()` per ref — from `push_thread` after it finishes using `ctx->ep`, or from `push_config_transport_only`'s own cleanup if a thread never started (pthread_create failure or a total threads[] alloc failure). A concurrent RDHUP teardown can now only drop the OWNER's reference while a push is in flight, never free `ep` out from under `push_thread`. New seam `push_thread_send_hook` (config_repl.c, same pattern as `xce_wal_stream_request_hook`) lets `test_configpush_transport_holds_endpoint_ref_across_send` (test_config_repl.c) park a send mid-flight and assert the refcount bracket; RED→GREEN verified via temporary revert (`Expected 2 Was 1`), full controller suite green (230/230) after restore. `g_drive_async_configpush_offload` remains default OFF — this closes the correctness blocker for the flip, but the flip itself (and its own soak validation) is a separate follow-up, not part of this fix. diff --git a/.kiro/specs/raft-ha-core-safety/design.md b/.kiro/specs/raft-ha-core-safety/design.md deleted file mode 100644 index 65dbd74a..00000000 --- a/.kiro/specs/raft-ha-core-safety/design.md +++ /dev/null @@ -1,392 +0,0 @@ -# Design Document — raft-ha-core-safety - -## Overview - -This feature makes repeated and chaotic primary failover on an XCE controller HA cluster provably safe and live. It is a **correction-and-verification** spec, not a build-from-scratch one: the `(term, lsn)` commit-identity, conflict-reconcile, Figure-8 commit, and vote-freshness machinery already exist and are validated (`raft-log-replication`, `raft-commit-safety`, `ha-term-safety`), yet the live chaotic soak still produces committed-prefix divergence (LSN 9), standby-rejoin failure, and intermittent election stalls — on this branch and on baseline `21e26a1`. - -Discovery (task 1.3, live real-fs release soak, independently verified — see `research.md`) root-caused the residual divergence to **cross-term catch-up payload loss**, *not* the term-blind-bypass hypothesis the early analysis proposed (that hypothesis is **refuted**: the divergent entry carries the *correct* term and arrives *through* the reconcile seam). At committed LSN 9 all members agree on `term`, but the minority member holds an **empty payload** (`PLEN=0`, `pcrc=0`) where the majority holds the real entry (`PLEN=3560`). The bisect: when a restarted standby catches up an entry whose stored term is below the current leader's term (the bug-1067 cross-term path), the leader-side streamer `ha_stream_wal_to_standby` (`config_repl.c` ~2154-2163) **silently ships `PLEN=0`** on a short `pread` or `malloc` failure instead of failing loud; the receiver faithfully persists the empty body; and the persistence layer's **`(term,lsn)`-only idempotency guard is payload-blind**, so it never detects that an existing-LSN entry now has a divergent payload. The result is a permanently divergent committed prefix that carries the right term but the wrong (empty) data. - -The design therefore: (1) collapses every entry-ingest path into the one reconcile seam and makes persistence `(term, lsn)`-conflict-aware instead of LSN-only-idempotent; (2) closes the snapshot-boundary and standby-reconvergence gaps; (3) guarantees election liveness under churn; and (4) delivers a deterministic chaotic harness that forces each defect RED before its fix and a green 6+ round real-fs soak as the acceptance gate. - -### Goals -- Committed log prefix is identical across all members under repeated/chaotic failover (zero `walcompare` divergence). -- Every entry entering a member's log passes the single `(term, lsn)` reconcile seam; no path stamps a non-leader term or silently skips conflict repair. -- A restarted node deterministically reconverges to 3-of-3 (committed + applied), and elections always complete under churn. -- Each defect is pinned by a deterministic RED→GREEN test; the spec closes only on a green 6+ round chaotic real-fs release soak with zero divergence and zero committed-job loss. - -### Non-Goals -- Rebuilding or re-deriving the Raft library consensus core (log replication, commit advance, vote freshness) — adopted as-is. -- Persist-before-ACK durability and the sync barrier — owned by `ha-commit-durability`. -- `raft-drive-async` async-drive/offload behavior (SlowLane, peer-control dispatch, snapshot/config-push offload, endpoint-quiescence) — out of scope and must not change. -- Client-side primary discovery (libxce bug-854) and the overlayfs `fdatasync` environment defect (bug-924) — excluded from the acceptance verdict, not fixed here. -- Cluster membership resize, multi-region, non-Raft replication. - -## Boundary Commitments - -### This Spec Owns -- The **single-seam ingest invariant**: routing every replicated/catch-up entry through `raft_on_append_entries_request` so logOk + `(term,lsn)` conflict-truncate + leader-term stamping always run. -- Making the WAL persistence idempotency/contiguity guard **`(term,lsn)`-conflict-aware** (detect same-LSN/different-term as a conflict to repair, not a no-op to skip). -- **Snapshot-boundary `(term,lsn)` consistency** so post-install AE intake reconciles correctly and no committed entry below the checkpoint can diverge. -- **Standby reconvergence completeness**: post-recovery `log_len`/applied sync and catch-up apply completeness before a node reports healthy/3-of-3. -- **Election liveness under churn**: a member that loses its primary always proceeds to a completed election; no permanent stall. -- The **deterministic chaotic harness** (unit/integration + `xce-test-ha` soak extension + `walcompare` use) and the **acceptance soak gate**. - -### Out of Boundary -- Raft library internals (`src/lib/raft/*`) beyond consuming the existing callbacks/seam — no new consensus primitives. -- Persist-before-ACK durability mechanism and the commit sync barrier (`ha-commit-durability`). -- `raft-drive-async` dispatch behavior. -- libxce client discovery (bug-854); overlayfs env (bug-924). -- Sequential-failover rejoin reconcile already delivered by `ha-convergence-recovery` (reused, not re-implemented). - -### Allowed Dependencies -- Raft library reconcile seam + five callbacks (`raft_on_append_entries_request`, `on_persist_entry`, `on_get_entry_term`, `on_truncate_from`, `on_commit_advanced`). -- `persist_set_term` / per-LSN term index / `WAL_LEGACY_TERM` mapping (`ha-term-safety`). -- `ha_rejoin_reconcile` intake hook and `ha_self_restart` convergence primitive (`ha-convergence-recovery`). -- `walcompare` `(term, lsn, pcrc)` detector; `xce-test-ha` soak driver; `run-ha-soak.sh` BIN/XT override + non-ASAN release build. -- The controller TLA+ spec `raft/raft.tla` and the invariant checklist in `src/controller/CLAUDE.md` as the correctness oracle. - -### Revalidation Triggers -- Any change to the Raft callback signatures or the reconcile-seam contract → revalidate `ha-term-safety`, `raft-log-replication`, `raft-commit-safety`. -- Any change to WAL header `(term, lsn)` layout or the term index → revalidate `walcompare` and `ha-commit-durability`. -- Any change to the snapshot install boundary contract → revalidate `ha-convergence-recovery` rejoin path. -- Closing this gate (green soak) is itself the revalidation trigger that **unblocks the `raft-drive-async` production flag-flip**. - -### Retroactive Boundary Crossings (2026-07-09 validate-impl finding) - -Two tasks materially crossed the "Out of Boundary" lines above. Both are self-labeled in tasks.md, non-conflicting with the specs that inherited them, and confirmed by a later cross-task validation pass — documented here so a future validate-impl doesn't re-flag them as new findings: - -- **Task 5.9** shipped a commit-durability fix (`persist_submit_wal_and_commit` extraction, hook-rc-clobber job-loss) — territory line 19/36 assigns to `ha-commit-durability`. Landed here as a "down payment" (tasks.md's own words) because `raft-apply-decoupling` had no tasks.md yet and this spec's Req 8 soak needed zero job-loss to close. `raft-apply-decoupling`'s own task 3.1/3.2 later built the canonical version on the same code path — confirmed non-duplicating successor, not a competing fix. -- **Task 5.8** shipped an endpoint-refcount/quiescence mechanism — territory line 20 explicitly excludes ("`raft-drive-async`... endpoint-quiescence — out of scope and must not change"). In practice it became the shared foundation `raft-drive-async`'s own task 3.2/3.4a cite as the reason their UAF hazards are "closed for real" — both specs' authors acknowledged the dependency in their own tasks.md, making this real, coherent, bidirectional coupling rather than silent scope creep. - -## Architecture - -### Existing Architecture Analysis -- **The real defect (task 1.3, verified):** the cross-term catch-up *payload* is lost at its source. `ha_stream_wal_to_standby` (`config_repl.c` ~2154-2163) reads the entry body with `pread` into a `malloc`'d buffer; on a short read or allocation failure it frees the buffer and ships the entry with `PLEN=0` **silently** (no error, no retry). The receiver persists the empty body, and the `(term,lsn)`-only persistence guard cannot tell that an existing-LSN entry now carries a divergent (empty) payload. Term and ingest-path are correct; the data is gone. -- **Single reconcile seam (preserve, do not duplicate):** all `(term,lsn)` conflict detection/repair lives inside `raft_on_append_entries_request` (raft.c:646–765). This seam is sound and is preserved as an invariant — but it is *not* where this bug lives (the divergent entry already passes through it with the correct term). The fix is upstream (streamer payload integrity) and underneath (payload-aware persistence guard). -- **Layering (must hold):** Raft library (pure, callback-driven) ← controller HA adapter (`ha.c`, `ha_convergence.c`, `config_repl.c`) ← persistence (`persist.c`, WAL). The adapter calls into the library and implements callbacks; the library never calls the adapter except through callbacks. The fix routes adapter ingest *into* the library, never around it. -- **Two distinct truncate contracts (must not be conflated):** - - **Promotion-truncate** (a node becoming leader): clamps at `max(safe_committed_lsn, durable_tip)` (ha.c:999–1000, bug-1060) — a fresh leader must not discard its own durable tip. - - **Follower-AE-conflict-truncate** (a follower/standby receiving an entry that conflicts at `(term, lsn)`): obeys Raft Log Matching — truncates the conflicting suffix **regardless of local `durable_tip`**. The bug-1060 promotion clamp does **not** apply here. Rationale: an entry that conflicts with the leader's `(term, lsn)` was, by Raft's Leader Completeness, never majority-committed, so it is safe — and required — to remove even if the local node had marked it durable/committed. Applying the promotion clamp on this path would block legitimate repair and convert repairable divergence into a permanent fence. - - The only entries the follower-conflict path must never cut are those that match the leader (no conflict) — guaranteed by logOk before any `on_truncate_from`. -- **Technical debt addressed:** the silent empty-payload ship in the cross-term catch-up streamer and the payload-blind persistence idempotency guard are the primary debt this spec removes. The term-blind ingest branch is retired as *defensive* hardening (it is a latent hazard, not the cause of the observed divergence). - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph Library [Raft Library pure] - AE[raft_on_append_entries_request] - LOGOK[logOk term lsn check] - TRUNC[on_truncate_from] - PERSISTCB[on_persist_entry] - AE --> LOGOK --> TRUNC - LOGOK --> PERSISTCB - end - subgraph Adapter [Controller HA Adapter] - STREAM[Catchup Streamer payload integrity] - INGEST[Unified Entry Ingest] - SNAP[Snapshot Boundary Reconcile] - CONV[Standby Reconvergence] - ELECT[Election Liveness] - end - subgraph Persist [Persistence WAL] - GUARD[term lsn payload conflict guard] - SETTERM[persist_set_term leader term] - TERMIDX[per lsn term index] - end - subgraph Verify [Verification] - HARNESS[Deterministic Chaotic Harness] - WALCMP[walcompare term lsn pcrc] - SOAK[xce-test-ha chaotic soak] - end - STREAM --> INGEST - INGEST --> AE - SNAP --> AE - CONV --> INGEST - PERSISTCB --> SETTERM --> GUARD --> TERMIDX - TRUNC --> GUARD - ELECT -.guarantees.-> AE - HARNESS --> WALCMP - HARNESS --> INGEST - SOAK --> WALCMP -``` - -**Architecture Integration** -- **Selected pattern:** payload-integrity-at-source + payload-aware-persistence, with the single reconcile seam preserved as an invariant. The cross-term catch-up streamer must never ship a non-empty entry as empty (fail-loud on short `pread`/`malloc`-fail); persistence enforces `(term, lsn, payload-crc)` conflict semantics so a divergent-payload entry at an existing LSN is detected, not idempotent-skipped. -- **Boundaries:** catch-up payload integrity (Adapter streamer), persistence-conflict (Persist), reconvergence + liveness (Adapter), verification (test-only) are separable seams under one acceptance gate. -- **Patterns preserved:** single reconcile seam, callback-driven library, truncate-clamp, persist-before-ACK. -- **New components rationale:** the harness is genuinely new code; the streamer fix hardens a silent-failure path; the persistence change extends an existing guard from `(term,lsn)` to `(term,lsn,payload-crc)`. -- **Steering compliance:** async-I/O (no blocking in vtask handlers; file I/O + restart on delaymgr, never callback thread), fail-loud over silent skip, controller-never-self-terminates-on-operational-condition. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Consensus core | `src/lib/raft` (in-repo) | reconcile seam + callbacks (consumed) | no library changes beyond callback use | -| Controller HA adapter | C (`ha.c`, `ha_convergence.c`, `config_repl.c`) | ingest routing, snapshot/rejoin, election | primary change surface | -| Persistence | C (`persist.c`, WAL segments) | `(term,lsn,payload-crc)`-aware guard, term index | harden guard, no format change | -| Verification | C unit (`controller_unit_test`), `walcompare`, `xce-test-ha` bash | RED→GREEN + soak gate | new `test_ha_core_safety.c`; chaotic soak mode | -| Runtime | real filesystem (ext4) release build | acceptance soak | overlayfs excluded (bug-924) | - -## File Structure Plan - -### Modified Files -- `src/controller/config_repl.c` — **Core fix.** Leader-side catch-up streamer `ha_stream_wal_to_standby` (~2154-2163): on a short `pread` or `malloc` failure when reading an entry body, **fail loud** (error + retry/abort the stream) instead of silently freeing the buffer and shipping `PLEN=0`. A non-empty entry must never be transmitted with an empty payload. Disambiguate and handle both send-side-empty (short read/alloc fail here) and any transitive-empty-propagation source uncovered during implementation. -- `src/controller/persist.c` — Extend the replicated-write guard (`persist_wal_write_replicated` persist.c:802, idempotency guard at :822/:839) from `(term,lsn)`-only to **`(term, lsn, payload-crc)`-aware**: an entry arriving for an existing LSN whose payload-crc differs (e.g. a real entry over a previously-stored empty one, or vice versa) is a conflict to surface for repair, not an idempotent `return 0`. Never accept an empty payload as the durable form of a known-non-empty entry. Preserve persist-before-ACK. -- `src/controller/ha.c` — Receiver-side handling so a recovered/divergent-payload entry is repaired through the reconcile seam (`ha_consume_append_entries` ha.c:2410-2440 body copy; `ha_apply_wal_catchup_entry` ha.c:2233). Snapshot post-install (`ha_handle_snapshot_chunk` ha.c:2492) establishes the `(term,lsn)` boundary. Standby reconvergence completion (applied/log_len sync, healthy gating). Election-liveness guarantee on the failover/vote path. **Defensive:** retire the term-blind `ha_apply_single_entry` (ha.c:2097) direct-persist branch so no future path can bypass the seam. -- `src/controller/ha_convergence.c` — Reuse the `ha_rejoin_reconcile` intake hook for chaotic-rejoin reconciliation through the seam; retain the `ha_self_restart` convergence primitive and once-guard. -- `src/controller/testsuite/meson.build` — Register `test_ha_core_safety.c`. -- `src/controller/testsuite/walcompare.c` — Extend to emit a machine-checkable `(lsn → term, pcrc, present)` per-member report consumable by the harness verdict (committed-vs-uncommitted classification). -- `.claude/skills/xce-test-ha/run-ha-soak.sh` (and `SKILL.md`) — Chaotic mode + isolated verdict (done, tasks 1.1/1.2). **Plus (task 1.3 follow-up):** make primary-detection robust to the release build's block-buffered `--logfile` — switch the soak's primary/convergence detection to a client probe (`ctrlstatus`) rather than log-grep, so the chaotic soak runs unattended on release binaries. -- `src/controller/` log init (only if the client-probe approach is insufficient) — ensure the release build flushes (line-buffers) its `--logfile` so operator-observable progress is not lost until signal. Prefer the soak-side client-probe fix first; treat a controller log-flush change as a fallback that must not alter production logging semantics. - -### New Files -- `src/controller/testsuite/test_ha_core_safety.c` — Deterministic RED→GREEN harness: forces the **cross-term catch-up payload-loss** divergence (a non-empty entry streamed/persisted as empty at an existing committed LSN), snapshot-boundary divergence, standby reconvergence-failure, and election-stall paths using existing gate/probe seams (pattern: `test_ha_dispatch_integration.c`); no sleeps. - -> Dependency direction (enforced): `persist.c` (term/lsn storage) ← `ha.c`/`config_repl.c`/`ha_convergence.c` (adapter ingest/election) → Raft library seam. Tests depend on all; production never depends on tests. - -## System Flows - -### Cross-term catch-up payload integrity (the fix) - -```mermaid -sequenceDiagram - participant L as Leader streamer - participant S as Standby ingest - participant Seam as raft_on_append_entries_request - participant P as persist term lsn payload guard - L->>L: read entry body pread into malloc buffer - alt short read or malloc fail - L->>L: FAIL LOUD error retry or abort stream - Note over L: never ship a non empty entry as PLEN 0 - else full body - L->>S: stream entry at LSN N with full payload and term - end - S->>Seam: route through reconcile seam - Seam->>P: on_persist_entry N term payload - alt existing LSN payload crc differs - P->>P: conflict surface for repair not idempotent skip - else identical term and payload crc - P->>P: idempotent no op safe - end - P-->>S: WAL holds full payload at N - Note over S,P: walcompare term and pcrc identical across members -``` - -Key decisions: the streamer fails loud on a short `pread`/`malloc` failure instead of silently shipping `PLEN=0`; the persistence guard compares `(term, lsn, payload-crc)` so an empty-over-real (or real-over-empty) write at an existing LSN is a conflict to repair, not an idempotent skip. The single reconcile seam is preserved; truncation (when a genuine `(term,lsn)` conflict exists) never cuts below `max(safe_committed_lsn, durable_tip)` on the promotion path and follows the follower-conflict contract on the receive path. - -### Standby reconvergence gating - -```mermaid -stateDiagram-v2 - [*] --> Rejoining - Rejoining --> CatchingUp: route entries through seam - CatchingUp --> AppliedSync: log tip reached at match level - AppliedSync --> Healthy: applied_lsn and log_len equal leader committed - AppliedSync --> CatchingUp: apply gap remains - Healthy --> [*]: report 3 of 3 -``` - -A node is reported healthy/3-of-3 only when committed **and** applied state match the leader — not at match-index alone (closes the bug-1042 reconvergence gap). - -## Requirements Traceability - -| Requirement | Summary | Components | Key Contracts | Flows | -|-------------|---------|------------|---------------|-------| -| 1.1–1.4 | Committed prefix identical; no divergent entry at a position; divergent treated non-authoritative; TLA+ State-Machine/Commit safety | Catch-up Payload Integrity; Persist Conflict Guard | streamer fail-loud; `(term,lsn,payload-crc)` guard | Payload integrity | -| 2.1–2.4 | Entry carries faithful identity+payload; never empty-over-real at a committed LSN; identity distinguishes conflicts; survives restart | Persist Conflict Guard; Catch-up Payload Integrity | `(term,lsn,payload-crc)` guard; `persist_set_term`; term index | Payload integrity | -| 3.1–3.4 | Detect conflict; repair to majority prefix; surface unrecoverable; never cut majority-committed | Unified Entry Ingest; Snapshot Boundary | `on_truncate_from`; follower-conflict truncate contract (logOk-gated) | Divergence repair | -| 4.1–4.3 | At most one committing primary/term; step-down on higher term; Election-Safety/Leader-Completeness | Election Liveness (step-down) | vote-once; pre-vote; vote freshness | — | -| 5.1–5.4 | Rejoin → 3-of-3 in bound; applied+log_len sync; catch-up completeness; not converged until applied matches | Standby Reconvergence | healthy-gating contract | Reconvergence gating | -| 6.1–6.3 | Elect higher-term primary in bound; keep electing under churn; retry on failed round | Election Liveness | election-restart guarantee | — | -| 7.1–7.5 | Force divergence/reconverge/stall RED on baseline; GREEN after; extend soak + unit/integration | Deterministic Chaotic Harness | RED→GREEN gate | all | -| 8.1–8.4 | 6+ round chaotic real-fs soak: zero divergence, zero job loss, reconverge+elect each round; gate for flag-flip | Acceptance Soak Gate | soak verdict (bug-854/924 isolated) | all | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| Catch-up Payload Integrity | Adapter | Stream a non-empty entry never as empty; fail-loud on short read/alloc-fail | 1,2,3 | catch-up streamer (P0); Persist guard (P0) | Service, State | -| Persist Conflict Guard | Persist | `(term,lsn,payload-crc)`-conflict-aware write; never accept empty-over-real | 1,2,3 | term index (P0); `persist_set_term` (P0) | State | -| Unified Entry Ingest | Adapter | Route every entry through the one reconcile seam; retire term-blind bypass (defensive) | 1,2,3 | Raft seam (P0); Persist guard (P0) | Service, State | -| Snapshot Boundary Reconcile | Adapter | Establish `(term,lsn)` at checkpoint; post-install reconcile | 3,5 | snapshot install (P0); seam (P1) | State | -| Standby Reconvergence | Adapter | Gate healthy/3-of-3 on committed+applied match | 5 | catch-up apply (P0); `ha_rejoin_reconcile` (P1) | Service, State | -| Election Liveness | Adapter | Always complete an election under churn; step-down on higher term | 4,6 | vote/election path (P0) | Service | -| Deterministic Chaotic Harness | Verify | Force each defect RED→GREEN; no sleeps | 7 | gate/probe seams (P0); `walcompare` (P0) | Batch | -| Acceptance Soak Gate | Verify | 6+ round chaotic real-fs soak verdict | 8 | `xce-test-ha`; release build | Batch | - -### Adapter - -#### Catch-up Payload Integrity -| Field | Detail | -|-------|--------| -| Intent | The cross-term catch-up streamer must transmit every non-empty entry with its full payload, or fail loud — never silently ship `PLEN=0`. | -| Requirements | 1.1, 1.2, 2.1, 2.3, 3.1 | - -**Responsibilities & Constraints** -- In `ha_stream_wal_to_standby` (`config_repl.c` ~2154-2163): when reading an entry body (`pread` into a `malloc`'d buffer), a short read or allocation failure must be handled as an error (log + retry/abort the stream), **not** by freeing the buffer and sending the entry with an empty payload. -- Disambiguate the two possible sources the reviewer flagged: (a) **send-side empty** — the short-read/alloc-fail here; (b) **transitive-empty propagation** — the streamed source WAL already holds an empty body (itself produced by an earlier instance of this defect). The fix must close (a) directly and, if (b) is observed, trace it to its own streamer instance rather than masking it. -- Cross-term entries (stored term < leader current term, the bug-1067 path) are the trigger seen live — ensure they are read and shipped identically to same-term entries. - -**Dependencies**: Inbound: walAck catch-up continuation (P0). Outbound: Persist Conflict Guard (P0) on the receiver; the reconcile seam (P1). - -**Contracts**: Service [x] / State [x] -- Precondition: the source WAL holds a non-empty body for the entry being streamed. -- Postcondition: the receiver persists the same `(term, lsn, payload-crc)` the leader holds; a transmit failure is surfaced, never silently degraded to an empty entry. -- Invariant: `walcompare` reports identical `(term, pcrc)` for the streamed LSN across leader and standby. - -**Implementation Notes** -- Validation: the RED test reproduces the live signature — a cross-term catch-up of an entry whose body read short/fails must NOT result in a persisted `PLEN=0` entry at a committed LSN; instead the stream errors and retries. Pin send-side vs transitive via an instrumented `payload_len` capture at send and at `on_persist_entry`. -- Risks: failing loud on the stream must not wedge catch-up permanently — pair the error with a bounded retry so a transient short read recovers, while a persistent failure is surfaced (fail-loud over silent skip; controller never self-terminates). - -#### Unified Entry Ingest -| Field | Detail | -|-------|--------| -| Intent | Every replicated/catch-up entry passes `raft_on_append_entries_request` (logOk + `(term,lsn)` conflict + leader-term stamp). The term-blind direct-persist branch is closed by reachability, not by removal — task 6.2 traced every production sender and found it unreachable with real data; documented SAFE-BY-GUARD rather than structurally retired (see task 6.2's Implementation Note below). | -| Requirements | 1.2, 2.3, 3.1, 3.2, 3.4 | - -**Responsibilities & Constraints** -- Defensive: the `entry->term == 0` direct-persist branch (`ha_apply_single_entry`) is NOT structurally removed (task 6.2 decision). Every real sender (`repl_broadcast` for live replication, `ha_stream_wal_to_standby` for catch-up) always supplies `n_entries>=1` with a non-zero term, so this branch is unreachable with production data; the sole function that could ever drive it (`repl_send_append_entries`) is invoked only by the raft library's `raft_send_append_entries`, which has zero production callers today. It remains documented and guarded in `ha.c`, not retired. (Not the cause of the observed divergence, but a latent bypass hazard — now shown to be dormant, not live.) -- Conflict repair on this (follower/standby) path uses the **follower-AE-conflict-truncate** contract (truncate the conflicting suffix per Raft Log Matching) — **not** the promotion clamp. logOk gates the truncate, so only genuinely-conflicting (never-majority-committed) entries are cut. The promotion clamp applies only to the leader-promotion path, which this component does not touch. -- File I/O / truncation / restart never on a callback or iomgr thread — defer to delaymgr (existing pattern). - -**Dependencies**: Inbound: catch-up/replication/rejoin call sites (P0). Outbound: Raft seam (P0); Persist Conflict Guard (P0). - -**Contracts**: State [x] / Service [x] - -##### State Management -- State model: per-member WAL log indexed by LSN, each entry carrying `(term, payload)`; library `log_len`/`commit_index` mirror the WAL. -- Consistency: post-ingest, `on_get_entry_term(N)` equals the leader's term for N across all members; `walcompare` reports identical `(term, pcrc)` on the committed prefix. -- Concurrency: ingest may run on iomgr; deferred truncate/persist follow the existing delaymgr one-shot pattern. - -**Implementation Notes** -- Integration: reuse `ha_apply_term_entry_via_library`/`ha_consume_append_entries` for every reachable ingest path. -- Validation (task 6.2, closing task 2.4's deferral): rather than retiring `ha_apply_single_entry`'s direct-persist path, task 6.2 traced every production sender of a `wal_catchup_entry_t` message and found the term-blind branches (the batch-mode `sub->term==0` fallthrough and the unconditional `entry->n_entries==0` single-entry mode in `ha_apply_wal_catchup_entry`) unreachable with real data: `repl_broadcast` always sends `n_entries==1` with the real stamped term; `ha_stream_wal_to_standby` always batches `n_entries>=1` and sources each sub-entry's term via `persist_entry_term`, whose legacy-zero rule maps a stored term of 0 to `WAL_LEGACY_TERM` for any present LSN (never 0). The only function that could emit the term-blind shape, `repl_send_append_entries`, is driven solely by the raft library's `raft_send_append_entries`, which has zero production call sites (only the pure library simulator and an `XCE_TESTING` test helper call it). Decision: **document safe-by-guard, do not structurally retire** — retiring would mean building a term-0-safe seam variant for a path with no live trigger, which is speculative work the evidence does not warrant. The disposition and reachability trace are recorded as comments at `ha_apply_single_entry` and both term-blind branches in `ha.c`, and pinned by `test_2_3_termblind_route_payload_guarded`, `test_2_3_unrepairable_catchup_raises_nonpromotable_fence`, and `test_6_2_real_senders_never_produce_termblind_shape` (`test_ha_core_safety.c`). -- Risks: the safety boundary on this path is **logOk**, not a durable-tip clamp — only entries that fail the `(term, lsn)` match are truncated, and a failed match proves the entry was never majority-committed (Leader Completeness). A unit test must assert that an entry which *matches* the leader is never truncated, and that a *conflicting* entry below local `durable_tip` *is* truncated. - -#### Standby Reconvergence -| Field | Detail | -|-------|--------| -| Intent | A recovered standby — including a former primary holding a divergent uncommitted tail above the new leader's committed point — reconverges to follower (truncates the divergent tail, catches up) and is reported healthy/3-of-3 only after committed **and** applied state match the leader. | -| Requirements | 5.1, 5.2, 5.3, 5.4 | - -**Responsibilities & Constraints**: -- **Divergent-tail reconvergence (the live-stall fix, task 1.3/Phase-4 investigation):** when a standby recovers with an uncommitted tail whose entries conflict with the leader's log (the former-primary `current_lsn > leader_committed` shape — e.g. ctrl-2 holding 12-15 at an old term while the leader committed 11), the AE log-matching backtrack must reach the divergence point and the follower must **truncate the conflicting suffix** (follower-AE-conflict-truncate contract) and accept the leader's entries — the same way a short-suffix standby already reconverges (`ae_reject_logmismatch` ×N → `ae_already_have`). Determine and remove whatever prevents a longer/former-primary divergent tail from reaching `ae_already_have` (candidate: the leader's `next_index` backtrack does not cross the divergence before the control channel drops; or the recovered tip is advertised in a way that stalls the backtrack). -- Post-recovery `log_len`/applied-lsn sync; complete outstanding catch-up apply before "healthy"; bounded recovery window. - -**Dependencies**: Inbound: rejoin path (P0). Outbound: Unified Entry Ingest (P0); `ha_rejoin_reconcile` hook (P1); the leader-side AE streamer `next_index` backtrack (P0). - -**Contracts**: State [x] / Service [x] -- Postcondition: a recovered standby with a divergent uncommitted tail truncates it and reaches `applied_lsn == log_len == leader_committed_lsn`; reported converged only then, otherwise `CatchingUp`. - -**Implementation Notes** -- Integration: reuse the `ha-convergence-recovery` rejoin hook; ensure the AE backtrack + follower-conflict-truncate path actually fires for a former-primary divergent tail, not only a 1-2 entry suffix. -- Validation: RED integration test — construct a recovered standby with a multi-entry divergent uncommitted tail above the leader's committed point and assert it reconverges to `ae_already_have` (no permanent `ae_reject_logmismatch` loop) and is not reported converged until applied state matches. - -#### Election Liveness -| Field | Detail | -|-------|--------| -| Intent | A member that loses its primary always proceeds to a completed election; at most one committing primary per term. | -| Requirements | 4.1, 4.2, 4.3, 6.1, 6.2, 6.3 | - -**Responsibilities & Constraints**: -- On *genuine* primary loss with quorum reachable, elect a higher-term primary within the failover window; restart a failed election round; step down on observing a higher/equal-term leader. No permanent stall under repeated rapid rounds. -- **Spurious-failover suppression (the live-stall fix, task 1.3/Phase-5 investigation):** a *transient* primary control-channel RDHUP must not launch an unwinnable PreVote-failover storm while a current-term primary is actually alive and reachable by the quorum. The observed stall: a recovered standby whose channel to the live primary flaps (~15s) treats each RDHUP as "primary died" → `ha_begin_failover` → PreVote that times out (peers see the primary alive, deny/ignore) → reconnect → repeat forever, so the node never rejoins. The fix: on a control-channel RDHUP, **attempt reconnect + catch-up first**, and/or back off failover when a node's own PreVotes repeatedly fail to reach a quorum because a current-term leader exists — only escalate to a real failover when the primary is genuinely unreachable to the quorum. The election *retry* already works (R6); the defect is retrying an unwinnable election instead of rejoining the live primary. - -**Dependencies**: Inbound: failover detection / `ha_on_primary_ctrl_fd_rdhup` (P0). Outbound: Raft vote/election + `raft_cb_vote_eligible` (P0); peer reconnect path (P0). - -**Contracts**: Service [x] -- Invariant: never two committing primaries in one term (Election Safety); a node without a *reachable* primary keeps initiating elections (liveness); a node WITH a reachable current-term primary rejoins it rather than launching repeated unwinnable failovers. - -**Implementation Notes** -- Validation (liveness, R6): RED integration test forces repeated rapid kill/restart rounds and asserts a higher-term `CTRL_PRIMARY` appears each round; a scripted single-round failure asserts a follow-on election starts. -- Validation (spurious-failover, the live stall): RED test/soak asserts a recovered standby whose control channel flaps while a current-term primary is alive does NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. -- Validation (safety, R4 — two-leaders-per-term): an explicit **Election Safety assertion** runs in the chaotic harness and soak — for every observed term, at most one member ever reaches committing-primary state. This is a *falsifiable verification*, not an assertion of correctness: it maps to `raft/raft.tla` Election Safety and must hold under overlapping kills. If it fails, the design's "library already enforces single-leader" assumption is disproven and a concrete guard (e.g. tighten the pre-vote/vote-once window at the adapter promotion gate) becomes required work rather than verification. -- Risks: over-eager step-down could itself cause a stall — step-down must be paired with election restart. The harness must distinguish a *liveness* failure (no primary) from a *safety* failure (two primaries) so a fix for one does not regress the other. - -### Persist - -#### Persist Conflict Guard -| Field | Detail | -|-------|--------| -| Intent | `(term, lsn, payload-crc)`-aware write: identical `(term, payload-crc)` at an LSN is an idempotent no-op; a *differing* payload-crc (or term) at an existing LSN is a conflict surfaced for repair — never a silent `return 0`. An empty payload is never accepted as the durable form of a known-non-empty entry. | -| Requirements | 1.1, 1.2, 2.1, 2.2, 2.4, 3.1, 3.4 | - -**Responsibilities & Constraints**: extend the LSN-only idempotency check (persist.c:822) to compare `(term, payload-crc)`; preserve persist-before-ACK; preserve the term index and `WAL_LEGACY_TERM` mapping. - -**Dependencies**: Inbound: `on_persist_entry`/`on_truncate_from` (P0). Outbound: term index `record_entry_term`/`persist_entry_term` (P0). - -**Contracts**: State [x] -- Precondition: caller has passed the entry through the seam (leader term known via `persist_set_term`; logOk already decided match vs conflict). -- Postcondition: WAL at LSN N holds exactly the leader's `(term, payload)`; no silent retention of a stale-term or empty-payload entry. -- Invariant: never write `term == 0` for a live-replicated entry, and never let a `PLEN=0` write overwrite or shadow a known-non-empty entry at the same LSN. On the seam-driven conflict path the guard honors `on_truncate_from` (follower-AE-conflict contract) — it does **not** re-apply the promotion durable-tip clamp, which belongs to the leader-promotion path only. The guard never *initiates* a truncate on its own; it only executes the seam's decision. - -**Implementation Notes** -- Integration: the guard distinguishes "already have identical entry (same term + payload-crc)" from "have a different-payload entry at this LSN"; the latter is surfaced for repair instead of `return 0`. -- Validation: unit test writes an empty-payload entry at LSN N, then the correct full-payload entry, and asserts the WAL ends with the full payload (no silent idempotent skip); and the reverse (full then empty) never degrades a stored entry. -- Risks: mis-tagging an identical `(term, payload-crc)` entry as a conflict would cause needless work — the guard compares both term and payload-crc and treats a true match as idempotent. - -### Verify - -#### Deterministic Chaotic Harness -| Field | Detail | -|-------|--------| -| Intent | Force each defect deterministically; RED on baseline, GREEN after fix; no sleeps. | -| Requirements | 7.1, 7.2, 7.3, 7.4, 7.5 | - -**Contracts**: Batch [x] -- Trigger: unit/integration run (`controller_unit_test`) + `xce-test-ha` chaotic soak. -- Input: injected divergent `(term,lsn)` entry; injected apply gap; scripted overlapping kills inside election windows. -- Output: pass/fail per path; `walcompare` `(term,lsn,pcrc)` report. -- Idempotency & recovery: deterministic gate/probe seams (pattern `test_ha_dispatch_integration.c`); teardown drains pending tasks (avoid the known teardown-race UAF class, bug-1224). - -**Hypothesis-validation discipline (mandatory):** -- The live-repro gate (task 1.3) has **already run and refuted the term-blind hypothesis** (verified): the real defect is cross-term catch-up payload loss (`research.md` §"Task 1.3"). The RED unit/integration cases must therefore reproduce the *observed* signature — a non-empty entry persisted as `PLEN=0` at a committed LSN under cross-term catch-up — not the synthetic term-conflict the earlier draft assumed. -- A green synthetic unit test over a still-red soak is not a pass: R1/R8 close only when the chaotic real-fs soak (task 7.2) reports zero `walcompare` divergence with the streamer + guard fixes in place. -- **Contingency**: if the soak still diverges after the streamer payload-integrity fix and the payload-aware guard land, treat it as a new finding — classify via `walcompare (term,lsn,pcrc)`, record in `research.md`, and add the next fix task; do **not** mark R1/R8 done on the strength of green unit tests alone. (The reviewer flagged one still-open sub-question: send-side-empty vs transitive-empty propagation — disambiguate during the streamer fix.) - -#### Acceptance Soak Gate -| Field | Detail | -|-------|--------| -| Intent | 6+ round chaotic failover on real fs (release build): zero divergence, zero job loss, reconverge+elect each round. | -| Requirements | 8.1, 8.2, 8.3, 8.4 | - -**Contracts**: Batch [x] -- Trigger: `run-ha-soak.sh` chaotic mode, real-fs (ext4) release binaries. -- Output: verdict that **isolates** bug-854 (client discovery) and bug-924 (overlayfs) from the divergence/job-loss result. -- Idempotency & recovery: verdict counts `walcompare` divergences and committed-job loss; gate passes only at zero across ≥6 rounds. This green result is the trigger that unblocks the `raft-drive-async` production flag-flip. - -## Error Handling - -### Error Strategy -- **Conflict at an LSN (same LSN, different term) on the follower/standby path:** logOk fails ⇒ seam truncates the conflicting suffix (follower-AE-conflict contract, no promotion clamp) and repersists the leader's `(term, payload)`; never a silent no-op (`fail-loud over silent skip`). This is the normal, recoverable case — a conflicting entry was never majority-committed, so cutting it loses no committed data. -- **Genuinely unrecoverable divergence:** only when a node cannot obtain the majority-agreed prefix at all (e.g. the leader's log no longer covers the needed range and no snapshot can backfill it) — surface a loud, non-promotable fence and keep retrying; the controller does **not** self-terminate on an operational condition (steering invariant). Operator-visible WARN/ERROR with both LSNs/terms. Note: a local entry conflicting below the node's own `durable_tip` is **not** in this category — it is repairable via truncate, because the conflict proves it was never committed cluster-wide. -- **Reconvergence apply gap:** remain `CatchingUp`, keep streaming via the seam; never report healthy with an open gap. -- **Election round failure:** start a further election; never park without a primary while quorum is reachable. - -### Monitoring -- `walcompare` divergence count (soak verdict), per-edge quorum transition WARN (existing), election round/term progression logs, reconvergence applied/log_len deltas. Each defect path emits an operator-observable line so the soak verdict is explainable. - -## Testing Strategy - -### Unit Tests (`test_ha_core_safety.c`, `controller_unit_test`) -- Streamer payload integrity: a cross-term catch-up whose entry-body read returns short / `malloc` fails must NOT persist a `PLEN=0` entry at a committed LSN — the stream errors and retries instead (1.1, 2.1, 3.1). -- Payload-aware guard: an empty-payload write at existing LSN N followed by the correct full-payload entry ⇒ WAL ends with the full payload (guard did not silently idempotent-skip); the reverse (full then empty) never degrades the stored entry (1.1, 1.2, 2.1, 2.2). -- Defensive term-blind retirement: a `term==0`-shaped catch-up entry routes through the seam and is reconciled (no direct-persist bypass) (1.2, 3.2). -- Follower-conflict truncate contract: a *conflicting* entry below local `durable_tip` **is** truncated and repaired (the promotion clamp does not block it); a *matching* entry is **never** truncated (3.2, 3.4). -- Reconvergence gate: injected apply gap ⇒ node not reported converged until closed (5.2, 5.3, 5.4). - -### Integration Tests -- Snapshot post-install reconcile: a divergent pre-snapshot entry is reconciled against the checkpoint `(term,lsn)` (3.1, 5.1). -- Election liveness under churn: repeated rapid kill/restart ⇒ higher-term `CTRL_PRIMARY` each round; scripted failed round ⇒ follow-on election (6.1, 6.2, 6.3). -- Election Safety (two-leaders-per-term): across overlapping kills, assert at most one member ever reaches committing-primary per term — falsifiable check mapped to `raft/raft.tla` Election Safety; failure promotes a concrete adapter-gate guard to required work (4.1, 4.3). - -### E2E / Soak (`xce-test-ha` chaotic mode, real fs, release) -- **Live-repro first**: the chaotic soak must first reproduce the actual `walcompare` divergence (RED) and bisect it to a seam before synthetic unit tests are accepted as representative (7.1). -- ≥6 rounds chaotic failover: zero `walcompare` divergence (8.1); zero committed-job loss (8.2); 3-of-3 reconverge + new primary each round (8.3); verdict isolates bug-854/bug-924 (8.4). - -### RED-first ordering -- Every implementation task is preceded by its failing test on the unfixed baseline (project TDD steering). The live-divergence repro (7.1) and harness/RED tests land before the ingest/persist/election fixes so each fix is pinned to a `walcompare`/invariant violation (7.1–7.4). A green synthetic unit test over a still-red soak does not satisfy R1/R8. - -## Security Considerations -- No new external surface; all changes are intra-cluster consensus/persistence. The single-seam invariant strengthens integrity (no unvalidated entry enters a member's committed log). Fenced unrecoverable-divergence prevents serving divergent committed state. No secrets, auth, or input-boundary changes. \ No newline at end of file diff --git a/.kiro/specs/raft-ha-core-safety/requirements.md b/.kiro/specs/raft-ha-core-safety/requirements.md deleted file mode 100644 index 2911c646..00000000 --- a/.kiro/specs/raft-ha-core-safety/requirements.md +++ /dev/null @@ -1,102 +0,0 @@ -# Requirements Document - -## Introduction - -**raft-ha-core-safety** fixes the pre-existing HA-core consistency, convergence, and liveness failures that the live failover soak (`xce-test-ha`) reproduces under repeated and chaotic primary failover on an XCE controller cluster. Operators and clients rely on Raft failover to keep committed jobs safe and the cluster available; today, under primary churn, the cluster can silently diverge committed state, fail to recover a healthy quorum, or stall electing a new primary. - -All three symptoms reproduce on the pre-`raft-drive-async` baseline (commit `21e26a1`), confirmed by a baseline-vs-current soak comparison, so they are pre-existing HA-core defects independent of the (complete and dormant) `raft-drive-async` async-drive/offload work. This spec is the gate that must close before `raft-drive-async` flips its dispatch flags ON in production. - -The defects, in protocol terms: - -1. **Committed-prefix divergence** (bug-851; bug-795/828 family). Under chaotic failover, `walcompare` reports a committed-prefix divergence: one member is missing a committed log entry the majority holds, and silently retains a divergent committed prefix. Known contributors: entry term left at `0` instead of the writing leader's term; two-leaders-per-term windows; and no per-entry commit identity or reconcile path, so a member cannot detect or repair a divergent committed prefix. -2. **Standby-rejoin / reconvergence failure** (bug-1042 family). A restarted, previously-downed node intermittently fails to return the cluster to a healthy 3-of-3 within bound, even after reaching the leader's log tip at the replication-match level, due to post-recovery log-length/applied-state desync and catch-up apply gaps. -3. **Round-N election stall.** After several rapid kill/restart rounds the cluster intermittently fails to elect a new primary after a primary kill within the failover window — a liveness regression under churn. - -Behavior is verified against `raft/raft.tla` and the controller TLA+ safety-invariant checklist in `src/controller/CLAUDE.md` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability). - -## Boundary Context - -- **In scope**: per-entry commit identity and divergence detect/repair; entry term always reflecting the writing leader's term; single-leader-per-term enforcement; deterministic standby-rejoin reconvergence to 3-of-3; election liveness under repeated churn; a deterministic reproduction harness (soak extension + unit/integration) that goes RED before the fix; and a green multi-round chaotic-failover soak as the acceptance gate. -- **Out of scope**: the `raft-drive-async` async-drive/offload work (SlowLane, peer-control vtask dispatch, snapshot/config-push offload, endpoint-quiescence). This spec must not change its behavior. New cluster-membership-resize, multi-region, or non-Raft replication features are also out of scope. -- **Adjacent expectations**: the cluster runs the existing 3-node Raft controller HA topology and persists committed state to a real filesystem. This spec expects `xce-test-ha` and the controller test harness to be the verification surface; it does not own the async dispatch path or the client-facing API contract beyond preserving committed-job safety. - -## Requirements - -### Requirement 1: Committed-prefix safety across members - -**Objective:** As an operator of an XCE controller HA cluster, I want every member to agree on the contents of the committed log prefix, so that no committed job is ever silently lost or diverged under failover. - -#### Acceptance Criteria -1. While the cluster is operating under repeated or chaotic primary failover, the Controller HA subsystem shall keep the committed log prefix identical across all members up to each member's commit point. -2. When a member holds a committed entry at a given log position, the Controller HA subsystem shall guarantee no other member holds a different committed entry at that same position. -3. If a member's committed prefix differs from the majority-agreed committed prefix at any position, then the Controller HA subsystem shall treat that member's committed state as divergent rather than authoritative. -4. The Controller HA subsystem shall satisfy the State Machine Safety and Commit Ordering/Durability invariants in the controller TLA+ safety checklist for every committed entry. - -### Requirement 2: Per-entry commit identity - -**Objective:** As an operator, I want each log entry to carry an unambiguous identity, so that members can distinguish entries written by different leaders and terms at the same position. - -#### Acceptance Criteria -1. When a leader writes a log entry, the Controller HA subsystem shall stamp that entry with the leader's current term. -2. The Controller HA subsystem shall never persist a committed entry whose term is `0` or otherwise does not identify the writing leader's term. -3. When two entries occupy the same log position, the Controller HA subsystem shall be able to determine from their identity whether they are the same entry or conflicting entries from different leaders. -4. While restoring persisted log entries after restart, the Controller HA subsystem shall preserve each entry's original term identity. - -### Requirement 3: Divergence detection and repair - -**Objective:** As an operator, I want a member with a divergent committed prefix to detect and repair it, so that the cluster self-heals instead of silently carrying a permanent inconsistency. - -#### Acceptance Criteria -1. When a member rejoins or replicates and its committed prefix conflicts with the majority-agreed prefix, the Controller HA subsystem shall detect the conflict. -2. When a committed-prefix conflict is detected, the Controller HA subsystem shall repair the divergent member's log to match the majority-agreed committed prefix. -3. If a member cannot repair its divergent committed prefix, then the Controller HA subsystem shall surface the unrecoverable divergence rather than continue serving divergent committed state. -4. The Controller HA subsystem shall not discard or overwrite any entry that is part of the majority-agreed committed prefix during repair. - -### Requirement 4: Single leader per term - -**Objective:** As an operator, I want at most one leader to be able to commit entries in any given term, so that conflicting writes cannot enter the committed prefix. - -#### Acceptance Criteria -1. The Controller HA subsystem shall allow at most one member to act as committing primary within any single term. -2. If a member detects another leader in the same or a higher term, then the Controller HA subsystem shall require that member to stop committing entries in its current term. -3. The Controller HA subsystem shall satisfy the Election Safety and Leader Completeness invariants in the controller TLA+ safety checklist across all failover transitions. - -### Requirement 5: Standby-rejoin reconvergence to 3-of-3 - -**Objective:** As an operator, I want a restarted node to deterministically rejoin and restore a full healthy quorum, so that the cluster regains its fault tolerance after recovery. - -#### Acceptance Criteria -1. When a previously-downed node restarts and reconnects, the Controller HA subsystem shall bring the cluster to a healthy 3-of-3 membership within the bounded recovery window. -2. When a rejoining node reaches the leader's log tip at the replication-match level, the Controller HA subsystem shall also bring that node's applied state and log length into agreement with the leader. -3. If a rejoining node has a catch-up apply gap, then the Controller HA subsystem shall complete the outstanding apply work before reporting the node as healthy. -4. While a node is rejoining, the Controller HA subsystem shall not report the cluster as fully converged until that node's committed and applied state matches the leader. - -### Requirement 6: Election liveness under churn - -**Objective:** As an operator, I want elections to always complete after a primary failure, so that the cluster never gets permanently stuck without a primary under repeated failover. - -#### Acceptance Criteria -1. When the current primary is killed and a quorum of members remains reachable, the Controller HA subsystem shall elect a new primary at a higher term within the bounded failover window. -2. While the cluster undergoes repeated rapid kill/restart rounds, the Controller HA subsystem shall continue to elect a new primary after each primary loss. -3. If an election round fails to produce a primary, then the Controller HA subsystem shall start a further election rather than remain permanently without a primary. - -### Requirement 7: Deterministic reproduction harness - -**Objective:** As a developer, I want a deterministic harness that forces the divergence, reconvergence, and election-stall paths, so that each fix is proven by a test that fails before it and passes after. - -#### Acceptance Criteria -1. The reproduction harness shall force the committed-prefix divergence path and fail (RED) on the unfixed baseline. -2. The reproduction harness shall force the standby-rejoin reconvergence-failure path and fail (RED) on the unfixed baseline. -3. The reproduction harness shall force the round-N election-stall path and fail (RED) on the unfixed baseline. -4. When run against the fixed system, the reproduction harness shall pass (GREEN) for the divergence, reconvergence, and election-liveness paths. -5. The reproduction harness shall extend the existing `xce-test-ha` soak and add unit/integration coverage for the same paths. - -### Requirement 8: Acceptance soak gate - -**Objective:** As a release owner, I want a green multi-round chaotic-failover soak on a real filesystem before this spec is accepted, so that `raft-drive-async` may flip its dispatch flags ON only after committed-job safety is proven. - -#### Acceptance Criteria -1. When a chaotic-failover soak of at least 6 rounds runs against a release build on a real filesystem, the Controller HA subsystem shall report zero committed-prefix divergence. -2. When that soak runs, the Controller HA subsystem shall report zero committed-job loss. -3. While the soak runs, the Controller HA subsystem shall reconverge to 3-of-3 and elect a new primary after every primary kill round. -4. The Controller HA subsystem shall not be accepted as the `raft-drive-async` production flag-flip gate until the soak passes all of the above. diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md deleted file mode 100644 index 953f42d9..00000000 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ /dev/null @@ -1,990 +0,0 @@ -# Gap Analysis — raft-ha-core-safety - -_Generated by `/kiro-validate-gap`. Information over decisions: options and findings, not final choices._ - -## Headline Finding (reframes the spec) - -The `(term, lsn)` commit-identity + reconcile + commit-safety machinery the requirements describe **already exists and is wired at the baseline**. Confirmed: - -- **Library half is 100% complete and validated**: `raft-log-replication` (24/24, G1/G2/G3/G5) and `raft-commit-safety` (18/18, G4 Figure-8 + G6 vote freshness). All five callbacks present in `src/lib/raft/raft.{c,h}`. -- **Controller adapter is 16/20 wired** by `ha-term-safety`: all five callbacks implemented in `s_real_cbs` (`ha.c:1480/1539/1656/1711/1739`), primary append routed through `raft_log_append` (`ha.c:3408+`), heartbeat clean (no `on_persist_entry` on `n_entries==0`), per-LSN term index (`persist.c:414/476/487/752`), config markers, and `walcompare` (`src/controller/testsuite/walcompare.c`). -- **Baseline `21e26a1` already contains all of it.** `git`: `21e26a1` is itself a `raft-drive-async` task-2.3 commit, ancestor of HEAD, and contains `s_real_cbs` + `walcompare`. So the machinery is NOT absent at baseline. - -**Yet the live chaotic-failover soak still fails on this branch AND on baseline `21e26a1`**: `walcompare FAIL: committed-prefix divergence at LSN 9` (round 3) and standby `ctrl-2` rejoin failure (round 4), with intermittent round-N election stall. - -**Therefore raft-ha-core-safety is NOT a build-the-machinery spec. It is a residual-defect + convergence/liveness + live-verification spec.** The work is: root-cause why divergence survives the existing reconcile path under overlapping chaotic failover; close the convergence and election-liveness gaps no completed spec actually closed; and finish the live chaotic soak that `ha-term-safety` left permanently BLOCKED on env. **Do not rebuild the `(term,lsn)`/callback machinery — extend and correct it.** - -## Requirement-to-Asset Map - -| Req | Capability | Existing asset | Gap tag | -|-----|-----------|----------------|---------| -| R1 Committed-prefix safety | quorum commit + Figure-8 guard | `raft_advance_commit` raft.c:769-821; `on_commit_advanced` ha.c:1739 | **Constraint** — machinery present but live soak still diverges at LSN 9 → residual defect | -| R2 Per-entry commit identity | term stamped from leader; per-LSN term index; legacy-0→1 | raft.c:571-580; `persist_set_term` persist.c:476; `record_entry_term` persist.c:414/752; `WAL_LEGACY_TERM` persist.c:456 | **Done (verify)** — confirm no path still emits term-0 under chaos | -| R3 Divergence detect + repair | logOk + conflict-truncate-then-persist on AE intake | raft.c:701-765; `on_truncate_from` ha.c:1711; `walcompare` detector | **Constraint** — repair fires on AE-intake path only; rejoin/catch-up path may bypass it (see Research Needed #1) | -| R4 Single leader per term | term++ on election, vote-once, step-down, pre-vote, vote freshness | raft.c:105-249; `raft_cb_vote_eligible` ha.c:342 | **Constraint** — two-leaders-per-term window under churn not yet proven closed live | -| R5 Standby rejoin → 3-of-3 | catch-up apply, applied-lsn floor, snapshot backfill, contiguity guard | `ha_apply_wal_catchup_entry` ha.c:2233; `ha_advance_applied_lsn` ha.c:2078; `ha_handle_snapshot_chunk` ha.c:2492; contiguity persist.c:839 | **Missing (effective)** — bug-1042 standby-never-converges has NO dedicated owning task; `ha-convergence-recovery` claims it but soak still fails round 4 | -| R6 Election liveness under churn | election restart, candidate advertise floor | `raft_start_election` raft.c:105; `ha_candidate_advertise_lsn` ha.c:3921 | **Missing (effective)** — round-N stall not closed; no deterministic churn test | -| R7 RED→GREEN harness | `xce-test-ha` soak, `walcompare`, `controller_unit_test`, `test_ha_dispatch_integration` | `.claude/skills/xce-test-ha/`; `walcompare.c`; `src/xce.build/controller/controller_unit_test` | **Missing** — no deterministic harness that *forces* chaotic divergence/rejoin/stall as a RED unit/integration gate (current soak is probabilistic) | -| R8 Acceptance soak gate | real-fs release soak; BIN/XT overridable | `run-ha-soak.sh` (BIN/XT env override added this branch); non-ASAN release build | **Unknown** — never passed at 6+ chaotic rounds; was BLOCKED on overlayfs `fdatasync` (bug-924) | - -## Existing-Code Constraints (must respect) - -- **Reconcile is single-seam**: divergence repair happens only inside `raft_on_append_entries_request` (logOk → `on_truncate_from` → re-persist, raft.c:730-765). Any code path that brings a node's log forward *without* routing through this function (e.g. a snapshot/catch-up stream that writes WAL directly) will not detect or repair a divergent committed prefix. -- **Truncate must never cut the majority-committed prefix** (raft.c monotonic commit; promotion truncate-clamp `max(safe_committed_lsn, durable_tip)` ha.c:999-1000, bug-1060 fix). R3.4 depends on this clamp holding under chaos. -- **Persist-before-ACK durability is owned by `ha-commit-durability`, not this spec.** R1/R2 must not weaken it. `on_commit_advanced` is a barrier trigger, not the durability mechanism. -- **`raft-drive-async` is out of scope and dormant.** Do not change SlowLane / peer-control dispatch / endpoint-quiescence behavior (Boundary Context). -- **Client-side bug-854** (libxce primary discovery) can fail the soak harness at EXIT-1 even when server data safety passes. Must be isolated so it does not mask/forge R8 verdicts. - -## Implementation Approach Options - -### Option A — Extend & correct existing machinery (recommended) -Keep all library + adapter machinery. Root-cause the residual divergence, route the rejoin/catch-up path through the reconcile seam, close the convergence + liveness gaps with targeted fixes, add a deterministic chaotic harness, finish the live soak. -- ✅ No rebuild of consensus-critical code that already passed library + clean-failover validation -- ✅ Smallest blast radius; aligns with TLA+ already-proven invariants -- ❌ Requires precise root-cause of a hard-to-repro chaotic defect before fixing -- ❌ Risk of "fix the symptom not the invariant violation" if harness stays probabilistic - -### Option B — New convergence/divergence-repair subsystem -Build a standalone post-failover reconcile/repair component separate from the AE-intake seam. -- ✅ Clean place to own rejoin reconcile + unrecoverable-divergence fencing -- ❌ Duplicates logic that already lives in `raft_on_append_entries_request`; two reconcile paths is exactly the class of bug being fixed -- ❌ High risk of new two-leaders/two-reconcile divergence - -### Option C — Hybrid (Option A + new deterministic harness component) -Extend/correct in place (A), but build the **deterministic chaotic harness** (R7) as genuinely new test components (forced overlapping kills, injected divergent prefix, scripted round-N stall) rather than relying on the probabilistic soak. -- ✅ Keeps consensus code as A; isolates the new surface to test infrastructure where new code is safe -- ✅ Gives a RED gate that pins each fix to an invariant violation -- ❌ Harness design is itself non-trivial (deterministic chaos, no sleeps, gate/probe seams like `test_ha_dispatch_integration.c`) - -## Effort & Risk - -| Item | Effort | Risk | Justification | -|------|--------|------|---------------| -| Root-cause residual LSN-9 divergence | M | High | Chaotic, intermittent; survives existing reconcile — invariant violation is subtle | -| Route rejoin/catch-up through reconcile seam (R3/R5) | M | High | Touches recovery + commit prefix; must not cut majority-committed | -| Election liveness under churn (R6) | M | Medium | Election machinery exists; needs churn-stall root cause + restart guarantee | -| Deterministic chaotic harness (R7) | L | Medium | New, but test-only; pattern exists (`test_ha_dispatch_integration.c`) | -| Live 6+round real-fs soak (R8) | M | High | Was permanently blocked on overlayfs `fdatasync` (bug-924); needs real-fs env + bug-854 isolation | -| **Total** | **L–XL** | **High** | Consensus correctness under chaos + env-gated live verification | - -## Research Needed (carry to design phase) - -1. **Why does divergence survive the reconcile seam?** Trace the rejoin/catch-up path (`ha_apply_wal_catchup_entry` ha.c:2233, snapshot `ha.c:2492`): does it route through `raft_on_append_entries_request` so logOk/`on_truncate_from` fire, or does it write WAL directly and bypass conflict repair? Hypothesis: catch-up streams `[watermark..committed]` directly (per memory bug-795/828 family) → no `(term,lsn)` conflict check on rejoin. -2. **Is LSN-9 divergence in the committed or uncommitted prefix?** If committed, the violation is upstream (two-leaders-per-term or premature commit-advance), not a repair gap. Use `walcompare` `(lsn→term,pcrc)` output to classify. -3. **Two-leaders-per-term under overlapping kills** — does pre-vote + vote-once fully close the window when kills overlap election timeouts? Map against `raft/raft.tla` Election Safety. -4. **bug-1042 ownership** — `ha-convergence-recovery` claims the rejoin path but soak still fails round 4. Determine whether its tasks are actually landed on this branch or stranded, to avoid duplicate work. -5. **R8 env** — confirm real-fs (ext4) soak runs now via `run-ha-soak.sh` BIN/XT override + non-ASAN release build; quantify how bug-924 (overlayfs) and bug-854 (client discovery) are excluded from the R8 verdict. - -## Recommendation for Design Phase - -- **Preferred: Option C** (extend/correct in place + new deterministic chaotic harness). Treat the spec as residual-defect + convergence + liveness + live-verification, explicitly **not** a machinery rebuild. -- **Key design decisions to make**: (a) route rejoin/catch-up through the single reconcile seam vs. add a guarded repair at promotion; (b) classify LSN-9 as committed vs uncommitted before choosing repair vs upstream fix; (c) harness determinism mechanism (forced-kill gates/probes) for R7; (d) how R8 isolates bug-854/bug-924 so the gate measures only server-side committed-job safety. -- **Carry forward**: Research Needed #1–#5; verify every change against `raft/raft.tla` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability) per `src/controller/CLAUDE.md`. - ---- - -# Design-Phase Discovery (kiro-spec-design) - -## ROOT CAUSE (term-blind catch-up bypass) — ⚠️ SUPERSEDED / REFUTED by task 1.3 - -> **This hypothesis was tested live (task 1.3) and REFUTED.** The real defect is cross-term catch-up *payload* loss (see §"Task 1.3 — Live divergence reproduction & bisect" below). The divergent entry carries the correct term and arrives through the reconcile seam; the term-blind path is a latent hazard but not the observed cause. Retained below for the diagnostic record. - -Deep-trace of the standby catch-up/rejoin paths found a **split** where one branch bypasses the single reconcile seam: - -| Path | Entry | Routes through `raft_on_append_entries_request`? | logOk | conflict-truncate | term stamped | -|------|-------|---|---|---|---| -| 1 Term-carrying | `entry->term != 0` → `ha_apply_term_entry_via_library` (ha.c:2173) → `ha_consume_append_entries` (ha.c:2383) → raft.c:646 | YES | YES (raft.c:701) | YES (raft.c:735) | leader term via `persist_set_term` | -| 2 **Term-blind (defect)** | `entry->term == 0` → `ha_apply_single_entry` (ha.c:2097) → `persist_wal_write_replicated` (persist.c:802) DIRECT | **NO** | **NO** | **NO** | **standby's own `p->term`** (persist.c:862) | -| 3 Snapshot | `ha_handle_snapshot_chunk` (ha.c:2492) → `persist_load_snapshot` sets applied_lsn=checkpoint | NO | N/A | N/A | N/A (canonical state) | -| 4 Stream+ACK | `ha_stream_wal_to_standby` (config_repl.c:2032) feeds Path 1 or 2 by `entry->term` | depends | depends | depends | depends | -| 5 `ha_rejoin_reconcile` | ha_convergence.c:254 — **now a no-op stub** (committed-watermark truncate retired; relies on library) | NO | N/A | retired | N/A | - -**Divergence mechanism (LSN 9):** -1. An entry reaches a standby via Path 2 (term-blind). `persist_wal_write_replicated` stamps `hdr.term = standby p->term` (wrong) and records it (`record_entry_term`, persist.c:902). No logOk, no conflict check. -2. The correct term-carrying entry for the same LSN later arrives via Path 1. The persist **idempotency guard** (`if lsn+1 <= p->current_lsn return 0`, persist.c:822) **silently no-ops** — the divergent entry is never overwritten/repaired. -3. `walcompare` (testsuite/walcompare.c:135) compares `(term, payload_crc)` per committed LSN; the member with the term-blind write shows a different `term` → `WALCMP_DIVERGENT`. `match_term` maps raw 0 → `WAL_LEGACY_TERM`, so legacy-0 vs genuine-term at the same LSN reads as divergence. - -**Why the existing `(term,lsn)` machinery does not catch it:** all conflict detection lives *inside* `raft_on_append_entries_request`; Path 2 never enters it, and the idempotency guard treats "LSN already present" as "done" regardless of term. - -## Synthesis - -> ⚠️ **D1/D3 partially superseded by task 1.3.** The "eliminate every bypass of the one seam / collapse Path 2 into Path 1" framing assumed the term-blind hypothesis, now refuted. The single-seam invariant (D2) and harness-first discipline (D4) still hold; but the actual fix is the cross-term catch-up streamer payload integrity + a `(term,lsn,payload-crc)`-aware guard (see §"Task 1.3"), not seam-bypass elimination. Term-blind retirement remains as defensive hardening only. - -- **Generalization (D1):** R1/R2/R3/R4 are one underlying invariant — *every* entry that enters a member's log must pass the single `(term,lsn)` reconcile seam. The fix is not four mechanisms but **eliminating every bypass of the one seam** + making persistence `(term,lsn)`-conflict-aware rather than LSN-only-idempotent. -- **Build vs adopt (D2):** Adopt the existing library seam (`raft_on_append_entries_request` + `on_truncate_from` + `persist_set_term`) — already validated (raft-log-replication/raft-commit-safety). Build nothing new in the consensus core; only route into it and harden the persist guard. Reuse `ha_rejoin_reconcile` intake hook (currently stub) rather than a second reconcile path. -- **Simplification (D3):** Do not add a separate "divergence repair subsystem" (gap-analysis Option B rejected) — that would create the exact second-reconcile-path class of bug being fixed. Collapse Path 2 into Path 1. -- **Harness-first (D4):** The live bug is intermittent/chaotic. Make it deterministic: a RED unit test injects a term-blind/divergent entry at an LSN and asserts the seam repairs it; a chaotic soak mode forces overlapping kills. RED before any fix (R7), pins each fix to a `walcompare`/invariant violation. - -## Open risks carried to tasks -- Confirm Path 2 is the *only* divergence source live (harness RED test will confirm/refute; if soak still diverges after Path-2 closure, classify next via `walcompare (term,lsn,pcrc)`). -- Snapshot boundary (Path 3): post-install AE intake must reconcile against the snapshot's `(term,lsn)`; verify no committed entry below the checkpoint can diverge. -- Standby reconvergence (R5) and election-liveness (R6) are separable seams from divergence-repair; phaseable but kept in one spec per the unified acceptance gate. -- bug-854 (client discovery) and bug-924 (overlayfs fdatasync) must be excluded from the R8 verdict so the gate measures server-side committed-job safety only. - ---- - -# Design Review Resolution (kiro-validate-design) - -Three critical issues raised and folded into design.md (GO): - -1. **Repair-vs-clamp conflation (correctness).** Split truncate into two contracts: *promotion-truncate* keeps the bug-1060 `max(safe_committed_lsn, durable_tip)` clamp; *follower-AE-conflict-truncate* obeys Raft Log Matching and truncates the conflicting suffix regardless of local `durable_tip` (a conflicting entry was never majority-committed → safe to cut). Applying the promotion clamp on the follower path would have converted repairable divergence into a permanent fence and failed R8. Updated: §Existing Architecture, §Unified Entry Ingest, §Persist Conflict Guard, §Error Handling, Traceability 3.x, Testing. -2. **R4 two-leaders-per-term unverified.** Added a falsifiable Election Safety assertion (≤1 committing-primary per term under overlapping kills, mapped to raft.tla); failure promotes a concrete adapter-gate guard from verification to required work. Updated: §Election Liveness, Testing Integration. -3. **Hypothesis-validation risk.** RED gate (7.1) must reproduce the *live* `walcompare` divergence and bisect to a seam before synthetic `(term,lsn)` injection counts; baseline 21e26a1 already retired live term-0 (ha-term-safety 3.4), so the term-blind path may not be the live trigger. Contingency: if soak still diverges after Path-2 closure, re-classify via `walcompare (term,lsn,pcrc)` and add a fix task — do not mark R1/R8 done on green unit tests alone. Updated: §Deterministic Chaotic Harness, Testing E2E + RED-first. - -Verdict: **GO** (unconditional after edits). Architecture unchanged; refinements were bounded contract clarifications. - ---- - -# Task 1.3 — Live divergence reproduction & bisect - -_Generated by the task-1.3 investigation subagent (REPRODUCE + CLASSIFY + BISECT). Evidence-backed; no production code changed. Real-fs (ext4) `~/tmp`, non-ASAN release binaries `src/build.release/controller/{xce_controller,xce_test}` (built 08:48 this day)._ - -## TL;DR / VERDICT - -**The design's primary "term-blind catch-up bypass" hypothesis is REFUTED for the reproduced live committed divergence.** The divergent committed entry has the **CORRECT term**, was ingested **through** the single library reconcile seam (`ae_append` → `on_persist_entry`), and the LSN-only idempotency no-op was **not** the mechanism. The real defect is **payload loss on cross-term catch-up**: a committed `JOB_SUBMIT` entry is persisted on the catching-up member with `payload_len=0` (empty body) while its `(term, lsn)` are correct. It is a sibling of the bug-795/828/1060 committed-prefix-loss family, located in the **catch-up replication payload path**, not in the `(term,lsn)`/idempotency-guard seam Phase 2 currently targets. - -## Commands run & reproduction status - -- Two `--chaotic` soak attempts via `run-ha-soak.sh` (rounds=6, `XCE_HA_BIN/XT` → release binaries, dir `~/tmp/ha-1p3`) **both aborted at cold-start** with `FAIL: no primary elected within 45s`. A `--sequential` attempt failed identically. -- **This cold-start "election stall" is a HARNESS/LOG-BUFFER ARTIFACT, not a consensus stall.** Evidence: in every aborted run a primary WAS elected on time (e.g. `ctrl-1 → CTRL_PRIMARY (term=2)` logged ~15s after start, `quorum_reached`), yet `wait_for_primary_log` never tripped. All three nodes' last log line is the identical cleanup-kill timestamp (= start+45s): the 08:48 release build **block-buffers its `--logfile` and only flushes on signal**, so the harness's log-grep primary detection (`latest_primary_term`) cannot observe progress live. The OLD binaries used by the prior passing sequential soaks (`~/tmp/xce-soak-{rel,bl}.log`, 02:56/03:01) did not have this buffering behaviour. The client (`ctrlstatus`) path is unaffected. -- **Reproduced cleanly** by driving the cluster with a client-based (log-buffer-immune) driver (`scratchpad/drive.sh`, fresh deploy `~/tmp/ha19`, sequential kill→restart→re-elect, 6 rounds). Committed-prefix divergence appeared by round 1 and persisted: round 1 LSN 8 (present=2), round 2 LSN 8 (present=2), **round 3 LSN 9 (present=3 — all members hold it but disagree)**. Driver wedged at round 4 on a transient quorum loss; the frozen WAL dirs were classified directly. - -## walcompare --report classification (frozen `~/tmp/ha19`, `--committed 25`) - -Decisive divergence — **committed LSN 9, class=committed, compared=3, present=3** (every member holds the entry; they disagree on payload): - -``` -WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=3 - member 0 (ctrl-1): term=3 pcrc=00000000 <-- MINORITY / divergent - member 1 (ctrl-2): term=3 pcrc=512dc5b5 <-- majority - member 2 (ctrl-3): term=3 pcrc=512dc5b5 <-- majority -``` - -Same pattern at committed LSN 10 (ctrl-1 pcrc=0 vs majority 06e88dac) and LSN 11 (ctrl-1 pcrc=0 vs majority fea75587). **Term is identical (3) across all three members; only the payload-CRC diverges.** (ctrl-3 additionally shows `status=MISSING` for LSN 12-19 — a separate catch-up-lag/convergence gap from the round-4 quorum loss, bug-1042 family, not the payload defect.) - -Raw `waldump` confirms the payload loss, not a term mismatch: - -``` -ctrl-1 (divergent): 9 | 3 | JOB_SUBMIT | PLEN=0 | 00000000 | jobid=? -ctrl-2/ctrl-3 (maj): 9 | 3 | JOB_SUBMIT | PLEN=3560 | 512dc5b5 | jobid=0-0-1782550813-0 ... -``` - -The origin is intact — both majority members hold the full 3560-byte body with a decodable jobid; **ctrl-1 lost the payload bytes during catch-up while keeping the correct type (JOB_SUBMIT) and term (3).** - -## Bisect — the concrete ingest path - -ctrl-1's `--logfile` (fully flushed post-kill) traces exactly how it acquired the divergent entries: - -1. ctrl-1 was the term-1 cold-start primary (wrote LSN 1-7), killed round 1, **restarted round 2** (`persist_recover done — recovered durable_floor=7 current_lsn=8`). It then needed LSN 8-onward. -2. `09:00:13` — term-3 leader ctrl-3 streamed catch-up `op 37 walCatchupEntry`, but starting at `prevIdx=8` (entry 9) while ctrl-1 only held up to 7 → library `ae_reject_logmismatch`, `ha_consume_append_entries … success=0`. Nothing written. (ctrl-3 then died — round 2 failover.) -3. `09:00:26` — new term-4 leader ctrl-2 streamed `op 37 walCatchupEntry` from `prevIdx=7`. The **library accepted and APPENDED**: `RAFT[p0] term=4 event=ae_append`, `ha_consume_append_entries … prevIdx=7 n=1 → success=1 match=8`, then match=9, 10, 11, 12. These were **fresh appends through the reconcile seam (`on_persist_entry`)**, not idempotent no-ops (current_lsn was 8 immediately prior). LSN 9-11 landed with `payload_len=0`. -4. In the SAME catch-up, LSN 13-15 (stored term=4, the streaming leader's OWN term) landed with FULL payloads. **Signature: same-term catch-up preserves payload; cross-term catch-up of prior-term entries (stored term 3 streamed under leader term 4) drops it.** - -Code-path mapping (read at the research anchors): -- Receiver `ha_consume_append_entries` (ha.c:2410-2440) copies the body **unconditionally** (`if (payload && plen>0) memcpy(...)`, ha.c:2438) from `e->raw_payload`/`e->raw_payload_len`. So the empty body **arrived empty on the wire** → the loss is SEND-side / source-side, not in the seam's persist. -- Both documented send paths nominally read the payload: batch streamer `ha_stream_wal_to_standby` (config_repl.c:2154-2161, `pread` when `hdr.payload_len>0`, and sets `entry->term = persist_entry_term(...)` + `entry->leader_current_term = leader_term`, config_repl.c:2130-2151) and the single-entry `on_send_append_entries` → `ha_read_wal_entry_at` (ha.c:1611-1640). Neither *statically* drops a cross-term payload — so the loss is either (a) the streaming leader (ctrl-2) **itself held an empty copy at stream time** and the full payload was reconciled into ctrl-2's WAL only later (a transitive empty-propagation across the failover chain), or (b) the wire encode/decode of the body for these messages. Distinguishing (a) vs (b) requires one instrumented run capturing per-entry `payload_len` at the leader's send and at the receiver's `on_persist_entry` for a cross-term LSN — see "Implied next work". - -## Why the term-blind hypothesis is refuted here - -- Predicted by design.md ROOT CAUSE: `entry->term==0` → `ha_apply_single_entry` (ha.c:2097) → `persist_wal_write_replicated` DIRECT, **wrong term** (standby's `p->term`), idempotency no-op of the later correct entry. -- Observed: the entry's **term is correct (3)**, it was ingested via `ha_apply_term_entry_via_library` → `ha_consume_append_entries` → `on_persist_entry` (the term-carrying LIBRARY seam, `event=ae_append`), and the catch-up was a **fresh append** (no idempotency-guard no-op involved). The streamer stamps a non-zero term (config_repl.c:2141), so the `term==0` → `ha_apply_single_entry` branch (ha.c:2253/2291 gate `sub->term != 0`) is not taken for these entries. - -## VERDICT - -- **Term-blind catch-up bypass (wrong-term + LSN-only idempotency no-op): REFUTED** as the trigger of the reproduced live committed divergence. -- **Actual trigger: payload-loss on cross-term catch-up** — committed `JOB_SUBMIT` entries persisted with `payload_len=0` (term/type correct) on a member that catches up prior-term entries from a higher-term leader; the divergence is a **payload-CRC mismatch at an identical `(term,lsn)`**, in the committed prefix. A member carrying this can (and in round 3 did) become primary, so it is a silent committed-state corruption in the bug-795/828/1060 family. - -## Implied next work (steers Phase 2) - -1. **Re-scope the Phase-2 fix away from the term-stamp/idempotency seam.** Collapsing Path-2 into Path-1 (design D3/D1) will NOT fix this divergence — the entry already goes through Path-1 with the correct term. The fix must target the **catch-up payload source-of-truth**: add an instrumented capture of `payload_len` at `ha_stream_wal_to_standby`/`on_send_append_entries` (send) and `on_persist_entry` (receive) for cross-term LSNs to prove send-side empty vs transitive-empty, then fix whichever drops/propagates the empty body. -2. **Harden the persist conflict guard to be `(term,lsn,payload)`-aware**, not term-only: a committed `(term,lsn)` whose payload-CRC disagrees with the quorum must be detected/repaired (or the member fenced), since term-only reconcile is blind to this class. -3. **Fix the divergence detector / RED gate to assert payload-CRC at committed LSN even when terms match** (it already does via `pcrc`; keep `walcompare --report (term,lsn,pcrc)` as the canonical RED classifier — it caught this). -4. **R6/Harness defect (secondary):** the release build's log buffering defeats `run-ha-soak.sh`'s log-based `wait_for_primary_log`/`primary_from_log` primary detection (false "election stall"). The chaotic acceptance gate (R8) needs client-based (`ctrlstatus`) primary detection, or the controller must line-buffer/flush its INFO log, before the soak can run unattended on release binaries. - -## Evidence artifacts (under ~/tmp, ephemeral) - -- `~/tmp/ha19/` — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (`controller/ctrl-{1,2,3}/data`, `logs/ctrl-1.log`). -- `~/tmp/ha19-drive.log` — driver round-by-round output (divergence first seen round 1). -- `~/tmp/ha-1p3-run{1,2}.log`, `~/tmp/ha-1p3-seq.log` — the cold-start log-buffer "stall" runs. - ---- - -# Phase-2 interim live validation (post-2.4, real-fs release rebuild) - -Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (`~/tmp/ha-verify-2p4`). The 4-round run **wedged (timed out ~9m40s) on an election stall** before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with `walcompare --report`: - -- **Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members** (`walcompare: OK — committed prefixes identical`). -- **No present-but-empty committed divergence** — the bug-1.3 signature (one member `pcrc=0/PLEN=0` PRESENT where the majority holds a full payload) is **absent**. The `pcrc=00000000` entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere). -- Remaining apparent divergences are all `status=MISSING` tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — **catch-up lag** on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated `--committed` boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up. - -**Conclusion:** the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A *clean end-to-end* `committed_divergence=0` soak verdict is now gated on: -1. **Phase 5 (election liveness under churn)** — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2). -2. **Phase 4 (standby reconvergence)** — lagging members must finish catching up so the tail `MISSING` resolves once the cluster stabilizes. - -**Steering implication:** Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3. - ---- - -# Phase 4/5 — Post-Phase-2 stall root cause (investigation, salvaged from a hung run) - -Reproduced the chaotic-soak wedge on real-fs release binaries (HEAD with Phase-2 fixes). The committed/payload prefix is clean (Phase-2 holds). The remaining wedge is a **reconvergence-triggered spurious-failover loop**, NOT a simple missing-election-restart. - -## Evidence (frozen `~/tmp/ha-estall` logs) -- **ctrl-1 = stable primary at term 6**: `became_leader`, repeated `repl_broadcast` / `term=6 event=log_append`. The cluster HAS a leader. -- **ctrl-3 reconverges (slow but works)**: `term=6 ae_recv → ae_reject_logmismatch` ×4, then `ae_already_have`. The AE log-matching backtrack converges for a short divergent suffix. -- **ctrl-2 = the stall**. It recovered as a former primary with an uncommitted higher tail (`durable_floor=15 current_lsn=16`, entries 12-15 at an old term) vs the leader's committed 11. Its log shows a ~15s flap cycle, repeating indefinitely: - 1. `handle_ctrlhello: primary control channel established (fd=6)` → receives heartbeats → re-attaches CTRL_STANDBY. - 2. ~12-15s later: `ha_on_primary_ctrl_fd_rdhup: primary_ctrl_fd RDHUP → begin PreVote failover (Req 1.1)`. - 3. `ha_start_prevote_async (prospective term=6)` → `ha_prevote_round_timeout_fn: PreVote round timed out before a decision — treating as LOSS, staying CTRL_STANDBY` (NO `pre_vote_reply_in` — peers see the primary alive and don't grant). - 4. Primary reconnects → loop repeats. (Last heartbeat and last prevote-timeout share the same timestamp — the channel is flapping, not cleanly down.) - -## Root cause -The primary↔ctrl-2 control channel keeps **RDHUP-flapping** (~15s period). Because "control-channel RDHUP = primary died → failover" is the designed trigger (CLAUDE.md), each flap spawns a spurious PreVote failover that **cannot win** (the primary is genuinely alive, so the other voters don't grant), times out, and re-arms. The likely driver of the flap: ctrl-2 cannot reconverge its **divergent uncommitted tail (12-15)** against the leader — the catch-up/AE path stalls or the leader drops the channel — so ctrl-2 never settles as a follower and the channel cycles. Net: the recovered node never rejoins → no stable 3-of-3 → soak `wait_for_full_convergence` times out (the observed wedge). - -## What the fix must address (Phase 4 + Phase 5, intertwined) -1. **Reconvergence (Phase 4 / R5):** a standby that recovers with a divergent *uncommitted* tail must truncate it (follower-AE-conflict-truncate contract) and catch up to the leader — the same way ctrl-3 does for a short suffix — so the control channel stays up. Determine why ctrl-2's longer/former-primary divergent tail does not reconcile while ctrl-3's short one does (candidate: the backtrack doesn't reach far enough before the channel drops, or the former-primary recovery advertises a tip that the leader's nextIndex backtracking never crosses). -2. **Spurious-failover suppression (Phase 5 / R6/R4):** a transient/flapping control-channel RDHUP must not spawn an unwinnable PreVote storm while the primary is demonstrably alive (peers still see it). A standby should prefer to **reconnect + catch up** on RDHUP before (or instead of) launching failover, or back off failover when its own PreVotes repeatedly get no quorum because a current-term leader exists. (The retry itself works — R6 retry is present — but it retries an election that can never win instead of rejoining.) - -## RED test direction (5.1 / 4.1) -- Force a recovered standby holding a divergent uncommitted tail (former-primary shape, tip above the new leader's committed) to rejoin a cluster with a live higher-term primary, and assert it **reconverges to follower (truncates tail, catches up) without entering a PreVote-failover loop** — i.e. no repeated `ha_begin_failover`/`PreVote timed out` while a current-term primary is reachable. This is deterministically constructible with the 1.4 kill/election + apply-gap seams plus a divergent-tail fixture. - -## Process note -The dispatched investigation subagent backgrounded the soak and hung (~no findings returned); this section was reconstructed by the parent directly from the frozen `~/tmp/ha-estall` controller logs. - ---- - -# Task 7.2 — Acceptance chaotic soak (first run with Phase-2+5 fixes): NO-GO + new finding - -Ran `run-ha-soak.sh --chaotic 6` on real-fs (ext4 `/home/claude/tmp/ha-7p2`) with the -**release binaries rebuilt at HEAD `01419e1`** (Phase-2 payload fix + Phase-5 5.2 -spurious-failover suppression). Frozen at `/home/claude/tmp/ha-7p2-frozen`. - -## What PASSED (job-safety + the landed fixes) -- **Zero committed-JOB loss.** Every round the job-verify confirmed all tracked jobs - present on the new primary (round1 6/6, round2 9/9, round3 12/12). -- **Zero TRUE committed divergence.** `walcompare --committed 11` over the frozen - WALs: **"OK — committed prefixes identical"** across all 3 members. The Phase-2 - payload-loss signature (present=3, one member `pcrc=0` vs majority real) is ABSENT. - `lsn=8` is `pcrc=0` *identically on all three* — a legit empty NOOP marker, not loss. -- **The counted `walcompare` divergences are catch-up LAG, not corruption.** The only - boundary-19 divergence is `lsn=12 present=2`: ctrl-3 (tip=11, restarted round 3) is - MISSING it; ctrl-1/ctrl-2 hold it with matching content. All "divergences" are - `present=2` MISSING tails of restarted-but-uncaught followers (tips ctrl-1=19, - ctrl-2=15, ctrl-3=11), which an inflated `--committed` boundary over-claims as - committed — they are uncommitted/in-flight catch-up tail, exactly the research.md - Phase-2-interim note. -- **5.2 spurious-failover suppression confirmed working.** Rounds 1–3 each elected a - strictly higher-term primary (ctrl-3 t3 → ctrl-2 t4 → ctrl-1 t5): liveness intact, - the fix did not over-suppress. The round-4 RDHUP logs show the GENUINE-loss path - ("primary_ctrl_fd RDHUP → begin PreVote failover"), NOT the suppression message — - correct, because the primary was truly dead (`ha_heartbeat_is_fresh==0`). - -## What FAILED — round-4 election stall (NEW residual liveness defect) -After 3 overlapping kills, round 4 kills ctrl-1 (tip 19, the only fully-up-to-date -node). The two survivors are both restarted-and-lagging (ctrl-2 tip=15, ctrl-3 tip=11), -and their PreVote rounds **repeatedly time out before a decision** (no grant quorum): -each loops `ha_start_prevote_async → ha_prevote_round_timeout_fn (LOSS) → RDHUP → -ha_begin_failover → …` indefinitely, prospective terms frozen (ctrl-2=5, ctrl-3=6, -never bumped because no PreVote ever WINS). The soak wedged here until the 10-min cap. - -**This is NOT the spurious-failover flap (5.2 closed that) and NOT a job-safety bug.** -A SAFE candidate exists — ctrl-2 holds the full committed prefix [1..11] (Leader -Completeness satisfied) and is the freshest survivor — yet the election never converges -on it under sustained overlapping churn. Observed contributors (need a dedicated -RED→bisect): freshness mutual-denial (`PreVote DENIED … reason=freshness, floor lsn=11`) -between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240 -"a catching-up node sends zero PreVote replies") with no live primary left to catch up -FROM, and/or peer-to-peer vote connections not established between two nodes that each -only held a control channel to the now-dead primary. - -## Verdict -**7.2 = NO-GO.** Job safety (R8.1/R8.2) holds; the acceptance liveness criterion -(R8.3 — reconverge 3-of-3 + elect after EVERY kill round) fails at round 4. Per the -design's chaotic-harness Contingency, this is a recorded new finding, not closure: -added follow-up **task 5.3** (overlapping-churn election stall — a lagging-but-safe -survivor must be electable; the catching-up/freshness vote gates must not deadlock when -the primary is gone). 7.2 stays blocked on 5.3. Re-run 7.2 after 5.3 lands. - ---- - -# Task 5.3 + 7.2 re-run — CORRECTED root cause: the soak stall is bug-924, not consensus - -After landing task 5.3 (commit bf88681 — gate the `in_catch_up` PreVote deny on a -LIVE primary), the chaotic soak was re-run on real-fs release binaries -(`/home/claude/tmp/ha-7p2b`, controller rebuilt 02:44 with all Phase-2+5.2+5.3 fixes). - -## Result: round-4 stalls AGAIN — but the cause is the persist fdatasync wedge (bug-924), NOT election logic -- Rounds 1–3 elect a higher-term primary each round (term 4→5→6); job safety holds. -- Round 4 "NO new primary within window" — and ctrl-2's PreVote rounds get **zero - replies** (timeout, not LOST), because the *other* survivor **ctrl-3 is wedged**: - its persist flusher spins on `flusher fallback fdatasync FAILED (errno=22 Invalid - argument)` from 02:45:49 onward and stops answering PreVotes. ctrl-3's process is - alive but non-responsive — jammed in the durability layer, not the election layer. -- This is the **bug-924** WAL-fdatasync-on-overlay artifact (memory obs 1476/1477): - the WAL segment's `fdatasync` returns EINVAL in this sandbox. A file-level repro of - the exact controller pattern (`open O_RDWR|O_CREAT|O_TRUNC` → `posix_fallocate(64MB)` - → write → `fdatasync`) SUCCEEDS on `/home/claude/tmp`, `/home/rshen/projects/xce` - (both ext4) and `/dev/shm` — so it is the controller's live overlay-backed WAL fd - under chaotic restart, not a plain fs limitation. bug-924 is a **Non-Goal of this - spec** (design §Non-Goals / §Out of Boundary: "the overlayfs `fdatasync` - environment defect (bug-924) — excluded from the acceptance verdict, not fixed - here"); durability sync is owned by `ha-commit-durability`. - -## Re-attribution of the round-4 stall (corrects the first §"Task 7.2" entry) -The first soak's round-4 stall was attributed to the `in_catch_up` mutual-deny. That -IS a real latent defect — 5.3 reproduces it deterministically at unit level (a -catching-up survivor with no live primary denied a fresh PreVote) and fixes it — but -it is NOT what blocks the soak. The soak's actual round-4 blocker in BOTH runs is the -bug-924 fdatasync wedge that makes a survivor non-responsive. 5.3 is kept as a valid -correctness fix (unit RED→GREEN, `test_ha_core_safety` 20 Tests / 0 Failures); its -6-round-soak observable clause shares 7.2's environment block. - -## 7.2 status: BLOCKED ON ENVIRONMENT (bug-924), not a consensus NO-GO -The consensus work this spec owns is unit-complete and validated: Phase-2 payload -integrity (committed prefix byte-identical across members), 5.2 spurious-failover -suppression, 5.3 catching-up-survivor electability. The live ≥6-round acceptance soak -cannot reach a clean verdict in this sandbox because the controller's WAL `fdatasync` -fails (bug-924) and wedges nodes mid-soak — the same env block that left -`ha-term-safety`'s live soak permanently BLOCKED. 7.2 (and 5.3's soak clause) require a -real-fs host/CI where the controller WAL `fdatasync` succeeds, OR the soak harness must -detect the fdatasync-wedge and exclude it from the liveness verdict (a Phase-1 harness -improvement), OR bug-924 is resolved by `ha-commit-durability`. The consensus fixes do -not need re-work for this; it is an environment/durability-layer blocker. - ---- - -# Task 7.2 on ~/tmp (non-overlay) — DEFINITIVE 6-round blocker: a controller restart SEGFAULT - -Re-ran the chaotic soak on the user-provided non-overlay path `~/tmp` (= `/home/claude/tmp`) -with the release binaries (Phase-2 + 5.2 + 5.3) and a harness change to **capture each -controller's stdout+stderr** (`start_node` previously sent it to `/dev/null`, hiding all -crash output — the key diagnostic gap). - -## What is now PROVEN -- **Elections work under chaotic churn.** A 2-round chaotic soak COMPLETES (both rounds - elect a higher-term primary). In 6-round runs, rounds 1–3 each elect (terms climb - 4→5→6). 5.2 + 5.3 hold live. Job safety holds (zero committed-job loss every run). -- **The 6-round blocker is a RELEASE-BUILD SEGFAULT on restart**, captured at last in the - per-node stderr: `start.sh: line 14: <pid> Segmentation fault (core dumped) … xce_controller - --myname ctrl-3 …`. A controller **segfaults a few seconds after being restarted into the - churning cluster** (during recovery→failover/PreVote). One node down → cluster at 1-of-3 → - the lone survivor cannot reach quorum → "NO new primary within window." The harness's other - stderr lines are the expected `kill -9` ("Killed"); only the restarted node shows SEGV. -- **This is NOT** the in_catch_up eligibility deadlock (5.3 fixes that, unit-proven), **NOT** - bug-924 fdatasync (that wedged a node in ONE earlier run but is absent here — zero - `fdatasync FAILED`), and **NOT** the consensus logic this spec owns. It is a restart/rejoin - stability crash. - -## Why it could not be root-caused in this sandbox -- The crash is a **heisenbug**: it does NOT reproduce under the ASAN controller build (all - ASAN-run stderr shows only harness `kill -9`, no SEGV / no ASAN report) — memory-layout / - timing dependent. -- No `gdb` and no usable core: `core_pattern` pipes to a non-running `systemd-coredump`, so - the "core dumped" core is discarded; `coredumpctl`/`gdb` are absent from the env. -- Root-causing requires a debug-non-ASAN build with core dumps + gdb (or a host where the - systemd-coredump pipeline is live), i.e. a tooling environment this sandbox lacks. - -## Harness accuracy note (separate, in-scope) -The soak counts `committed_divergence` via `walcompare --report --committed $((1 + 2*njobs))` -(run-ha-soak.sh:543/663). That boundary is the OPTIMISTIC job-count estimate, not the true -majority-committed LSN, so a lagging follower's not-yet-replicated tail (uncommitted on that -member) is mis-counted as committed divergence. The completed 2-round soak reported -`committed_divergence=3 committed_job_loss=0` — and a frozen-WAL `walcompare` at the true -boundary shows the committed prefix byte-identical (the 3 are catch-up-lag MISSING tails, not -corruption). `ctrlstatus` already reports `committedLsn`; the boundary should use it. (Even -fixed, the 6-round verdict stays blocked by the segfault above.) - -## 7.2 status (corrected): blocked by a restart SEGFAULT, not consensus, not bug-924 -Consensus correctness (Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 -catching-up-survivor electability) is unit-complete and confirmed live for every round that -completes. The clean ≥6-round acceptance verdict is blocked by the restart segfault — added -follow-up **task 5.4** (root-cause+fix the chaotic-restart crash; needs core+gdb / debug env) -and **bug-1281**. The harness `committedLsn`-boundary fix is a separate Phase-1 accuracy -improvement. - ---- - -# Task 7.2/5.4 — UNIFIED root cause via gdb: round-N stall is a restart-stability bug (I/O wedge / SIGSEGV), not a consensus-logic gap - -Wrapped the release `xce_controller` under `gdb -batch` (handles RT signals; dumps all-thread -bt on SIGSEGV) via `XCE_HA_BIN=<gdb-wrapper>` and re-ran the chaotic soak on `~/tmp`. - -## Decisive findings -- **Two round-4 faces, ONE root cause.** Plain release: a restarted node **SIGSEGVs** - (captured `Segmentation fault (core dumped) … --myname ctrl-3`). Under gdb the crash does NOT - fire (ptrace perturbs the memory bug); instead the same restarted node's **inbound-accept / - iomgr path WEDGES** — ctrl-3's last `cluster.c:737 server sock accepted` is 03:42:03, yet it - keeps firing its own election timer (`pre_vote_start`) to 03:51:59. It campaigns but can - neither accept peers' PreVote connections nor get answered; ctrl-2's outbound PreVote connects - hang on the dead listener (zero connect-fail logs). The two restarted survivors can't exchange - votes → permanent round-4 stall. Release SEGV and gdb accept-hang = the same restart-time - corruption manifesting differently. -- **Restart-STABILITY bug, not a consensus-logic gap.** PreVote/eligibility/freshness (incl. the - 5.3 fix) are correct; the node is simply I/O-dead after restart. Predates this session — - baseline `21e26a1` failed round 3–4 (requirements.md symptom 3). 5.2/5.3 sit in the - restart/failover path so 5.4 must rule them out during the backtrace, but they did not - introduce the instability. -- **Cannot backtrace in this sandbox.** Heisenbug: invisible under ASAN (no SEGV) AND under gdb - (becomes a hang, not a catchable signal); `core_pattern` is read-only (pipes to a non-running - systemd-coredump) so the release core is discarded. Needs a non-ASAN DEBUG build (-O0 -g) run - directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipe. - -## Net for 7.2 -Elections + job safety are sound for every round that COMPLETES (2-round soak PASSES; 6-round -rounds 1–3 elect higher-term primaries; committed prefix byte-identical; zero job loss). The -≥6-round acceptance verdict is blocked by the restart-stability bug (bug-1281 / task 5.4), -reachable at round 4 (kill the up-to-date node → two restarted survivors). Spec-owned consensus -correctness (Phase-2/5.2/5.3) is unaffected. - -## Harness wins -`start_node` now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV. -The gdb-wrapper (`XCE_HA_BIN=<wrapper>` → `gdb -batch … --args <release ctrl>`) is the reusable -technique to catch a restarted node's fault once a debug+core env is available. - ---- - -# Task 7.2/5.4 — ROOT CAUSE NAILED via gdb: ABBA lock-order deadlock in src/lib async dispatch (UPSTREAM raft-drive-async) - -Wrapped the release controller under gdb -batch with a SIGQUIT handler that dumps all-thread -backtraces; reproduced the round-4 hang; kill -QUIT'd the two wedged survivors. ctrl-2 and ctrl-3 -showed the IDENTICAL two-thread deadlock. - -## The deadlock (both nodes) -- io_worker (netio.c:612): takes ep->lock (netio.c:672) and, still holding it, runs - vtask_enq -> dataq_enq which blocks on q->lock (dataq.c:81). Order: ep->lock -> q->lock. -- vdispatcher_entry: dataq_deq (dataq.c:65) takes q->lock for the WHOLE match scan and runs the - match callback vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) which - blocks on ep->lock. Order: q->lock -> ep->lock. -- ABBA: io_worker holds ep wants q; vdispatcher holds q wants ep -> permanent deadlock. io_worker - dies -> the node stops accepting / answering PreVotes -> two restarted survivors can't elect -> - round-4 stall. In release the same ep/q race under churn corrupts -> SIGSEGV (the earlier crash). - -## Exact defect + minimal fix -dataq_deq invokes a match callback that acquires a SECOND lock (ep->lock via the -endpoint_is_task_blocked eligibility gate) while holding q->lock — inverting the ep->lock->q->lock -order io_worker uses. Minimal fix: make ep->busy atomic and read it locklessly in -endpoint_is_task_blocked (remove the ep->lock acquire from under q->lock); or io_worker drops -ep->lock before dataq_enq; or impose one global lock order. - -## Ownership: UPSTREAM (raft-drive-async / core libxce async-io), NOT this spec -Entirely in src/lib async-I/O dispatch (netio.c io_worker/endpoint, vtaskmgr.c eligibility, -dataq.c) — the VTASK dispatch / endpoint-blocked eligibility layer raft-drive-async owns and this -spec's Non-Goals exclude. Per upstream-ownership protocol: route to raft-drive-async (bug-1282), -keep 7.2 blocked until the dispatch lock-order is repaired, then re-run the gate. This is the gate -working as designed: 7.2 must be GREEN before raft-drive-async flips dispatch flags ON, and it just -caught a real dispatch-layer deadlock. - -## Consensus correctness (this spec) — unaffected -Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are -correct and unit-validated; elections + job safety hold for every round that completes. The -round-4 blocker is the upstream dispatch deadlock, not consensus logic. - ---- - -# bug-1282 FIXED + bug-1283 isolated (the round-4 stall had TWO causes) - -Implemented the bug-1282 fix (commit 25e741d): promoted `endpoint_t.busy` to `_Atomic int` and -made `endpoint_block_tasks`/`endpoint_unblock_tasks`/`endpoint_is_task_blocked` lockless, removing -the `ep->lock` acquire from the vtask-eligibility path that ran under the dataq `q->lock`. - -## Verified -- **Deadlock gone:** post-fix gdb soak shows ZERO `__lll_lock_wait` threads across all 3 nodes and - `endpoint_is_task_blocked` no longer appears in any lock path. The ABBA inversion is eliminated. -- **No regression:** `controller_unit_test` all green (incl. `test_ha_core_safety` 20/0); libxce + - controller + unit test build clean. - -## But the round-4 stall had TWO independent causes -A plain release soak WITH the fix STILL segfaults a restarted standby at round 4 (ctrl-3, -"Segmentation fault (core dumped)"). **Deadlocks hang; they do not SIGSEGV** — so this crash is a -SEPARATE memory bug (now bug-1283), not bug-1282. It is most likely an endpoint-lifetime UAF under -chaotic restart (bug-1197 family): the dispatch eligibility path reads `vdata->u.iodata.ep` / -endpoint state while io_worker tears down or mutates the same endpoint. It is a heisenbug — -invisible under ASAN and under gdb (gdb perturbs it into a cold-start timeout / no crash) — and the -sandbox has no usable core (read-only `core_pattern` → non-running systemd-coredump), so a -backtrace needs a non-ASAN DEBUG (-O0 -g) build under gdb or a host with a live coredump pipeline. - -## Net for 7.2 -The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wedge). The remaining -blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint -quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and -elections + job safety hold for every round that completes. - ---- - -# bug-1286 (the round-4 segfault) ROOT-CAUSED — NULL ep->owner — via the new /xce-build debug profile - -The `/xce-build ... debug` profile added this session (non-ASAN `-O0 -g3`, no sanitizer → `build.dbg`) -reproduced the heisenbug AND gave a symbolized backtrace under gdb (ASAN and gdb-on-release both -perturbed it away). The captured SIGSEGV: - -``` -Thread received SIGSEGV -#0 ha_dispatch_gate (ha=0x3ff0, opcode=37) controller/ha.c:3194 (atomic_load &ha->role) -#1 myreq_handler controller/cluster.c:251 -#2 vtask_entry controller/cluster.c:693 -#3 run_task / vcpu_entry lib/vtaskmgr.c -``` - -Root cause: `myreq_handler` does `cluster_t* c = (cluster_t*)ep->owner;` then -`ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a -`walCatchupEntry` (op 37) vtask while `ep->owner == NULL`, so `c == NULL` and -`&c->ha == NULL + offsetof(cluster_t,ha) == 0x3ff0` — a garbage pointer `ha_dispatch_gate` -dereferences → SIGSEGV. (DISTINCT from the bug-1282 deadlock, which is fixed; deadlocks hang, -they do not crash.) Fix: NULL-guard `c = ep->owner` in `myreq_handler` (drop/reject the request if -the endpoint's owner is not yet attached) instead of dereferencing; deeper, ensure `ep->owner` is -set at accept time before the endpoint is monitored for input and never cleared while a vtask is in -flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bug-1283). - -7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock) -is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected. - ---- - -# Both fixes verified (bug-1282 + bug-1286) — round-4 now a CLEAN election-liveness cross-deny - -6-round chaotic soak on real-fs release WITH bug-1282 (deadlock) + bug-1286 (NULL ep->owner crash) -both fixed: rounds 1-3 elect; round 4 has NO segfault, NO fdatasync, NO iomgr deadlock (both fixes -held, confirmed). The round-4 stall is now the GENUINE consensus-liveness edge the crash/deadlock -were masking: - -- ctrl-1 (primary term 6) killed round 4. -- ctrl-2 (recovered durable_floor=15, the FRESHER survivor) denies ctrl-3 on FRESHNESS (ctrl-3 - advertises 12 < 15) - correct. -- ctrl-3 (floor 12) denies the fresher ctrl-2 on ELIGIBILITY (raft_cb_vote_eligible), repeatedly - for 2+ minutes - because ctrl-3 still believes its primary (the dead ctrl-1) is alive - (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd still >=0). So the legitimate winner - (ctrl-2) never gets ctrl-3's grant -> stall. - -This is requirements.md symptom #3 (round-N election stall) in pure form. Contributors to chase: -(a) ctrl-3's "primary alive" view not expiring within the round - the heartbeat-fresh/in_catch_up -eligibility deny (gated on primary_ctrl_fd>=0 && ha_heartbeat_is_fresh by task 5.3) does not lift -because last_activity is not going stale; SUSPECT task 5.2's ha_arm_heartbeat_timeout_vtask re-arm -resetting last_activity=now on the RDHUP-suppress path keeps a node falsely-fresh after the primary -dies - revisit whether the suppress path should reset last_activity. (b) a stale primary_ctrl_fd to -the dead primary not cleared promptly. Next Phase-5 task: a node denying on heartbeat-freshness/ -in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so -the freshest survivor is electable. - -## Session net -bug-1282 (ABBA iomgr deadlock) and bug-1286 (round-4 restart SIGSEGV, NULL ep->owner) are FIXED and -VERIFIED (no crash / no deadlock / no fdatasync across round-4 churn). The remaining 7.2 blocker is -this election-liveness cross-deny - the hard core of Phase 5, no longer masked. - ---- - -# Task 5.5 fix VERIFIED as progress; round-4 now reaches a DEEPER layer (dueling-candidate term escalation) - -Landed the 5.5 fix (commit b8453b2: advertise persist_entry_term(p, log_len) as lastLogTerm instead -of the stale c->persist.term) and re-ran the 6-round soak. Round-4 frozen state: ctrl-2 tip 15 @ -term 5 (freshest), ctrl-3 tip 11 @ term 4 (staler). - -## 5.5 is correct and made real progress -BEFORE the fix the freshest survivor advertised a too-low lastLogTerm and EVERY voter's logOk -rejected it — it could not even win a PreVote. AFTER the fix ctrl-2 WINS PreVote (pre_vote_won) AND -enters the real election ("PreVote round WON — entering real election"). The advertised-term -asymmetry is fixed; the fresher candidate is now electable in principle. - -## The deeper layer it exposed (next task) -ctrl-2 wins PreVote -> real election (term N) -> RequestVote -> replies but NO became_leader -> -retries at term N+1, N+2 ... prospective term climbing 21->25->28->29 unbounded. Mechanism: ctrl-2's -primary (dead ctrl-1) keeps RDHUP'ing ~every 15s; 5.2 does NOT suppress (primary genuinely dead), -so each RDHUP re-triggers ha_begin_failover -> fresh PreVote -> real election -> term++. ctrl-2 and -ctrl-3 duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), -and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally -resolves a split vote. Neither reaches majority; the term escalates without converging. - -Classic Raft split-vote / dueling-candidates liveness problem, amplified by the RDHUP failover -re-trigger. SEPARATE layer from 5.5. Next: break the duel (per-node randomized failover backoff / -election jitter that desynchronizes the two survivors; and/or do not re-trigger a fresh failover -via RDHUP while a real election the node just entered is still within its timeout window). - -## Session net (4 layers peeled on the round-4 stall) -1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d). -2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a). -3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified: the fresher survivor now - wins PreVote + enters the real election (could not before). -4. REMAINING: dueling-candidate split-vote + unbounded term escalation under the RDHUP failover - cadence — the round-4 election round still does not complete. New follow-up. -Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is -purely election LIVENESS under sustained churn. - ---- - -# Task 5.6 fix VERIFIED (unit) + reduces churn; round-4 exposes a 5th layer (vote-reply term-lag escalation) - -Landed 5.6 (commit 81cb09a: ha_on_primary_ctrl_fd_rdhup drops an RDHUP while role==CTRL_ELECTING -or pre_vote_in_progress, instead of restarting failover). Unit RED→GREEN (test_ha_core_safety 22 -Tests / 0 Failures); the soak's term churn is slower (peaked ~24-31 vs ~28-29+ before at the same -elapsed time) — the fix does prevent RDHUP from preempting an in-flight election. - -## The 5th layer (next task 5.7) -Round 4 still stalls. The freshest survivor ctrl-2 (15@t5) repeatedly wins PreVote, enters the real -election, AND ctrl-3 GRANTS it (vote_granted) — ctrl-3 is voting FOR ctrl-2, not leapfrogging -(ctrl-3 wins only 1 prevote total). But ctrl-2 never tallies a majority: its log shows every grant -as **vote_reply_stale_drop** — the reply's term is LOWER than ctrl-2's current election term. -Mechanism: ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can process -the request and reply, so ctrl-3's grant (stamped at ctrl-3's lower term) is always one term behind -ctrl-2's current term → DropStaleResponse → not counted → election times out → term++ → the grant -for the new term is again stale. Self-perpetuating term escalation; ctrl-2 never wins despite a -willing grantor. - -election_timeout_ms = 5000, jitter = 200ms (HA_DEFAULT_*). The 5s window should suffice for a warm -RTT, so the lag points at: a just-restarted ctrl-2 with COLD peer connections (each real-vote opens -a fresh endpoint_connect + SSL handshake; raft_cb_send_vote_request), plus parallel connect attempts -to the dead ctrl-1 (DEFAULT_CONNECT_TIMEOUT_MS) — the grant from ctrl-3 lands after ctrl-2 has moved -on. Fix directions (5.7): (a) don't bump the election term while a vote round-trip for the current -term is still outstanding (give the grant time to land before re-escalating); (b) warm/reuse the -peer vote connection so the grant RTT < election_timeout; (c) larger election-timeout jitter so the -two survivors desynchronize and one's term stops racing the other's reply. Preserve Election Safety -(DropStaleResponse itself is correct Raft — the bug is the escalation cadence, not the drop). - -## Session net (5 layers peeled on the round-4 stall; 4 fixed) -1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d). -2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a). -3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified prevote-win. -4. 5.6 RDHUP preempts in-flight election — FIXED (81cb09a), unit-verified, reduces churn. -5. REMAINING (task 5.7): vote-reply term-lag / unbounded term escalation under cold-connection churn - — the freshest survivor is granted but its term out-races the grant. New follow-up. -Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every -completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn. - ---- - -## Task 5.7 fix VERIFIED + round-4 blocker re-rooted (2026-06-30) - -**5.7 (election-timeout resonance) — FIXED, commit 74784b2.** - -The 5.6 note guessed the 5.7 cause as "term re-escalates faster than the grant -can land." Reading the code pinned the *mechanism* precisely: the per-CAMPAIGN -election watchdog (`raft_start_election` → `on_schedule_timeout` → -`raft_cb_schedule_timeout`, ha.c) was armed with a **FIXED** `election_timeout_ms` -— **no randomization**. Only the *startup-discovery* path (`ha_wait_for_heartbeat_or_timeout`) -jittered; the recurring campaign watchdog did not. raft.c:157 passes the bare -`s->election_timeout_ms` straight through. - -A fixed deadline + a roughly-constant cold-connection grant RTT **RESONATE**: if -ctrl-3's grant consistently lands a hair *after* a constant deadline, every -campaign loses by the *same* margin forever → the campaign times out, the term -re-escalates, and the now-stale grant is dropped by `DropStaleResponse` -(raft.c:453, which is *correct* Raft). Terms climb 25→30+. The bug was never -"RTT genuinely > 5 s"; it was that the deterministic cadence locks the grant -permanently just outside every window. - -**Fix (Ongaro thesis §3.4 — randomized election timeout, re-drawn each campaign):** -new `ha_jittered_election_delay(ha, base)` returns `base + rand[0, base]` (the -canonical [T, 2T] interval), wired into `raft_cb_schedule_timeout` so the watchdog -is re-randomized per campaign. Done in the HA adapter so the pure, sim-tested raft -library stays deterministic. Independent of `election_jitter_max_ms` (which tunes -only the sub-second startup/stagger jitter) so it spreads even at jmax=200; never -below the floor so genuine-loss failover is not sped up past `election_timeout_ms`. - -Unit: `test_5_7_campaign_timeout_is_randomized` (CORE) + `test_5_7_floor_preserved_and_zero_safe` -(PIN). RED verified by stubbing the jitter to 0 → the randomization test fails with -the resonance message while the PIN stays green. `test_ha_core_safety` 24 Tests / 0 -Failures. (Aside: the full aggregate `controller_unit_test` binary cannot reach the -ha_core_safety suite because an EARLIER suite, `test_ha_vep_teardown`, SIGSEGVs in -`iomgr_fini`→`SSL_CTX_free` under openssl 3.6 — unrelated to this change; run the -suite in isolation with `-t test_ha_core_safety`.) - -**Soak confirmation:** a fresh ASAN 6-round chaotic soak now elects with **BOUNDED -terms** — rounds 1→3 elect ctrl-3/ctrl-2/ctrl-1 at terms **4→5→6** (not 25→30+). -The unbounded escalation is gone; zero job loss on every completed round (3/6/9/12 -tracked jobs all present on each new primary). The election-liveness layer is DONE. - -**Round-4 blocker RE-ROOTS to a non-election crash (task 5.8 / bug-1298).** -Round 4 still stalls, but no longer for any election reason. ctrl-3 **ABORTs** on -an ASAN heap-use-after-free: - -``` -UAF READ size 4 endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3) - vtask_eligible vtaskmgr.c:255 ← dataq_deq_first_match ← vdispatcher_entry -FREED BY (T2) endpoint_killer cluster.c:813 (io_worker) - handle_endpoint_error cluster.c:108 ← handle_error ← io_worker -ALLOC endpoint_get netio.c:748 ← handle_connection cluster.c:741 -``` - -io_worker (T2) tears down a connection on error and `free()`s the endpoint via -`endpoint_killer` while the vtask dispatcher (T3) concurrently reads `ep->busy` on -that just-freed endpoint during its dataq eligibility scan. The endpoint is freed -while still referenced by a pending vtask; nothing drains its queued vtasks before -the free and no refcount keeps it alive across the dispatcher read. ctrl-3 dies → -the cluster drops to 1/3 (the round-4 survivors are the just-restarted ctrl-2 and a -now-dead ctrl-3) → no quorum → "NO new primary within window." ctrl-2's logs show -only PreVote-round timeouts (`prospective term=6`, never wins) — the textbook -symptom of a lost quorum peer, not an election defect. - -Same netio surface and routing precedent as task 5.4/bug-1282: an async-io dispatch -concurrency bug, **UPSTREAM-owned** by raft-drive-async / core libxce. Likely a -latent UAF that the bug-1282 lockless `ep->busy` read unmasked (the dispatcher now -reaches the read without contending on `ep->lock`). Logged as **bug-1298**; tracked -as task 5.8. Consensus safety (Phase-2/5.2/5.3/5.5/5.6/5.7) is sound; elections and -job safety hold for every round that completes. - ---- - -# Cross-Spec Gap Analysis (2026-06-30) - -Generated by `/kiro-validate-gap ha-term-safety raft-drive-async raft-ha-core-safety ha-consistency-invariants`. - -## Summary - -4-spec HA safety family: **ha-term-safety COMPLETE**, **raft-drive-async DORMANT** (all tasks done, production flags OFF), **raft-ha-core-safety IN PROGRESS** (12/26 tasks, Phase 5 critical path — task 5.7), **ha-consistency-invariants DESIGN ONLY**. One blocker to production: raft-ha-core-safety 7.2 chaotic soak PASS. - -## Dependency Graph - -``` -ha-consistency-invariants (umbrella — defines invariants) - ├── ha-term-safety (term recording + reconcile) ← COMPLETE - ├── raft-ha-core-safety (payload safety + election liveness + soak gate) ← IN PROGRESS - └── raft-drive-async (non-blocking drive threads) ← DORMANT - gates on: raft-ha-core-safety 7.2 PASS -``` - -## Implementation State - -| Spec | Tasks Done | Tasks Remaining | Status | -|------|-----------|----------------|--------| -| ha-term-safety | 20/20 | 0 | COMPLETE (build green, 175 tests, clean failover verified) | -| raft-ha-core-safety | 12/26 | 14 (5.7 critical) | IN PROGRESS (Phases 1-2+4+5.1-5.4 done; Phase 3 not started) | -| raft-drive-async | 17/17 | 0 (dormant flags) | DORMANT (production flags OFF; gates on raft-ha-core-safety 7.2) | -| ha-consistency-invariants | 0/0 | design only | DESIGN ONLY (no tasks generated) | - -## Critical Path - -``` -raft-ha-core-safety 5.7 (vote-reply term-lag) → 7.2 (soak PASS) → raft-drive-async flags ON -``` - -## Remaining Gaps - -1. **Task 5.7** (NOT STARTED): vote-reply term-lag — candidate's term escalates faster than grant RTT, all grants arrive stale. Fix: don't bump election term while vote round-trip outstanding, or warm peer connections. -2. **Task 7.2** (BLOCKED on 5.7): acceptance chaotic soak gate — ≥6 rounds, zero divergence, zero job loss. Currently: rounds 1-6 pass, round 7 stalls on PreVote cross-denial (bug-1296). Election liveness is the LAST remaining data-path gap; payload safety (Phase 2) and reconvergence (Phase 4) are proven. -3. **Phase 3** (snapshot-boundary reconcile): lowest priority — snapshot install is rare (16-byte sentinel today). -4. **raft-drive-async flags**: peerctrl/snapshot/configpush all default OFF. Flag-flip gate = raft-ha-core-safety 7.2 PASS + borrowed-ep refcount for snapshot/configpush lane. -5. **ha-consistency-invariants**: umbrella spec with approved design, no tasks. Close by verifying all invariants are satisfied by the three implementation specs. - -## Risk - -| Risk | Likelihood | Impact | -|------|-----------|--------| -| Task 5.7 reveals further election liveness layers (5.5→5.6→5.7 chain) | HIGH | Schedule slip; budget 2-3 more iterations | -| Chaotic soak still fails after 5.7 | MEDIUM | 7.2 blocks raft-drive-async; consider sequential-only fallback | -| Raft-drive-async flag-flip reveals new races | LOW | Gradual rollout: peerctrl first | - -## Recommendations - -1. **Immediate**: Fix task 5.7 (vote-reply term-lag) — approach (a): don't bump term while vote outstanding. -2. **Short-term**: Complete raft-ha-core-safety 5.7→7.2→7.3, then flip raft-drive-async flags. -3. **Medium**: Generate tasks from ha-consistency-invariants design; close the umbrella loop. - ---- - -## Task 5.8 fix VALIDATED + round-4..6 reveals pre-existing commit-loss (2026-06-30) - -**5.8 (endpoint teardown-vs-dispatch UAF, bug-1298) — FIXED, commit 6e05dc1.** - -Chosen fix (user picked the principled option over the targeted deferred-teardown -patch): an endpoint REFERENCE COUNT consolidating the prior ad-hoc lifetime -patches. endpoint_t.refcnt (_Atomic) + heap_alloc; endpoint_get → refcnt=1 (owner); -vtask_enq takes a per-vtask ref BEFORE publishing the task to the dispatcher, -run_task / taskq_item_killer release it; every heap-ep teardown drops its owner ref -via endpoint_unref (last unref fini+frees). Embedded eps (client ctx->ep, -heap_alloc=0) are never freed by unref. So the dispatcher reads ep->busy on a queued -vtask without the io_worker freeing it underneath — the UAF is structurally -impossible. Also fixed several latent ep-struct leaks (endpoint_fini-only sites). - -Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown -OFF-path test updated — it asserted the now-fixed struct leak). 6-round ASAN chaotic -soak: ZERO AddressSanitizer reports, ctrl-3 survives round 4, ALL 6 rounds elect at -bounded terms (5→6→7→8). The round-4 crash-stall is gone. - -**The soak now reaches rounds 4-6 for the FIRST TIME — and reveals a pre-existing -committed-job-loss (task 5.9, NOT a regression).** - -Verdict: committed_job_loss=4, committed_divergence=7 → FAIL. But this is NOT a -regression of 5.8 nor of any previously-fixed bug. Proof: - -1. ONE specific committed job (0-0-1782829181-1) goes missing at round 4 and stays - missing through rounds 5 and 6, across THREE different elected primaries - (ctrl-3, ctrl-2, ctrl-3). A permanent committed-prefix gap that propagates as - catch-up streams FROM the gapped node — the exact bug-795/828/851 signature. -2. Rounds 1-3 are CLEAN (all jobs present, zero divergence). Loss begins precisely - at round 4 — the first round whose overlapping kills elect a JUST-RESTARTED node - (ctrl-2 restarted, ctrl-1 killed, ctrl-3 — itself recently restarted — elected). - A protocol/catch-up condition, not a memory condition. -3. This session changed ZERO WAL/commit/catch-up/replication code (persist.c, - config_repl.c, commit.c, repl*). The files touched are cluster.c, ha.c/.h, - netio.c, vtaskmgr.c (ep refcount + the 5.7 election timeout) + 2 test files. The - commit-durability path is byte-identical to the session start — it CANNOT have - regressed because it did not change. -4. ASAN is fully clean — no refcount-induced UAF/double-free/leak. A refcount bug - would crash (caught by ASAN) or corrupt broadly; it cannot selectively delete - one committed job's payload from a WAL. -5. No chaotic soak EVER reached rounds 4-6 before (every prior run crashed/stalled - at round 3-4: bug-851 cross-deny → bug-1282 deadlock → bug-1286 segv → 5.5/5.6/ - 5.7 election stalls → bug-1298 UAF). There is no prior green baseline for rounds - 4-6 to regress FROM — this is a first observation. - -Conclusion: bug-795/828/851 FAILOVER CATCH-UP GAP — a restarted node rejoins and is -elected primary without backfilling the full committed prefix (empty-rejoin catch-up -streams [watermark..committed], no snapshot backfill → permanent gap). EXTERNAL spec -ha-commit-durability (+ raft term/log-safety). Tracked as task 5.9; route there. - -## Phase 1 closeout — 10-round chaotic soak VALIDATES the election layer (2026-06-30) - -Re-ran the chaotic soak at the full 10-round budget (`run-ha-soak.sh --chaotic 10`, -deploy `/home/claude/tmp/ha-1296`) to confirm bug-1296 (the round-7 RDHUP→PreVote→timeout -liveloc) is gone after tasks 5.5/5.6/5.7. - -**Election liveness: PASS (10/10).** Every kill round elected a new primary in a single -term increment — terms walked 4→5→6→7→8→9→10→11→12→13, one step per round, ZERO escalation. -Round 7 (the historical bug-1296 liveloc point) and rounds 8–10 all elected cleanly. The -per-campaign jittered watchdog (5.7) + don't-preempt-in-flight-election (5.6) fully resolve -the resonance. bug-1296 marked FIXED. - -**Consistency: FAIL (expected, task 5.9 / EXTERNAL).** Final verdict -`committed_divergence=11 committed_job_loss=5 result=FAIL`. Job `0-0-1782854622-0` went -missing at round 8 and stayed absent on the reachable primary through rounds 8/9/10 and the -settle — a permanent committed-prefix gap. This is the documented bug-795/828/851/1306 -failover catch-up gap (a just-restarted node elected primary without backfilling the full -committed prefix), NOT an election fault and NOT a regression: rounds 1–7 are job-loss-free, -the loss onset tracks kill timing, and the election machinery is perfect across all 10 rounds. -The per-round `compared=3 present=2` divergence before round 8 is the just-restarted node -lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2) closes. - -Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 / -ha-commit-durability catch-up backfill). - -## Task 5.10(b) — round-4 election-escalation tail: root cause (2026-06-30) - -The bug-1306-fix soak (`/home/claude/tmp/ha-1306fix`, committed_job_loss=0) hit a round-4 -election-escalation storm: terms laddered 6→~25 in one round window, overrunning the 40s -harness window ("NO new primary within window"), before converging (elected round 5). - -**Not RNG-correlation.** `ha_seed_jitter` seeds from `/dev/urandom` (good entropy); the two -survivors' xorshift streams are independent. So the jitter `[T,2T]` is working as designed. - -**Mechanism = a FRESHNESS-DENY escalation loop under a just-restarted-survivor overlap.** -Round 4 killed the term-6 leader (ctrl-1) while ctrl-2 was JUST RESTARTED (low/behind log). -ctrl-2 campaigned and was denied `reason=freshness` at EVERY term 18→24 (waldump/log: ctrl-2 -log shows freshness at 18,19,20,21,22,23,24), timing out each campaign → term++ → re-campaign. -The fresher survivor did not promptly win either (ctrl-3 led at 4, then not again until 27), -so the two survivors churned terms for ~18 increments before one converged. 5.7's per-campaign -jitter bounds the retry RATE but NOT the escalation, because the driver is repeated freshness -rejection of a behind candidate, not a timing collision the jitter can desync. - -**Fix direction (NOT a quick patch — same 5.5/5.6/5.7 election-liveness family):** a candidate -that is repeatedly freshness-denied must recognize it is BEHIND and stop laddering its term — -defer to the fresher survivor (catch up first) rather than re-campaign at term+1 forever. I.e. -gate the real-election term increment on "could I actually win the freshness check?" (a behind -node should PreVote-fail and back off to catch-up, not inflate the cluster term). Must preserve -genuine-loss liveness (R6), the 5.2 spurious-failover suppression, and Election Safety. - -**Validation requires a DETERMINISTIC 2-survivor repro** (one behind-log survivor + one fresher, -primary killed; assert the cluster elects within a BOUNDED number of terms, no unbounded -freshness-deny escalation) before any fix — speculative tuning here risks regressing 5.5/5.6/5.7. -Stochastic: this tail appeared in 1 of 2 ten-round soaks (the other laddered single-increment -terms 4→13). Tracked as task 5.10(b). - -### 5.10(b) DEEPER (2026-06-30) — it's an RDHUP-triggered failover STORM, not a lone behind-node loop - -Log forensics on the escalating survivor (ctrl-2, PID 511891) over the round-4 window: -`begin_failover` ×48, `pre_vote_won` ×24, `event=election_start` ×24, `rdhup` ×27. So the -term ladder is driven by REPEATED `ha_on_primary_ctrl_fd_rdhup → ha_begin_failover → -ha_start_prevote_async → PreVote WON → enter real election (term++)` cycles — roughly one term -bump per RDHUP from the dead primary's socket. PreVote WINS each time (so PreVote's freshness -gate is not the lever here) and the real election then churns; the driver is at the -FAILOVER-TRIGGER level. - -Task 5.6 added "drop an RDHUP while role==CTRL_ELECTING || pre_vote_in_progress" (commit -81cb09a) to stop exactly this. Under the round-4 overlap it is INSUFFICIENT: 48 begin_failovers -got through, so the node must be repeatedly OUTSIDE the electing/prevote window when an RDHUP -lands (e.g. brief returns to CTRL_STANDBY between a lost real election and the next trigger, or -the RDHUP arrives on a re-established fd the guard doesn't cover). Net: a sustained RDHUP cadence -from a dead primary defeats the point-in-time suppression and ladders the term. - -Fix direction (5.6-family, NOT a quick patch): make the RDHUP→failover suppression robust to a -SUSTAINED cadence — e.g. a per-failover cooldown / debounce so a fresh RDHUP cannot re-trigger a -new PreVote+election until the prior attempt has fully resolved OR a minimum interval elapsed; -and/or hold the "failover in progress" latch across the whole prevote→real-election→settle -window (not just the instantaneous role check). Must preserve genuine-loss liveness (R6) and not -re-introduce the stall 5.2/5.5/5.6/5.7 fixed. Needs a deterministic repro: a dead-primary RDHUP -cadence + 2 survivors → assert bounded term growth (≤ K bumps) to convergence. Stochastic in the -soak (1 of 2 ten-round runs). Tracked as 5.10(b). - -## Task 7.3 — Hypothesis-validation closeout (2026-07-01) - -**Original hypothesis (Headline Finding, above):** the `(term,lsn)` commit-identity machinery -already existed at baseline, but the live chaotic soak still diverged — `walcompare FAIL: -committed-prefix divergence at LSN 9` (round 3) — so this spec was scoped as a residual-defect + -convergence/liveness + live-verification spec, not a from-scratch build. - -**Resolution chain, in the order landed:** -- **Payload loss (Phase 2, done 2026-06-27):** cross-term catch-up streamer silently shipped - `PLEN=0` on a short `pread`/`malloc`-fail (task 2.4); persist guard hardened to - `(term,lsn,payload-crc)` (task 2.2). Closed the LSN-9 divergence signature from the Headline - Finding. -- **Election liveness (Phase 5, done 2026-06-30/07-01):** five layers peeled on the round-4 stall - — spurious-failover suppression (5.2), lagging-survivor eligibility deadlock (5.3), an upstream - ABBA deadlock + release SIGSEGV (5.4, routed to `raft-drive-async`), advertised-term cross-deny - (5.5), dueling-candidate split-vote (5.6), vote-reply term-lag resonance (5.7), an - endpoint-teardown UAF (5.8), and the hook-rc-clobber committed-job-loss bug (5.9, bug-1306). - Each was root-caused to a concrete site before any fix, per this spec's own discipline. -- **Committed-divergence residual (5.10a) + snapshot boundary (Phase 3, done 2026-07-01):** the - last standing divergence source was a member that caught up via full-checkpoint snapshot — - `ha_handle_snapshot_chunk`/`persist_load_snapshot` advanced `checkpoint_lsn`/`current_lsn`/ - `applied_lsn` but never reconciled a *term* at the checkpoint boundary, so a stale local term - from a divergent pre-snapshot entry survived install unrepaired (task 3.1 RED: `Expected 7 Was - 3`). Fixed in task 3.2 by carrying the checkpoint's term on the wire (`SnapshotTransfer. - checkpointTerm`, append-only field) and reconciling it into the receiver's per-LSN term index - on install, before `applied_lsn` advances. -- **End-to-end + single-seam verification (Phase 6, done 2026-07-01):** task 6.1 proved the - (term,payload) fidelity guarantee holds across two real `persist_t`/WAL instances through the - real production ingest seam, for both live replication and cross-term catch-up. Task 6.2 - confirmed the one remaining term-blind direct-persist path (`ha_apply_single_entry`) is - unreachable with a real non-zero term in any current production path (its sole trigger, - `raft_send_append_entries`, has zero production call sites) — documented safe-by-guard rather - than speculatively retired, and pinned that `(term,lsn)` identity survives a restart (`replay_wal` - already reconstructs the term index via `record_entry_term`). -- **Full regression sweep (task 7.1, done 2026-07-01):** the first-ever full 69-group ASAN sweep - of `controller_unit_test` (previously only run suite-by-suite in fragments) surfaced exactly 2 - stale tests — both asserting a PRE-fix contract that an earlier, already-approved task in this - same spec (5.3's PreVote-gate relaxation; 5.9's hook-to-explicit-commit-barrier extraction) had - deliberately superseded. Fixed test-only; 69/69 groups green, 0 ASAN reports. - -**Live soak closure (task 7.2, done 2026-07-01):** a ≥6-round chaotic (overlapping-kill) soak -against real-fs release binaries built from the full fix chain above — `committed_divergence=0`, -`committed_job_loss=0`, 3-of-3 reconvergence and a new primary elected every round (terms -2→4→5→6→7→8→9, monotonic, no stalls), independently reproduced 3 times (implementer, adversarial -reviewer, and once more after a harness-hardening pass that replaced a static -`1+2*njobs` divergence-boundary estimate — which had produced a spurious `divergence=7` on a -provably byte-identical WAL set, per `waldump` forensics — with the live authoritative -`committedLsn`, gated on genuine cluster convergence). **The hypothesis is confirmed resolved by -the live soak, not by synthetic unit tests alone**, satisfying this task's own bar. - -**`raft/raft.tla` invariant review against final behavior:** -- **Election Safety** (≤1 leader/term): held throughout every soak round — `probe_primary` - reported one committing primary at each of the 7 distinct terms observed; task 5.3's PreVote - eligibility gate and the vote-once-per-term contract (unchanged by this spec, exercised - end-to-end by the live soak) are the enforcing mechanism. -- **Leader Completeness** (committed entries survive election): the promotion truncate clamp - (`max(safe_committed_lsn, durable_tip)`, bug-1060) was never triggered to cut a leader's own - durable log in any soak round; task 3.2's snapshot-boundary reconcile closes the one path - (post-snapshot-install AE intake) that previously lacked a verifiable `(term,lsn)` anchor to - check completeness against. -- **State Machine Safety** (no divergent committed entries): `walcompare` reported zero divergence - across all three post-hardening soak runs; `waldump` forensics on the pre-hardening run - independently confirmed byte-identical, gap-free, CRC-intact WALs across all 3 members — the - one observed "divergence" was a harness measurement artifact, not a state-machine-safety - violation, and the harness itself is now fixed (task 7.2). -- **Commit Ordering / Durability** (contiguous prefix, ack ⇒ durable-on-majority): unaffected by - this spec's changes (owned by `ha-commit-durability`/`raft-apply-decoupling`); indirectly - reconfirmed by zero committed-job loss and gapless WAL chains across every soak round. - -**Closure:** no residual divergence remains open under this spec. Two items are explicitly -NOT part of this closure and are tracked to their owning specs, per this spec's established -upstream-routing discipline: (1) the dormant `raft_send_append_entries`/heartbeat-driver path -flagged in task 6.2 as a re-evaluation trigger for any future task that wires it up; (2) the soak -harness's `probe_primary`/`ctrlstatus` transient `committedLsn=0` read (guarded against in task -7.2, root cause of the transient read itself not further chased — the guard is sufficient for the -harness's correctness and the transient does not affect production behavior). diff --git a/.kiro/specs/raft-ha-core-safety/spec.json b/.kiro/specs/raft-ha-core-safety/spec.json deleted file mode 100644 index e338608c..00000000 --- a/.kiro/specs/raft-ha-core-safety/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "raft-ha-core-safety", - "created_at": "2026-06-27T03:15:23Z", - "updated_at": "2026-07-09T21:00:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md deleted file mode 100644 index 01d262c6..00000000 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ /dev/null @@ -1,227 +0,0 @@ -# Implementation Plan: raft-ha-core-safety - -> **TDD (project steering, mandatory):** every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first, then implements to GREEN. The deterministic harness (Phase 1) is the RED infrastructure that pins every later fix to a `walcompare`/invariant violation. -> **No `(P)` markers:** Phases 2–6 modify shared files (`ha.c`, `persist.c`, `config_repl.c`) and the RED cases share `test_ha_core_safety.c`, so the plan is sequential — concurrent edits would contend (same constraint as `ha-convergence-recovery`). Foundation Phase 1 establishes shared prerequisites and is also sequential. -> **Correctness oracle:** verify every change against `raft/raft.tla` and the invariant checklist in `src/controller/CLAUDE.md` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability). -> **Verify anchors before editing** (files drift): the design and `research.md` carry 2026-06-27 file:line anchors; re-grep before each edit. -> **Root cause (task 1.3, DONE + verified):** the term-blind hypothesis is **refuted**. The real defect is **cross-term catch-up payload loss** — `ha_stream_wal_to_standby` silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Phase 2 targets the streamer + payload-aware guard (`research.md` §"Task 1.3"). A green synthetic unit test over a still-red soak does not satisfy 1.1/8.1 — closure requires the real-fs chaotic soak (7.2) clean. - -- [x] 1. Foundation: deterministic chaotic harness and live-divergence reproduction -- [x] 1.1 Extend the divergence detector to a machine-checkable per-member report - - Extend `walcompare` to emit a parseable per-member `(lsn -> term, payload-crc, present/MISSING)` report over the committed prefix, and classify each divergence as committed vs uncommitted - - Observable completion: `walcompare` prints a parseable per-member report and a committed-vs-uncommitted classification; the existing soak still detects the round-3 LSN divergence through it - - _Requirements: 7.1, 7.5_ - - _Boundary: Deterministic Chaotic Harness_ - -- [x] 1.2 Add a chaotic soak mode with an isolated verdict - - Add an overlapping-kill mode to the soak driver (kills landing inside election windows, not only sequential kill→wait→rejoin); produce a verdict that counts `walcompare` divergences and committed-job loss while excluding client-discovery (bug-854) and overlayfs `fdatasync` (bug-924) failures from the divergence/job-loss result - - Observable completion: the soak runs chaotic mode against real-fs release binaries and emits a verdict whose divergence/job-loss counts are unaffected by bug-854/bug-924 - - _Requirements: 7.5, 8.4_ - - _Boundary: Acceptance Soak Gate, Deterministic Chaotic Harness_ - -- [x] 1.3 Reproduce and classify the live divergence, then bisect to a seam - - Run the chaotic soak to capture the actual `walcompare` committed-prefix divergence (RED), classify it via `(term, lsn, pcrc)` (which member, which term, committed vs uncommitted), and bisect it to the concrete ingest path; confirm or refute the term-blind-bypass hypothesis - - Observable completion: a recorded RED divergence with member/term/lsn classification and the identified seam, written to `research.md`; the hypothesis is confirmed or a new trigger is named before any fix - - _Requirements: 7.1, 1.1_ - - _Boundary: Deterministic Chaotic Harness_ - -- [x] 1.4 Stand up the unit/integration harness infrastructure - - Create `test_ha_core_safety.c` with the deterministic gate/probe seams (no sleeps; teardown drains pending delaymgr tasks to avoid the bug-1224 teardown-race UAF), the `(term, lsn)`/apply-gap/kill-injection fixtures, and register it in the controller unit-test build; include one smoke-RED case proving the harness can observe a divergence. The per-defect RED bodies are owned by their phases (2.3 ingest, 4.1 reconvergence, 5.1 election) — this task provides the shared seams they build on, not those cases - - Observable completion: `test_ha_core_safety` builds and runs, the injection fixtures and probe seams are usable by later phases, and the smoke-RED case FAILs on the baseline with no ASAN teardown errors - - _Requirements: 7.1, 7.4_ - - _Boundary: Deterministic Chaotic Harness_ - -- [x] 1.5 Make soak primary-detection robust to the release-build log buffer - - Task 1.3 found the release build block-buffers its `--logfile` and flushes only on signal, so the soak's log-grep primary/convergence detection cannot see an elected primary and false-aborts. Switch the soak's primary + convergence detection to a client probe (`ctrlstatus`/`xce_test`) instead of log-grep, so the chaotic soak runs unattended on release binaries. (Only if a client probe is insufficient, fall back to a controller release log-flush change that does not alter production logging semantics.) - - Observable completion: `run-ha-soak.sh --chaotic` on the `~/tmp` real-fs path with the `src/build.release` binaries detects the elected primary each round via the client probe and no longer false-aborts on "no primary within Ns"; a short (2-round) run reaches its verdict line - - _Requirements: 7.5, 8.3_ - - _Boundary: Acceptance Soak Gate, Deterministic Chaotic Harness_ - - _Depends: 1.2_ - -- [x] 2. Committed-prefix divergence fix: cross-term catch-up payload integrity + payload-aware guard - - _Root cause (task 1.3, verified): the cross-term catch-up streamer silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Term and ingest-path are correct — the payload is lost._ -- [x] 2.1 RED: payload-aware persistence guard tests - - Add cases: an empty-payload (`PLEN=0`) entry written at an existing LSN N followed by the correct full-payload entry must end with the FULL payload (today the `(term,lsn)`-only idempotency guard silently no-ops the corrective write); the reverse (full then empty at the same `(term,lsn)`) must NOT degrade the stored entry to empty; an identical `(term, payload-crc)` re-write is a safe idempotent no-op - - Observable completion: the new persistence cases FAIL on the baseline because the guard compares only `(term, lsn)` and cannot see the payload difference - - _Requirements: 1.1, 1.2, 2.1, 2.2, 3.1, 3.4_ - - _Boundary: Persist Conflict Guard_ - -- [x] 2.2 Make the persistence guard `(term, lsn, payload-crc)`-aware - - Extend the LSN-only idempotency check (persist.c:822): identical `(term, payload-crc)` at an LSN is an idempotent no-op; a differing payload-crc (or term) at an existing LSN is surfaced for repair, never a silent `return 0`; never accept a `PLEN=0` write as the durable form of a known-non-empty entry; preserve persist-before-ACK, the term index, and `WAL_LEGACY_TERM` - - Observable completion: the 2.1 cases pass; an empty-over-real write at an existing committed LSN no longer silently survives; identical re-writes stay idempotent - - _Requirements: 1.1, 1.2, 2.1, 2.2, 2.4, 3.1_ - - _Boundary: Persist Conflict Guard_ - -- [x] 2.3 RED: cross-term catch-up streamer payload-integrity tests - - Add a case reproducing the live signature: a cross-term catch-up (stored term < leader current term, bug-1067 path) whose entry-body `pread` returns short / `malloc` fails must NOT result in a persisted `PLEN=0` entry at a committed LSN — instead the stream must error and retry. Also add the defensive case that a `term==0`-shaped catch-up entry routes through the reconcile seam (no direct-persist bypass), and the unrecoverable-fence case (a node that cannot obtain the majority-agreed committed prefix refuses promotion instead of serving divergent state). Instrument `payload_len` at send and at `on_persist_entry` to disambiguate send-side-empty vs transitive-empty propagation - - Observable completion: the streamer case FAILs on the baseline because the short-read/malloc-fail path silently ships `PLEN=0`; the seam-routing and fence cases FAIL because those paths/fence do not exist yet - - _Requirements: 1.1, 1.2, 2.1, 3.1, 3.2, 3.3, 7.1_ - - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - - _Depends: 2.2_ - -- [x] 2.4 Fix the streamer payload integrity + defensive seam routing + fence - - Core: in `ha_stream_wal_to_standby` (config_repl.c ~2154-2163), fail loud on a short `pread`/`malloc`-fail with a bounded retry instead of freeing the buffer and shipping `PLEN=0`; a non-empty entry is never transmitted empty. Disambiguate and handle send-side-empty vs transitive-empty propagation per the 2.3 instrumentation. Defensive: retire the term-blind `ha_apply_single_entry` direct-persist branch so all ingest routes through the seam; surface genuinely-unrecoverable divergence as a loud non-promotable fence (never self-terminate); defer file I/O to delaymgr (never the callback/iomgr thread) - - Observable completion: 2.3 passes (streamer never ships empty; seam-routing and fence cases green); on the 1.3 reproduction path (real-fs release chaotic soak) `walcompare` reports zero committed divergence at the previously-divergent LSNs - - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.3, 3.1, 3.2, 3.3, 3.4_ - - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - - _Depends: 2.2, 2.3_ - -- [x] 3. Snapshot-boundary reconciliation -- [x] 3.1 RED: snapshot post-install reconcile test - - Add a case where a divergent pre-snapshot entry must be reconciled against the installed checkpoint's `(term, lsn)` boundary (today snapshot install sets applied-lsn without a `(term, lsn)` reconcile) - - Observable completion: the case FAILS on the baseline because no `(term, lsn)` reconcile occurs at the snapshot boundary - - _Requirements: 3.1, 5.1_ - - _Boundary: Snapshot Boundary Reconcile_ - -- [x] 3.2 Establish the snapshot `(term, lsn)` boundary - - Make snapshot install establish a verifiable `(term, lsn)` at the checkpoint so post-install AE intake reconciles correctly and no committed entry below the checkpoint can diverge - - Observable completion: 3.1 passes; a member that installed a snapshot matches the leader's `(term, payload-crc)` at and below the checkpoint boundary - - _Requirements: 3.1, 5.1_ - - _Boundary: Snapshot Boundary Reconcile_ - - _Depends: 2.4_ - -- [x] 4. Standby reconvergence completeness - - _Root cause (Phase-4/5 investigation, `research.md`): a recovered standby holding a divergent uncommitted tail above the new leader's committed point (former-primary shape, e.g. ctrl-2 with 12-15 at an old term while leader committed 11) never reaches `ae_already_have` — it loops on `ae_reject_logmismatch`, its control channel flaps, and it never rejoins. A short-suffix standby (ctrl-3) DOES reconverge, so the backtrack works for small divergence but not the former-primary tail._ -- [x] 4.1 RED: divergent-tail reconvergence + apply-gap gate tests - - Primary case: construct a recovered standby with a MULTI-entry divergent uncommitted tail above the leader's committed point and assert it reconverges to follower — truncates the conflicting suffix and reaches `ae_already_have` (no permanent `ae_reject_logmismatch` loop). Secondary case: a rejoining node with an injected catch-up apply gap must not be reported converged/healthy until `applied_lsn` and `log_len` equal the leader's committed boundary - - Observable completion: both cases FAIL on the baseline (the divergent-tail case loops on log-mismatch / never truncates; the apply-gap case reports healthy at match-index alone) - - _Requirements: 5.1, 5.2, 5.3, 5.4, 7.2_ - - _Boundary: Standby Reconvergence_ - -- [x] 4.2 Verify divergent-tail reconvergence + applied-agreement gate (no production fix needed) - - **Discriminator outcome (4.1, adversarially confirmed):** the follower-side AE log-matching backtrack + follower-AE-conflict-truncate path ALREADY reconverges a multi-entry former-primary divergent tail (truncates 12-15 old-term, accepts the leader's term-6 entries, preserves committed prefix 1..11) when driven deterministically WITHOUT a channel flap; and the applied-agreement healthy gate (`ha_process_standby_state`) already blocks "converged" until `applied_lsn == committed_lsn`. Neutering `ha_consume_append_entries` makes the test fail (`reached=0`), proving the real path does the work. **No production change required in Phase 4** — the live stall is NOT follower-side reconvergence. The fix moves to Phase 5 (channel-flap suppression + `in_catch_up`-blocks-PreVote-reply). - - Observable completion: 4.1's divergent-tail + apply-gap tests pass as regression pins; no production diff in Phase 4 (verified by the discriminator) - - _Requirements: 5.1, 5.2, 5.3, 5.4_ - - _Boundary: Standby Reconvergence_ - - _Depends: 2.4_ - -- [x] 5. Election liveness and safety under churn - - _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._ -- [x] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests - - Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`) - - Observable completion: the spurious-failover case FAILS on the baseline (unbounded prevote-timeout loop while a live primary exists); the liveness cases pass/fail per the stall; the Election Safety assertion runs and reports per-term primary counts - - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_ - - _Boundary: Election Liveness_ - -- [x] 5.2 Suppress unwinnable failover on transient RDHUP; keep genuine-loss liveness and single-leader safety - - On a control-channel RDHUP, attempt reconnect + catch-up first, and/or back off failover when a node's own PreVotes repeatedly fail to reach quorum because a current-term leader exists — only escalate to a real election when the primary is genuinely unreachable to the quorum. Preserve: genuine primary loss still elects a higher-term primary (no permanent stall while quorum reachable); step-down-on-higher-term paired with election restart; at-most-one-committing-primary-per-term - - Observable completion: 5.1 spurious-failover case passes (a recovered standby rejoins a live primary instead of prevote-spinning); genuine-loss liveness still completes each round; the Election Safety assertion still holds - - _Requirements: 4.1, 4.2, 4.3, 6.1, 6.2, 6.3_ - - _Boundary: Election Liveness_ - - _Depends: 4.2_ - -- [x] 5.3 Lagging-but-safe survivor must be electable under overlapping churn (NEW — surfaced by the 7.2 soak) - - _Root cause (task 7.2 first soak, `research.md` §"Task 7.2"): distinct from the 5.2 flap. After ≥3 overlapping kills the only fully-up-to-date node is killed and the two survivors are both restarted-and-lagging (e.g. tips 15 and 11 after the leader at 19 dies). Their PreVote rounds repeatedly time out with no grant quorum — prospective terms frozen, never bumped — so no primary is elected, even though the fresher survivor holds the FULL committed prefix (Leader Completeness satisfied) and is a SAFE candidate. Observed contributors: freshness mutual-denial between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240) with no live primary left to catch up FROM, and/or peer-to-peer vote connections never established between two nodes that each only held a control channel to the now-dead primary._ - - RED: deterministically construct the round-4 shape (kill the up-to-date leader; two survivors each holding the full committed prefix but at different lagging tips, both flagged in_catch_up, no live primary) and assert the freshest survivor reaches CTRL_PRIMARY at a higher term within the failover window (no permanent PreVote-timeout loop). Reuse the 1.4 kill/election + apply-gap seams + the 5.1 PreVote dispatch seam. - - Fix: the catching-up/freshness vote gates must not deadlock when the primary is gone — a survivor that holds the majority-agreed committed prefix must be able to grant/win a PreVote once no live primary remains (drop in_catch_up for vote-eligibility when the primary is unreachable AND the candidate is ≥ the voter's committed floor; ensure the peer vote connection is established during failover). Preserve Election Safety (≤1 primary/term), Leader Completeness (never elect a candidate below the committed floor), and the 5.2 spurious-failover suppression. - - Observable completion: the RED passes; a 6-round chaotic soak elects a new primary after EVERY kill round (no round-N stall) with zero committed-job loss; Election Safety still holds - - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - - _Boundary: Election Liveness_ - - _Depends: 5.2_ - - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._ - -- [x] 5.4 Root-cause the chaotic-restart round-N blocker (DONE: it is an UPSTREAM dispatch deadlock — route to raft-drive-async) - - **ROOT-CAUSED via gdb (bug-1282):** the round-4 hang is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (netio.c:672) holds `ep->lock` then blocks on `q->lock` (`dataq_enq`), while `vdispatcher`'s `dataq_deq` (dataq.c:65) holds `q->lock` then blocks on `ep->lock` via the `vtask_eligible`→`endpoint_is_task_blocked` (netio.c:1001) match callback. io_worker dies → the node stops accepting/answering PreVotes → two restarted survivors can't elect → stall. The release-build SIGSEGV is the same `ep`/`q` race corrupting under churn. Confirmed by identical all-thread backtraces on two independently-wedged nodes (gdb SIGQUIT-bt; `research.md` §"ROOT CAUSE NAILED"). **NOT a consensus-logic gap** (PreVote/eligibility/freshness incl. 5.3 are correct). - - **Ownership = UPSTREAM raft-drive-async / core libxce async-io dispatch** (vtaskmgr/dataq/netio), which this spec's Non-Goals exclude. Per the upstream-ownership protocol, the FIX is routed there (bug-1282; minimal: make `ep->busy` atomic + lockless in `endpoint_is_task_blocked`, or io_worker drops `ep->lock` before `dataq_enq`, or one global lock order). 7.2 stays blocked until the upstream lock-order is repaired, then re-run the gate. The 5.4 obligation (root-cause to a concrete site + assign ownership) is met. - - _Requirements: 5.1, 6.2, 7.4, 8.3_ - - _Boundary: Standby Reconvergence, Election Liveness (verification only — fix is upstream)_ - - _Depends: 5.3_ - - -- [x] 5.5 Release the heartbeat-fresh/in_catch_up vote deny on genuine primary death within the failover window (NEW — the unmasked round-4 stall) - - _CLOSED (commit b8453b2; advertised-lastLogTerm fix live-verified). Checkbox flipped: the election-liveness layer is validated end-to-end by the 5.7 6-round soak (bounded-term election every completed round). Kept open historically only because its observable was the green soak, which 5.7 delivered._ - - _Root cause (7.2 soak with bug-1282+bug-1286 fixed, `research.md` §"Both fixes verified"): with the crash + deadlock gone, round 4 stalls on a pure election-liveness cross-deny. The freshest survivor (ctrl-2, floor 15) is the legitimate winner, but the other survivor (ctrl-3, floor 12) denies it on ELIGIBILITY (`raft_cb_vote_eligible`) for minutes because ctrl-3 still believes its primary (the dead ctrl-1) is alive (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd≥0). The "primary alive" view does not expire within the round → the freshest survivor never gets a grant._ - - Investigate/fix: a node denying a PreVote on heartbeat-freshness or in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so the freshest survivor is electable. Prime suspect: task 5.2's `ha_arm_heartbeat_timeout_vtask` re-arm resets `last_activity=now` on the RDHUP-suppress path, keeping a node falsely-fresh after the primary dies — revisit whether the suppress path should reset last_activity (or should only re-arm the watchdog without bumping the freshness clock). Also ensure a stale `primary_ctrl_fd` to a dead primary is cleared promptly. RED: two divergent-freshness survivors, primary killed; assert the freshest reaches CTRL_PRIMARY within the failover window (no multi-round eligibility cross-deny). Preserve Election Safety + the 5.2 spurious-failover suppression (do not re-introduce the flap). - - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no round-N cross-deny stall) with bug-1282/bug-1286 fixes in place - - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - - _Boundary: Election Liveness_ - - _Depends: 5.2, 5.3_ - - _ACTUAL root cause (NOT the last_activity hypothesis) + PARTIAL FIX (commit b8453b2): the cross-deny was a stale **advertised lastLogTerm** — the candidate advertised `c->persist.term` (the persist running term, stale/0 after restart) instead of `persist_entry_term(p, log_len)` (the term of the last WAL entry, the same source voters use via on_get_entry_term). The freshest survivor advertised a too-low term → every voter's logOk rejected it. bug-851 family. FIX VERIFIED AS PROGRESS: the fresher survivor now WINS PreVote + enters the real election (could not before). But the round still does not COMPLETE → a deeper layer remains (task 5.6). Detail in `research.md` §"Task 5.5 fix VERIFIED"._ - -- [x] 5.6 Break the dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence (NEW — the layer under 5.5) - - _CLOSED (commit 81cb09a, don't-preempt-in-flight-election; reduced term churn). Checkbox flipped: 5.7's per-campaign jittered watchdog is the final fix that bounds the term escalation this task targeted; 6-round soak elects with bounded terms. Layer DONE._ - - _Root cause (`research.md` §"Task 5.5 fix VERIFIED"): with 5.5's advertised-term fix in, round-4 ctrl-2 (freshest) wins PreVote and enters the real election, but never reaches majority — it retries at ever-higher terms (21→25→28→29…). Mechanism: the dead primary keeps RDHUP'ing ~every 15s; 5.2 correctly does not suppress (primary genuinely dead), so each RDHUP re-triggers `ha_begin_failover` → fresh PreVote → real election → term++. The two survivors duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally resolves a split vote._ - - Fix direction: introduce/strengthen per-node randomized failover backoff (election jitter) so two surviving candidates DESYNCHRONIZE and one wins; and/or do not re-trigger a fresh failover via RDHUP while a real election the node just entered is still within its election-timeout window (let the in-progress round complete or time out before re-arming). Preserve genuine-loss liveness (R6), spurious-failover suppression (5.2), and Election Safety. RED: two divergent-freshness survivors + a periodic RDHUP cadence; assert a primary is elected within a bounded number of rounds (term does not escalate unbounded). - - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no unbounded term escalation / split-vote stall) with all prior fixes in place — the green 7.2 gate - - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - - _Boundary: Election Liveness_ - - _Depends: 5.5_ - - _PARTIAL FIX (commit 81cb09a): the RDHUP-entry now drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress (don't preempt an in-flight election). Unit RED→GREEN (22 Tests/0 Failures); reduces the term churn. But the soak still stalls at round 4 → a deeper 5th layer (task 5.7): the freshest survivor is GRANTED but its term out-races the grant (vote_reply_stale_drop). Detail in `research.md` §"Task 5.6 fix VERIFIED"._ - -- [x] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6) - - _FIXED + VERIFIED (commit 74784b2, `research.md` §"Task 5.7 fix VERIFIED"). Actual root cause was narrower than the 5.6 note guessed: the per-CAMPAIGN election watchdog was armed with a FIXED `election_timeout_ms` (`raft_start_election`→`on_schedule_timeout`→`raft_cb_schedule_timeout`); only the startup-discovery path jittered. A fixed deadline + a roughly-constant cold-connection grant RTT RESONATE — the single needed grant lands a hair past every deadline forever, the campaign times out, the term re-escalates, and the now-stale grant is dropped (DropStaleResponse correctly). Fix (Ongaro §3.4): re-randomize the watchdog per campaign to the canonical [T, 2T] interval, injected in the HA adapter so the pure (sim-tested) raft library stays deterministic; independent of `election_jitter_max_ms`; never below the floor. New `ha_jittered_election_delay()` + 2 unit tests (RED verified — stubbing jitter to 0 fails `test_5_7_campaign_timeout_is_randomized` with the resonance message; PIN guards floor + modulo-by-zero). `test_ha_core_safety` 24 Tests / 0 Failures. SOAK-CONFIRMED: a fresh 6-round chaotic soak now elects with BOUNDED terms (rounds 1→3 at terms 4→5→6, NOT 25→30+); zero job loss across completed rounds. The election-liveness layer is DONE._ - - _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. election_timeout_ms=5000, jitter=200ms._ - - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate ✅ (election layer met; 7.2 now blocked on a SEPARATE non-election crash, task 5.8) - - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - - _Boundary: Election Liveness_ - - _Depends: 5.6_ - -- [x] 5.8 Endpoint teardown-vs-dispatch heap-use-after-free crash (NEW — the round-4 7.2 blocker after 5.7) - - _FIXED + VALIDATED (commit 6e05dc1, `research.md` §"Task 5.8 fix VALIDATED"). With 5.7's election fix in, the round-4 stall RE-ROOTED off elections entirely: ctrl-3 ABORTed on an ASAN heap-use-after-free — the vtask dispatcher read `ep->busy` (endpoint_is_task_blocked, netio.c) on a queued vtask while the io_worker freed that ep via `endpoint_killer` (cluster.c, from handle_endpoint_error). ctrl-3 died → cluster 1/3 → no quorum → "no new primary."_ - - _FIX (chosen by user over the targeted deferred-teardown patch): a principled endpoint REFERENCE COUNT, consolidating the prior ad-hoc lifetime patches (bug-1282 lockless busy, bug-1197 dispatch_inflight). endpoint_t.refcnt (_Atomic) + heap_alloc; endpoint_get starts refcnt=1 (owner); vtask_enq takes a per-vtask ref BEFORE publishing the task to the dispatcher, run_task/taskq_item_killer release it; every heap-ep teardown site drops its owner ref via endpoint_unref (last unref fini+frees). Embedded eps (client ctx->ep, heap_alloc=0) are never freed by unref. So the dispatcher reads ep->busy on a queued vtask without the io_worker freeing it underneath — UAF eliminated. Also fixed several latent ep-struct leaks (endpoint_fini-only sites)._ - - _VALIDATION: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown updated — its OFF-path test asserted the now-fixed struct leak). 6-round ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 survives round 4, ALL 6 rounds elect at bounded terms (5→6→7→8). The round-4 crash-stall is gone._ - - Observable completion: the 6-round chaotic soak runs all rounds with NO controller crash ✅ (met). 7.2 still NOT green — see below. - - _Requirements: 6.1, 6.2, 6.3_ - - _Boundary: Async-IO dispatch (endpoint lifetime)_ - - _Depends: 5.7_ - -- [x] 5.9 Committed JOB LOSS under chaotic failover — FIXED (the durability half; divergence tracked separately in 5.10) - - _ROOT CAUSE (sharpened 2026-06-30, waldump+log bisect — NOT the catch-up prefix-gap the earlier note guessed): `xce_run_hooks` (src/lib/hook.c:134-157) returns only the LAST-registered hook's rc. The durable-majority gate worked (a term-9 leader's job at LSN 30 timed out commit → `commit_submit` -ETIMEDOUT → `persist_job_hook_fun` returned -ETIMEDOUT), but a later JOB AFTER_EVENT hook (sched/apiserver) returned 0 and CLOBBERED it → `handle_jobsubmit` replied `jobsubmit ok` for an uncommitted entry → a term-10 leader legally overwrote LSN 30 → committed-job-loss. Violated the Commit Durability invariant (reply-success ⇒ majority-durable). All 3 WALs were gapless/CRC-intact/full-payload (rules out prefix-trim + bug-847); lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority._ - - _FIX (raft-apply-decoupling item #1 down payment, commit pending): extracted the WAL append + majority-commit barrier out of `persist_job_hook_fun`'s WAIT branch into an explicit `persist_submit_wal_and_commit(p, jlist)` (persist.c/.h); `handle_jobsubmit` (job.c) calls it directly and reads the commit rc DIRECTLY — rejects on <0 BEFORE firing the now notification-only AFTER hook (canonical commit→notify→reply). The commit verdict is no longer a clobber-prone hook return. Regression pin `test_job_hook_new_is_notify_only` + 4 relocated hook tests; persist_hooks 19/0, e2e_hooks 3/0, commit_barrier_reply 22/0, commit_durability 38/0._ - - _PROVEN: 10-round chaotic soak `committed_job_loss` 5 → **0** (same election code, A/B isolates the durability fix)._ - - _Boundary: Commit durability (raft-apply-decoupling item #1). Full canonical inversion (async keystone + apply_committed_entry seam + facts + restart validation) = the follow-on ha spec._ - - _Depends: 5.8_ - -- [x] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag [5.10a, CLOSED] + election-escalation tail [5.10b, FIX LANDED] - - _5.10b DONE (bug-1310, commit 0d70d5e): RDHUP failover DEBOUNCE. Root cause = a dead primary's fd RDHUPs faster than an election settles; the 5.6 point-in-time guard misses RDHUPs that land after a lost election (role briefly back to STANDBY) → term++ per RDHUP (48 begin_failovers, terms 6→24). Fix: ha_state_t.last_failover_ms stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2×election_timeout of the last trigger. Unit-proven (test_rdhup_within_cooldown_is_debounced); soak 10/10 elect, terms 4→14 bounded, begin_failover 14/14/12 (was 48), no regression._ - - _(a) committed_divergence>0 from the just-restarted node's catch-up LAG (walcompare `present=2`: entry on the 2 live nodes, not yet backfilled on the restarted one). Transient, settles; NOT lost data. (b) Election-escalation TAIL: under an adversarial overlap (kill inside a split-vote window) a round can still stall + ladder terms (observed round-4 term 6→25 before electing) despite 5.7's jitter — liveness held (elected next round) but degraded._ - - _2026-07-01 investigation, part 1: task 7.2's own 3-round-clean soak history never triggered a real snapshot install (waldump confirmed `snap/` empty every run — all rejoins caught up via incremental WAL, never a full checkpoint), so 5.10a's exact scenario remained UNEXERCISED despite the acceptance gate passing. Enabling fixes: (1) `walcompare` gained per-member checkpoint-floor awareness (commit `f3c86a6`) — it previously had no concept of a member's own compaction floor and would always flag a snapshot-caught-up member's pre-checkpoint range as MISSING/divergent regardless of correctness (the false-positive `ha-entity-snapshots` task 7.1 diagnosed as `committed_divergence=11`); proven via 2 dedicated unit tests. (2) a `--max-wal-records` CLI flag + a `--force-snapshot` soak mode (commit `9f4292d`) to deterministically force a checkpoint. This surfaced a delaymgr bug: `schedule_checkpoint()` fired correctly but the delaymgr-scheduled `checkpoint_task_fn` never executed._ - - _2026-07-01 investigation, part 2 (delaymgr fix + real green force-snapshot soak): root-caused the delaymgr stall — `delay_task_t` carried two INCOMPARABLE deadline representations (`time_t nextrun` wall-clock epoch for `delay_task()` seconds-callers vs `struct timespec nextrun_ts` CLOCK_MONOTONIC for `delay_task_ms()` callers sharing the same `cluster_t.delaymgr`); `task_cmp()`'s mixed-type fallback always sorted the ms-task sentinel (`nextrun=0`) ahead, and since `delay_worker()` only ever inspects the queue HEAD while `ha.c` continuously repopulates it with recurring ms-tasks (heartbeats, election timers), any seconds-scheduled task (the checkpoint) was starved indefinitely. Fixed (commit `b9ad018`) by normalizing to one flat `uint64_t` CLOCK_MONOTONIC-ms deadline (`delay.h`/`delay.c`); `delay_task()` is now a thin seconds→ms wrapper over `delay_task_ms()`. Checkpoint now fires and a real snapshot file is written — but the soak STILL failed the same log-grep check on the restarted node, because of a separate, third issue: WAL segment rotation (`open_new_segment`/`rotate_segment`) was gated on a hardcoded 64 MB compile-time `WAL_SEGMENT_SIZE_BYTES`, and `trim_wal_segments()` never deletes the currently-open (active) segment — so at soak job counts (~113 records) the single segment never rotates, nothing is ever physically trimmable, and `ha_evaluate_repl_action` always chose STREAM_WAL over SEND_SNAPSHOT for the empty-rejoining node. Fixed by threading a `wal_segment_bytes` override through `persist_config_t` (mirroring the `--max-wal-records` pattern exactly: `config.h` DEFAULT_WAL_SEGMENT_BYTES, `persist.h` setter, `persist.c` 3 usage sites + minimum-floor enforcement, `main.c`/`myconf.h`/`myapp.c`/`cluster.c` 6-touch-point CLI wiring) and a new `--wal-segment-bytes` CLI flag; the soak's `--force-snapshot` mode now sets `WAL_SEGMENT_BYTES=4096` so ~113 tiny records force multiple real rotations. Re-running then surfaced a FOURTH, genuinely new bug — the first-ever live exercise of a node receiving a real inbound snapshot transfer while mid-election: a heap-use-after-free in `INVOKE_MSG_HANDLER` (`include/netio.h`) — the macro calls `ep->msg_handler(ep)` synchronously (not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing does not cover it) then writes `ep->fsm_in.pkt = NULL`; a stale vote-reply handler chain (`raft_on_vote_reply` → `raft_cb_become_follower` → `endpoint_unref`) could drop the endpoint's last ref and free it mid-handler, so the trailing write hit freed memory (ASAN-confirmed: `handle_input`/netio.c:472, freed by the same thread inside its own call stack). Fixed by holding a ref across the synchronous call (`endpoint_ref`/`endpoint_unref` bracketing the handler invocation), the same pattern task 5.8 established for the async vtask path. With all four fixes in place, `--force-snapshot` now PASSES live: `VERDICT mode=force-snapshot rounds=1 jobs=113 committed_divergence=0 committed_job_loss=0 result=PASS`, with the restarted node's log confirming a real `HA: snapshot` install line before the verdict is trusted. A 3-round `--chaotic` regression run after the `INVOKE_MSG_HANDLER` fix also passed clean (0 divergence, 0 job loss). Full 217-test controller suite green (0 failures) before and after every fix in this chain. 5.10a is CLOSED with direct live evidence, not just synthetic unit tests. (Full 10-round chaotic re-confirmation of 5.10b's debounce, beyond the 3-round regression check run here, was not re-executed this session — its prior 10-round proof stands unless a future soak surfaces a regression.)_ - - Observable completion: MET — `--force-snapshot` soak (the exact 5.10a scenario) is committed_divergence=0 committed_job_loss=0 result=PASS with a confirmed real snapshot install; `--chaotic` regression (3 rounds) is also clean after the netio.h fix - - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - - _Boundary: catch-up backfill + election-liveness tail (5.7 family)_ - - _Depends: 5.9_ - -- [x] 6. Integration: single-seam invariant end-to-end -- [x] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end - - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path - - Observable completion: an end-to-end test (or instrumented soak round) shows leader and standby hold identical `(term, payload-crc)` for every replicated and cross-term-caught-up LSN; `walcompare --report` clean across members - - _Requirements: 2.1, 2.3, 3.2_ - - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - - _Depends: 2.4_ - -- [x] 6.2 Verify the single-seam invariant and restart identity - - Assert no ingest path (live replication, catch-up batch, snapshot post-install, rejoin) writes the WAL outside the reconcile seam, and that each entry's `(term, lsn)` identity survives a restart/recovery. **Carry-forward from 2.4:** the term-blind `ha_apply_single_entry` direct-persist branch was NOT structurally retired in 2.4 (its empty-over-full safety is enforced by the 2.2 payload guard and pinned by `test_2_3_termblind_route_payload_guarded`/`test_2_3_unrepairable_catchup_raises_nonpromotable_fence`). This task must either structurally retire that bypass (route term-0 catch-up through the seam, with its own RED first) OR formally document it as safe-by-persist-guard and adjust the invariant wording accordingly - - Observable completion: an integration test confirms every ingest path reconciles through the seam and that entry terms are preserved across restart; the term-blind bypass is either removed (no direct-persist path remains) or documented safe-by-guard with the pin tests cited - - _Requirements: 1.1, 1.2, 2.4, 3.1_ - - _Depends: 2.4, 3.2, 4.2_ - -- [x] 7. Validation: regression and acceptance soak gate -- [x] 7.1 Full controller suite under ASAN with the new harness green - - Build and run the full controller unit suite under ASAN (suite-by-suite if the aggregate hits the known 300s cap); fix any regression from the ingest/persist/election changes - - Observable completion: the full suite is green under ASAN and every `test_ha_core_safety` case passes with no ASAN errors - - _Requirements: 7.4_ - - _Depends: 6.2, 5.2_ - -- [x] 7.2 Acceptance chaotic-failover soak gate - - Run the chaotic soak (≥6 rounds, overlapping kills) against real-fs release binaries and assert zero `walcompare` divergence, zero committed-job loss, and 3-of-3 reconvergence plus a new primary after every kill round, with bug-854/bug-924 isolated from the verdict - - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); 5.6 (don't preempt an in-flight election) is FIXED (81cb09a, unit-verified, reduces churn); the remaining 7.2 blocker is now **task 5.7** — vote-reply term-lag / unbounded term escalation under cold-connection churn (the freshest survivor is granted but its term out-races the grant). Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._ - -- [x] 7.3 Hypothesis-validation closeout - - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure - - Observable completion: the soak divergence count is zero with closure justified by the live soak, OR a recorded new finding plus a follow-up task; `raft/raft.tla` invariants (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering) reviewed against the final behavior - - _Requirements: 1.1, 1.4, 7.1, 8.1_ - - _Depends: 7.2_ - -## Implementation Notes - -- **Task 2.2 — payload repair must be TIP-ONLY (consensus-safety, caught in review round 1).** The `(term,lsn,payload-crc)`-aware guard repairs an empty-over-real entry by truncate-to-`lsn-1`+re-append, but this is safe ONLY when `lsn == current_lsn-1` (the conflict is at the tip with nothing committed above). For a BELOW-tip conflict (`lsn < current_lsn-1` — the actual live LSN-9 case with 10/11/12 above), truncating the tail would drop committed entries and roll `committed_lsn` backward (State Machine Safety violation). Below-tip is handled as a loud non-promotable fence (reuses `recovery_unrecoverable` sticky flag), never mutating any boundary; the real reconvergence comes from the 2.4 streamer fix + re-stream. The tip path snapshots/restores `committed_lsn` under `p->lock` so the boundary is never observably lowered. Any future in-WAL repair must preserve: never drop a committed entry above the conflict, never lower `committed_lsn`. - -- **Task 5.1 — RED lands (3 cases in `test_ha_core_safety.c`).** CORE RED `test_5_1_spurious_failover_suppressed_while_primary_alive` FAILs on baseline (`Expected 0 Was 5`): it drives the REAL `ha_on_primary_ctrl_fd_rdhup` (unit-test `c==NULL` path) 5× with a FRESH heartbeat (`ha_heartbeat_is_fresh()==1`) and counts PreVote launches — baseline launches one unwinnable round per flap. **Root-cause asymmetry the fix targets:** the DEFERRED PreVote path (`ha_failover_stagger_fn`, ha.c ~6664) ALREADY guards with `ha_heartbeat_is_fresh` and skips, but the RDHUP-INITIATION path (`ha_on_primary_ctrl_fd_rdhup`→`ha_begin_failover`→`ha_start_prevote_async` synchronous/`c==NULL` branch) has NO such guard. **Task 5.2 GREEN = add the `ha_heartbeat_is_fresh` guard at the RDHUP initiation path** (reconnect+rejoin instead of campaigning) WITHOUT regressing the two PASSing pins: `test_5_1_genuine_loss_elects_and_advances_term` (genuine loss — stale heartbeat — must STILL launch; failed round re-arms; rapid rounds advance term) guards against over-suppression, and `test_5_1_election_safety_one_primary_per_term` (vote-once via the real `handle_request_vote`, A=3/B=0 → ≤1 primary/term) is the falsifiable Election Safety check mapped to raft.tla. Note: `handle_request_vote` already denies same-term challengers when `primary_ctrl_fd>=0 && ha_heartbeat_is_fresh` — the VOTE path is safe; only failover-INITIATION is not. - -- **Task 5.2 — GREEN (the one-guard fix, `ha.c:ha_on_primary_ctrl_fd_rdhup`).** Added the spurious-failover-suppression guard between the fd-close and the failover INFO: `if (rdhup_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) { log; if (c) ha_arm_heartbeat_timeout_vtask(ha,c); return; }`. A transient RDHUP while a current-term primary is alive (fresh heartbeat) now stays CTRL_STANDBY and awaits reconnect instead of launching an unwinnable PreVote — the same guard `ha_failover_stagger_fn` already applies on the deferred path. Genuine loss (stale heartbeat) falls through unchanged; no term bump / no role change, so Election Safety is untouched. The watchdog re-arm (production-only, `c!=NULL`) resets `last_activity` to now for a fresh grace window, so a primary that truly never heartbeats again still fires a real failover after `heartbeat_timeout_ms`. **Why one guard suffices over the `in_catch_up`-denies-PreVote facet (obs 1240):** that deny is correct Raft (Req 2.5) and only deadlocks if catch-up never finishes — suppressing the flap keeps the standby attached so catch-up completes, `in_catch_up` clears, and votes flow. Validation: `test_ha_core_safety` 17 Tests / 0 Failures (spurious `launches=0/5` GREEN; genuine-loss `final_term=8`; Election Safety A=3/B=0→1 primary); zero failing suites binary-wide; LSan unchanged (174379 B / 1553 allocs = pre-fix baseline — suppression path allocates a liveness ctx only under real `c`, untouched by the c==NULL unit tests). Live ≥6-round chaotic soak proof is task 7.2. - -- **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2. diff --git a/.kiro/specs/raft-library/design.md b/.kiro/specs/raft-library/design.md deleted file mode 100644 index 0b3bf691..00000000 --- a/.kiro/specs/raft-library/design.md +++ /dev/null @@ -1,662 +0,0 @@ -# Design Document: raft-library - -## Overview - -This feature extracts the Raft-style election state machine from `ha.c` into a self-contained -C library (`src/lib/raft/`) that is decoupled from the iomgr/SSL/capnp transport layer. The -library exposes a pure state machine with a callback interface for all transport effects, enabling -deterministic unit testing without a live cluster. - -**Purpose**: Provide a correct, independently verifiable Raft election implementation that eliminates -split-brain bugs traceable to entangled state transitions in `ha.c`. - -**Users**: Controller developers will use this to wire the election protocol into the controller -via a thin adapter, and to verify protocol correctness via a deterministic N-node simulator. - -**Impact**: `ha.c` becomes an adapter over `raft.h` rather than the owner of election state. -`ha_state_t.term`, `ha_state_t.voted_for_*`, and quorum logic move to `raft_state_t`. - -### Goals -- Correct Raft invariants: one vote per term, quorum safety, no split-brain in 3-node deterministic tests -- Transport-free public API: `raft.h` has zero dependency on `endpoint_t`, `SSL_CTX`, `iomgr_t`, `cluster_t` -- Exhaustive protocol-level test cases covering all invariants specified in requirements -- Drop-in wiring: the controller adapter preserves existing XCE_TESTING hook seams - -### Non-Goals -- Full Raft log replication (XCE is single-writer; only leader election is needed) -- WAL truncation on failover (remains in `ha.c`/`persist.c`) -- Config replication push (remains in `config_repl.c`) -- Disk persistence of term/voted_for (caller's responsibility via on_persist callback) -- Multi-Raft or multi-group election - ---- - -## Boundary Commitments - -### This Spec Owns -- `src/lib/raft/raft.h` — public API: `raft_state_t`, `raft_callbacks_t`, all function signatures -- `src/lib/raft/raft.c` — pure state machine implementation -- `src/lib/raft/raft_sim.c` — deterministic N-node simulator (test harness only) -- `src/lib/testsuite/test_raft.c` — protocol-level test suite (≥ 20 test cases) -- `src/lib/meson.build` additions for `raft.c` and `test_raft.c` -- The definition of `raft_callbacks_t` and the semantic contract for each callback - -### Out of Boundary -- `ha.c` adapter wiring (separate task, touches ha.c but does not own raft.h) -- SSL/iomgr transport implementation of callbacks -- WAL truncation, config push, cluster_init_phase2 -- Persistence to disk (ha-term.bin) — caller implements via on_persist callback -- Controller startup sequencing (`myapp_run`) - -### Allowed Dependencies -- **`raft.h` and `raft.c`: `stdint.h`, `string.h`, `errno.h` only** — no pthread, no OS primitives -- The library has zero threading dependencies; concurrency is entirely the caller's responsibility -- No libxce dependencies from `raft.h` (headers that include `endpoint_t` or similar are forbidden) -- Test file may include `<unity/unity.h>` (project test framework) and `<pthread.h>` (for test harness only) -- `raft_sim.c` may include `<string.h>` and `<stdio.h>` for the simulator and dump log - -### Revalidation Triggers -- Changes to `raft_callbacks_t` shape (adding/removing fields) require re-check of `ha.c` adapter -- Changes to quorum formula require re-running protocol-level tests -- Changes to role enum values require scanning `ha.c` for role-comparison code - ---- - -## Architecture - -### Existing Architecture Analysis - -`ha.c` currently mixes three concerns: -1. **Raft state** — term, voted_for, role, quorum tracking (`ha_elect_ctx_t`) -2. **Transport** — `endpoint_connect()`, `iomgr_monitor()`, SSL setup, capnp encoding/decoding -3. **Post-election actions** — `cluster_init_phase2()`, `push_config_to_standbys()`, WAL truncation - -The split-brain bugs arise from concern #1 being modified across threads without a consistent lock discipline: the transport layer (concern #2) and post-election hooks (concern #3) call into Raft state functions without a clear ownership model. - -### Architecture Pattern - -```mermaid -graph TB - subgraph libxce_raft - RaftState[raft_state_t] - RaftAPI[raft.c Pure SM] - RaftSim[raft_sim.c N-node sim] - end - - subgraph ha_adapter - HaAdapter[ha.c adapter] - end - - subgraph transport - Iomgr[iomgr endpoint_t SSL] - Persist[ha_persist_term] - DelayMgr[delaymgr timeout] - end - - HaAdapter -->|calls| RaftAPI - RaftAPI -->|callbacks| HaAdapter - HaAdapter -->|drives| Iomgr - HaAdapter -->|drives| Persist - HaAdapter -->|drives| DelayMgr - RaftSim -->|calls| RaftAPI -``` - -**Key decisions:** -- **No threads in the library**: `raft.c` is fully synchronous. The caller holds a lock before calling any `raft_*` function. -- **Callbacks fire synchronously**: The library calls back into the adapter within the same call frame. The adapter is responsible for not holding library locks during slow I/O. -- **Role enum is self-contained**: `raft_role_t` in `raft.h` is independent of `ctrl_role_t` in `ha.h`; the adapter maps between them. - -### Technology Stack - -| Layer | Choice | Role | -|-------|--------|------| -| Language | C gnu17 | Matches project standard | -| OS dependencies | **None** (`stdint.h`, `string.h`, `errno.h` only) | Library is OS-free; concurrency is entirely caller-owned | -| Threading | Caller-owned (pthread or none) | `raft_state_t` is not thread-safe by itself; caller holds whatever lock fits their threading model | -| Atomics | None in library | Caller uses atomics for ha_state_t integration if needed | -| Test framework | Unity (existing) | Consistent with controller testsuite | -| Build | Meson (existing) | `static_library('raft', 'raft.c')` added to `src/lib/meson.build` | - ---- - -## File Structure Plan - -``` -src/lib/ -├── raft/ -│ ├── raft.h # Public API: raft_state_t, raft_callbacks_t, all function signatures -│ ├── raft.c # Pure state machine: term, voted_for, role, quorum evaluation -│ └── raft_sim.c # N-node deterministic simulator (compiled only under XCE_TESTING) -├── testsuite/ -│ └── test_raft.c # Protocol-level test suite (≥ 20 cases) -└── meson.build # Add raft.c to xcelib; add test_raft.c to xcelib_test -``` - -**Modified files:** -- `src/lib/meson.build` — add `raft/raft.c` to xcelib sources; add `testsuite/test_raft.c` -- `src/controller/ha.h` — add `#include "raft/raft.h"` and `raft_state_t raft` field to `ha_state_t` (adapter task) -- `src/controller/ha.c` — delegate election logic to `raft_*` API (adapter task, separate from library task) - ---- - -## System Flows - -### Election State Machine - -```mermaid -stateDiagram-v2 - [*] --> FOLLOWER : raft_init - FOLLOWER --> CANDIDATE : raft_start_election - CANDIDATE --> LEADER : quorum reached - CANDIDATE --> FOLLOWER : timeout or higher term - LEADER --> FOLLOWER : higher term seen - FOLLOWER --> FOLLOWER : raft_on_leader_lost clears voted_for - FOLLOWER --> CANDIDATE : raft_on_election_timeout -``` - -### Vote Request Processing - -```mermaid -sequenceDiagram - participant Caller - participant RaftSM as raft.c - participant CB as callbacks - - Caller->>RaftSM: raft_on_vote_request(state, from, term) - alt term < current_term - RaftSM-->>Caller: RAFT_DENY stale term - else term > current_term - RaftSM->>RaftSM: update term, clear voted_for, role=FOLLOWER - end - alt already voted for different candidate same term - RaftSM-->>Caller: RAFT_DENY already voted - else on_vote_eligible returns 0 - RaftSM->>CB: on_vote_eligible(ctx, from, term) - CB-->>RaftSM: 0 deny - RaftSM-->>Caller: RAFT_DENY policy - else all checks pass - RaftSM->>CB: on_vote_eligible(ctx, from, term) - CB-->>RaftSM: 1 grant - RaftSM->>CB: on_persist(term, from) - RaftSM->>RaftSM: voted_for = from - RaftSM-->>Caller: RAFT_GRANT - end -``` - -**on_vote_eligible contract** (implemented by ha.c adapter): -- Return 0 if local role is LEADER or CTRL_PRIMARY_COMMITTING -- Return 0 if STANDBY with fresh heartbeat (primary still alive) -- Return 0 if in-catch-up state -- Return 0 if candidate config_tag < local config_tag, or equal tag with lower LSN -- Return 1 otherwise (candidate is eligible) - -### RDHUP Failover (Priority Jitter) - -```mermaid -sequenceDiagram - participant Caller - participant RaftSM as raft.c - participant CB as callbacks - - Caller->>RaftSM: raft_on_leader_lost(state, my_priority) - RaftSM->>RaftSM: clear voted_for, role=FOLLOWER - RaftSM->>CB: on_start_election_after_delay(priority * election_timeout_ms) - note over Caller: delay elapses in caller's timer - Caller->>RaftSM: raft_check_and_start_election(state) - alt voted_for set to non-self (received vote during delay) - RaftSM-->>Caller: RAFT_SKIP_ELECTION - else - RaftSM->>RaftSM: raft_start_election internally - RaftSM->>CB: on_send_vote_request per peer - end -``` - ---- - -## Components and Interfaces - -### Component Summary - -| Component | Layer | Intent | Req Coverage | Contracts | -|-----------|-------|--------|--------------|-----------| -| `raft_state_t` | Data | Raft node state | 1.1, 1.2 | State | -| `raft_callbacks_t` | Interface | Transport/effect boundary | 7.1, 7.2, 7.3 | Service | -| `raft.c` State Machine | Logic | Pure election logic | 1–6 | Service | -| `raft_sim.c` Simulator | Test | Deterministic N-node harness | 8.1, 8.2 | Service | -| `ha.c` Adapter | Integration | Wire raft.c to iomgr/SSL | 9.1, 9.2, 9.3 | Service | - ---- - -### Core Data Structures - -```c -/* raft_role_t: self-contained role enum (does not depend on ctrl_role_t) */ -typedef enum { - RAFT_FOLLOWER = 0, - RAFT_CANDIDATE = 1, - RAFT_LEADER = 2, -} raft_role_t; - -/* raft_state_t: pure Raft consensus state for one node. - * - * Contains only the Raft invariant state. Application-specific fields - * (config_tag, log_lsn, heartbeat timestamps, ctrl_role) stay in ha_state_t; - * they are accessed by the caller via the on_vote_eligible callback. */ -typedef struct { - uint64_t term; /* current term (monotonically increasing) */ - raft_role_t role; /* current role */ - int voted_for; /* peer index voted for this term; -1=none; self_idx=self-vote */ - uint64_t voted_for_term; /* term in which voted_for was set */ - int n_peers; /* total cluster size (including self) */ - int self_idx; /* this node's index in [0, n_peers) */ - int votes_granted; /* votes collected in current election */ - int election_decided; /* 1 once on_become_leader/follower fired */ - uint32_t election_timeout_ms; - int my_priority; /* priority index for jitter (0 = highest) */ -} raft_state_t; - -/* raft_callbacks_t: all side effects the library fires. - * - * The library owns Raft invariants (term ordering, one-vote-per-term, quorum). - * All application-specific policy (freshness gate, role gate, catch-up state, - * heartbeat liveness) is delegated to on_vote_eligible so the library is - * fully independent of XCE domain logic. - */ -typedef struct { - /* Send a RequestVote RPC to peer peer_idx for the given term. - * The caller encodes config_tag, log_lsn, and any other fields in the wire RPC. */ - int (*on_send_vote_request)(void *ctx, int peer_idx, uint64_t term); - - /* Application-specific vote eligibility gate. - * Called after Raft invariants pass (correct term, voted_for unset). - * Returns 1 to grant, 0 to deny. Receives the candidate's peer_idx and - * the incoming term so the caller can check freshness (config_tag, log_lsn), - * role gate (LEADER/COMMITTING deny), catch-up status, heartbeat liveness. - * The library grants the vote only if this callback returns 1. */ - int (*on_vote_eligible)(void *ctx, int from_idx, uint64_t term); - - /* Node has won the election */ - void (*on_become_leader)(void *ctx, uint64_t term); - /* Node lost election or saw higher term; leader_idx = -1 if unknown */ - void (*on_become_follower)(void *ctx, uint64_t term, int leader_idx); - /* Persist term + voted_for before granting vote (durability guarantee) */ - int (*on_persist)(void *ctx, uint64_t term, int voted_for); - /* Schedule election timeout (fires raft_on_election_timeout after delay_ms) */ - void (*on_schedule_timeout)(void *ctx, uint32_t delay_ms); - /* Cancel pending election timeout */ - void (*on_cancel_timeout)(void *ctx); - /* Request delayed election start after RDHUP (priority-based jitter) */ - void (*on_start_election_after_delay)(void *ctx, uint32_t delay_ms); - /* Trace callback for observability (optional, may be NULL) */ - void (*on_trace)(void *ctx, const char *event, const char *detail); -} raft_callbacks_t; -``` - ---- - -### Public API (`raft.h`) - -```c -/* - * raft_config_t: initialization context for a Raft node. - * - * Encapsulates all configuration so raft_init() has a single, extensible - * parameter rather than a flat list. Matches the existing XCE pattern - * (ha_config_t, persist_config_t). Zero-initialize and set only the - * fields you need — the library fills in safe defaults for unset fields. - * - * Persistence model: - * initial_term / initial_voted_for are restored by the caller from disk - * before raft_init(). The library never reads storage; it only writes - * via on_persist callbacks. Pass {0, -1} on first boot. - */ -typedef struct { - int n_peers; /* total cluster size (including self) */ - int self_idx; /* this node's index in [0, n_peers) */ - uint32_t election_timeout_ms; /* base election timeout; 0 → default 2000ms */ - int my_priority; /* priority index for RDHUP jitter (0 = highest) */ - uint64_t initial_term; /* term restored from disk; 0 on first boot */ - int initial_voted_for; /* voted_for restored from disk; -1 if none */ - const raft_callbacks_t *cbs; /* all callbacks; required callbacks must be non-NULL */ - void *ctx; /* opaque caller context passed to every callback */ -} raft_config_t; - -/* - * raft_init: initialize raft_state_t from a raft_config_t. - * on_vote_eligible and on_trace may be NULL (optional policy / observability). - * All other callbacks in cbs must be non-NULL. - * Returns 0 on success; -EINVAL if a required callback is NULL or n_peers < 1. - */ -int raft_init(raft_state_t *s, const raft_config_t *cfg); - -/* - * raft_start_election: begin a new election. - * Increments term, records self-vote (voted_for=self_idx), transitions to - * CANDIDATE, fires on_send_vote_request for each peer, fires on_schedule_timeout. - * Returns 0 on success; -EALREADY if already CANDIDATE/LEADER; - * RAFT_ELECTED_IMMEDIATELY for single-node (n_peers==1). - */ -int raft_start_election(raft_state_t *s); - -/* - * raft_on_vote_request: process an incoming RequestVote RPC. - * Applies Raft invariants (term comparison, one-vote-per-term). - * If invariants pass, calls on_vote_eligible to allow application policy. - * Fires on_persist before granting. - * Returns RAFT_GRANT (1) or RAFT_DENY (0); sets *reply_term to current term. - */ -int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term, - uint64_t *reply_term); - -/* - * raft_on_vote_reply: process a reply to our RequestVote. - * Returns RAFT_BECAME_LEADER, RAFT_BECAME_FOLLOWER, or RAFT_PENDING. - */ -int raft_on_vote_reply(raft_state_t *s, int from_idx, int granted, uint64_t term); - -/* - * raft_on_election_timeout: called by the timer when no quorum was reached. - * Transitions CANDIDATE → FOLLOWER; fires on_become_follower, on_cancel_timeout. - */ -void raft_on_election_timeout(raft_state_t *s); - -/* - * raft_on_leader_lost: primary control channel dropped (RDHUP). - * Clears voted_for, transitions to FOLLOWER, fires on_start_election_after_delay. - */ -void raft_on_leader_lost(raft_state_t *s); - -/* - * raft_check_and_start_election: call after the delay from raft_on_leader_lost. - * Starts election only if we have not already voted for a non-self candidate - * in the new term (i.e., another node won during the jitter window). - * Returns RAFT_SKIP_ELECTION or RAFT_ELECTION_STARTED. - */ -int raft_check_and_start_election(raft_state_t *s); - -/* Return codes */ -#define RAFT_OK 0 -#define RAFT_GRANT 1 -#define RAFT_DENY 0 -#define RAFT_BECAME_LEADER 1 -#define RAFT_BECAME_FOLLOWER 2 -#define RAFT_PENDING 3 -#define RAFT_ELECTED_IMMEDIATELY 4 -#define RAFT_SKIP_ELECTION 5 -#define RAFT_ELECTION_STARTED 6 -``` - -**Preconditions for all functions:** Caller must hold a mutex protecting `raft_state_t`. -**Postconditions:** Callbacks may fire synchronously within the call; caller must not hold transport locks during callback execution. - ---- - -### Deterministic Simulator (`raft_sim.c`) - -The simulator replaces real network I/O with synchronous in-process callback routing. - -```c -/* raft_sim_node_t: one simulated cluster node */ -typedef struct { - raft_state_t state; - raft_callbacks_t cbs; - void *ctx; /* points back to sim_cluster_t */ - int idx; - int partition; /* 1 = isolated from other nodes */ -} raft_sim_node_t; - -/* raft_sim_cluster_t: N-node simulation cluster */ -typedef struct { - raft_sim_node_t nodes[RAFT_SIM_MAX_NODES]; - int n; - int leader_count; /* incremented on each on_become_leader */ -} raft_sim_cluster_t; - -int raft_sim_init(raft_sim_cluster_t *c, int n_nodes, uint32_t election_timeout_ms); -void raft_sim_start_all_elections(raft_sim_cluster_t *c); /* triggers all nodes simultaneously */ -void raft_sim_deliver_pending(raft_sim_cluster_t *c); /* routes buffered RPCs */ -void raft_sim_fire_timeout(raft_sim_cluster_t *c, int node_idx); -void raft_sim_partition(raft_sim_cluster_t *c, int node_idx, int isolated); -``` - ---- - -## Protocol-Level Use Cases (Testing Strategy) - -The following test cases verify Raft invariants at the protocol level, independent of transport. -Each test uses `raft_sim_cluster_t` and runs to completion synchronously. - -### Group A: Basic Election Correctness (Req 1–4, 7, 8) - -| TC | Scenario | Nodes | Expected | -|----|----------|-------|----------| -| A1 | Single-node cluster starts election | 1 | Immediately LEADER, on_become_leader fires once | -| A2 | 3-node simultaneous election, node 0 has priority | 3 | Exactly 1 LEADER; nodes 1,2 grant to node 0 and become FOLLOWER | -| A3 | 3-node election, node 0 wins, node 1 starts late election same term | 3 | Node 1's election denied (voted_for set); still 1 LEADER | -| A4 | 5-node election, quorum = 3 | 5 | Leader wins after 3 grants; 2 remaining nodes become FOLLOWER | -| A5 | Node receives vote request in LEADER role | 3 | Denies regardless of term; role stays LEADER | - -### Group B: Voted_For Invariant (Req 3, one-vote-per-term) - -| TC | Scenario | Expected | -|----|----------|----------| -| B1 | Node A and B both request vote from node C in same term | C grants only the first requester; second is denied | -| B2 | Node sees vote request with term > current_term | Updates term, clears voted_for, re-evaluates request | -| B3 | Node sees vote request with term < current_term | Denies immediately, returns current term | -| B4 | Node already voted for X in term T; receives request from Y in term T | Denies Y; voted_for stays X | -| B5 | Node granted vote for X in term T; receives request from X again in term T | Grants (idempotent re-grant to same candidate) | - -### Group C: on_vote_eligible Policy Callback (Req 3.4, 3.5) - -The library calls `on_vote_eligible` after Raft invariants pass. These tests use a -mock `on_vote_eligible` that returns a configurable value: - -| TC | Scenario | on_vote_eligible returns | Expected | -|----|----------|--------------------------|----------| -| C1 | All Raft invariants pass; on_vote_eligible returns 0 | 0 | Denied (policy) | -| C2 | All Raft invariants pass; on_vote_eligible returns 1 | 1 | Granted | -| C3 | on_vote_eligible is NULL at init | — | All eligible candidates granted (no policy gate) | -| C4 | Raft invariant fails (stale term); on_vote_eligible NOT called | N/A | Denied before callback fires | -| C5 | Voted-for already set; on_vote_eligible NOT called | N/A | Denied before callback fires | - -*Note*: The freshness gate (config_tag, log_lsn), role gate (LEADER/COMMITTING deny), -and heartbeat-freshness gate are ha.c adapter responsibilities implemented inside -`on_vote_eligible` — not tested in the library test suite. - -### Group H: Persistence and Restart (Req 1.2, on_persist) - -| TC | Scenario | Expected | -|----|----------|----------| -| H1 | raft_init with cfg.initial_term=5, cfg.initial_voted_for=1 | state.term=5, state.voted_for=1 immediately | -| H2 | raft_init with cfg.initial_term=0, cfg.initial_voted_for=-1 (first boot) | state.term=0, state.voted_for=-1 | -| H3 | Vote granted; on_persist fires before voted_for is recorded | on_persist called with (term, from_idx) before grant returns | -| H4 | on_persist returns non-zero (disk failure) | Vote denied; voted_for unchanged | -| H5 | raft_start_election: on_persist fires for self-vote | on_persist called with (new_term, self_idx) | - -### Group D: Quorum and Result (Req 4) - -| TC | Scenario | Expected | -|----|----------|----------| -| D1 | 3-node: 1 grant received (self + 1 = 2 = quorum) | LEADER immediately | -| D2 | 3-node: both peers deny | election_decided=1, FOLLOWER, on_become_follower fires | -| D3 | on_become_leader fires: second vote reply arrives | No-op (election_decided guard) | -| D4 | Reply with term > current_term | Update term, FOLLOWER, on_become_follower fires | - -### Group E: Election Timeout (Req 6) - -| TC | Scenario | Expected | -|----|----------|----------| -| E1 | Timeout fires before quorum | on_become_follower; on_cancel_timeout | -| E2 | Timeout fires after quorum | No-op (election already decided) | -| E3 | New election after timeout | term increments; on_schedule_timeout fires again | - -### Group F: RDHUP Failover (Req 5) - -| TC | Scenario | Expected | -|----|----------|----------| -| F1 | 3-node: node 0 killed; nodes 1,2 call raft_on_leader_lost | Node 1 (priority=1) waits 1×timeout; node 2 (priority=2) waits 2×timeout | -| F2 | During node 1's delay, node 1 receives vote request from another election starter | voted_for set; raft_check_and_start_election returns RAFT_SKIP_ELECTION | -| F3 | During node 1's delay, no vote request arrives | raft_check_and_start_election starts election; 1 LEADER | -| F4 | Simultaneous RDHUP (all remaining nodes fire at same time, no delay difference) | Only one LEADER (self-vote prevents double grant) | - -### Group G: Split-Brain Prevention (Req 4, overall correctness) - -| TC | Scenario | Expected | -|----|----------|----------| -| G1 | 3-node simultaneous elections all terms (100 random election rounds via simulator) | Never more than 1 LEADER per term across all rounds | -| G2 | Network partition: nodes 0+1 isolated from node 2 | Partition of 2 can elect; partition of 1 cannot | - ---- - -## Requirements Traceability - -| Req | Summary | Component | Interface/Contract | -|-----|---------|-----------|-------------------| -| 1.1 | State fields | `raft_state_t` | Data struct | -| 1.2 | Init to FOLLOWER | `raft_init` | Function | -| 1.3 | No transport types in header | `raft.h` boundary | Boundary commit | -| 2.1 | Election: incr term, self-vote, send RPCs | `raft_start_election` | Function | -| 2.2 | Single-node shortcircuit | `raft_start_election` | `RAFT_ELECTED_IMMEDIATELY` | -| 2.3 | Error if already CANDIDATE/LEADER | `raft_start_election` | `-EALREADY` | -| 3.1 | Higher term → update, re-evaluate | `raft_on_vote_request` | Function | -| 3.2 | Lower term → deny | `raft_on_vote_request` | Return value | -| 3.3 | Same term, different voted_for → deny | `raft_on_vote_request` | voted_for check | -| 3.4 | Freshness gate → grant + persist | `raft_on_vote_request` | `on_persist` callback | -| 3.5 (5) | LEADER/COMMITTING → deny | `raft_on_vote_request` | Role check | -| 4.1 | Quorum → LEADER | `raft_on_vote_reply` | `RAFT_BECAME_LEADER` | -| 4.2 | Higher term in reply → FOLLOWER | `raft_on_vote_reply` | `RAFT_BECAME_FOLLOWER` | -| 4.3 | Post-decision no-op | `raft_on_vote_reply` | `election_decided` guard | -| 4.4 | All replied, no quorum → FOLLOWER | `raft_on_vote_reply` | vote counting | -| 5.1 | Leader lost → clear voted_for + delay callback | `raft_on_leader_lost` | `on_start_election_after_delay` | -| 5.2 | Already voted during delay → skip | `raft_check_and_start_election` | `RAFT_SKIP_ELECTION` | -| 5.3 | Not voted → start election | `raft_check_and_start_election` | `RAFT_ELECTION_STARTED` | -| 6.1 | Start election → schedule timeout | `raft_start_election` | `on_schedule_timeout` | -| 6.2 | Timeout fires, undecided → follower | `raft_on_election_timeout` | `on_become_follower` | -| 6.3 | Timeout fires, decided → no-op | `raft_on_election_timeout` | `election_decided` guard | -| 7.1 | raft_callbacks_t struct definition | `raft.h` | Data struct | -| 7.2 | void* ctx threading | All callbacks | Function signatures | -| 7.3 | NULL callback → error | `raft_init` | `-EINVAL` | -| 8.1 | Deterministic simulator | `raft_sim.c` | Simulator API | -| 8.2 | 4 invariant test cases | `test_raft.c` | Groups A–G | -| 8.3 | ASAN clean | Build config | Meson test suite | -| 9.1 | Adapter implements callbacks | `ha.c` adapter (separate task) | raft_callbacks_t impl | -| 9.2 | ha.c delegates state to library | `ha.c` adapter (separate task) | ha_state_t.raft field | -| 9.3 | XCE_TESTING hooks preserved | `ha.c` adapter (separate task) | Existing seams | - ---- - -## Error Handling - -| Error | Source | Response | -|-------|--------|----------| -| NULL callback at init | `raft_init` | Return `-EINVAL` before any state is set | -| `raft_start_election` called while CANDIDATE | State check | Return `-EALREADY` | -| `on_persist` callback returns error | `raft_on_vote_request` | Do not grant vote; return RAFT_DENY | -| `on_send_vote_request` fails for one peer | Election start | Count as no-reply for that peer; election proceeds with remaining peers | - ---- - -## Observability and Debug Tracing - -Protocol correctness requires that every state transition is observable without a live cluster. -The library provides two tracing mechanisms: - -### 1. Trace Callback (production + test) - -Add an optional `on_trace` field to `raft_callbacks_t`: - -```c -/* on_trace: fired on every state transition or significant decision. - * event: short string literal, e.g. "vote_granted", "became_leader", "deny_stale_term" - * detail: formatted human-readable context string (caller-allocated) - * May be NULL to suppress tracing. */ -void (*on_trace)(void *ctx, const char *event, const char *detail); -``` - -Every call to a public `raft_*` function fires `on_trace` at entry and at each decision point: - -| Trace event | When fired | -|-------------|-----------| -| `election_start` | `raft_start_election` begins | -| `self_vote` | Self-vote recorded | -| `vote_request_in` | `raft_on_vote_request` called | -| `vote_granted` | Vote granted to candidate | -| `vote_denied_stale_term` | Denied: term < current | -| `vote_denied_already_voted` | Denied: voted_for already set | -| `vote_denied_role` | Denied: LEADER or COMMITTING | -| `vote_denied_freshness` | Denied: candidate not fresh | -| `vote_reply_in` | `raft_on_vote_reply` called | -| `quorum_reached` | Votes_granted >= quorum | -| `became_leader` | on_become_leader fired | -| `became_follower` | on_become_follower fired | -| `election_timeout` | `raft_on_election_timeout` called | -| `leader_lost` | `raft_on_leader_lost` called | -| `jitter_skip` | `raft_check_and_start_election` → SKIP | -| `term_update` | term incremented or bumped by peer | - -In production (`ha.c` adapter), `on_trace` routes to `INFO(...)` logging. -In tests, `on_trace` appends to a per-node trace buffer; test assertions can inspect the -event sequence to verify protocol correctness beyond role/term checks. - -### 2. State Snapshot (`raft_snapshot`) - -```c -/* raft_snapshot_t: point-in-time read of all Raft state for logging/debugging */ -typedef struct { - uint64_t term; - raft_role_t role; - int voted_for; - uint64_t voted_for_term; - int votes_granted; - int election_decided; -} raft_snapshot_t; - -/* Fills snap with current state. Thread-safe (caller must hold lock). */ -void raft_snapshot(const raft_state_t *s, raft_snapshot_t *snap); -``` - -The `raft_snapshot` function is used by: -- The simulator to print cluster state after each test step -- The adapter to log state on RDHUP for post-mortem debugging -- The ASAN test runner to dump state on test failure - -### 3. Simulator Trace Output - -`raft_sim_cluster_t` records a timestamped event log for each RPC and state transition. -After each test, `raft_sim_dump_log(cluster, FILE*)` prints the protocol trace in a -human-readable format, e.g.: - -``` -[step 0] node0: election_start term=2 -[step 1] node0→node1: vote_request term=2 tag=0 lsn=0 -[step 1] node0→node2: vote_request term=2 tag=0 lsn=0 -[step 2] node1: vote_granted to node0 term=2 -[step 2] node2: vote_granted to node0 term=2 -[step 3] node0: quorum_reached votes=3 → became_leader term=2 -``` - -This trace is printed to stderr on test failure and can be enabled unconditionally -with `RAFT_SIM_VERBOSE=1`. - -### File structure additions - -``` -src/lib/raft/ -├── raft.h # + on_trace, raft_snapshot_t, raft_snapshot() -├── raft.c # + trace emission at every decision point -└── raft_sim.c # + raft_sim_dump_log, per-node trace buffer -``` - ---- - -## Key Decisions - -1. **Zero OS dependencies.** `raft.h` and `raft.c` depend only on `stdint.h`, `string.h`, and `errno.h`. No `pthread.h`, no atomics, no OS primitives. The library is a pure function of its `raft_state_t` input — all threading, locking, and scheduling are the caller's responsibility. In the controller adapter, the caller holds `ha->lock` around every `raft_*` call. In the deterministic simulator tests, no lock is needed (single-threaded). This makes the library embeddable in any threading model without modification. - -2. **Self-vote recorded immediately in `raft_start_election`.** This prevents a concurrent node from granting its vote to two candidates (the root cause of observed split-brain). The self-vote is `voted_for = self_idx` at term increment time, before any RPC is sent. - -3. **Role enum is separate from `ctrl_role_t`.** The Raft roles (FOLLOWER/CANDIDATE/LEADER) are a subset of XCE's controller states (INIT/STANDBY/ELECTING/PRIMARY_COMMITTING/PRIMARY). The adapter maps between them; the library owns only Raft roles. - -4. **Priority-based jitter is part of the library via `raft_on_leader_lost`.** The jitter formula (`priority_idx * election_timeout_ms`) is in the library because it is a Raft-level correctness concern. The caller implements the actual sleep/timer. - -5. **Simulator routes callbacks synchronously.** For test determinism, `raft_sim` delivers vote RPCs synchronously (no goroutines, no nanosleep). Timing tests verify term-based invariants, not wall-clock timing. diff --git a/.kiro/specs/raft-library/requirements.md b/.kiro/specs/raft-library/requirements.md deleted file mode 100644 index 318893d4..00000000 --- a/.kiro/specs/raft-library/requirements.md +++ /dev/null @@ -1,97 +0,0 @@ -# Requirements Document - -## Introduction -XCE controller developers need a correct, independently-testable election consensus library. -The existing election logic in ha.c is tightly coupled to the iomgr/SSL/capnp transport -layer, preventing deterministic unit testing without a live cluster. This feature extracts -the pure Raft-style election state machine into a self-contained C library (libxce) with -callback interfaces, so election correctness can be verified with a deterministic simulator -and then wired into the controller via thin adapters. - -## Boundary Context -- **In scope**: Election state machine (term, voted_for, role, quorum evaluation), vote - request/reply processing, leader-lost failover trigger, callback interface definition, - deterministic simulator test harness. -- **Out of scope**: Network transport (SSL, iomgr, capnp), WAL truncation on failover, - config replication push, cluster_init_phase2, persistence to disk (remains in caller). -- **Adjacent expectations**: The library fires callbacks when it needs to send a vote RPC - or when role transitions occur; the controller adapter implements those callbacks using - the existing iomgr/SSL transport. Disk persistence of term/voted_for remains the caller's - responsibility via an on_persist callback. - -## Requirements - -### 1. Election State -**Objective:** As a controller developer, I want the election state machine to be self-contained, so that I can test Raft invariants without network infrastructure. - -#### Acceptance Criteria -1. The Raft Library shall maintain per-node state: current term (uint64_t), role (FOLLOWER/CANDIDATE/LEADER), voted_for candidate index (or -1), and n_peers count. -2. When initialized, the Raft Library shall set term=0, role=FOLLOWER, voted_for=-1. -3. The Raft Library shall expose no dependency on endpoint_t, SSL_CTX, iomgr_t, cluster_t, or capnp types in its public header. - -### 2. Election Start -**Objective:** As a controller developer, I want to start an election by calling a single function, so that the library handles the Raft protocol steps. - -#### Acceptance Criteria -1. When raft_start_election() is called, the Raft Library shall increment term, record self-vote (voted_for = self index), set role to CANDIDATE, and invoke the on_send_vote_request callback for each peer. -2. When raft_start_election() is called with n_peers == 1 (single-node), the Raft Library shall skip peer RPCs and immediately transition to LEADER via the on_become_leader callback. -3. If raft_start_election() is called while role is already CANDIDATE or LEADER, the Raft Library shall return an error without modifying state. - -### 3. Vote Request Handling -**Objective:** As a controller developer, I want incoming vote requests to be evaluated correctly, so that only one candidate wins per term. - -#### Acceptance Criteria -1. When raft_on_vote_request() is called with term > current_term, the Raft Library shall update current_term, clear voted_for, set role to FOLLOWER, and evaluate the request. -2. When raft_on_vote_request() is called with term < current_term, the Raft Library shall deny the vote and return the current term to the caller. -3. When raft_on_vote_request() is called with term == current_term and voted_for is already set to a different candidate, the Raft Library shall deny the vote (one vote per term invariant). -4. When raft_on_vote_request() is called and the candidate passes the freshness gate (candidate config_tag ≥ local config_tag AND candidate log LSN ≥ local applied LSN), the Raft Library shall grant the vote, record voted_for = candidate index, and invoke the on_persist callback. -5. While role is LEADER or CANDIDATE_COMMITTING, the Raft Library shall deny all incoming vote requests regardless of term. - -### 4. Vote Reply Handling and Quorum -**Objective:** As a controller developer, I want vote replies to be accumulated correctly so that the library detects quorum and reports the election result. - -#### Acceptance Criteria -1. When raft_on_vote_reply() is called with a granted vote and votes_granted reaches quorum (ceil((n_peers+1)/2)), the Raft Library shall set role to LEADER and invoke the on_become_leader callback exactly once. -2. When raft_on_vote_reply() is called with term > current_term, the Raft Library shall update current_term, set role to FOLLOWER, and invoke on_become_follower. -3. When raft_on_vote_reply() is called after the election is already decided (on_become_leader or on_become_follower already fired), the Raft Library shall be a no-op. -4. When all peers have replied and votes_granted < quorum, the Raft Library shall set role to FOLLOWER and invoke on_become_follower. - -### 5. Leader Lost (RDHUP Failover) -**Objective:** As a controller developer, I want to notify the library when the primary control channel is lost, so that a new election is triggered correctly. - -#### Acceptance Criteria -1. When raft_on_leader_lost() is called, the Raft Library shall clear voted_for, set role to FOLLOWER, and invoke the on_start_election_after_delay callback with the priority-based delay (priority_index * election_timeout_ms). -2. When the delay elapses and voted_for is set to a non-self candidate for the new term (another node won during the delay), the Raft Library shall skip starting its own election. -3. When the delay elapses and voted_for is unset or set to self, the Raft Library shall invoke raft_start_election() internally. - -### 6. Election Timeout -**Objective:** As a controller developer, I want an election timeout to fire if no leader is elected within the configured window, so that the cluster does not stall. - -#### Acceptance Criteria -1. When raft_start_election() is called, the Raft Library shall invoke the on_schedule_timeout callback with election_timeout_ms. -2. When the timeout fires (caller invokes raft_on_election_timeout()) and the election is not yet decided, the Raft Library shall invoke on_become_follower and cancel any pending vote tracking. -3. When the timeout fires and the election is already decided, the Raft Library shall invoke on_cancel_timeout and be a no-op. - -### 7. Callback Interface -**Objective:** As a controller developer, I want the library to communicate all side effects through callbacks, so that I can inject a deterministic test double without network I/O. - -#### Acceptance Criteria -1. The Raft Library shall define a raft_callbacks_t struct with the following function pointer fields: on_send_vote_request, on_become_leader, on_become_follower, on_persist (term/voted_for), on_schedule_timeout, on_cancel_timeout. -2. The Raft Library shall accept a user-supplied void* context pointer at init time and pass it unchanged to every callback invocation. -3. If any required callback pointer is NULL at raft_init() time, the Raft Library shall return a non-zero error code. - -### 8. Deterministic Test Harness -**Objective:** As a controller developer, I want a simulator that exercises the Raft Library without network I/O, so that I can verify correctness under controlled timing. - -#### Acceptance Criteria -1. The Raft Library test suite shall include a deterministic N-node simulator that routes on_send_vote_request callbacks to the target node's raft_on_vote_request/reply functions synchronously, without real network I/O. -2. The test suite shall verify the following invariants: (a) exactly one LEADER per election in a 3-node cluster with no failures (no split-brain), (b) a second vote request in the same term is denied, (c) a single-node cluster self-elects immediately, (d) election timeout causes re-election. -3. The Raft Library shall compile and all tests shall pass under ASAN with ASAN_OPTIONS=detect_leaks=0. - -### 9. Integration with Controller -**Objective:** As a controller developer, I want a thin adapter in ha.c that connects the Raft Library to the existing iomgr/SSL transport, so that ha.c no longer owns election state directly. - -#### Acceptance Criteria -1. The adapter shall implement all raft_callbacks_t callbacks using existing endpoint_t and iomgr transport without duplicating election state management. -2. After integration, ha_start_election_async shall delegate term management, voted_for tracking, and quorum evaluation to the Raft Library; it shall no longer directly write ha->term or ha->voted_for fields outside of the adapter callbacks. -3. The existing XCE_TESTING hook seams (ha_phase2_hook, ha_rdhup_election_hook) shall remain functional after integration so that existing test cases continue to pass. diff --git a/.kiro/specs/raft-library/research.md b/.kiro/specs/raft-library/research.md deleted file mode 100644 index 2c9634c5..00000000 --- a/.kiro/specs/raft-library/research.md +++ /dev/null @@ -1,296 +0,0 @@ -# Research & Design Decisions: raft-library - -## Discovery Scope -- Type: Complex new library (brownfield extraction from ha.c) -- Process: Full discovery — codebase analysis of ha.c, ha.h, test files - -## Key Findings - -### Existing Election State (ha.c) -- ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32) -- ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore -- Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new() -- Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys() - -### Root Cause of Split-Brain -- Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived -- This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request -- Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC - -### Build vs. Adopt Decision -- No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative -- Decision: build a minimal Raft election library in src/lib/raft/ -- Scope is election only (not log replication), which is simpler than full Raft - -### Synthesis Outcomes -1. **Generalization**: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases. -2. **Build vs. Adopt**: Build; no suitable C library exists. -3. **Simplification**: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING. - -### Priority Jitter Design -- The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport -- Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback -- Caller implements the actual timer/sleep - -### Tracing Decision -- Added on_trace callback to raft_callbacks_t for full protocol observability -- Simulator dump log enables post-mortem analysis of split-brain scenarios -- raft_snapshot() provides safe point-in-time state read for crash/failure logging - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **raft-library** is **~100% complete**. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are `[x]` in tasks.md. The `ready_for_implementation` flag is set in spec.json. -- **No remaining gaps**: The library (`src/lib/raft/raft.h`, `raft.c`, `raft_sim.c`) exposes `raft_state_t` + `raft_callbacks_t` with 9 callback hooks. The deterministic simulator (`raft_sim.c`) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library. -- **Cross-spec integration**: `election-iomgr-refactor` uses `raft_start_election` instead of the old `am_i_primary_quorum`. The adapter callbacks in ha.c wire `on_send_vote_request` → iomgr async connect, `on_become_leader` → `cluster_init_phase2` + config push, `on_become_follower` → `CTRL_STANDBY`. The `on_vote_eligible` callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (`CTRL_PRIMARY_COMMITTING` → deny). -- **Test coverage**: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN. -- **Recommendation**: The spec can be marked `implemented`. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -> Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree -> (uncommitted changes to `src/lib/raft/raft.c`, `src/controller/ha.c`, `ha.h`, `myapp.c` analyzed as-is). -> spec.json phase: `tasks-generated`, all tasks.md checkboxes `[x]` except parent rollups 2/3/4/5/6. - -## 1. Current State Summary - -- Library exists and is build-integrated: `src/lib/raft/raft.h` (276 lines), `raft.c` (290 lines), - `raft_sim.c` (207 lines); wired in `src/lib/meson.build` (raft.c at line 108; test_raft.c + raft_sim.c - under the test executable at lines 175–176, simulator guarded by `XCE_TESTING`). -- Test suite: `src/lib/testsuite/test_raft.c` (2990 lines, 47 test functions, 79 RUN_TEST invocations) - covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios. -- Controller adapter live: `ha_state_t` embeds `raft_state_t raft` (`ha.h:183`); 9 callbacks implemented - in `ha.c` (`s_real_cbs` at ha.c:735); `raft_init` called in ha init (ha.c:854) and re-initialized with - real topology by `ha_raft_reinit` (ha.c:~2981–3008). -- Production dispatch fully delegated: vote requests → `raft_on_vote_request` (cluster.c:373), - vote replies → `raft_on_vote_reply` (ha.c:2361), RDHUP → `raft_on_leader_lost` (ha.c:3290), - election start → `raft_start_election` (ha.c:3146). - -## 2. Requirement-to-Asset Map - -| Req | Criterion | Asset(s) | Status | -|---|---|---|---| -| 1.1 | Per-node state (term/role/voted_for/n_peers) | `raft_state_t` (raft.h:23–38) | Implemented | -| 1.2 | Init defaults term=0/FOLLOWER/-1 | `raft_init` (raft.c:13–55); H2 test | Implemented (extended: restores `initial_term`/`initial_voted_for` from config) | -| 1.3 | No endpoint_t/SSL/iomgr/cluster_t/capnp in header | raft.h includes only errno/stdint/string (+stdio under XCE_TESTING) | Implemented | -| 2.1 | start_election: term++, self-vote, CANDIDATE, send RPCs | raft.c:57–104 | Implemented (also persists self-vote before RPCs — stronger than spec) | -| 2.2 | Single-node immediate LEADER | raft.c:83–88; tests A1, single_node | Implemented | -| 2.3 | -EALREADY if CANDIDATE/LEADER | raft.c:60–61; tests already_candidate/leader | Implemented | -| 3.1 | term > current: upgrade, clear vote, FOLLOWER, evaluate | raft.c:121–130; B2 | Implemented (extra: fires `on_become_follower` if was LEADER — split-brain fix, uncommitted change) | -| 3.2 | term < current: deny, return current term | raft.c:110–115; B3 | Implemented | -| 3.3 | One vote per term | raft.c:157–164; B1/B4/B5 | Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3) | -| 3.4 | Freshness gate (config_tag/LSN) before grant | Split: caller-side gate in cluster.c:352–356 + `on_vote_eligible` hook (raft.c:167–173) | Partial — behavior present but located in the dispatcher, not in `on_vote_eligible` as tasks.md 6.2 states; `raft_cb_vote_eligible` (ha.c:202) intentionally skips freshness. Doc/code drift. | -| 3.5 | LEADER or COMMITTING deny ALL vote requests regardless of term | raft.c:135–148 (same-term deny only); COMMITTING gate in `on_vote_eligible` adapter | Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated. | -| 4.1 | Quorum (n/2)+1 → LEADER, on_become_leader once | raft.c:222–234; D1, A4 | Implemented | -| 4.2 | Reply term > current → FOLLOWER + on_become_follower | raft.c:197–206; D4 | Implemented | -| 4.3 | Reply after decided → no-op | raft.c:209–213; D3 | Implemented | -| 4.4 | All replied, no quorum → FOLLOWER | raft.c:237–244; D2 | Implemented | -| 5.1 | leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout) | raft.c:268–276; F1 | Implemented | -| 5.2 | Skip election if voted for non-self during delay | `raft_check_and_start_election` raft.c:280–284; F2 | Implemented | -| 5.3 | Else start election internally | raft.c:286–289; F3 | Implemented | -| 6.1 | start_election schedules timeout | raft.c:101; E3 | Implemented | -| 6.2 | Timeout undecided → on_become_follower, cancel tracking | raft.c:260–265; E1 | Implemented | -| 6.3 | Timeout after decided → on_cancel_timeout no-op | raft.c:252–257; E2 | Implemented | -| 7.1 | raft_callbacks_t with 6 required fields | raft.h:66–96 (9 fields: 7 required + 2 optional) | Implemented (superset: adds `on_start_election_after_delay` as required, `on_vote_eligible`/`on_trace` optional) | -| 7.2 | void* ctx passed to every callback | raft_config_t.ctx → s->ctx, all call sites | Implemented | -| 7.3 | NULL required callback → error at init | raft.c:23–38 (-EINVAL); Hnull test | Implemented | -| 8.1 | Deterministic N-node simulator, synchronous routing | raft_sim.c + raft.h:197–274; sim tests | Implemented | -| 8.2 | Invariants: one-leader/3-node, dup deny, single-node, timeout re-election | A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition | Implemented | -| 8.3 | Compile + all tests pass under ASAN | tasks.md task 7 `[x]`; commits e10cf9a/361397a fixed ASAN issues | Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run | -| 9.1 | Adapter implements all callbacks via endpoint_t/iomgr | ha.c:107–744 (8 raft_cb_* + s_real_cbs) | Implemented | -| 9.2 | ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter | ha.c:3038–3161 delegates; only atomic loads + persist/restore remain | Partial — production path clean, but legacy `handle_request_vote()` (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct `ha->voted_for_term`/`voted_for_name` writes. It is production-dead (only `test_ha_vote_gate.c` calls it) but duplicates election state management contrary to the spirit of 9.2. | -| 9.3 | ha_phase2_hook / ha_rdhup_election_hook still functional | ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup) | Implemented | - -**Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).** - -### Constraints / behavior added beyond requirements -- **C1 (Constraint)**: `raft_init` rejects even `n_peers > 1` with -EINVAL (raft.c:21–22). Not in - requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches - deployment expectations (`deploy-ha.sh`, cluster config validation) or it becomes a runtime landmine. -- **C2 (Constraint)**: `voted_for_term` is unconditionally reset to 0 in `raft_init` (raft.c:43) even when - `initial_voted_for`/`initial_term` are restored from disk. After a restart, the duplicate-vote check - (`voted_for_term == term`, raft.c:157) cannot match the restored term, so a node that voted for X in - term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could - grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks `term`/`voted_for` - but not `voted_for_term`. Potential one-vote-per-term violation across restart — needs confirmation. - -## 3. Implementation Approach Options (for the remaining gaps) - -### Option A: Extend existing components (close gaps in place) — RECOMMENDED -- raft.c: initialize `voted_for_term = (initial_voted_for >= 0) ? initial_term : 0` in `raft_init` (C2); - add an H6 restart-double-vote test. -- requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate - enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text. -- ha.c: delete or fold legacy `handle_request_vote()` into the raft path; retarget `test_ha_vote_gate.c` - at `raft_on_vote_request` + `on_vote_eligible` (closes 9.2). -- Re-run `/xce-build lib` + `/xce-build controller` under ASAN to refresh 8.3. -- Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a - legacy test file; requirements edit needs human re-approval per workflow. - -### Option B: Create new components (formal vote-record module) -- Introduce a `raft_persisted_vote_t {term, voted_for}` pair persisted/restored atomically, replacing the - separate initial_term/initial_voted_for fields; new restart-focused test group. -- Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in - raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI). - -### Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up - cleanup spec/task since it is production-dead. -- Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication - in tree longer; two passes of review. - -## 4. Effort & Risk - -| Gap | Effort | Risk | Justification | -|---|---|---|---| -| C2 voted_for_term restore fix + test | S | Low | One-line init change + one Unity test in an established harness | -| Requirements/tasks doc sync (3.4, 3.5) | S | Low | Documentation only; needs approval round-trip | -| Legacy handle_request_vote removal (9.2) | S–M | Medium | Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline | -| ASAN re-verification (8.3) | S | Low | Mechanical `/xce-build` run; fixes only if new findings | -| C1 even-n_peers policy confirmation | S | Medium | If 2-node deployments are expected, this is a hard init failure discovered only at runtime | - -Overall remaining work: **S–M total, Low–Medium risk** — the library and integration are functionally complete. - -## 5. Recommendations for Design/Closure Phase - -1. Adopt **Option A**. Fix C2 (`voted_for_term` restore) first — it is the only candidate correctness - gap; everything else is documentation or hygiene. -2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on - higher term; freshness gate enforced caller-side before `raft_on_vote_request` with COMMITTING/liveness - in `on_vote_eligible`. Re-approve, then run `/kiro-verify-completion raft-library`. -3. Remove or refactor legacy `handle_request_vote` (ha.c:1791) to eliminate the last duplicated election - state path (Req 9.2), retargeting `test_ha_vote_gate.c`. -4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state - predates the uncommitted raft.c Step-3/Step-4 rewrite). - -## Research Needed - -- **R1**: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment? - If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the - failure surfaces at startup with a clear message rather than as a silent raft_init failure. -- **R2**: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does - `ha_raft_reinit` run before the listener accepts vote requests? If unreachable, document why; fix anyway. -- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code - (raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an - inbound vote request flips role between RDHUP and the delayed check). - ---- - -## Gap Re-Audit (2026-06-18) - -Post-implementation audit. Spec phase = implemented, tasks 24/24 [x]. This pass re-verifies every -EARS requirement against the working tree and reconciles the prior audit's open items (C2, R1–R3). -Assets are tagged: **[CLOSED]** verified met in code, **[CLOSED-AS-BUILT]** met but behavior differs -from literal req text (improvement), **[RESIDUAL]** doc/hygiene gap, **[SCOPE-DRIFT]** code beyond spec. - -### Requirement-to-Asset Map - -| Req | Acceptance | Asset (file:line) | Status | -|-----|-----------|-------------------|--------| -| 1.1 | per-node state fields | `raft.h:28-50` raft_state_t (term/role/voted_for/voted_for_term/n_peers) | CLOSED | -| 1.2 | init term=0 role=FOLLOWER voted_for=-1 | `raft.c:40-56` raft_init; defaults via cfg {0,-1} | CLOSED | -| 1.3 | no transport types in header | `raft.h:4-6` includes only errno/stdint/string; no endpoint_t/SSL_CTX/cluster_t | CLOSED | -| 2.1 | election: incr term + self-vote + send RPCs | `raft.c:68-71,98-102` | CLOSED | -| 2.2 | single-node → immediate LEADER | `raft.c:87-92` returns RAFT_ELECTED_IMMEDIATELY | CLOSED | -| 2.3 | already CANDIDATE/LEADER → error, no state change | `raft.c:64-65` -EALREADY | CLOSED | -| 3.1 | term>cur → update,clear vote,FOLLOWER,evaluate | `raft.c:125-137` (also steps down leader via on_become_follower) | CLOSED-AS-BUILT | -| 3.2 | term<cur → deny + return cur term | `raft.c:114-119` | CLOSED | -| 3.3 | same term, diff voted_for → deny | `raft.c:161-168` (gated on voted_for_term==term) | CLOSED | -| 3.4 | freshness gate pass → grant + persist | `raft.c:170-190`; freshness split: tag/lsn floor enforced caller-side (cluster.c:363), COMMITTING/liveness in on_vote_eligible (`ha.c:299` raft_cb_vote_eligible) | CLOSED-AS-BUILT | -| 3.5 | LEADER/COMMITTING → deny all | `raft.c:148-152` LEADER deny; COMMITTING enum removed (ha-single-leader-role 93dfc02), now an on_vote_eligible policy | CLOSED-AS-BUILT | -| 4.1 | quorum ceil((n+1)/2) → LEADER once | `raft.c:372-383`, quorum=(n_peers/2)+1 == ceil for odd n | CLOSED | -| 4.2 | reply term>cur → FOLLOWER + on_become_follower | `raft.c:346-355` | CLOSED | -| 4.3 | post-decision no-op | `raft.c:358-362` election_decided guard → RAFT_PENDING | CLOSED | -| 4.4 | all replied, no quorum → FOLLOWER | `raft.c:386-393` replies_received==n_peers-1 | CLOSED | -| 5.1 | leader-lost → clear vote + delay(priority*timeout) | `raft.c:417-425` | CLOSED | -| 5.2 | voted non-self during delay → skip | `raft.c:429-433` RAFT_SKIP_ELECTION | CLOSED | -| 5.3 | not voted / self → start election | `raft.c:435-438` RAFT_ELECTION_STARTED | CLOSED | -| 6.1 | start election → on_schedule_timeout | `raft.c:105` | CLOSED | -| 6.2 | timeout undecided → on_become_follower + cancel | `raft.c:409-414` | CLOSED | -| 6.3 | timeout decided → on_cancel_timeout, no-op | `raft.c:401-407` | CLOSED | -| 7.1 | raft_callbacks_t with required fn ptrs | `raft.h:79-117` (9 fields incl. on_send_pre_vote_request) | CLOSED | -| 7.2 | void* ctx passed unchanged | `raft.h` ctx threaded; `raft.c` passes s->ctx everywhere | CLOSED | -| 7.3 | NULL required callback → non-zero | `raft.c:23-38` 7 required-cb checks → -EINVAL | CLOSED | -| 8.1 | deterministic N-node simulator | `raft_sim.c`, `raft.h:264-342` (XCE_TESTING) | CLOSED | -| 8.2 | 4 invariant cases (a-d) | `test_raft.c`: A2 one-leader, B1/B4 deny-second, A1 single-node, E*/sim timeout | CLOSED | -| 8.3 | compiles + tests pass under ASAN (leaks=0) | meson `raft/raft.c` (l.109) + `test_raft.c` (l.179); 209 RUN_TEST/test fns | CLOSED (re-run, see V1) | -| 9.1 | adapter implements all callbacks, no dup state | `ha.c:1309-1320` s_real_cbs wires all 10; production state in raft_state_t | CLOSED | -| 9.2 | ha delegates term/voted_for/quorum to library | `ha.c:4382-4522` ha_start_election_async→raft_start_election; `cluster.c:418` vote req; `ha.c:3237` vote reply | CLOSED-AS-BUILT | -| 9.3 | XCE_TESTING hooks still functional | `ha_rdhup_election_hook` (ha.c:4528), ha_phase2 seams retained | CLOSED | - -**Score: 24/24 acceptance-criteria groups CLOSED.** No requirement is unimplemented. 5 are CLOSED-AS-BUILT -(real code is a strict superset/improvement over literal req text — see residual R-DOC). - -### Reconciliation of Prior Audit Open Items - -- **C2 (voted_for_term restore double-vote) — NOW FIXED.** `raft.c:47` sets - `voted_for_term = (initial_voted_for != -1) ? initial_term : 0` at init, closing the post-restart - second-vote window. Regression test `test_proto_H6_restored_vote_blocks_second_vote` (test_raft.c:2100, - registered l.3978) guards it. Prior research's "only candidate correctness gap" is resolved (commit 9821ec4). -- **R3 / Req 9.2 dual vote path — DOWNGRADED.** Legacy `handle_request_vote` (ha.c:2608) is now - `#ifdef XCE_TESTING`-only — the production vote path is exclusively `cluster.c:418 → raft_on_vote_request`. - The duplicated election-state logic no longer runs in production; it remains only as a test fixture for - `test_ha_vote_gate.c` / `test_ha_election.c`. No production split-brain risk from the dual path. Residual: - the test-only copy can drift from raft.c semantics over time (R-DUP below). -- **R1 (even-n_peers policy) — STILL OPEN.** `raft_init` rejects even n_peers>1 with `-EINVAL` - (raft.c:21-22), test `test_raft_init_even_peers_returns_einval`. Not stated in requirements; a 2-node HA - deploy would silently fail raft_init. Needs requirements note + startup-time config validation. - -### Residual Gaps (non-blocking) - -- **R-DOC (Low):** requirements.md 3.4/3.5 still describe COMMITTING as an in-library role gate and the - freshness gate as in-library; as-built, COMMITTING enum is removed and freshness is split caller-side + - on_vote_eligible. Requirements text lags the (better) implementation. Update for traceability. -- **R-DUP (Low):** test-only `handle_request_vote` duplicates vote-gate logic; risk of semantic drift from - raft.c. Consider retargeting `test_ha_vote_gate.c` onto `raft_on_vote_request` directly. -- **R1 (Medium):** even-n_peers rejection undocumented; surfaces as silent raft_init -EINVAL rather than a - clear startup error. Add config-load validation OR a requirement clause. -- **R-SCOPE / SCOPE-DRIFT (informational, not a gap):** PreVote (raft_begin_pre_vote, - raft_on_pre_vote_request, raft_on_pre_vote_reply, on_send_pre_vote_request, RAFT_PRE_VOTE_* codes; - raft.c:193-336, raft.h:160-259) lives in the raft-library files but belongs to the separate **ha-prevote** - spec (commits 449b8e9–e6b234a). It is purely additive, mutates no persistent Raft state, and does not - violate any raft-library requirement. Flag only so the raft-library boundary doc reflects the co-tenancy: - raft.h/raft.c now host two specs' surfaces. - -### Integration Risks - -- **IR-1 (Low):** PreVote tally arrays are bounded by `RAFT_MAX_PEERS=8` (raft.h:18); `raft_on_pre_vote_reply` - range-guards (raft.c:287). Real-vote path has no equivalent fixed array (uses scalar counters) so it is - unbounded-safe. Mismatch is intentional but worth a one-line note in design. -- **IR-2 (Low):** `raft_check_and_start_election` discards `raft_start_election` return (raft.c:436-437); - comment claims -EALREADY unreachable because raft_on_leader_lost forces FOLLOWER. Valid for the current - adapter, but an inbound vote/AppendEntries that flips role between RDHUP and the delayed fire could make it - reachable. Low likelihood; assert instead of (void) for defense-in-depth. - -### Open Work — Option Analysis - -**Option A — Doc-only closure (recommended).** Update requirements.md 3.4/3.5/9.2 to as-built; add R1 -even-n_peers clause; note PreVote co-tenancy + IR-1/IR-2 in design.md. Then `/kiro-verify-completion`. -- Effort: **S** (~1-2h, docs only). Risk: **Low**. No code change, no regression surface. - -**Option B — Doc closure + hygiene.** Option A plus: (1) startup config validation for even n_peers with a -loud error, (2) retarget test-only `handle_request_vote` onto raft_on_vote_request (kill R-DUP), (3) convert -IR-2 `(void)rc` to an assert. -- Effort: **M** (~half day). Risk: **Low-Medium** (touches ha.c test seam + config load; rerun full - controller + lib test suites). - -**Option C — Defer, accept as-is.** Ship implemented; track R-DOC/R-DUP/R1 as backlog. The only real -correctness item (C2) is already fixed and tested, so functional risk of deferring is minimal. -- Effort: **none**. Risk: **Low** functionally; **Medium** for spec/code traceability drift over time. - -### Validation To Re-Run - -- **V1:** `/xce-build lib` and `/xce-build controller` with `ASAN_OPTIONS=detect_leaks=0` — no build dir - present in the worktree at audit time; the test_raft.c suite (209 cases) and controller tests should be - re-run to confirm green on the current tree (which now includes the PreVote additions + C2 fix, post-dating - the last recorded task-7 ASAN run). - -**Bottom line:** raft-library is functionally complete — 24/24 met, the sole prior correctness gap (C2) is -fixed and regression-tested, and the production dual-vote-path risk is eliminated (legacy path is test-only). -Remaining work is documentation/hygiene (Option A, S effort, Low risk) plus a clean ASAN re-run (V1). diff --git a/.kiro/specs/raft-library/spec.json b/.kiro/specs/raft-library/spec.json deleted file mode 100644 index 9e43cd9a..00000000 --- a/.kiro/specs/raft-library/spec.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "feature_name": "raft-library", - "created_at": "2026-05-31T04:00:00Z", - "updated_at": "2026-05-31T04:45:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "bug-339 (voted_for_term restore) fixed 2026-06-09. handle_request_vote guarded under XCE_TESTING \u2014 production path is cluster.c \u2192 raft_on_vote_request. Controller suite green. Req 3.4/3.5 text reflects current behavior (freshness gate in cluster.c dispatcher, higher-term leader step-down)." -} diff --git a/.kiro/specs/raft-library/tasks.md b/.kiro/specs/raft-library/tasks.md deleted file mode 100644 index 97233927..00000000 --- a/.kiro/specs/raft-library/tasks.md +++ /dev/null @@ -1,185 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: build system and file scaffold - - Create `src/lib/raft/` directory with placeholder files: `raft.h`, `raft.c`, `raft_sim.c` - - Add `static_library('raft', 'raft/raft.c')` to `src/lib/meson.build` and link it into `xcelib` - - Add `testsuite/test_raft.c` to the xcelib test executable block; guard `raft_sim.c` compilation under `XCE_TESTING` define - - `/xce-build lib` compiles to completion without link errors - - _Requirements: 8.3_ - -- [x] 2. Public API definition (raft.h) -- [x] 2.1 Define core data structures: raft_role_t, raft_state_t, raft_snapshot_t - - `raft_role_t` enum: `FOLLOWER=0`, `CANDIDATE=1`, `LEADER=2` - - `raft_state_t` struct: `term`, `role`, `voted_for`, `voted_for_term`, `n_peers`, `self_idx`, `votes_granted`, `replies_received`, `election_decided`, `election_timeout_ms`, `my_priority` - - `raft_snapshot_t`: point-in-time read of `term`, `role`, `voted_for`, `voted_for_term`, `votes_granted`, `election_decided`; `raft_snapshot()` signature declared - - Header includes only `stdint.h`, `string.h`, `errno.h` — no `iomgr_t`, `endpoint_t`, `SSL_CTX`, `cluster_t`, or capnp types visible anywhere - - _Requirements: 1.1, 1.3_ - -- [x] 2.2 Define callback interface (raft_callbacks_t, raft_config_t) and all public function signatures - - `raft_callbacks_t` with 9 function pointers: `on_send_vote_request`, `on_vote_eligible` (optional/NULL-safe), `on_become_leader`, `on_become_follower`, `on_persist`, `on_schedule_timeout`, `on_cancel_timeout`, `on_start_election_after_delay`, `on_trace` (optional/NULL-safe) - - `raft_config_t` struct: `n_peers`, `self_idx`, `election_timeout_ms` (0 → 2000ms default), `my_priority`, `initial_term`, `initial_voted_for`, `cbs`, `ctx` - - All public function signatures: `raft_init`, `raft_start_election`, `raft_on_vote_request`, `raft_on_vote_reply`, `raft_on_election_timeout`, `raft_on_leader_lost`, `raft_check_and_start_election`, `raft_snapshot` - - Return code defines: `RAFT_OK`, `RAFT_GRANT`, `RAFT_DENY`, `RAFT_BECAME_LEADER`, `RAFT_BECAME_FOLLOWER`, `RAFT_PENDING`, `RAFT_ELECTED_IMMEDIATELY`, `RAFT_SKIP_ELECTION`, `RAFT_ELECTION_STARTED` - - Header compiles cleanly as a standalone include in a file with no other XCE headers - - _Requirements: 1.2, 7.1, 7.2, 7.3_ - -- [x] 3. Core state machine implementation (raft.c) -- [x] 3.1 Implement raft_init and raft_snapshot - - `raft_init` validates all required callback pointers are non-NULL (all except `on_vote_eligible` and `on_trace`); returns `-EINVAL` without modifying state if any required callback is NULL or `n_peers < 1` - - Initializes from `raft_config_t`: copies `n_peers`, `self_idx`, `election_timeout_ms` (apply 2000ms default if 0), `my_priority`, `initial_term`, `initial_voted_for`; sets `role=FOLLOWER`, `votes_granted=0`, `replies_received=0`, `election_decided=0` - - `raft_snapshot()` fills snap with current field values (called while caller holds lock) - - `raft_init` with a NULL required callback returns `-EINVAL`; subsequent `raft_start_election` is not callable - - _Requirements: 1.2, 7.2, 7.3_ - -- [x] 3.2 Implement raft_start_election - - Guard: returns `-EALREADY` if `role` is `RAFT_CANDIDATE` or `RAFT_LEADER` without modifying state - - Increments `term`, sets `role=RAFT_CANDIDATE`, records `voted_for=self_idx`, `voted_for_term=term`, resets `votes_granted=1`, `replies_received=0`, `election_decided=0` - - Fires `on_persist(ctx, new_term, self_idx)` for self-vote durability before sending any RPC - - Single-node shortcircuit: if `n_peers==1`, sets `role=RAFT_LEADER`, `election_decided=1`, fires `on_become_leader(ctx, term)`, returns `RAFT_ELECTED_IMMEDIATELY` (no RPCs, no timeout) - - For `n_peers>1`: fires `on_send_vote_request(ctx, peer_idx, term)` for each peer index ≠ `self_idx`, then fires `on_schedule_timeout(ctx, election_timeout_ms)` - - Fires `on_trace` events: `election_start`, `self_vote`, `term_update` if NULL-safe checks pass - - _Requirements: 2.1, 2.2, 2.3, 6.1_ - -- [x] 3.3 Implement raft_on_vote_request (incoming RequestVote processing) - - Stale term: `term < current_term` → set `*reply_term`, fire `on_trace(vote_denied_stale_term)`, return `RAFT_DENY` - - Term upgrade: `term > current_term` → update `current_term=term`, clear `voted_for=-1`, set `role=RAFT_FOLLOWER`, fire `on_trace(term_update)` - - Role gate: if `role == RAFT_LEADER` → return `RAFT_DENY` (fire `on_trace(vote_denied_role)`); note: COMMITTING state is delegated to `on_vote_eligible` in the adapter - - Duplicate deny: if `voted_for != -1` AND `voted_for != from_idx` AND `voted_for_term == term` → return `RAFT_DENY` (fire `on_trace(vote_denied_already_voted)`) - - Policy gate: if `on_vote_eligible` is non-NULL, call it; if it returns 0 → return `RAFT_DENY` (fire `on_trace(vote_denied_freshness)`); if NULL, treat as eligible - - Grant: fire `on_persist(ctx, term, from_idx)`; if non-zero, return `RAFT_DENY` without updating `voted_for`; on success, set `voted_for=from_idx`, `voted_for_term=term`, fire `on_trace(vote_granted)`, return `RAFT_GRANT` - - Sets `*reply_term = current_term` before every return path - - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_ - -- [x] 3.4 Implement raft_on_vote_reply (quorum tracking and election decision) - - Term upgrade: if `reply_term > current_term`, update `current_term`, set `role=RAFT_FOLLOWER`, `election_decided=1`, fire `on_become_follower(ctx, current_term, -1)`, `on_cancel_timeout`, `on_trace(became_follower)`, return `RAFT_BECAME_FOLLOWER` - - No-op guard: if `election_decided`, fire `on_trace(vote_reply_noop)`, return `RAFT_PENDING` - - Increment `replies_received` on every call past the guard (tracks total peer replies) - - Grant accumulation: if `granted`, increment `votes_granted`; quorum = `(n_peers / 2) + 1` - - Quorum reached: set `role=RAFT_LEADER`, `election_decided=1`, fire `on_become_leader(ctx, term)`, `on_cancel_timeout`, `on_trace(quorum_reached)`, `on_trace(became_leader)`, return `RAFT_BECAME_LEADER` - - All-replied-no-quorum: if `replies_received == n_peers - 1` and `votes_granted < quorum`, set `role=RAFT_FOLLOWER`, `election_decided=1`, fire `on_become_follower(ctx, term, -1)`, `on_trace(became_follower)`, return `RAFT_BECAME_FOLLOWER` - - _Requirements: 4.1, 4.2, 4.3, 4.4_ - -- [x] 3.5 Implement raft_on_election_timeout, raft_on_leader_lost, raft_check_and_start_election - - `raft_on_election_timeout`: if `election_decided`, fire `on_cancel_timeout`, `on_trace(election_timeout_noop)`, return; otherwise set `role=RAFT_FOLLOWER`, `election_decided=1`, fire `on_become_follower(ctx, term, -1)`, `on_cancel_timeout`, `on_trace(election_timeout)` - - `raft_on_leader_lost`: clear `voted_for=-1`, set `role=RAFT_FOLLOWER`, fire `on_start_election_after_delay(ctx, my_priority * election_timeout_ms)`, fire `on_trace(leader_lost)` - - `raft_check_and_start_election`: if `voted_for != -1` AND `voted_for != self_idx` (another node won during the delay), fire `on_trace(jitter_skip)`, return `RAFT_SKIP_ELECTION`; otherwise call `raft_start_election(s)` internally, return `RAFT_ELECTION_STARTED` - - After `raft_on_leader_lost`, calling `raft_check_and_start_election` with no intervening vote grant starts a new election and returns `RAFT_ELECTION_STARTED` - - _Requirements: 5.1, 5.2, 5.3, 6.2, 6.3_ - -- [x] 4. (P) Deterministic N-node simulator (raft_sim.c) - _Depends: 3_ -- [x] 4.1 Implement simulator data structures and initialization - - `raft_sim_node_t`: contains `raft_state_t state`, `raft_callbacks_t cbs`, `void *ctx` (points back to cluster), `int idx`, `int partition`, per-node event log buffer for `on_trace` capture - - `raft_sim_cluster_t`: array of nodes up to `RAFT_SIM_MAX_NODES`, node count `n`, `leader_count` (incremented by each `on_become_leader` firing) - - `raft_sim_init(cluster, n_nodes, election_timeout_ms)`: initializes `n` nodes with indices 0..n-1, priority=idx, wires `on_send_vote_request` to buffer the RPC for routing, wires `on_become_leader` to increment `leader_count` - - `raft_sim_dump_log(cluster, FILE*)`: prints `[step N] nodeX: event` lines; `RAFT_SIM_VERBOSE=1` enables unconditional output; prints to stderr on Unity test failure - - `raft_sim_init(c, 3, 2000)` completes without error; all 3 nodes have `role=FOLLOWER`, `term=0` - - _Requirements: 8.1_ - - _Boundary: raft_sim.c_ - -- [x] 4.2 Implement simulator network operations and test driving - - `raft_sim_deliver_pending(cluster)`: routes all buffered RPCs synchronously — for each buffered `(src, dst, term)` entry, calls `raft_on_vote_request` on `dst` then routes the reply back to `src` via `raft_on_vote_reply`; skips delivery if `dst` has `partition=1` - - `raft_sim_start_all_elections(cluster)`: calls `raft_start_election` on each node in order 0..n-1 - - `raft_sim_fire_timeout(cluster, node_idx)`: calls `raft_on_election_timeout` on the specified node - - `raft_sim_partition(cluster, node_idx, isolated)`: sets/clears `partition` flag; subsequent `raft_sim_deliver_pending` drops RPCs to/from that node - - Running `raft_sim_start_all_elections` + `raft_sim_deliver_pending` on a 3-node cluster yields `leader_count == 1` - - _Requirements: 8.1, 8.2_ - - _Boundary: raft_sim.c_ - -- [x] 5. Protocol-level test suite (test_raft.c) - _Depends: 4_ -- [x] 5.1 Test Groups A and B: basic election correctness and voted_for invariant (10 cases) - - A1: single-node cluster → LEADER immediately; `on_become_leader` fires once; no `on_send_vote_request` calls - - A2: 3-node simultaneous election → `leader_count == 1` after `raft_sim_deliver_pending` - - A3: node 0 wins, node 1 starts late election same term → node 1 denied; total `leader_count` stays 1 - - A4: 5-node election → leader wins after 3 grants (quorum = 3); remaining 2 become FOLLOWER - - A5: vote request arrives at a LEADER node with any term → denied; role stays `RAFT_LEADER` - - B1: two candidates request vote from node C same term → only first requester granted - - B2: vote request `term > current_term` → node updates term, clears `voted_for`, re-evaluates; test confirms `state.term` updated - - B3: vote request `term < current_term` → denied immediately; `*reply_term` equals node's current term - - B4: node voted for X; receives request from Y same term → denied; `voted_for` stays X - - B5: node voted for X; receives request from X again same term → re-granted (idempotent) - - All 10 Unity test cases pass; `raft_sim_dump_log` confirms correct event sequence for A2 - - _Requirements: 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.5, 8.2, 8.3_ - -- [x] 5.2 Test Groups C and H: on_vote_eligible policy and persistence/restart (11 cases) - - C1: all Raft invariants pass, `on_vote_eligible` returns 0 → denied - - C2: all Raft invariants pass, `on_vote_eligible` returns 1 → granted - - C3: `on_vote_eligible` is NULL at init → eligible candidates granted when Raft invariants pass - - C4: stale term → `on_vote_eligible` never called (verified via mock call counter) - - C5: `voted_for` already set → `on_vote_eligible` never called - - H1: `raft_init` with `initial_term=5`, `initial_voted_for=1` → `state.term==5`, `state.voted_for==1` immediately after init - - H2: `raft_init` with `initial_term=0`, `initial_voted_for=-1` → first-boot defaults (`role=FOLLOWER`, `term=0`) - - H3: vote granted → `on_persist` fires (and records its call) before `voted_for` is set in state; verified by checking call order in mock - - H4: `on_persist` returns non-zero → vote denied; `voted_for` unchanged in state - - H5: `raft_start_election` → `on_persist` fires with `(new_term, self_idx)` for self-vote - - Null-callback: `raft_init` with NULL required callback returns `-EINVAL`; state unmodified - - All 11 Unity test cases pass - - _Requirements: 1.2, 3.4, 3.5, 7.2, 7.3, 8.3_ - -- [x] 5.3 Test Groups D and E: quorum evaluation and election timeout (7 cases) - - D1: 3-node cluster, 1 peer grant received (self + 1 = quorum 2) → `role==RAFT_LEADER`; `on_become_leader` fired once - - D2: 3-node, both peers deny → `election_decided==1`, `role==RAFT_FOLLOWER`; `on_become_follower` fired - - D3: `on_become_leader` fired; second vote reply arrives → returns `RAFT_PENDING`; `leader_count` stays 1 - - D4: reply with `term > current_term` → `role==RAFT_FOLLOWER`; `on_become_follower` fires; `state.term` updated - - E1: timeout fires before quorum → `on_become_follower` fires; `on_cancel_timeout` fires; `election_decided==1` - - E2: timeout fires after quorum reached → `on_cancel_timeout` fires; `role` stays `RAFT_LEADER`; `on_become_follower` not fired - - E3: start new election after timeout → `state.term` incremented again; `on_schedule_timeout` fires again - - All 7 Unity test cases pass; trace buffer confirms `on_cancel_timeout` fires in E1 and E2 - - _Requirements: 4.1, 4.2, 4.3, 4.4, 6.1, 6.2, 6.3, 8.2, 8.3_ - -- [x] 5.4 Test Groups F and G: RDHUP failover and split-brain prevention (6 cases) - - F1: nodes 1 and 2 call `raft_on_leader_lost`; mock `on_start_election_after_delay` receives `1×timeout` for node 1 and `2×timeout` for node 2 (priority-based jitter) - - F2: during node 1's delay, node 1 receives and grants a vote request from another node → `voted_for != self_idx`; `raft_check_and_start_election` returns `RAFT_SKIP_ELECTION` - - F3: `raft_on_leader_lost` followed by `raft_check_and_start_election` with no intervening vote grant → returns `RAFT_ELECTION_STARTED`; `state.role==RAFT_CANDIDATE` - - F4: simultaneous RDHUP on all 3 remaining nodes with identical priority → exactly 1 LEADER after delivery (self-vote prevents any node from granting two votes) - - G1: 100-round randomized 3-node election loop via simulator → `leader_count` never exceeds 1 per round; loop completes without assertion failure - - G2: partition node 2 from nodes 0+1 → 2-node partition can elect (`leader_count==1`); isolated node 2 cannot win (`votes_granted < quorum`) - - All 6 Unity test cases pass; G1 confirms split-brain freedom across 100 rounds - - _Requirements: 5.1, 5.2, 5.3, 8.1, 8.2, 8.3_ - -- [x] 6. (P) Controller adapter: wire raft.h into ha.c - _Depends: 3_ -- [x] 6.1 Add raft_state_t to ha_state_t and initialize at startup - - Add `#include "raft/raft.h"` to `ha.h`; add `raft_state_t raft` field to `ha_state_t` - - In the ha initialization path, call `raft_init(&ha->raft, &cfg)` with term/voted_for loaded from `ha_persist_load()`, all ha-implemented callbacks, and `ha_state_t*` as context - - `/xce-build controller` compiles without errors after the change - - _Requirements: 9.1, 9.2_ - - _Boundary: ha.h, ha.c adapter init_ - -- [x] 6.2 Implement all raft_callbacks_t callbacks in ha.c - - `on_send_vote_request`: encode capnp `VoteRequest` and send via existing `endpoint_t`/iomgr path - - `on_become_leader`: invoke existing ha leadership promotion path (phase2 trigger) - - `on_become_follower`: invoke existing ha standby transition path - - `on_persist`: call `ha_persist_term(ha, term, voted_for)` for durable write before granting vote - - `on_schedule_timeout`: schedule election watchdog timer via existing delaymgr mechanism - - `on_cancel_timeout`: cancel pending election timer - - `on_start_election_after_delay`: schedule RDHUP retry timer with the received priority-based delay - - `on_vote_eligible`: implement freshness gate (candidate config_tag ≥ local config_tag AND candidate log_lsn ≥ applied_lsn), role gate (CTRL_PRIMARY_COMMITTING → return 0), heartbeat liveness check (active heartbeat → return 0 on standby); return 1 if all pass - - All 8 callbacks compile and link; adapter initializes in the existing ha startup flow - - _Requirements: 9.1_ - - _Boundary: ha.c callbacks_ - -- [x] 6.3 Delegate election state management to raft_* API - - Remove direct `ha->term++` and `ha->voted_for = ...` writes from `ha_start_election_async`; replace with `raft_start_election(&ha->raft)` - - Replace vote request handling in the capnp vote-request handler with `raft_on_vote_request(&ha->raft, from_idx, term, &reply_term)` - - Replace vote reply accumulation with `raft_on_vote_reply(&ha->raft, from_idx, granted, term)` - - Replace RDHUP election trigger with `raft_on_leader_lost(&ha->raft)` - - No direct writes to `ha->term` or `ha->voted_for` remain outside `on_persist` and `on_vote_eligible` callbacks after the refactor - - _Requirements: 9.2_ - - _Boundary: ha.c election dispatch_ - -- [x] 6.4 Verify XCE_TESTING hook seams remain functional - - Confirm `ha_phase2_hook` and `ha_rdhup_election_hook` still compile and are invoked at the same protocol points as before the refactor - - Run `/xce-build controller`; all pre-existing controller tests pass without regression - - Test pass count for `test_config_repl.c` and other ha-related tests matches or exceeds pre-refactor baseline - - _Requirements: 9.3_ - - _Boundary: ha.c hook seams_ - -- [x] 7. Validation: ASAN build and full protocol verification - _Depends: 5, 6_ - - `/xce-build controller` with `ASAN_OPTIONS=detect_leaks=0` produces zero sanitizer errors or warnings - - All `test_raft.c` groups (A–H, ≥ 20 cases + H null-callback case) pass under ASAN - - All pre-existing controller tests (`test_config_repl.c` and others) remain green under ASAN - - lib + controller combined test pass count meets or exceeds pre-refactor numbers - - _Requirements: 8.3_ diff --git a/.kiro/specs/raft-log-replication/brief.md b/.kiro/specs/raft-log-replication/brief.md deleted file mode 100644 index 32ae1c3d..00000000 --- a/.kiro/specs/raft-log-replication/brief.md +++ /dev/null @@ -1,41 +0,0 @@ -# Brief: raft-log-replication - -## Problem -`src/lib/raft/raft.{c,h}` is an **election-only** Raft library (terms, votes, leader election, PreVote) with a clean callback interface and a deterministic simulator (`raft_sim.c`). It has **no log, no AppendEntries, no commit**. The replicated log is hand-rolled in the controller (`persist.c` WAL, `repl.c` catch-up streaming, `ha.c`, `ha_convergence.c`) and **violates the Raft log invariants of `raft/raft.tla`**, which root-caused **bug-851** (silent committed-prefix divergence under chaotic failover). Authoritative gap analysis: `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`. - -This spec closes the four **log-half** gaps by making the library own the log, strictly following `raft.tla`: -- **G1** — per-entry term is dead (WAL `hdr.term` always 0; `p->term` never set from `ha->term`). -- **G2** — replication matches LSN contiguity only; no `prevLogTerm` check (TLA+ `AppendEntries`/`HandleAppendEntriesRequest` `logOk`). -- **G3** — no `(term,lsn)` conflict truncation (TLA+ conflict branch: delete the divergent entry + all after). -- **G5** — a node adopts a higher term only on **vote** RPCs; the AppendEntries/heartbeat path never advances `currentTerm` (TLA+ `UpdateTerm` fires on **any** higher-term RPC). Confirmed live: a rejoined standby reported `term:2` under a term-3 primary → term reuse → two leaders per term. - -## Approach -Extend the library to own the replicated log, **strictly mirroring the `raft.tla` actions**, keeping the core **pure / synchronous / deterministic** (no I/O, no threads) so it stays simulator-verifiable. All side effects go through new callbacks the controller adapter implements over the existing WAL/repl transport. - -### TLA+ actions implemented (raft.tla line refs) -- `ClientRequest(i,v)` (246) → `raft_log_append(...)`: append `[term ↦ currentTerm, value]`. -- `AppendEntries(i,j)` (204) → emit via `on_send_append_entries` with `prevLogIndex`, `prevLogTerm`, ≤1+ entries, `leaderCommit`. Per-peer `nextIndex`/`matchIndex` (TLA+ leaderVars). -- `HandleAppendEntriesRequest(i,j,m)` (327) → `raft_on_append_entries_request(...)`: `logOk` = `prevLogIndex==0 ∨ log[prevLogIndex].term==prevLogTerm`; reject (stale term / `¬logOk`), candidate→follower on equal term, accept (already-have / conflict-truncate-one / append). -- `HandleAppendEntriesResponse(i,j,m)` (393) → `raft_on_append_entries_reply(...)`: success → advance `nextIndex`/`matchIndex`; fail → decrement `nextIndex`. -- `UpdateTerm(i,j,m)` (406) → applied on **every** incoming RPC including AppendEntries/heartbeat (closes G5): higher term ⇒ adopt term, clear vote, step down to follower. - -### Authoritative shared contract (downstream specs MUST reference this verbatim) -Log index model: the library log index is 1-based (TLA+) and maps 1:1 to the controller WAL **LSN**. Per-entry `term` is `uint64_t`. New REQUIRED callbacks (pure; controller implements I/O): -- `int on_persist_entry(void *ctx, uint64_t index, uint64_t term, const void *entry, size_t len)` — durably append/overwrite at index. -- `uint64_t on_get_entry_term(void *ctx, uint64_t index)` — term of local entry at index (0 if absent). -- `int on_send_append_entries(void *ctx, int peer_idx, uint64_t prev_log_index, uint64_t prev_log_term, uint64_t leader_commit, /* entries handle */ ...)` — transmit. -- `int on_truncate_from(void *ctx, uint64_t index)` — delete local entries at `index..end`. -New library entry points: `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`. Existing election callbacks/behaviour unchanged. - -## Scope -- **In**: library log model with per-entry term; `ClientRequest` append; AppendEntries send (per-peer next/match index); `HandleAppendEntriesRequest` accept/reject/conflict-truncate/append by `(term,index)`; `UpdateTerm` on every RPC; controller adapter that maps callbacks onto the existing WAL/repl transport (WAL `hdr.term` stamped from `ha->term`, carried in the repl envelope, read back on recovery); simulator coverage asserting **Log Matching**. -- **Out**: `AdvanceCommitIndex` / commit-index safety + `(term,lsn)` vote freshness → `raft-commit-safety`. The full conformance harness → `raft-sim-conformance`. Config-change markers + upgrade snapshot → `ha-term-safety`. Async I/O, fan-out scheduling, the sync commit barrier, snapshot transport → controller callback adapters (not the pure core). - -## Preserve as policy (do NOT canonicalize away) -PreVote, priority leadership, persist-before-ACK durability + synchronous commit barrier (consumed in `raft-commit-safety`), "controller never self-terminates" (library returns statuses; controller owns liveness/fencing). - -## Verification -Extend `raft_sim.c`: drive AppendEntries across partitions, message drop/dup/reorder, and restarts; assert the **Log Matching** invariant (same `(index,term)` ⇒ identical prefix) and that a follower truncates a divergent suffix. Replace the controller's LSN-only reconcile with `(term,index)` matching delegated to the library. - -## Dependencies -`raft-library` (election core, implemented `[x]`). Feeds `raft-commit-safety`, `raft-sim-conformance`, and `ha-term-safety` integration. Closes bug-851 G1/G2/G3/G5. diff --git a/.kiro/specs/raft-log-replication/design.md b/.kiro/specs/raft-log-replication/design.md deleted file mode 100644 index ee78fbf1..00000000 --- a/.kiro/specs/raft-log-replication/design.md +++ /dev/null @@ -1,414 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: This feature extends the election-only Raft library (`src/lib/raft/raft.{c,h}`) into a strict implementation of the **log half** of `raft/raft.tla` (Ongaro canonical), giving the library ownership of the replicated log. It closes the four log-half gaps of bug-851 (G1/G2/G3/G5): per-entry term, `prevLogTerm` log-matching, `(term, index)` conflict truncation, and term adoption on the AppendEntries/heartbeat path. - -**Users**: The XCE controller (as a Raft node), the deterministic simulator (`raft_sim.c`), and three downstream specs that consume this seam verbatim — `raft-commit-safety`, `raft-sim-conformance`, and `ha-term-safety`. - -**Impact**: Today the replicated log is hand-rolled across the controller (`persist.c`/`repl.c`/`ha.c`/`ha_convergence.c`) and matches by LSN contiguity only. After this feature, the library owns log growth, AppendEntries emission/handling, per-peer progress, and `(term, index)` truncation as pure, synchronous, deterministic state transitions; the controller becomes a thin adapter that performs I/O behind four new callbacks. The on-disk WAL format is unchanged (the term is a value fix, not a layout change). - -### Goals -- Add a library-owned log model where every entry carries a `uint64_t` term, index 1-based mapping 1:1 to the WAL LSN (G1). -- Implement `ClientRequest`, `AppendEntries` emission, `HandleAppendEntriesRequest`, and `HandleAppendEntriesResponse` exactly as `raft.tla` defines them (G2/G3). -- Apply `UpdateTerm` on **every** RPC including AppendEntries/heartbeat (G5). -- Define the four new callbacks **verbatim** per the brief's Authoritative shared contract. -- Extend `raft_sim.c` to assert the **Log Matching** invariant and divergent-suffix truncation under partitions, drop/dup/reorder, and restart. - -### Non-Goals -- **No commit-index advance** (`AdvanceCommitIndex`, G4) and **no Figure-8 current-term commit rule** — owned by `raft-commit-safety`. -- **No `(lastLogTerm, lastLogIndex)` vote freshness** (G6) — owned by `raft-commit-safety`. -- **No config-change markers / upgrade snapshot** — owned by `ha-term-safety`. -- **No async I/O, fan-out scheduling, commit barrier, or snapshot transport** — these live in controller-side callback adapters, not the pure core. -- **No heavy controller WAL/repl/recovery wiring** — this spec defines only the seam contract and a thin adapter description; the integration is `ha-term-safety`. - -## Boundary Commitments - -### This Spec Owns -- The library log model: per-entry `uint64_t` term, 1-based index, the in-library leader progress vectors (`next_index[]`, `match_index[]`). -- New entry points: `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`. -- The four new callbacks in `raft_callbacks_t`: `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`. -- `UpdateTerm`-on-any-RPC applied on the AppendEntries/heartbeat path. -- The `(term, index)` reconcile seam: the library decides match / truncate / append; the controller routes the decision. -- Simulator extensions in `raft_sim.c` + tests in `src/lib/testsuite/test_raft.c` asserting Log Matching. - -### Out of Boundary -- Commit-index advance and current-term commit safety (`raft-commit-safety`, G4). -- Vote-freshness `(term, index)` comparison in `RequestVote`/PreVote (`raft-commit-safety`, G6). -- Configuration markers in the unified log + the change-boundary snapshot (`ha-term-safety`). -- The controller's WAL byte layout, repl envelope serialization, recovery loop, commit barrier, fsync/durability, and snapshot transport internals (`ha-term-safety` adapter). This spec specifies only **what the adapter must route**, not its internal mechanics. - -### Allowed Dependencies -- The implemented election core (`raft-library`): `raft_state_t`, `raft_callbacks_t`, `raft_init`, the vote/PreVote path. All unchanged. -- `errno.h`, `stdint.h`, `string.h` only (the library has no other dependencies and must stay free-standing for the simulator). -- The simulator harness (`raft_sim.c`) for verification. -- Must NOT depend on any controller header, WAL type, capnp type, or transport — the core stays pure. - -### Revalidation Triggers -Dependent specs (`raft-commit-safety`, `raft-sim-conformance`, `ha-term-safety`) must re-check integration if any of the following change: -- The signature or semantics of any of the four new callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`). -- The signature or return semantics of `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`. -- The 1-based-index ↔ WAL-LSN identity, or the "term `0` means absent" convention. -- The per-peer `next_index[]` / `match_index[]` initialization or update rules. -- The placement of `UpdateTerm` before `logOk` on the AppendEntries path. - -## Architecture - -### Existing Architecture Analysis - -The library is a pure, callback-driven state machine (`raft.c`, ~430 lines). `raft_state_t` holds only Raft invariant state; application policy is delegated to `on_vote_eligible`. Persistence is write-only via `on_persist` — **the library never reads storage**; the caller restores `(term, voted_for)` into `raft_config_t` before `raft_init`. This feature follows the same discipline for the log: the library never reads the WAL directly; it reads entry terms only through `on_get_entry_term`, and writes/truncates only through callbacks. The pattern preserved: - -- **Pure / synchronous / deterministic core** — no I/O, no threads, no allocation in the hot path. The log model uses fixed bookkeeping (a `Len(log)` counter `log_len` plus per-peer vectors); the entry **bytes** live in the adapter (WAL) / simulator (in-memory array), reached via callbacks. The library never owns entry storage — consistent with how it already does not own `(term, voted_for)` storage. -- **Persist-before-act ordering** — `on_persist` is called before granting a vote / sending RPCs; `on_persist_entry` is called before reporting an append success, identically. -- **Status returns, never self-terminate** — the library returns codes; the controller owns liveness and fencing. No new code path aborts. - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TB - subgraph PureCore["src/lib/raft/raft.c — PURE CORE (this spec)"] - APP["raft_log_append()<br/>ClientRequest (tla:246)"] - SEND["AppendEntries emit<br/>(tla:204) next/match index"] - REQ["raft_on_append_entries_request()<br/>HandleAppendEntriesRequest (tla:327)<br/>UpdateTerm (tla:406) first"] - REPLY["raft_on_append_entries_reply()<br/>HandleAppendEntriesResponse (tla:393)"] - end - subgraph CBs["raft_callbacks_t — NEW seam (verbatim contract)"] - C1["on_persist_entry(index,term,entry,len)"] - C2["on_get_entry_term(index) -> term|0"] - C3["on_send_append_entries(peer,prevIdx,prevTerm,leaderCommit,...)"] - C4["on_truncate_from(index)"] - end - subgraph Consumers["ADAPTERS — implement callbacks (NOT this spec's heavy wiring)"] - CTRL["controller adapter (ha-term-safety):<br/>WAL hdr.term=ha->term, repl envelope,<br/>recovery read-back, (term,idx) route"] - SIM["raft_sim.c: in-memory log,<br/>synchronous routing (this spec)"] - end - APP --> C1 - SEND --> C3 - REQ --> C2 & C1 & C4 - REPLY -.updates.-> SEND - C1 & C2 & C3 & C4 --> CTRL - C1 & C2 & C3 & C4 --> SIM -``` - -**Architecture Integration**: -- **Selected pattern**: Pure state machine + callback ports (Hexagonal/Ports-and-Adapters), already established by the election core. New log actions are new ports; adapters (controller, simulator) implement them. -- **Domain boundaries**: protocol decisions (what to persist/send/truncate, accept/reject/conflict) stay in the library; I/O (WAL write, transport, recovery, in-memory backing) stays in adapters. -- **Existing patterns preserved**: `raft_config_t` extensibility, persist-before-act, NULL-checked optional callbacks, status-code returns, `XCE_TESTING`-gated simulator. -- **Steering compliance**: matches `src/lib/CLAUDE.md` (free-standing C, gnu17), `ha-async-io.md` (no blocking in the core; async lives in adapters), `testing.md` (TDD: simulator/test first), "controller never self-terminates" (library returns statuses only). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Library core | C (gnu17), `errno.h`/`stdint.h`/`string.h` only | Pure log state machine | Free-standing; no controller/capnp deps | -| Callbacks | `raft_callbacks_t` function pointers | Seam for I/O | 4 new required callbacks | -| Verification | `raft_sim.c` (XCE_TESTING) + `test_raft.c` | Deterministic Log Matching | In-memory log backing | -| Adapter (downstream) | controller `persist.c`/`repl.c` (`ha-term-safety`) | WAL/transport I/O | Not built here | - -## File Structure Plan - -### Modified Files -- `src/lib/raft/raft.h` — Add log-model fields to `raft_state_t` (`log_len`, `next_index[RAFT_MAX_PEERS]`, `match_index[RAFT_MAX_PEERS]`); add the four callbacks to `raft_callbacks_t`; declare `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`; add an `raft_append_result_t` reply struct; extend the simulator section (an in-memory log per `raft_sim_node_t`, an AppendEntries RPC buffer/route). -- `src/lib/raft/raft.c` — Implement the three new entry points + a private AppendEntries emit helper; add `on_persist_entry` ordering on `raft_init` validation; apply `UpdateTerm` on the AppendEntries path before `logOk`; initialize `next_index`/`match_index` in `raft_start_election`/`on_become_leader` path. -- `src/lib/raft/raft_sim.c` — In-memory per-node log; wire the four callbacks; add `raft_sim_deliver_append_entries` (synchronous request→reply routing honoring partitions); drop/dup/reorder + restart drivers; the Log Matching assertion helper. -- `src/lib/testsuite/test_raft.c` — New tests for each entry point and the simulator invariants (TDD-first). - -### Directory Structure (unchanged) -``` -src/lib/raft/ -├── raft.h # + log fields, 4 callbacks, 3 entry points, sim log -├── raft.c # + ClientRequest / AppendEntries / Handle* / UpdateTerm-on-AE -└── raft_sim.c # + in-memory log, AE routing, Log Matching assert -src/lib/testsuite/ -└── test_raft.c # + log-replication + Log Matching tests -``` - -> Each file keeps one responsibility: `raft.c` = pure protocol, `raft_sim.c` = deterministic harness + in-memory adapter, `test_raft.c` = assertions. No new file is introduced; the existing module pattern is extended. - -## System Flows - -### AppendEntries: leader emit → follower handle → leader reply (TLA+ actions in nodes) - -```mermaid -sequenceDiagram - participant L as Leader (raft.c) - participant LC as Leader adapter (cb) - participant FC as Follower adapter (cb) - participant F as Follower (raft.c) - Note over L: raft_log_append(v) — ClientRequest tla:246 - L->>LC: on_persist_entry(idx, term, v) - Note over L: AppendEntries tla:204<br/>prevLogIndex=nextIndex[j]-1 - L->>LC: on_send_append_entries(j, prevIdx, prevTerm, leaderCommit, entry) - LC-->>FC: (transport / sim route) - Note over F: raft_on_append_entries_request()<br/>UpdateTerm tla:406 FIRST (G5) - F->>FC: on_get_entry_term(prevIdx) %% logOk (G2) - alt term < currentTerm OR (follower AND not logOk) - F-->>L: reply success=false, matchIndex=0 - else conflict: local[idx].term != entry.term (G3) - F->>FC: on_truncate_from(idx) - F->>FC: on_persist_entry(idx, term, entry) - F-->>L: reply success=true, matchIndex=idx - else no-conflict append OR already-have - F->>FC: on_persist_entry(idx, term, entry) %% append case - F-->>L: reply success=true, matchIndex=prevIdx+len - end - Note over L: raft_on_append_entries_reply() tla:393 - alt success - Note over L: nextIndex[j]=matchIndex+1; matchIndex[j]=reply.matchIndex - else failure - Note over L: nextIndex[j]=Max(nextIndex[j]-1, 1) %% walk back - end -``` - -Flow-level decisions: -- **UpdateTerm fires first** on the request path (before `logOk`), so a strictly-higher-term request is never spuriously rejected for a log mismatch (R6.4, closes G5). -- **`logOk`** is computed via `on_get_entry_term(prevLogIndex)`; a return of `0` is the "absent" sentinel consistent with `prevLogTerm == 0` when `prevLogIndex == 0` (R1.6, R4.1). -- **Conflict truncation** deletes `index..end` via `on_truncate_from` then appends in the same handler invocation (R4.5/R4.6, closes G3) — an intentional, safe collapse of `raft.tla`'s truncate-only conflict step plus the following no-conflict append; see **Note on conflict branch** below. -- **Commit is opaque** — `leaderCommit` is passed through untouched; this spec never advances commit (R3.4, R5.4). - -## Requirements Traceability - -| Requirement | Summary | Design Elements | raft.tla action | -|-------------|---------|-----------------|-----------------| -| 1.1–1.6 | Library log model, per-entry term | `raft_state_t.log_len`; `on_get_entry_term`; `on_persist_entry` | `ClientRequest` (246), `log[i].term` | -| 2.1–2.5 | ClientRequest append | `raft_log_append()` | `ClientRequest` (246) | -| 3.1–3.5 | AppendEntries emit, next/match | `next_index[]`/`match_index[]`; `on_send_append_entries` | `AppendEntries` (204), `BecomeLeader` (229) | -| 4.1–4.7 | logOk / reject / step-down / accept / conflict | `raft_on_append_entries_request()`; `on_get_entry_term`; `on_truncate_from`; `on_persist_entry` | `HandleAppendEntriesRequest` (327) | -| 5.1–5.4 | reply advance/decrement | `raft_on_append_entries_reply()` | `HandleAppendEntriesResponse` (393) | -| 6.1–6.5 | UpdateTerm on any RPC | term-adoption block in request handler (before logOk); `on_persist`; `on_become_follower` | `UpdateTerm` (406) | -| 7.1–7.7 | Callback contract + pure split | `raft_callbacks_t` additions; `raft_init` validation; adapter description | (seam, not a TLA+ action) | -| 8.1–8.6 | Simulator Log Matching | `raft_sim.c` AE routing, in-memory log, assert helpers | Log Matching invariant | - -## Components and Interfaces - -| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | -|-----------|-------|--------|--------------|------------------|-----------| -| Log model (`raft_state_t`) | Library core | Per-entry term + per-peer progress | 1, 3 | election core (P0) | State | -| `raft_log_append` | Library core | ClientRequest append | 2 | on_persist_entry (P0) | Service | -| `raft_on_append_entries_request` | Library core | Handle+logOk+conflict+UpdateTerm | 4, 6 | on_get_entry_term, on_truncate_from, on_persist_entry, on_persist, on_become_follower (P0) | Service | -| `raft_on_append_entries_reply` | Library core | Advance/decrement progress | 5 | (internal state) | Service | -| New callbacks | Seam | I/O ports | 7 | adapters (P0) | Service | -| Simulator | Verification | Deterministic Log Matching | 8 | all of the above (P0) | State | - -### Library Core - -#### Log model additions to `raft_state_t` - -| Field | Detail | -|-------|--------| -| Intent | Hold the library's view of log length and per-peer replication progress (TLA+ `logVars` length + `leaderVars`) | -| Requirements | 1.1, 1.2, 3.1 | - -**Responsibilities & Constraints** -- `uint64_t log_len;` — `Len(log)`; the highest 1-based index the library believes is persisted locally. Incremented on append, set by truncate. -- `uint64_t next_index[RAFT_MAX_PEERS];` — per-peer `nextIndex` (1-based). -- `uint64_t match_index[RAFT_MAX_PEERS];` — per-peer `matchIndex` (0 = nothing matched). -- The library stores **no entry bytes and no per-entry term array** — entry terms are obtained via `on_get_entry_term`; bytes live in the adapter/simulator. This keeps `raft_state_t` fixed-size and the core pure (consistent with not storing `(term, voted_for)` bytes). -- Invariant: `match_index[j] ≤ next_index[j] - 1` for every peer; `log_len` is the 1:1 WAL LSN. - -**State Management** -- `log_len` restored at init from the adapter's recovered WAL length (added to `raft_config_t` as `initial_log_len`, default 0; the library never reads storage). -- `next_index[j] = log_len + 1`, `match_index[j] = 0` set when the node becomes leader (mirrors `BecomeLeader`, tla:229). - -#### `raft_log_append` — ClientRequest (raft.tla:246) - -| Field | Detail | -|-------|--------| -| Intent | Append one entry stamped with `currentTerm` on the leader | -| Requirements | 2.1, 2.2, 2.3, 2.4, 2.5 | - -##### Service Interface -```c -/* - * raft_log_append: leader appends one client entry (raft.tla ClientRequest, 246). - * Stamps the entry with s->term, assigns index Len(log)+1, persists it via - * on_persist_entry BEFORE returning success (persist-before-act). - * Does NOT advance any commit index (owned by raft-commit-safety). - * - * out_index: receives the assigned 1-based index (== WAL LSN) on success. - * Returns RAFT_OK on success; -EPERM if role != RAFT_LEADER; the on_persist_entry - * return is propagated on persist failure (no log mutation on failure). - */ -int raft_log_append(raft_state_t *s, const void *entry, size_t len, - uint64_t *out_index); -``` -- Preconditions: `s->role == RAFT_LEADER`. -- Postconditions: `log_len` incremented by 1; `*out_index == log_len`; entry persisted at `(log_len, s->term)`. -- Invariants: appended entry term == `s->term`; no commit-index change. - -#### `raft_on_append_entries_request` — HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (406) - -| Field | Detail | -|-------|--------| -| Intent | Apply UpdateTerm first, then logOk + reject/step-down/accept (already-have / conflict-truncate / append) | -| Requirements | 4.1–4.7, 6.1–6.5 | - -##### Service Interface -```c -/* Result of handling an AppendEntries request — the adapter sends this back. */ -typedef struct { - uint64_t reply_term; /* current term to return to the leader */ - int success; /* 1 = accepted, 0 = rejected */ - uint64_t match_index; /* on success: prevLogIndex + len(entries); else 0 */ -} raft_append_result_t; - -/* - * raft_on_append_entries_request: handle an incoming AppendEntries / heartbeat. - * Step order (must not be reordered): - * 1. UpdateTerm (raft.tla:406, G5): if term > s->term, adopt term, clear vote, - * step down to FOLLOWER, on_persist(term,-1), fire on_become_follower if was - * leader. Applied on EVERY RPC including a 0-entry heartbeat, BEFORE logOk. - * 2. logOk (raft.tla:328, G2): prevLogIndex==0 || (prevLogIndex<=log_len && - * on_get_entry_term(prevLogIndex)==prevLogTerm). - * 3. Reject (raft.tla:333): term<s->term || (term==s->term && FOLLOWER && !logOk) - * -> success=0, match_index=0. - * 4. Candidate step-down (raft.tla:346): term==s->term && CANDIDATE -> FOLLOWER. - * 5. Accept (raft.tla:351), index = prevLogIndex+1: - * already-have: entries empty, or local[index].term == entry.term -> success. - * conflict (raft.tla:375, G3): local[index] exists, term differs -> - * on_truncate_from(index), then on_persist_entry(index, entry). - * append (raft.tla:383): log_len == prevLogIndex -> on_persist_entry(index). - * entries_len is 0 (heartbeat) or 1 (this spec, like raft.tla). - * Returns RAFT_OK; fills *out. - */ -int raft_on_append_entries_request(raft_state_t *s, int leader_idx, - uint64_t term, uint64_t prev_log_index, - uint64_t prev_log_term, uint64_t leader_commit, - const void *entry, size_t entry_len, - uint64_t entry_term, int entries_len, - raft_append_result_t *out); -``` -- Preconditions: none beyond a valid `s`. -- Postconditions: local log matches the leader through `match_index` on success; divergent suffix truncated on conflict; term adopted if higher. -- Invariants: `on_truncate_from` is always followed by re-`on_persist_entry` of the leader's entry; commit is never advanced here (`leader_commit` is opaque, passed for the downstream commit spec to consume — ignored now). - -> **Note on `leader_commit`**: accepted into the signature now (so the wire/sim shape is stable for `raft-commit-safety`) but **not acted on** in this spec — R3.4/R5.4. This avoids a later signature change (a Revalidation Trigger). - -> **Note on conflict branch (intentional, safe divergence from `raft.tla`)**: `raft.tla` (lines 375–388) models the conflict as a **truncate-only** step — `HandleAppendEntriesRequest` deletes the conflicting suffix and returns, and the leader's entry is appended on a *later* AppendEntries via the no-conflict branch (raft.tla:383). XCE deliberately **collapses** the truncate and the immediately-following no-conflict append into a single handler invocation: `on_truncate_from(index)` **then** `on_persist_entry(index, term, entry)`. This is a safe optimization — `raft.tla` itself notes implementations may batch these steps — and Log Matching is preserved because any entry kept after truncation still matches the leader's `(term, index)`, and the freshly-persisted entry is exactly the leader's. On this conflict-and-append path the returned `match_index` is `prevLogIndex + len(entries)` (= `index`), identical to the value the success/no-conflict branch returns, so the leader's `nextIndex`/`matchIndex` accounting is unchanged. - -#### `raft_on_append_entries_reply` — HandleAppendEntriesResponse (raft.tla:393) - -| Field | Detail | -|-------|--------| -| Intent | Advance per-peer progress on success; walk back `nextIndex` on failure | -| Requirements | 5.1, 5.2, 5.3, 5.4 | - -##### Service Interface -```c -/* - * raft_on_append_entries_reply: process a follower's AppendEntries response - * (raft.tla:393). Applied only when reply_term == s->term (stale-term drop, 415). - * success: next_index[peer]=match_index+1; match_index[peer]=match_index. - * failure: next_index[peer]=Max(next_index[peer]-1, 1); match_index unchanged. - * Does NOT advance commit (owned by raft-commit-safety). - * Returns RAFT_OK. - */ -int raft_on_append_entries_reply(raft_state_t *s, int peer_idx, int success, - uint64_t match_index, uint64_t reply_term); -``` -- Preconditions: `s->role == RAFT_LEADER` for the update to matter (stale replies are ignored). -- Postconditions: `next_index`/`match_index` for `peer_idx` updated per branch. -- Invariants: `next_index[peer] ≥ 1` always (floor); commit untouched. - -#### AppendEntries emit helper (raft.tla:204) - -| Field | Detail | -|-------|--------| -| Intent | Compute `prevLogIndex`/`prevLogTerm` and emit ≤1 entry via `on_send_append_entries` | -| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 | - -**Responsibilities & Constraints** -- For peer `j`: `prevLogIndex = next_index[j] - 1`; `prevLogTerm = (prevLogIndex > 0) ? on_get_entry_term(prevLogIndex) : 0`. -- `lastEntry = Min(log_len, next_index[j])`; send `SubSeq(log, next_index[j], lastEntry)` — at most one entry; a heartbeat when `next_index[j] > log_len`. -- Calls `on_send_append_entries(ctx, j, prevLogIndex, prevLogTerm, leaderCommit, /* entries handle */ ...)`. `leaderCommit` is supplied opaquely by the caller (the controller/sim), since commit lives outside this spec. -- Emitted only while `s->role == RAFT_LEADER` (R3.5). - -> Exposed as `raft_send_append_entries(raft_state_t *s, int peer_idx, uint64_t leader_commit)` so the controller/simulator can drive replication per-peer (fan-out scheduling itself is an adapter concern). - -### Seam — `raft_callbacks_t` additions (verbatim Authoritative shared contract) - -These are appended to `raft_callbacks_t` and validated **required** in `raft_init`. Reproduced **verbatim** from the brief so downstream specs depend on the exact shapes: - -```c -/* REQUIRED: durably append/overwrite the log entry at `index` with `term`. */ -int on_persist_entry(void *ctx, uint64_t index, uint64_t term, - const void *entry, size_t len); - -/* REQUIRED: term of the local entry at `index` (0 if absent). */ -uint64_t on_get_entry_term(void *ctx, uint64_t index); - -/* REQUIRED: transmit one AppendEntries to peer_idx with prevLog{Index,Term}, - * leaderCommit, and n_entries (0 = heartbeat, 1 = one entry). The library holds - * no entry bytes, so when n_entries==1 the adapter sources the single entry at - * prev_log_index+1 from its own log. [Concretizes the original varargs "entries - * handle" placeholder; impl 2026-06-21.] */ -int on_send_append_entries(void *ctx, int peer_idx, - uint64_t prev_log_index, uint64_t prev_log_term, - uint64_t leader_commit, int n_entries); - -/* REQUIRED: delete local entries at index..end. */ -int on_truncate_from(void *ctx, uint64_t index); -``` - -**Contract rules (R7.5)**: the library decides *what / whether* to persist, send, or truncate; the callback only performs I/O. `on_get_entry_term` returning `0` is the canonical "absent" signal (used by `logOk` and by `prevLogTerm` computation). All four are NULL-checked in `raft_init` and required (return `-EINVAL` if NULL), matching how the election-required callbacks are validated today. - -### Controller adapter (seam only — heavy wiring is `ha-term-safety`) - -This spec describes the adapter **only at the seam**; the full controller integration is the downstream `ha-term-safety` consumer. - -- `on_persist_entry` → controller WAL write that stamps `hdr.term` from `ha->term` (closing G1 at the byte level), at the LSN equal to `index`. -- `on_get_entry_term` → read the recorded `hdr.term` for a given LSN (from the in-memory WAL index / recovery read-back). -- `on_send_append_entries` → enqueue a repl envelope carrying `prevLogIndex`, `prevLogTerm`, the entry, and its term, onto the existing repl transport. -- `on_truncate_from` → truncate the WAL at/after the LSN, replacing the LSN-only reconcile in `ha_convergence.c` with the library's `(term, index)` decision. -- On recovery, the controller restores `log_len` (= recovered WAL length) and each entry's term, feeding `on_get_entry_term`; `initial_log_len` is passed into `raft_config_t`. - -## Data Models - -### Domain Model -- **Log entry (logical)**: `(index: uint64, term: uint64, value: bytes)`. The library holds only `index` accounting (`log_len`) and obtains `term` via callback; `value` bytes are owned by the adapter (WAL) / simulator (in-memory array). Index is 1-based, identical to the controller WAL LSN. -- **Per-peer progress**: `(next_index: uint64, match_index: uint64)` per peer, the TLA+ `leaderVars`. - -### Logical Data Model -- `raft_state_t` gains `log_len`, `next_index[RAFT_MAX_PEERS]`, `match_index[RAFT_MAX_PEERS]`. -- `raft_config_t` gains `initial_log_len` (restored WAL length at boot; 0 on first boot — the library never reads storage, consistent with `initial_term`/`initial_voted_for`). -- `raft_append_result_t` is the request-handler reply payload. -- **No on-disk format change**: the WAL header already carries a `hdr.term` field (today always 0); this feature only requires the adapter to write the real term. The library imposes no new persisted layout (R7 in requirements / unchanged on-disk format per `ha-term-safety` R1.7). - -### Data Contracts & Integration -- **Entries handle**: `on_send_append_entries` and `raft_on_append_entries_request` carry one entry as `(const void *entry, size_t len, uint64_t entry_term)` plus an `entries_len ∈ {0,1}`. This mirrors `raft.tla`'s "up to 1 entry" simplification and keeps the simulator and controller envelope shapes identical. -- **Term `0` sentinel**: `on_get_entry_term` returns `0` for an absent index; legacy pre-feature entries with a zero term are handled by the controller adapter (mapped to the lowest-known term) per `ha-term-safety` R1.8 — out of this library's scope, but the `0`-means-absent convention is honored consistently here. - -## Error Handling - -### Error Strategy -The library returns status codes only and **never aborts** (preserves "controller never self-terminates"). Callback failures propagate as negative returns; the caller decides liveness. - -### Error Categories and Responses -- **Caller misuse**: `raft_log_append` while not leader → `-EPERM`, no log change. NULL required callback at `raft_init` → `-EINVAL`. -- **Persist failure**: `on_persist_entry` non-zero return → propagated; the library does **not** increment `log_len` on a failed append (no phantom entry), preserving the persist-before-act guarantee. -- **Truncate failure**: `on_truncate_from` non-zero return on the conflict branch → reported as a non-success request result so the leader retries (the follower has not corrupted its log; it simply did not converge this round). -- **Stale RPCs**: a request with `term < currentTerm` is rejected (success=0, matchIndex=0); a reply with `reply_term < currentTerm` is ignored (DropStaleResponse, tla:415). -- **Higher-term RPC**: always adopted first (UpdateTerm) — never rejected for a log mismatch (R6.4). - -### Monitoring -Reuse the optional `on_trace` callback for new events: `"ae_recv"`, `"ae_reject_stale"`, `"ae_reject_logmismatch"`, `"ae_conflict_truncate"`, `"ae_append"`, `"term_adopt_ae"`, `"ae_reply_advance"`, `"ae_reply_backoff"`. NULL-checked exactly like existing trace calls. - -## Testing Strategy - -TDD-mandatory (`testing.md`): tests and simulator drivers are written **before** the `raft.c` implementation (RED → GREEN). All tests are `XCE_TESTING`-gated and live in `src/lib/testsuite/test_raft.c` driving `raft_sim.c`. - -### Unit Tests (entry points, against in-memory sim log) -1. `raft_log_append` on a leader stamps `currentTerm`, assigns `Len(log)+1`, calls `on_persist_entry` once, returns the index (R2.1–2.4); on a follower returns `-EPERM` with no log change (R2.2). -2. `raft_on_append_entries_request` `logOk`: accepts when `prevLogIndex==0`; accepts when `on_get_entry_term(prevLogIndex)==prevLogTerm`; rejects (success=0, matchIndex=0) when terms differ (R4.1, R4.2 — G2). -3. Conflict branch: a follower with `(index=k, term=A)` receiving `(index=k, term=B)` calls `on_truncate_from(k)` then `on_persist_entry(k, B)` and reports success with `matchIndex=k` (R4.5, R4.6 — G3). -4. `raft_on_append_entries_reply`: success sets `next_index=matchIndex+1`, `match_index=reply`; failure decrements `next_index` flooring at 1; stale `reply_term` ignored (R5.1–5.3). -5. `UpdateTerm` on AppendEntries: a follower/leader at term T receiving an AE/heartbeat at term T+1 adopts T+1, clears vote, persists, and (if was leader) fires `on_become_follower` — applied before `logOk`, including a 0-entry heartbeat (R6.1–6.4 — G5). - -### Integration / Simulator Tests (Log Matching invariant) -1. Two-phase replication: leader appends 3 entries, replicates to a quorum; assert every node's prefix is byte-identical (Log Matching, R8.4). -2. Partition + heal: isolate a follower during several appends, then heal; assert AppendEntries walks `nextIndex` back, the follower catches up, and Log Matching holds (R8.1, R8.3). -3. Divergent-suffix truncation: hand a follower a manufactured divergent tail (same indices, different term), replicate from a higher-term leader; assert the follower truncates and converges to the leader's prefix (R8.5 — G3). -4. Chaotic churn / G5 regression: a stale-term node (term lagging) rejoins under a higher-term leader; assert it adopts the higher term from the AppendEntries/heartbeat path and does **not** re-elect at a reused term (R8.6 — closes the bug-851 G5 mechanism in the deterministic harness). - -### Message-fault Tests -1. Drop, duplicate, and reorder AppendEntries requests/replies in the sim queue; assert idempotent convergence and Log Matching after the faults clear (R8.3) — the already-have accept branch (R4.4) makes a duplicated AE a no-op. diff --git a/.kiro/specs/raft-log-replication/requirements.md b/.kiro/specs/raft-log-replication/requirements.md deleted file mode 100644 index 3b1fcf26..00000000 --- a/.kiro/specs/raft-log-replication/requirements.md +++ /dev/null @@ -1,143 +0,0 @@ -# Requirements Document - -## Introduction - -`src/lib/raft/raft.{c,h}` is today an **election-only** Raft library: it implements terms, votes, leader election, and PreVote with a pure callback interface (`raft_callbacks_t`, `raft_state_t`) and a deterministic simulator (`raft_sim.c`). It has **no log, no AppendEntries, and no commit**. The replicated log is instead hand-rolled across the controller (`persist.c` WAL, `repl.c` catch-up streaming, `ha.c`, `ha_convergence.c`) and **violates the log invariants of `raft/raft.tla`** (Ongaro's canonical Raft TLA+ spec). That split root-caused **bug-851**: silent committed-prefix divergence under chaotic failover. The authoritative gap analysis is `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` (gaps G1–G6). - -This feature extends the library so it **owns the replicated log**, strictly mirroring the `raft.tla` log actions, while keeping the core **pure / synchronous / deterministic** (no I/O, no threads) so it remains independently verifiable in the simulator against the Raft safety invariants. All side effects flow through new callbacks that a controller adapter implements over the existing WAL/repl transport. The feature closes exactly the four **log-half** gaps: - -- **G1** — per-entry term is dead today (WAL `hdr.term` always 0; `p->term` never set from `ha->term`). The library log model gives every entry a real `uint64_t` term, set to the leader's current term at append time (`raft.tla` `ClientRequest`, line 246). -- **G2** — replication matches LSN contiguity only; there is no `prevLogTerm` check. The library enforces the `logOk` predicate (`raft.tla` `AppendEntries` line 204 / `HandleAppendEntriesRequest` line 327): `prevLogIndex == 0 ∨ log[prevLogIndex].term == prevLogTerm`. -- **G3** — no `(term, index)` conflict truncation; reconcile truncates by a committed-boundary heuristic and cannot detect same-index/different-term divergence. The library implements the TLA+ conflict branch: delete the divergent entry and everything after it. -- **G5** — a node adopts a higher term only on **vote** RPCs; the AppendEntries/heartbeat path never advances `currentTerm`. The library applies `raft.tla` `UpdateTerm` (line 406) on **every** incoming RPC including AppendEntries/heartbeat: a strictly higher term ⇒ adopt term, clear vote, step down to follower. - -Commit-index advance (`AdvanceCommitIndex`, G4) and `(lastLogTerm, lastLogIndex)` vote freshness (G6) are **out of scope** and are closed by the downstream `raft-commit-safety` spec. The full conformance harness is `raft-sim-conformance`. The controller-side wiring of WAL / commit-barrier / snapshot onto these callbacks is owned by `ha-term-safety`. This spec's contract (the new callbacks and entry points below) is consumed **verbatim** by all three downstream specs. - -## Boundary Context - -- **In scope**: - - A per-node log model **inside the library** where every entry carries a `uint64_t` term (G1), with the log index 1-based (TLA+) and mapping 1:1 to the controller WAL LSN. - - `ClientRequest` append: `raft_log_append(...)` appends `[term ↦ currentTerm, value]` on the leader (`raft.tla` line 246). - - `AppendEntries` send: per-peer `nextIndex` / `matchIndex` leader state; emission of `prevLogIndex`, `prevLogTerm`, ≤1 entry, and `leaderCommit` via the `on_send_append_entries` callback (`raft.tla` line 204). - - `HandleAppendEntriesRequest`: the `logOk` `prevLogTerm` match (G2), reject (stale term / `¬logOk`), candidate→follower on equal term, and the accept branch's three sub-cases — already-have, conflict-truncate, append-by-`(term, index)` (G3) (`raft.tla` line 327). - - `HandleAppendEntriesResponse`: success advances `nextIndex` / `matchIndex`; failure decrements `nextIndex` (floored at 1) (`raft.tla` line 393). - - `UpdateTerm` applied on **every** incoming RPC including AppendEntries/heartbeat (G5) (`raft.tla` line 406). - - The shared callback contract: `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from` (pure protocol decisions in the library; I/O in the controller adapter). - - The controller adapter only at the seam: it stamps the WAL `hdr.term` from `ha->term`, carries the term in the repl envelope, reads it back on recovery, and routes `(term, index)` match/truncate decisions back into the library. - - Simulator coverage in `raft_sim.c` that drives AppendEntries across partitions, message drop/dup/reorder, and restarts, asserting the **Log Matching** invariant and divergent-suffix truncation. - -- **Out of scope** (owned elsewhere): - - `AdvanceCommitIndex` / commit-index advance and the Figure-8 current-term commit rule (G4) → `raft-commit-safety`. - - `RequestVote` / PreVote freshness becoming `(lastLogTerm, lastLogIndex)` instead of LSN-only (G6) → `raft-commit-safety`. - - The full conformance / TLC trace-check harness → `raft-sim-conformance`. - - Configuration-change markers in the unified log + the upgrade snapshot → `ha-term-safety`. - - Async vtask I/O, fan-out replication scheduling, the synchronous commit barrier, persist-before-ACK durability, and snapshot transport → controller callback adapters (`ha-term-safety`), not the pure core. - - The heavy controller wiring of WAL / repl / recovery onto the callbacks → `ha-term-safety` (this spec defines only the seam contract and a thin adapter description). - -- **Adjacent expectations**: - - Builds on the implemented election core (`raft-library`); all existing election callbacks and behaviour are unchanged. - - Preserves as policy (must NOT be canonicalized away): PreVote, priority leadership, persist-before-ACK durability + the synchronous commit barrier (consumed downstream), and "controller never self-terminates" (the library returns statuses; the controller owns liveness/fencing — the library never aborts a process). - - Feeds `raft-commit-safety` (commit + freshness), `raft-sim-conformance` (verification harness), and `ha-term-safety` (XCE integration + regression). Its protocol invariants R1–R5 there are realized by this library spec rather than by patching the controller directly. - -## Requirements - -### Requirement 1: Library-owned log model with per-entry term (closes G1) - -**Objective:** As a consumer of the raft library, I want the library to own a log where every entry carries the term under which it was created, so that entries from different terms are distinguishable and Raft log-matching becomes possible. - -#### Acceptance Criteria - -1. The Raft Library shall maintain, per node, a 1-based log index whose value at index `i` includes the `uint64_t` term recorded for the entry at that index, mirroring `raft.tla` `log[i][...].term`. -2. The Raft Library shall expose the log index as identical to the controller WAL LSN (index 1 ↔ LSN 1), so callbacks and the adapter address entries by a single shared `(term, index)` identity. -3. When the leader appends a client value, the Raft Library shall record the leader's current `currentTerm` as that entry's term, mirroring `raft.tla` `ClientRequest` (`entry == [term ↦ currentTerm[i], value ↦ v]`). -4. The Raft Library shall not record a fixed or zero placeholder term for any entry created while a valid term is in effect (the G1 defect). -5. While persisting or overwriting an entry, the Raft Library shall delegate the durable write to the `on_persist_entry(ctx, index, term, entry, len)` callback and shall not itself perform any I/O. -6. When the library needs the term of a local entry at a given index, it shall obtain it via the `on_get_entry_term(ctx, index)` callback, treating a return of `0` as "no entry at that index" consistent with `raft.tla` `prevLogTerm == 0` when `prevLogIndex == 0`. - -### Requirement 2: ClientRequest append on the leader (raft.tla ClientRequest, line 246) - -**Objective:** As a controller acting as primary, I want a single library entry point to append a new log entry stamped with the current term, so that the library — not hand-rolled controller code — owns log growth. - -#### Acceptance Criteria - -1. When `raft_log_append(...)` is called while the node's role is `RAFT_LEADER`, the Raft Library shall append one entry at index `Len(log) + 1` with term equal to `currentTerm`, mirroring `raft.tla` `ClientRequest`. -2. If `raft_log_append(...)` is called while the node's role is not `RAFT_LEADER`, then the Raft Library shall reject the append and report a non-success status without mutating the log. -3. When appending, the Raft Library shall invoke `on_persist_entry` for the new `(index, term, entry)` before reporting success, preserving the persist-before-act ordering used by the existing election path's `on_persist`. -4. The Raft Library shall return the assigned 1-based index to the caller on a successful append, so the controller adapter can map it to the WAL LSN. -5. The Raft Library shall not, in this spec, advance any commit index as a side effect of append (commit advance is owned by `raft-commit-safety`). - -### Requirement 3: AppendEntries emission with per-peer next/match index (raft.tla AppendEntries, line 204) - -**Objective:** As a primary replicating to standbys, I want the library to own per-peer replication progress and emit correctly-formed AppendEntries, so that replication carries the `(prevLogIndex, prevLogTerm)` needed for log-matching. - -#### Acceptance Criteria - -1. The Raft Library shall maintain, per peer, a `nextIndex` and a `matchIndex` (the `raft.tla` leaderVars), initialized on becoming leader to `nextIndex = Len(log) + 1` and `matchIndex = 0` for every peer, mirroring `raft.tla` `BecomeLeader`. -2. When the library emits an AppendEntries to peer `j`, it shall compute `prevLogIndex = nextIndex[j] - 1` and `prevLogTerm = (prevLogIndex > 0) ? term-at-prevLogIndex : 0`, mirroring `raft.tla` `AppendEntries`. -3. When the library emits an AppendEntries to peer `j`, it shall send at most one entry, constrained by the end of the log (`SubSeq(log, nextIndex[j], Min(Len(log), nextIndex[j]))`), via the `on_send_append_entries(ctx, peer_idx, prevLogIndex, prevLogTerm, leaderCommit, ...)` callback. -4. The Raft Library shall pass the leader commit value through `on_send_append_entries` as an opaque `leaderCommit` argument and shall not itself interpret or advance commit (commit handling is owned downstream). -5. While the node's role is not `RAFT_LEADER`, the Raft Library shall not emit AppendEntries for that node. - -### Requirement 4: HandleAppendEntriesRequest — logOk, reject, step-down, accept (raft.tla line 327; closes G2, G3) - -**Objective:** As a standby receiving replication, I want the library to accept, reject, or conflict-truncate strictly by `(term, index)` matching, so that Log Matching holds and divergent suffixes are truncated. - -#### Acceptance Criteria - -1. When `raft_on_append_entries_request(...)` is processed, the Raft Library shall compute `logOk = (prevLogIndex == 0) ∨ (prevLogIndex ≤ Len(log) ∧ on_get_entry_term(prevLogIndex) == prevLogTerm)`, mirroring the `raft.tla` `logOk` predicate (closes G2). -2. If the request term is less than `currentTerm`, or the term equals `currentTerm` and the node is a follower and `¬logOk`, then the Raft Library shall reject the request and report failure with `matchIndex = 0`, mirroring the `raft.tla` reject branch. -3. When the request term equals `currentTerm` and the node is a candidate, the Raft Library shall transition the node to follower (return-to-follower branch) without otherwise mutating the log. -4. When the request term equals `currentTerm`, the node is a follower, and `logOk` holds, and the incoming entry already matches the local entry at `index = prevLogIndex + 1` (same term, or no entries sent), the Raft Library shall treat the request as already-satisfied and report success with `matchIndex = prevLogIndex + len(entries)` without rewriting the log. -5. When `logOk` holds and the local entry at `index = prevLogIndex + 1` exists with a **different** term than the incoming entry, the Raft Library shall delete the divergent entry and all entries after it via `on_truncate_from(ctx, index)`, mirroring the `raft.tla` conflict branch (closes G3). -6. When `logOk` holds and the local log ends exactly at `prevLogIndex` (no entry at `prevLogIndex + 1`), the Raft Library shall append the incoming entry via `on_persist_entry` and report success, mirroring the `raft.tla` no-conflict append branch. -7. The Raft Library shall return a reply carrying success/failure and the resulting `matchIndex` so the leader can update its per-peer progress, and shall not perform transport I/O itself (the adapter sends the reply). - -### Requirement 5: HandleAppendEntriesResponse — advance/decrement progress (raft.tla line 393) - -**Objective:** As a primary processing replication replies, I want the library to advance or back off per-peer progress, so that a lagging or divergent follower is walked back to the matching point. - -#### Acceptance Criteria - -1. When `raft_on_append_entries_reply(...)` reports success, the Raft Library shall set `nextIndex[peer] = matchIndex + 1` and `matchIndex[peer] = reply.matchIndex`, mirroring `raft.tla` `HandleAppendEntriesResponse` success. -2. If `raft_on_append_entries_reply(...)` reports failure, then the Raft Library shall set `nextIndex[peer] = Max(nextIndex[peer] - 1, 1)` and leave `matchIndex[peer]` unchanged, mirroring the `raft.tla` failure branch. -3. The Raft Library shall apply a reply only when the reply term equals `currentTerm`, ignoring stale-term responses, consistent with `raft.tla` `HandleAppendEntriesResponse` / `DropStaleResponse`. -4. The Raft Library shall not, in this spec, recompute or advance any commit index from updated `matchIndex` values (owned by `raft-commit-safety`). - -### Requirement 6: UpdateTerm on every RPC including AppendEntries/heartbeat (raft.tla line 406; closes G5) - -**Objective:** As any cluster member, I want a strictly-higher term on **any** incoming RPC — vote, AppendEntries, or heartbeat — to advance my term and step me down, so that a rejoined node can never act or re-elect at a stale term and two leaders can never share a term. - -#### Acceptance Criteria - -1. When an incoming AppendEntries or heartbeat RPC carries a term strictly greater than `currentTerm`, the Raft Library shall set `currentTerm` to that term, clear `voted_for`, and transition to follower, mirroring `raft.tla` `UpdateTerm` (closes G5). -2. While adopting a higher term from any RPC, the Raft Library shall persist the new `(term, voted_for)` via the existing `on_persist` callback before processing the rest of the RPC, preserving the durability ordering already used on the vote path. -3. When a node that was leader adopts a higher term from an AppendEntries/heartbeat RPC, the Raft Library shall fire `on_become_follower` so the controller HA FSM steps down (mirroring the existing higher-term step-down already wired for vote RPCs). -4. The Raft Library shall apply the term-adoption check on the AppendEntries path **before** the `logOk` / accept-reject logic, so a higher-term request is never rejected merely because the local log does not match. -5. The Raft Library shall not, on equal or lower term, alter `currentTerm` or `voted_for` from the AppendEntries path (term adoption is strictly-greater-than only), consistent with `raft.tla`. - -### Requirement 7: Shared callback contract and pure-core / adapter split - -**Objective:** As a downstream spec author (`raft-commit-safety`, `raft-sim-conformance`, `ha-term-safety`), I want the new callbacks and entry points fixed verbatim and the library kept pure, so that I can depend on a stable seam without re-deriving the protocol. - -#### Acceptance Criteria - -1. The Raft Library shall define the required callback `int on_persist_entry(void *ctx, uint64_t index, uint64_t term, const void *entry, size_t len)` to durably append or overwrite the entry at `index`. -2. The Raft Library shall define the required callback `uint64_t on_get_entry_term(void *ctx, uint64_t index)` returning the term of the local entry at `index`, or `0` when absent. -3. The Raft Library shall define the required callback `int on_send_append_entries(void *ctx, int peer_idx, uint64_t prev_log_index, uint64_t prev_log_term, uint64_t leader_commit, ...)` carrying an entries handle, to transmit one AppendEntries. -4. The Raft Library shall define the required callback `int on_truncate_from(void *ctx, uint64_t index)` to delete local entries in `index..end`. -5. The Raft Library shall keep all four callbacks free of protocol decisions: the library decides what/whether to persist, send, or truncate; the controller adapter only performs the I/O (WAL write, transport, recovery read-back). -6. The Raft Library shall leave all existing election callbacks (`on_send_vote_request`, `on_become_leader`, `on_become_follower`, `on_persist`, `on_schedule_timeout`, `on_cancel_timeout`, `on_start_election_after_delay`, optional `on_vote_eligible` / `on_send_pre_vote_request` / `on_trace`) and their behaviour unchanged. -7. The controller adapter shall implement the four callbacks over the existing WAL/repl transport — stamping `hdr.term` from `ha->term`, carrying the term in the repl envelope, reading it back on recovery, and routing `(term, index)` match/truncate decisions to the library — while the heavy controller wiring lives in `ha-term-safety` as a downstream consumer. - -### Requirement 8: Deterministic Log Matching verification in the simulator - -**Objective:** As a maintainer, I want the deterministic simulator to drive replication under adversarial conditions and assert Raft Log Matching, so that the log-half implementation is verifiable without a live cluster. - -#### Acceptance Criteria - -1. The Raft Simulator shall route AppendEntries requests and responses synchronously between nodes (mirroring its existing vote-RPC routing), honoring per-node partition flags. -2. The Raft Simulator shall provide an in-memory per-node log backing the four new callbacks, so a node can append, read entry terms, send, and truncate deterministically. -3. The Raft Simulator shall exercise message drop, duplication, and reordering of AppendEntries, plus node restart, while replication proceeds. -4. The Raft Simulator shall assert the **Log Matching** invariant after replication: for any two nodes and any index `i`, if both hold an entry at `i` with the same term, then their entire prefixes through `i` are byte-identical. -5. When a follower holds a divergent suffix (same index, different term) and then receives the leader's AppendEntries, the Raft Simulator shall assert that the follower truncates the divergent suffix and converges to the leader's prefix. -6. While driving chaotic leadership churn (a stale-term node rejoining under a higher-term leader), the Raft Simulator shall assert that the rejoining node adopts the higher term from the AppendEntries/heartbeat path (G5) and does not re-elect at a reused term. diff --git a/.kiro/specs/raft-log-replication/research.md b/.kiro/specs/raft-log-replication/research.md deleted file mode 100644 index 995f4930..00000000 --- a/.kiro/specs/raft-log-replication/research.md +++ /dev/null @@ -1,34 +0,0 @@ -# Research Log: raft-log-replication - -## Discovery Scope - -Extension of an existing, implemented pure-C library (`src/lib/raft/raft.{c,h}` + `raft_sim.c`). Discovery type: **Extension / integration-focused** — the architecture pattern (pure state machine + callback ports + deterministic simulator) already exists and is authoritative; this feature adds the log half of `raft/raft.tla` to it. No external dependency research needed (the library is free-standing C with only `errno.h`/`stdint.h`/`string.h`). - -## Authoritative Sources - -| Source | Use | -|--------|-----| -| `.kiro/specs/raft-log-replication/brief.md` | Problem, approach, **Authoritative shared callback contract** (binding, reproduced verbatim in design), TLA+ action→entry-point map | -| `raft/raft.tla` | Canonical log actions: `ClientRequest` (246), `AppendEntries` (204), `HandleAppendEntriesRequest` (327), `HandleAppendEntriesResponse` (393), `UpdateTerm` (406), `BecomeLeader` (229) | -| `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` | Gaps G1–G6; this spec closes G1/G2/G3/G5 | -| `src/lib/raft/raft.{c,h}` | Existing election core style: `raft_config_t` extensibility, persist-before-act, NULL-checked required callbacks, status-code returns, `XCE_TESTING` sim | -| `.kiro/steering/roadmap.md` (Raft TLA+ Conformance series) | Family policy: preserve PreVote, priority leadership, sync commit-barrier + persist-before-ACK, controller-never-self-terminates | - -## Key Findings - -1. **The library must NOT store entry bytes.** The election core already declines to store `(term, voted_for)` bytes — the caller restores them via `raft_config_t` and the library writes via `on_persist`. The log follows the same discipline: the library tracks only `log_len` + per-peer `next_index`/`match_index`; entry terms come from `on_get_entry_term`, bytes live in the adapter/simulator. This keeps `raft_state_t` fixed-size and the core pure/deterministic (simulator-verifiable). **Decision**: no per-entry array in `raft_state_t`. - -2. **Index ↔ LSN is 1:1 and 1-based.** `raft.tla` logs are 1-based; the controller WAL LSN is the natural mapping. Settling this in the contract (R1.2) means the controller adapter and simulator address entries identically. `on_get_entry_term(0)` / `prevLogIndex==0` → term `0` is the absent sentinel, matching `raft.tla`'s `prevLogTerm == 0` when `prevLogIndex == 0`. - -3. **UpdateTerm must precede logOk on the AE path (G5).** The bug-851 mechanism: a rejoined standby never adopted the leader's term from heartbeats, re-elected at a stale+1 term, and reused a term → two leaders/term. `raft.tla` `Receive` applies `UpdateTerm` before any handler. **Decision**: the request handler applies term adoption first, unconditionally, including for 0-entry heartbeats — otherwise a higher-term request could be rejected for a log mismatch and never advance the term (R6.4). - -4. **`leader_commit` is carried but not acted on.** Commit advance (G4) is `raft-commit-safety`. To avoid a later signature change (a Revalidation Trigger for all downstream consumers), the AppendEntries entry points accept `leader_commit` opaquely **now** and ignore it. This stabilizes the wire/sim shape across the two specs. **Decision (build-vs-defer)**: include the parameter, defer the behavior. - -5. **`(term, index)` truncation replaces the LSN-only reconcile.** The controller's `ha_convergence.c` truncates by a committed-boundary heuristic (G3) and cannot see same-LSN/different-term divergence. The library's conflict branch (`on_truncate_from(index)` + re-append) is the exact `raft.tla` semantics. The heavy controller swap is `ha-term-safety`; this spec owns the library decision + the simulator proof. - -## Synthesis Outcomes - -- **Generalization found**: the existing "pure core, I/O via callbacks, restore-state-via-config, never-read-storage" pattern generalizes cleanly from election state to log state. No new architectural concept is introduced — only new ports and new pure transitions. -- **Build-vs-adopt**: adopt the canonical `raft.tla` algorithm verbatim (it is the verified reference). Build only the C translation + the in-memory simulator backing. No third-party Raft library is imported (the project's constraint is a free-standing, simulator-verifiable core). -- **Simplification**: keep the "≤1 entry per AppendEntries" `raft.tla` simplification rather than batching — it minimizes atomic regions, matches the spec exactly, and keeps the simulator's Log Matching assertion tractable. Batching is an adapter-side optimization that can be layered later without changing the core contract. -- **Risk**: a future signature change to the four callbacks or the three entry points cascades to three downstream specs (logged as Revalidation Triggers). Mitigated by reproducing the brief's contract verbatim and by accepting `leader_commit` now. diff --git a/.kiro/specs/raft-log-replication/spec.json b/.kiro/specs/raft-log-replication/spec.json deleted file mode 100644 index 5556247c..00000000 --- a/.kiro/specs/raft-log-replication/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "raft-log-replication", - "created_at": "2026-06-21T11:00:00Z", - "updated_at": "2026-06-25T00:00:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "implementation_status": "implemented — library log half landed + validated (24/24 tasks, raft_sim GO per memory project_raft_log_replication_done). Phase reconciled 2026-06-25 (was mislabeled tasks-generated).", - "note": "Auto-approved (-y mode). Extends election-only src/lib/raft into a strict raft.tla log-half implementation: ClientRequest/AppendEntries/HandleAppendEntriesRequest/HandleAppendEntriesResponse/UpdateTerm-on-any-RPC. Closes bug-851 G1/G2/G3/G5. Authoritative shared callback contract (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from) consumed verbatim by raft-commit-safety, raft-sim-conformance, and ha-term-safety. Boundary: owns src/lib/raft log additions + (term,index) reconcile seam only; NOT commit-index advance, vote freshness, config, or controller WAL/transport internals." -} diff --git a/.kiro/specs/raft-log-replication/tasks.md b/.kiro/specs/raft-log-replication/tasks.md deleted file mode 100644 index e7dd2608..00000000 --- a/.kiro/specs/raft-log-replication/tasks.md +++ /dev/null @@ -1,205 +0,0 @@ -# Implementation Plan - -> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/lib/testsuite/test_raft.c` (and the simulator backing in `raft_sim.c`) that fails before the `raft.c` change and passes after. Build/run library tests via `/xce-build lib`. -> -> **Spec boundary** (applies to ALL tasks): this spec owns the `src/lib/raft` log additions (per-entry term in the library model, `ClientRequest`/`AppendEntries`/`Handle*`/`UpdateTerm`-on-AE) and the `(term, index)` reconcile seam (the four callbacks). It does **NOT** own commit-index advance, `(lastLogTerm,lastLogIndex)` vote freshness, config markers/upgrade snapshot, or the controller's WAL/repl/recovery internals — those are `raft-commit-safety` and `ha-term-safety`. No task here edits any controller file; the controller adapter is described, not built. - -## 1. Header contract: log model, callbacks, entry points - -- [x] 1.1 Extend `raft.h` with the log model state and config restore field - - Add `uint64_t log_len;`, `uint64_t next_index[RAFT_MAX_PEERS];`, `uint64_t match_index[RAFT_MAX_PEERS];` to `raft_state_t` - - Add `uint64_t initial_log_len;` to `raft_config_t` (restored WAL length at boot, default 0; library never reads storage) - - Observable completion: `raft.h` compiles; `sizeof(raft_state_t)` grows by the new fixed fields; no entry-byte storage added - - _Requirements: 1.1, 1.2, 3.1_ - - _Boundary: raft.h_ - -- [x] 1.2 Add the four new REQUIRED callbacks to `raft_callbacks_t` (verbatim contract) - - Declare `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from` with the exact signatures from design.md / brief - - Document each as REQUIRED and the `on_get_entry_term` → `0`-means-absent convention - - Observable completion: the four function-pointer members exist in `raft_callbacks_t` with the verbatim signatures; header doc comments match the Authoritative shared contract - - _Requirements: 7.1, 7.2, 7.3, 7.4_ - - _Boundary: raft.h_ - - _Depends: 1.1_ - -- [x] 1.3 Declare the three new entry points and `raft_append_result_t` - - NOTE (impl 2026-06-21): `on_send_append_entries`'s design "entries handle `...`" varargs placeholder was concretized to an explicit `int n_entries` (0=heartbeat, 1=one entry at `prev_log_index+1`; adapter sources the bytes since the library holds none). design.md §"Seam — raft_callbacks_t additions" updated to match. Downstream `raft-commit-safety` / `raft-sim-conformance` consume this concrete shape. - - Declare `raft_append_result_t`; declare `raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`, `raft_on_append_entries_reply` with the design.md signatures - - Observable completion: header declares all new entry points; a stub TU including `raft.h` compiles - - _Requirements: 2.1, 3.3, 4.7, 5.1_ - - _Boundary: raft.h_ - - _Depends: 1.2_ - -## 2. Init validation and leader-progress initialization - -- [x] 2.1 RED+GREEN: `raft_init` requires the four new callbacks - - Test: `raft_init` returns `-EINVAL` when any of the four new callbacks is NULL; returns 0 when all present - - Implement the NULL checks in `raft_init` (mirroring the existing required-callback checks); restore `s->log_len = cfg->initial_log_len` - - Observable completion: new `-EINVAL` test cases pass; existing election init tests still green - - _Requirements: 7.1, 7.2, 7.3, 7.4, 1.1_ - - _Boundary: raft.c, test_raft.c_ - - _Depends: 1.3_ - -- [x] 2.2 RED+GREEN: leader progress initialized on becoming leader - - Test: after a node becomes leader, `next_index[j] == log_len + 1` and `match_index[j] == 0` for every peer (raft.tla BecomeLeader) - - Implement the `next_index`/`match_index` initialization on the win path (`raft_start_election` single-node + the multi-node become-leader transition) - - Observable completion: assertion on `next_index`/`match_index` after election passes; no regression in election tests - - _Requirements: 3.1_ - - _Boundary: raft.c, test_raft.c_ - - _Depends: 2.1_ - -## 3. ClientRequest — raft_log_append (raft.tla:246) - -- [x] 3.1 RED: append tests (leader stamps term, follower rejected) - - Test: leader `raft_log_append` assigns index `log_len+1`, stamps `currentTerm`, calls `on_persist_entry` exactly once with `(index, term, entry)`, returns the index; follower call returns `-EPERM` with no log change and no callback - - Observable completion: tests compile and FAIL (no implementation yet) - - _Requirements: 2.1, 2.2, 2.3, 2.4_ - - _Boundary: test_raft.c_ - - _Depends: 2.2_ - -- [x] 3.2 GREEN: implement `raft_log_append` - - Role guard (`-EPERM` if not leader); persist-before-act (`on_persist_entry` before incrementing `log_len`); set `*out_index`; do NOT advance any commit index - - Observable completion: 3.1 tests pass; `log_len` only advances on a successful persist (no phantom entry on persist failure) - - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_ - - _Boundary: raft.c_ - - _Depends: 3.1_ - -## 4. AppendEntries emission — raft_send_append_entries (raft.tla:204) - -- [x] 4.1 RED: emit tests (prevLogIndex/prevLogTerm, ≤1 entry, leader-only) - - Test: for peer `j`, emit computes `prevLogIndex = next_index[j]-1` and `prevLogTerm` via `on_get_entry_term` (0 when `prevLogIndex==0`), sends ≤1 entry, passes `leader_commit` through opaquely; a non-leader emits nothing - - Observable completion: tests compile and FAIL - - _Requirements: 3.2, 3.3, 3.4, 3.5_ - - _Boundary: test_raft.c_ - - _Depends: 2.2_ - -- [x] 4.2 GREEN: implement `raft_send_append_entries` - - Compute prev index/term; `lastEntry = Min(log_len, next_index[j])`; one entry or heartbeat; call `on_send_append_entries` with the opaque `leader_commit`; leader-only guard - - Observable completion: 4.1 tests pass; heartbeat (0 entries) emitted when `next_index[j] > log_len` - - _Requirements: 3.2, 3.3, 3.4, 3.5_ - - _Boundary: raft.c_ - - _Depends: 4.1_ - -## 5. UpdateTerm-on-AppendEntries (raft.tla:406, closes G5) - -- [x] 5.1 RED: higher-term AE/heartbeat adopts term before logOk - - Test: a follower and a leader at term T receiving an AE (and separately a 0-entry heartbeat) at term T+1 adopt T+1, clear `voted_for`, call `on_persist(T+1,-1)`, and (was-leader case) fire `on_become_follower`; assert this happens even when the local log would fail `logOk` - - Observable completion: tests compile and FAIL - - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_ - - _Boundary: test_raft.c_ - - _Depends: 1.3_ - -- [x] 5.2 GREEN: term-adoption block at the top of the request handler - - Implement the `UpdateTerm` block as the FIRST step of `raft_on_append_entries_request`, before `logOk`; persist-before-process; fire `on_become_follower` if was leader; no change on equal/lower term - - Observable completion: 5.1 tests pass; the block is unmistakably before `logOk` in `raft.c` - - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_ - - _Boundary: raft.c_ - - _Depends: 5.1_ - -## 6. HandleAppendEntriesRequest — logOk / reject / step-down / accept (raft.tla:327, closes G2/G3) - -- [x] 6.1 RED: logOk + reject + candidate step-down tests - - Test: `logOk` true when `prevLogIndex==0` or `on_get_entry_term(prevLogIndex)==prevLogTerm`; reject (success=0, matchIndex=0) on stale term and on `follower && !logOk`; candidate at equal term transitions to follower - - Observable completion: tests compile and FAIL - - _Requirements: 4.1, 4.2, 4.3_ - - _Boundary: test_raft.c_ - - _Depends: 5.2_ - -- [x] 6.2 RED: accept branch tests (already-have / conflict-truncate / append) - - Test: already-have (empty entries or matching term) → success, no log rewrite, `matchIndex = prevLogIndex + entries_len`; conflict (`local[index].term != entry.term`) → `on_truncate_from(index)` then `on_persist_entry(index, entry)`, success `matchIndex=index`; append (`log_len == prevLogIndex`) → `on_persist_entry(index)`, success - - Observable completion: tests compile and FAIL - - _Requirements: 4.4, 4.5, 4.6, 4.7_ - - _Boundary: test_raft.c_ - - _Depends: 6.1_ - -- [x] 6.3 GREEN: implement `raft_on_append_entries_request` (logOk + all branches) - - Implement (after the 5.2 UpdateTerm block): `logOk` via `on_get_entry_term`; reject/candidate-step-down/accept; the three accept sub-cases exactly per raft.tla:351-388; fill `raft_append_result_t`; never advance commit (`leader_commit` ignored); never perform transport I/O - - Observable completion: 6.1 and 6.2 tests pass; `on_truncate_from` is always followed by re-persist of the leader entry on the conflict branch - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7_ - - _Boundary: raft.c_ - - _Depends: 6.2_ - -## 7. HandleAppendEntriesResponse — raft_on_append_entries_reply (raft.tla:393) - -- [x] 7.1 RED: reply advance/decrement/stale-drop tests - - Test: success sets `next_index[peer]=matchIndex+1`, `match_index[peer]=reply.matchIndex`; failure sets `next_index[peer]=Max(next_index[peer]-1,1)` (floor at 1), `match_index` unchanged; a reply with `reply_term != currentTerm` is ignored - - Observable completion: tests compile and FAIL - - _Requirements: 5.1, 5.2, 5.3_ - - _Boundary: test_raft.c_ - - _Depends: 2.2_ - -- [x] 7.2 GREEN: implement `raft_on_append_entries_reply` - - Stale-term drop; success/failure branches; `next_index` floored at 1; do NOT recompute or advance any commit index from `match_index` - - Observable completion: 7.1 tests pass - - _Requirements: 5.1, 5.2, 5.3, 5.4_ - - _Boundary: raft.c_ - - _Depends: 7.1_ - -## 8. Simulator backing: in-memory log + AppendEntries routing - -- [x] 8.1 Add an in-memory per-node log and wire the four callbacks in `raft_sim.c` - - NOTE (impl 2026-06-21): pulled forward and landed together with 2.1 — task 2.1's stricter `raft_init` (four callbacks now REQUIRED) makes `raft_sim_init` return -EINVAL, so ~15 `test_sim_*`/`test_proto_*` tests go red the instant 2.1 lands. 8.1's callback wiring only needs the header signatures (not 6.3/7.2), so it compiles standalone; co-committing 2.1+8.1 keeps the suite green at every commit. AE delivery (8.2) still deferred. - - Add a per-`raft_sim_node_t` in-memory entry array (`{term, bytes}` per index); implement `on_persist_entry` (append/overwrite at index), `on_get_entry_term` (term or 0), `on_truncate_from` (drop index..end), and buffer `on_send_append_entries` into a new AE RPC queue - - Observable completion: a sim node can append, read a term back, and truncate deterministically; `raft_sim_init` wires the four callbacks for every node - - _Requirements: 8.2, 7.5_ - - _Boundary: raft_sim.c_ - - _Depends: 6.3, 7.2_ - -- [x] 8.2 Add synchronous AppendEntries delivery honoring partitions - - NOTE (impl 2026-06-21): `raft_sim_ae_t` now carries `leader_term` captured at SEND time (in `sim_send_append_entries`) and delivery uses `ae->leader_term`, NOT the live source `state.term`. Required so reorder (9.3) and stale-term churn (9.4) deliver the sending-epoch term — using the live term would bypass UpdateTerm/stale-drop and defeat the G5 regression. Caught in review. - - Add `raft_sim_deliver_append_entries`: for each buffered AE, call `raft_on_append_entries_request` on the (non-partitioned) destination, then route the reply back via `raft_on_append_entries_reply`; skip partitioned nodes (mirrors the existing vote-RPC routing) - - Observable completion: driving one append + delivery replicates the entry to a connected follower; partitioned follower receives nothing - - _Requirements: 8.1_ - - _Boundary: raft_sim.c_ - - _Depends: 8.1_ - -- [x] 8.3 Add a Log Matching assertion helper - - Add a helper asserting: for any two nodes and any index `i`, if both hold an entry at `i` with equal term, their prefixes through `i` are byte-identical - - Observable completion: the helper returns pass on a converged cluster and fails on a hand-divergent one (self-test) - - _Requirements: 8.4_ - - _Boundary: raft_sim.c_ - - _Depends: 8.1_ - -## 9. Simulator invariant tests (Log Matching, truncation, faults, G5 regression) - -- [x] 9.1 Replication + partition catch-up Log Matching test - - Test: leader appends 3 entries, replicates to a quorum, assert Log Matching; then isolate a follower across more appends, heal, assert `nextIndex` walks back, the follower catches up, and Log Matching holds - - Observable completion: test passes end-to-end via the sim - - _Requirements: 8.1, 8.3, 8.4_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 8.2, 8.3_ - -- [x] 9.2 Divergent-suffix truncation test (G3) - - Test: give a follower a manufactured divergent tail (same indices, different term), replicate from a higher-term leader; assert the follower truncates the divergent suffix (`on_truncate_from`) and converges to the leader's prefix; Log Matching holds - - Observable completion: test passes; follower's post-converge prefix equals the leader's - - _Requirements: 8.5_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 9.1_ - -- [x] 9.3 Message drop/dup/reorder convergence test - - Test: drop, duplicate, and reorder AE requests/replies in the sim queue; assert idempotent convergence (a duplicated AE is a no-op via the already-have branch) and Log Matching after faults clear - - Observable completion: test passes despite injected faults - - _Requirements: 8.3_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 9.1_ - -- [x] 9.4 Chaotic-churn G5 regression test - - Test: a stale-term node (term lagging) rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does NOT re-elect at a reused term; Log Matching holds across the churn - - Observable completion: test passes; the rejoining node's term equals the leader's after the first heartbeat, and no second leader appears at the reused term - - _Requirements: 8.6_ - - _Boundary: test_raft.c, raft_sim.c_ - - _Depends: 9.1_ - -## 10. Verification - -- [x] 10.1 Full library suite green under ASAN - - Run `/xce-build lib`; all existing election tests plus the new log-replication and simulator tests pass under ASAN with zero leaks - - Observable completion: test binary exits 0; ASAN clean; no regression in the election-core tests - - _Requirements: 1.1, 2.1, 3.2, 4.3, 5.1, 6.1, 7.1, 8.4_ - - _Boundary: src/lib/raft, src/lib/testsuite_ - - _Depends: 9.2, 9.3, 9.4_ - -## Implementation Notes - -- **Validation (2026-06-21): GO.** Clean ASAN build green from scratch (`xce-build.sh lib`, Fail 0); 8/8 requirement sections covered (40/41 acceptance criteria); boundary ALIGNED (no controller files; no commit advance / vote freshness / transport I/O / entry-bytes in library state); out-of-scope items confirmed absent. -- **Req 8.3 "node restart while replication proceeds" — CLOSED (2026-06-21):** added `raft_sim_restart_node` (durable log[]/(term,voted_for) survive; volatile consensus state cleared via raft_init → FOLLOWER) + `test_sim_9_5_node_restart_resumes_replication` (restart a follower mid-replication with a partial log; assert log_count preserved — load-bearing: would fail if the log were wiped — role/term restored, replication resumes, Log Matching holds). drop/dup/reorder remain covered by 9.3. All 8/8 sections now fully covered. -- **Adapter invariant (from validation LOW note):** the AE accept already-have branch treats `on_get_entry_term(index) == entry_term` as a match, which is also true when both are 0. A correct leader never sends `entry_term == 0` for a real entry (`raft_log_append` stamps `s->term >= 1`), so this is safe in-library — but the controller adapter (raft-commit-safety) MUST guarantee it never delivers `entry_term == 0` with `entries_len == 1`. diff --git a/.kiro/specs/raft-sim-conformance/brief.md b/.kiro/specs/raft-sim-conformance/brief.md deleted file mode 100644 index ebf0e23c..00000000 --- a/.kiro/specs/raft-sim-conformance/brief.md +++ /dev/null @@ -1,26 +0,0 @@ -# Brief: raft-sim-conformance - -## Problem -Once the library owns the full Raft protocol (election via `raft-library`, log via `raft-log-replication`, commit safety via `raft-commit-safety`), XCE needs an **independent verification harness** that proves the consensus core conforms to `raft/raft.tla` — without a live cluster, threads, or sockets. Today `raft_sim.c` exists but only exercises the election half; there is no oracle for the four Raft safety invariants. This is the "use the verified spec to help" payoff: a deterministic regression oracle that would have caught **bug-851** before it shipped. - -## Approach -Extend `src/lib/raft/raft_sim.c` into a **conformance harness** that drives N library instances purely in-process through transitions equivalent to the `raft.tla` `Next` actions, and continuously checks the TLA+ safety invariants. - -### TLA+ actions to drive (raft.tla `Next`, line 454) -`Restart`, `Timeout`, `RequestVote`, `BecomeLeader`, `ClientRequest`, `AppendEntries`, `AdvanceCommitIndex`, `Receive` (vote/AE request+response, `UpdateTerm`, drop-stale), `DuplicateMessage`, `DropMessage`. The harness owns a message bag and per-node state mirroring `raft.tla`'s variables. - -### Invariants asserted every step (the four Raft safety properties) -1. **Election Safety** — at most one leader per term. -2. **Log Matching** — if two logs share an entry at `(index, term)`, all prior entries are identical. -3. **Leader Completeness** — an entry committed in term T is present in every leader's log for terms > T. -4. **State Machine Safety** — no two nodes apply different commands at the same index. - -## Scope -- **In**: a deterministic, seedable driver (randomized + targeted scenarios: partitions, restarts, message drop/dup/reorder, Figure-8); per-step invariant checks; failure reproduction (print the action trace that broke an invariant); CI-runnable test target alongside the existing library tests. Optionally: emit a trace consumable by TLC, or compare against TLC-generated traces of `raft.tla`. -- **Out**: the protocol implementation itself (owned by the three upstream specs); the live controller / `/xce-deploy-ha` integration (that is `ha-term-safety`'s chaotic-failover regression); config markers/snapshot. - -## Verification -The harness IS the verification. Acceptance: it reproduces a bug-851-class divergence on a deliberately-broken build (e.g. term stamping disabled, or `(term,lsn)` truncation removed) and passes on the correct build; it runs M randomized seeds in CI with zero invariant violations. - -## Dependencies -`raft-log-replication`, `raft-commit-safety` (HARD — the harness checks invariants over the log + commit behaviour they add). Complements `ha-term-safety`'s live regression (this is the in-process oracle; that is the end-to-end one). diff --git a/.kiro/specs/raft-sim-conformance/design.md b/.kiro/specs/raft-sim-conformance/design.md deleted file mode 100644 index d2ab97e9..00000000 --- a/.kiro/specs/raft-sim-conformance/design.md +++ /dev/null @@ -1,594 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: This feature turns `src/lib/raft/raft_sim.c` (today an election-only RPC -router) into an **in-process TLA+ conformance harness** for the XCE Raft core. It drives -N library instances synchronously through transitions equivalent to the `raft.tla` -`Next` actions (`raft.tla:454`) and asserts the four Raft safety invariants after every -step. It delivers an independent regression oracle for the consensus core — the "use the -verified spec to help" payoff — that would have caught **bug-851** (the silent -committed-prefix divergence) before it shipped. - -**Users**: Raft library maintainers (verifying `raft.c`), the `raft-log-replication` and -`raft-commit-safety` authors (whose entry points this harness exercises), and CI (which -runs the harness as a first-class library test target). - -**Impact**: Replaces ad-hoc election-only simulation with a complete in-process model of -the `raft.tla` state space: a shared message bag mirroring `messages`, per-node state -mirroring `serverVars`/`logVars`/`leaderVars`, an action scheduler, four invariant -checkers, and a failure-trace recorder. The protocol implementation is unchanged — the -harness drives it only through the published entry points and callbacks from the two -upstream specs; the harness itself **is** the in-memory adapter (it implements the WAL/ -transport callbacks against in-memory structures). - -### Goals - -- A deterministic, seedable driver that mirrors the `raft.tla` `messages` bag and the - per-node variables this harness tracks, driving the library entry points only. -- Per-step assertion of all four safety invariants (Election Safety, Log Matching, Leader - Completeness, State Machine Safety) computed purely from observable node state. -- Targeted adversarial scenarios (partition/heal, restart, drop/dup/reorder, Figure-8, - bug-851 G5 term-reuse) plus randomized seed sweeps. -- A replayable failure trace (seed + ordered action list) and a CI-runnable target. -- A deliberately-broken-build negative test proving the oracle catches a bug-851-class - divergence and passes on the correct build. -- Optionally, a TLC-consumable trace export. - -### Non-Goals - -- **No protocol semantics.** logOk, conflict truncation, the current-term commit rule, - `(lastLogTerm, lastLogIndex)` freshness, and `UpdateTerm` adoption live in `raft.c` - (owned by `raft-log-replication` / `raft-commit-safety`). The harness drives and checks - them; it never re-implements them. -- **No live cluster / controller integration.** The `/xce-deploy-ha` chaotic-failover - regression, controller WAL/repl/recovery/commit-barrier internals, config markers, and - the upgrade snapshot are `ha-term-safety`. -- **No changes to `raft/raft.tla`** and **no TLC model-checker harness** for the spec - (only an *optional export* the maintainer can feed to TLC manually). -- **No async I/O, threads, or sockets.** The harness is synchronous and deterministic by - construction. - -## Boundary Commitments - -### This Spec Owns - -- `src/lib/raft/raft_sim.c` — the harness body: the shared message bag, per-node - in-memory log + commit + applied state, the in-memory adapter callbacks, the seeded - action scheduler, the four invariant checkers, the trace recorder, the scenario - generators, and the deliberately-broken-build fault switch wiring (test-only). -- The harness public surface in `raft.h`'s `XCE_TESTING` section: new sim types - (message-bag entry, per-node log array, action record) and the new `raft_sim_*` driver - functions. These extend — never alter — the protocol declarations. -- The conformance test entry point + scenarios in `src/lib/testsuite/test_raft.c` - (`XCE_TESTING`-gated), including the negative (broken-build) test. - -### Out of Boundary - -- The protocol entry points and callback **semantics** (`raft_log_append`, - `raft_send_append_entries`, `raft_on_append_entries_request/reply`, - `raft_advance_commit`, the vote path, `on_persist_entry`, `on_get_entry_term`, - `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) — owned by - `raft-log-replication` / `raft-commit-safety`. This spec consumes them verbatim. -- Any `raft.c` change to protocol logic. The harness must not need one; if it does, that - is an upstream-spec gap, not a harness change. -- The controller-side adapters, WAL byte layout, repl transport, recovery, commit - barrier, config markers, snapshot — `ha-term-safety`. -- `raft/raft.tla` itself and any TLC tooling. - -### Allowed Dependencies - -- The `raft-log-replication` shared callback contract **verbatim**: `on_persist_entry`, - `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`; the 1-based index = - WAL LSN identity; the `term 0` = absent convention; per-peer `next_index[]`/ - `match_index[]`; entry points `raft_log_append`, `raft_send_append_entries`, - `raft_on_append_entries_request`, `raft_on_append_entries_reply`, - `raft_append_result_t`. -- The `raft-commit-safety` additions: `raft_advance_commit` (current-term-gated - `AdvanceCommitIndex`), `on_commit_advanced`, and the `(lastLogTerm, lastLogIndex)` vote - freshness already enforced inside the vote handlers. -- The implemented election core (`raft-library`): `raft_init`, `raft_start_election`, - `raft_on_vote_request/reply`, `raft_on_election_timeout`, PreVote, `raft_snapshot`. -- Only `errno.h`, `stdint.h`, `string.h`, `stdio.h`, `unity/unity.h` (test) — the - simulator stays free-standing, like the rest of the library, and is `XCE_TESTING`-gated. - -### Revalidation Triggers - -Re-check this design if any upstream changes: - -- The signature/semantics of any of the four log-replication callbacks or - `on_commit_advanced`. -- The signature/return semantics of `raft_log_append`, `raft_send_append_entries`, - `raft_on_append_entries_request`, `raft_on_append_entries_reply`, `raft_advance_commit`, - or `raft_append_result_t`. -- The 1-based-index ↔ WAL-LSN identity or the `term 0` = absent convention. -- The placement of `UpdateTerm` / `DropStaleResponse` inside the handlers (the harness - relies on the handlers performing these, not the harness). - -## Architecture - -### Existing Architecture Analysis - -`raft_sim.c` already establishes the harness idiom this design extends: - -- A `raft_sim_cluster_t` owning `nodes[RAFT_SIM_MAX_NODES]`, a `pending[]` RPC queue, and - cluster-level counters. -- Each `raft_sim_node_t` carries `raft_state_t state`, a `raft_callbacks_t cbs` wired to - sim routing functions, a back-pointer `ctx` to the cluster, an `idx`, a `partition` - flag, and a `trace_buf`. -- Callbacks cast `ctx` back to `raft_sim_node_t*`, reaching the cluster via `node->ctx`. -- Routing is synchronous: `raft_sim_deliver_pending` drains the queue, calling - `raft_on_vote_request` on the destination and routing the reply back. - -This design keeps that idiom and adds: a richer message bag (vote **and** AppendEntries, -requests **and** responses), an in-memory log + commit + applied state per node, the four -new adapter callbacks, a seeded action scheduler, invariant checkers, and a trace -recorder. No new file is introduced; the existing module is extended (consistent with how -`raft-log-replication` extends `raft_sim.c`). - -### TLA+ variable → harness state mapping - -The harness mirrors exactly the `raft.tla` variables it needs to drive actions and check -invariants. The protocol-internal portions of `serverVars`/`leaderVars` (`currentTerm`, -`votedFor`, `state`, `nextIndex`, `matchIndex`) live inside each `raft_state_t` and are -**read** through `raft_snapshot` and the library's own fields — the harness does not -duplicate them. - -| `raft.tla` variable | Harness home | Read/owned | -|---|---|---| -| `messages` (bag of in-flight RPCs) | `cluster.bag[]` (requests + responses) | harness-owned | -| `currentTerm[i]`, `state[i]`, `votedFor[i]` | `node.state` (`raft_state_t`) | library-owned, read via snapshot/fields | -| `log[i]` (sequence of `(index, term, value)`) | `node.log[]` in-memory array | harness-owned (the adapter) | -| `commitIndex[i]` | `node.commit_index` | harness-owned, advanced via `raft_advance_commit` + `on_commit_advanced` | -| `nextIndex[i][j]`, `matchIndex[i][j]` | `node.state.next_index[]/match_index[]` | library-owned | -| applied state machine (for SM Safety) | `node.applied[]` (value per committed index) | harness-owned | -| `allLogs` (history of every log) | `cluster.all_logs` snapshot used by checkers | harness-owned | - -### Component / data-flow map - -```mermaid -flowchart TB - subgraph SCHED["Action Scheduler (raft_sim.c) — seeded PRNG"] - PICK["pick action ~ raft.tla Next (454)"] - REC["record action -> trace buffer (R1.4)"] - end - subgraph NODES["N library instances (raft.c — UNCHANGED)"] - ST["raft_state_t (term, role, votedFor, next/match)"] - EP["entry points: start_election, on_vote_*,<br/>log_append, send_append_entries,<br/>on_append_entries_*, advance_commit"] - end - subgraph ADAPT["In-memory adapter callbacks (raft_sim.c = the adapter)"] - A1["on_persist_entry -> node.log[]"] - A2["on_get_entry_term -> node.log[idx].term | 0"] - A3["on_send_append_entries -> bag.push(AE req)"] - A4["on_truncate_from -> node.log truncate"] - A5["on_commit_advanced -> node.commit_index + apply()"] - A6["on_send_vote_request -> bag.push(vote req)"] - end - subgraph BAG["Message bag (cluster.bag[]) — mirrors raft.tla messages"] - B["vote/AE requests + responses,<br/>Drop/Duplicate/Reorder live here"] - end - subgraph CHK["Invariant checkers (after every step, R3)"] - I1["Election Safety"] - I2["Log Matching"] - I3["Leader Completeness"] - I4["State Machine Safety"] - end - PICK --> REC --> EP - EP --> A1 & A2 & A3 & A4 & A5 & A6 - A3 & A6 --> BAG - BAG -->|Receive/Deliver| EP - EP --> CHK - CHK -->|violation| FAIL["halt + dump seed & trace (R5)"] -``` - -**Architecture integration** - -- **Pattern**: model-based test harness over a pure state machine. The library is the - system-under-test (Ports-and-Adapters: the harness implements all callback ports - against in-memory structures). The scheduler is a seeded interleaving engine over the - `raft.tla` action set; the checkers are pure observers. -- **Domain boundary**: protocol decisions stay in `raft.c`; *driving* (which action, - which message) and *checking* (the four invariants) stay in the harness. -- **Preserved patterns**: `ctx`-back-pointer callbacks, synchronous routing, - `XCE_TESTING` gating, per-node trace buffers, the `raft_sim_*` naming. -- **Steering compliance**: `src/lib/CLAUDE.md` (free-standing gnu17, tests in - `testsuite/`), `ha-async-io.md` (no blocking — and here, no async at all), - `testing.md` (TDD: checkers/scenarios written before they pass), "controller never - self-terminates" (the harness `abort()`s a test on violation, never the library). - -### Technology Stack - -| Layer | Choice | Role | Notes | -|---|---|---|---| -| Harness core | C (gnu17), `string.h`/`stdint.h`/`stdio.h` | message bag, scheduler, checkers, trace | `XCE_TESTING`-gated; free-standing | -| PRNG | small inlined deterministic generator (e.g. SplitMix64/xorshift) seeded by a `uint64_t` | all nondeterministic choices | no libc `rand()` (not reproducible across platforms) | -| Adapter | in-memory `node.log[]`, `commit_index`, `applied[]` | implements the 6 callbacks | the harness IS the WAL/transport adapter | -| Test runner | Unity (`unity/unity.h`) in `test_raft.c` | scenario + seed-sweep + negative test | built by `src/lib/meson.build` test target | -| Fault switch | compile-time macro (e.g. `RAFT_SIM_BREAK_*`) in the test build only | negative test | never set in default CI run | - -## File Structure Plan - -### Modified Files - -- `src/lib/raft/raft.h` (`XCE_TESTING` section only) — extend the sim types: a unified - message-bag entry (`raft_sim_msg_t`: kind = vote-req/vote-rsp/ae-req/ae-rsp, src, dst, - and a union of payload fields incl. term, prev_log_index/term, leader_commit, entry - bytes + entry_term, match_index, granted/success); a per-node in-memory log entry - (`raft_sim_log_entry_t { uint64_t term; uint64_t value; }`); an action record - (`raft_sim_action_t`: kind + actor + target/message identity); extend - `raft_sim_node_t` with `log[]`, `log_len`, `commit_index`, `applied[]`; extend - `raft_sim_cluster_t` with the bag, the PRNG state, the trace buffer, and the seed; - declare the new `raft_sim_*` driver/scenario/check functions. **No protocol - declaration is changed.** -- `src/lib/raft/raft_sim.c` — the harness implementation (sections below): adapter - callbacks, bag operations (push/deliver/drop/duplicate/reorder), the seeded action - scheduler (`raft_sim_step` / `raft_sim_run`), the four invariant checkers - (`raft_sim_check_*` + `raft_sim_check_all`), the trace recorder + dumper, the targeted - scenario drivers, and the fault-switch hooks (compile-time-gated). -- `src/lib/testsuite/test_raft.c` — the conformance test cases: targeted scenarios, - the randomized M-seed sweep, the replay-reproduction test, and the negative - (broken-build) test. - -### Directory Structure (unchanged) - -``` -src/lib/raft/ -├── raft.h # + sim message bag, in-memory log, action record, driver decls (XCE_TESTING) -├── raft.c # UNCHANGED by this spec (protocol owned upstream) -└── raft_sim.c # + bag, scheduler, 4 invariant checkers, trace, scenarios, fault hooks -src/lib/testsuite/ -└── test_raft.c # + conformance scenarios, M-seed sweep, replay test, negative test -``` - -> Single-responsibility kept: `raft.c` = pure protocol (untouched), `raft_sim.c` = -> deterministic harness + in-memory adapter + checkers, `test_raft.c` = assertions and -> the CI batch. - -## System Flows - -### One scheduler step (the `Next` interleaving) - -```mermaid -sequenceDiagram - participant S as Scheduler (seeded PRNG) - participant L as Library node (raft.c) - participant A as Adapter cb (raft_sim.c) - participant B as Message bag - participant C as Checkers - S->>S: action = pick(Next set) from PRNG - S->>S: record(action) into trace (R1.4) - alt node-local (Timeout / RequestVote / BecomeLeader / ClientRequest) - S->>L: call entry point (start_election / log_append / ...) - L->>A: on_persist_entry / on_send_* (drives bag/log) - A->>B: push request(s) - else AppendEntries (leader replicate) - S->>L: raft_send_append_entries(peer, leader_commit) - L->>A: on_send_append_entries -> bag.push(AE req) - else AdvanceCommitIndex - S->>L: raft_advance_commit() - L->>A: on_commit_advanced -> commit_index advanced, apply() - else Receive (deliver one bag message) - S->>B: pop selected message - S->>L: route to handler (on_vote_request / on_append_entries_request / *_reply) - L->>A: UpdateTerm/persist/truncate/append + reply push - else DropMessage / DuplicateMessage - S->>B: discard / re-enqueue (no node mutation) (R2.5) - else Restart - S->>L: raft_init preserving persisted term/votedFor + restore log_len (R2.1) - end - S->>C: raft_sim_check_all() (R3) - alt any invariant violated - C-->>S: fail -> dump seed + trace + invariant (R5), then halt - else ok - C-->>S: continue - end -``` - -Flow-level decisions: - -- **The library performs the protocol; the harness performs the interleaving.** - `UpdateTerm` and `DropStaleResponse` happen *inside* the handlers (R2.4); the harness - never pre-filters by term — that would mask the very bugs it must catch. -- **Commit is library-gated.** The harness calls `raft_advance_commit`; the current-term - Figure-8 rule is enforced by `raft.c`. The harness only *applies* whatever the library - reports committed via `on_commit_advanced`, then checks State Machine Safety. -- **Faults are bag-only.** Drop/Duplicate/Reorder mutate `cluster.bag[]` exclusively - (R2.5), mirroring `raft.tla:443`/`448` (`UNCHANGED` of all non-message vars). -- **Restart preserves persistence.** The harness keeps `node.log[]`, the recovered - `log_len`, and the last-persisted `(term, votedFor)` across a `raft_init`, matching - `raft.tla` `Restart` (keeps `currentTerm`, `votedFor`, `log`; resets volatile state). - -### Invariant checker construction (observable-state-only) - -```mermaid -flowchart LR - subgraph IN["inputs: per-node observable state"] - R["roles+terms (raft_snapshot)"] - LG["node.log[] (index, term, value)"] - CM["node.commit_index"] - AP["node.applied[]"] - end - R --> ES["Election Safety: |{i: role==LEADER, term==t}| <= 1, all t (R3.1)"] - LG --> LM["Log Matching: pairwise, shared (idx,term) => identical prefix (R3.2)"] - LG --> LC["Leader Completeness: committed entry in every higher-term leader log (R3.3)"] - CM --> LC - AP --> SM["State Machine Safety: applied[i][k] == applied[j][k] for all k (R3.4)"] -``` - -## Requirements Traceability - -| Requirement | Summary | Design elements | raft.tla anchor | -|---|---|---|---| -| 1.1–1.6 | Deterministic seedable driver | seeded PRNG; `cluster.bag[]`; per-node `log[]`/`commit_index`/`applied[]`; trace buffer; no threads/sockets/clock; deterministic replay | `messages`, `serverVars`, `logVars` | -| 2.1–2.6 | Drive Next-equivalent actions via library entry points | scheduler step dispatch; `raft_send_append_entries`; `raft_advance_commit`; handler routing (UpdateTerm/DropStale inside lib); bag-only Drop/Dup; entry-points-only rule | `Next` (454), `Restart`/`Timeout`/`RequestVote`/`BecomeLeader`/`ClientRequest`/`AppendEntries`/`AdvanceCommitIndex`/`Receive`/`Duplicate`/`Drop` | -| 3.1–3.6 | Per-step four-invariant assertion | `raft_sim_check_election_safety/log_matching/leader_completeness/state_machine_safety`; `raft_sim_check_all`; halt-on-fail; observable-state-only | Election Safety / Log Matching / Leader Completeness / State Machine Safety | -| 4.1–4.5 | Targeted scenarios | `raft_sim_scenario_partition_heal/restart_churn/drop_dup_reorder/figure8/stale_rejoin` | partition / Restart / message faults / Figure-8 / G5 | -| 5.1–5.4 | Failure-trace capture + replay | trace recorder; `raft_sim_dump_trace` (seed + ordered actions); replay from seed; quiet-on-success | (verification) | -| 6.1–6.4 | CI-runnable target | `test_raft.c` conformance entry; M-seed batch + scenarios; ASAN/UBSAN clean; bounded steps/seeds | (verification) | -| 7.1–7.4 | Deliberately-broken-build negative test | compile-time `RAFT_SIM_BREAK_*` fault switch (test build only); both-direction assertion | bug-851 G1/G3 mechanism | -| 8.1–8.3 | Optional TLC-consumable trace export | stable machine-readable record named per `raft.tla`; off by default | `raft.tla` actions/vars | - -## Components and Interfaces - -| Component | Layer | Intent | Req coverage | Key dependencies | Contract | -|---|---|---|---|---|---| -| Message bag | Harness | Mirror `messages`; carry vote/AE req+rsp | 1, 2 | `raft_sim_msg_t` | State | -| In-memory adapter callbacks | Harness | Implement the 6 ports against `node.log[]`/commit | 1, 2 | upstream callback contract (P0) | Service | -| Action scheduler | Harness | Seeded interleaving over the Next set | 1, 2 | PRNG, entry points (P0) | Service | -| Invariant checkers | Harness | Assert the four safety properties | 3 | observable node state | Service | -| Trace recorder | Harness | Record + dump replayable action trace | 1, 5 | scheduler | Service | -| Scenario drivers | Harness | Targeted adversarial cases | 4 | scheduler, bag | Service | -| Fault switch | Test build | Deliberately-broken build | 7 | compile-time macro | Service | -| Conformance test | Test suite | CI batch + negative test | 6, 7 | all of the above | Test | - -### In-memory adapter callbacks (the harness as adapter) - -The harness implements the upstream callbacks against per-node in-memory state, exactly -as the `raft-log-replication` design anticipates ("bytes live in the adapter (WAL) / -simulator (in-memory array)"). Signatures are consumed verbatim from upstream; behavior: - -| Callback | Harness behavior | -|---|---| -| `on_persist_entry(ctx, index, term, entry, len)` | Write/overwrite `node.log[index-1] = {term, value}`; grow `log_len`. Persist-before-act preserved (called before the library reports success). | -| `on_get_entry_term(ctx, index)` | Return `node.log[index-1].term`, or `0` if `index==0` or `index > log_len` (the `term 0` = absent convention). | -| `on_send_append_entries(ctx, peer, prev_log_index, prev_log_term, leader_commit, ...)` | Push an AE-request `raft_sim_msg_t` onto `cluster.bag[]` honoring partitions. | -| `on_truncate_from(ctx, index)` | Set `node.log_len = index - 1` (delete `index..end`). | -| `on_commit_advanced(ctx, commit_index)` | Set `node.commit_index = commit_index`; apply `node.log[k].value` into `node.applied[k]` for each newly committed `k`. | -| `on_send_vote_request(ctx, peer, term)` (existing) | Push a vote-request message onto the bag. | - -### Message bag (mirrors `raft.tla` `messages`) - -```c -typedef enum { - RAFT_SIM_MSG_VOTE_REQ, RAFT_SIM_MSG_VOTE_RSP, - RAFT_SIM_MSG_AE_REQ, RAFT_SIM_MSG_AE_RSP, -} raft_sim_msg_kind_t; - -typedef struct { - raft_sim_msg_kind_t kind; - int src, dst; - uint64_t term; /* RPC term (all kinds) */ - /* AE request */ - uint64_t prev_log_index, prev_log_term, leader_commit; - uint64_t entry_term, entry_value; int entries_len; /* 0=heartbeat, 1=one entry */ - /* responses */ - int granted; /* vote rsp */ - int success; /* AE rsp */ - uint64_t match_index; /* AE rsp */ -} raft_sim_msg_t; -``` - -- `raft_sim_bag_push` / `raft_sim_bag_deliver(i)` (route message `i` to the matching - handler and push any reply) / `raft_sim_bag_drop(i)` / `raft_sim_bag_duplicate(i)` / - reorder (deliver in PRNG-chosen order). Delivery honors `node.partition` (a partitioned - endpoint's messages are not delivered, mirroring a dropped network edge). -- A handler reply is itself pushed onto the bag as a `*_RSP` message, so responses are - first-class bag members (drop/dup/reorder apply to them too). - -### Action scheduler - -```c -/* One step of the Next interleaving. Picks an enabled action from the PRNG, - * records it, applies it through the library entry points / bag, then runs - * raft_sim_check_all. Returns 0 on success, <0 (and dumps the trace) on the - * first invariant violation. */ -int raft_sim_step(raft_sim_cluster_t *c); - -/* Run up to max_steps (or until quiescent). Returns 0 if no invariant was - * violated through the whole run; <0 on the first violation. */ -int raft_sim_run(raft_sim_cluster_t *c, int max_steps); - -/* Seeded init: builds N nodes wired to the in-memory adapter + bag. */ -int raft_sim_conformance_init(raft_sim_cluster_t *c, int n_nodes, - uint32_t election_timeout_ms, uint64_t seed); -``` - -The action set the scheduler chooses from (each maps to a `raft.tla` `Next` disjunct): -`Restart`, `Timeout`, `RequestVote`(deliver vote-req), `BecomeLeader`(deliver enough -vote-rsp), `ClientRequest`, `AppendEntries`, `AdvanceCommitIndex`, `Receive`(deliver an -AE req/rsp incl. `UpdateTerm`/`DropStaleResponse`), `DuplicateMessage`, `DropMessage`. -Only *enabled* actions are eligible (e.g. `ClientRequest`/`AppendEntries` require a -leader; `Receive` requires a non-empty bag). Note that `BecomeLeader` is **not** a -standalone library entry point: it is realized by delivering the quorum-completing -vote-response to `raft_on_vote_reply`, which returns `RAFT_BECAME_LEADER` when that -response carries the candidate over quorum. - -### Invariant checkers - -```c -int raft_sim_check_election_safety(const raft_sim_cluster_t *c); /* R3.1 */ -int raft_sim_check_log_matching(const raft_sim_cluster_t *c); /* R3.2 */ -int raft_sim_check_leader_completeness(const raft_sim_cluster_t *c); /* R3.3 */ -int raft_sim_check_state_machine_safety(const raft_sim_cluster_t *c); /* R3.4 */ -int raft_sim_check_all(const raft_sim_cluster_t *c); /* R3.5: first failure */ -``` - -Checker semantics (observable-state-only, R3.6): - -- **Election Safety** — for each term `t`, count nodes with `role==LEADER && term==t`; - fail if any count > 1. -- **Log Matching** — for every node pair `(i, j)` and every index `k` present in both: - if `log_i[k].term == log_j[k].term`, then every entry at indices `1..k` must be equal - `(term, value)` in both logs; fail otherwise. Note: the library collapses `raft.tla`'s - truncate-only conflict step and the immediately-following no-conflict append into a - single AppendEntries handler invocation (truncate **then** append), so the - Log-Matching / divergent-suffix checker authors should expect the divergent suffix to - be truncated *and* replaced with the leader's entry in one step rather than over two. -- **Leader Completeness** — track the highest committed entry observed cluster-wide; for - every node currently `LEADER` (and across forced leader changes, the highest-term - leader to date), every committed `(index, term)` must be present in its log; fail - otherwise. (Committedness is taken from `node.commit_index` produced by the - library-gated `raft_advance_commit`.) -- **State Machine Safety** — for every index `k` applied by two nodes, `applied_i[k] == - applied_j[k]`; fail otherwise. - -### Trace recorder - -`raft_sim_action_t { kind, actor, msg_index, term, index }` recorded per step into -`cluster.trace[]`; `raft_sim_dump_trace(c, fp)` prints `seed`, `n`, the failing invariant, -and the ordered actions in a human-readable form (R5.1, R5.2). Replay = re-run -`raft_sim_conformance_init` with the same `(n, seed)` and the same `max_steps` (R5.3); -quiet on success unless `RAFT_SIM_VERBOSE` (R5.4). - -### Deliberately-broken-build fault switch (test-only) - -A **runtime** fault-injection flag (`cluster.fault_inject`, set via -`raft_sim_set_fault_inject`) gates **harness-side adapter** corruptions that emulate the -bug-851 defects without touching `raft.c`. The flag is compiled only under `XCE_TESTING` -(never in the shipped build) and defaults to OFF (never enabled in the default CI run), so -it satisfies R7.3 by the same substance a compile-time macro would — but, unlike a -`#ifdef`, a runtime flag can be toggled within a SINGLE binary, which R7.4 requires (the -negative test must assert the broken-config FAILURE and the correct-config PASS in one -test). The `RAFT_SIM_BREAK_*` names are the flag's bit values: - -- `RAFT_SIM_BREAK_TERM_STAMP` — `on_persist_entry` writes `term = 0` instead of the - supplied term (emulates G1: entry term always 0), so `on_get_entry_term` can no longer - distinguish `(term=A, idx=k)` from `(term=B, idx=k)`. -- `RAFT_SIM_BREAK_TRUNCATE` — `on_truncate_from` is a no-op (emulates G3: the divergent - suffix is never deleted). - -With a fault bit set the run must trip Log Matching / Leader Completeness / State Machine -Safety; with the flag off (the default), the same run is clean. This keeps the negative -test entirely inside the harness build configuration (R7.3) — the shipped protocol and the -default CI run are unaffected. (Implementation note: `RAFT_SIM_BREAK_TERM_STAMP` is the -flag used by the task-7.2 negative test because the library collapses conflict-truncate -and the immediately-following append into one AppendEntries handler invocation, so a no-op -truncate alone is overwritten by the re-persist and does not diverge through normal -replication — the term-stamp corruption reliably trips Log Matching instead.) - -> **Why adapter-side and not a `raft.c` edit:** the harness owns only `raft_sim.c`/tests. -> Emulating G1 by having the *adapter* drop the term reproduces the exact observable -> divergence (same-index/different-term entries become indistinguishable) that the real -> bug caused, while honoring the spec boundary (no protocol modification). It still proves -> the *checkers* have teeth: they catch the divergence regardless of where it originates. - -### Optional TLC-consumable trace export - -`raft_sim_export_trace(c, fp)` (off unless `RAFT_SIM_EXPORT`) emits, per step, the action -and the resulting tracked variables (`messages` summary, per-node `currentTerm`/`state`/ -`log`/`commitIndex`) named per `raft.tla` (R8.1, R8.2), in a stable line-oriented format -the maintainer can post-process for TLC cross-checking. Off by default (R8.3). - -## Data Models - -### Domain Model - -- **Bag message** `raft_sim_msg_t` — a single in-flight RPC (vote/AE, request/response), - the unit Drop/Duplicate/Reorder operate on. -- **Log entry (sim)** `raft_sim_log_entry_t { uint64_t term; uint64_t value; }` — index - is the 1-based array position (= WAL LSN); `value` is a synthetic monotone client value - so State Machine Safety divergence is observable. -- **Action record** `raft_sim_action_t` — `(kind, actor, msg_index, term, index)`, the - replayable trace unit. - -### Logical Data Model - -- `raft_sim_node_t` gains `log[RAFT_SIM_MAX_LOG]`, `log_len`, `commit_index`, - `applied[RAFT_SIM_MAX_LOG]`, and a persisted-`(term, votedFor)` shadow for Restart. -- `raft_sim_cluster_t` gains `bag[RAFT_SIM_MAX_MSGS]` + `bag_count`, the PRNG state, the - `seed`, and `trace[RAFT_SIM_MAX_STEPS]` + `trace_len`. -- Fixed-size arrays only (no allocation), bounded by `RAFT_SIM_MAX_LOG`/`MSGS`/`STEPS`, - keeping the harness allocation-free and ASAN-clean. - -### Data Contracts & Integration - -- **Entries handle**: one entry per AE (`entries_len ∈ {0,1}`), matching the upstream - `raft.tla` "≤1 entry" simplification and the log-replication wire shape — the harness - reuses that shape exactly. -- **`term 0` sentinel**: honored by `on_get_entry_term` for absent indices. -- **1-based index = LSN**: `node.log[index-1]` is the canonical mapping. - -## Error Handling - -### Error Strategy - -The harness is test code: on an invariant violation it dumps the seed + trace and fails -the Unity assertion (the test aborts), **never** the library (`raft.c` returns status -codes only — "controller never self-terminates" is preserved; the harness's halt is a -test failure, not a library abort). - -### Error Categories and Responses - -- **Invariant violation** → `raft_sim_check_all` returns the failing invariant; the - scheduler dumps `(seed, n, invariant, trace)` and returns `<0`; the test fails (R3.5, - R5.1). -- **Bag overflow / log overflow** (a scenario exceeded a fixed bound) → the harness - fails loud (a distinct, non-invariant test error) rather than silently truncating, so - bounds bugs are visible. -- **Misuse** (e.g. `ClientRequest` driven on a non-leader) → the scheduler only selects - *enabled* actions, so the library's `-EPERM` path is exercised intentionally (a - targeted unit test), never as an accidental scheduler state. - -### Monitoring - -Reuse the existing per-node `trace_buf` and the library `on_trace` events -(`"ae_recv"`, `"ae_conflict_truncate"`, `"term_adopt_ae"`, `"ae_reply_advance"`, etc.) -plus the new harness-level action trace, so a failing run shows both the library's view -and the scheduler's interleaving. - -## Testing Strategy - -TDD-mandatory (`testing.md`): every checker and scenario is written as a RED test in -`test_raft.c` (with the `raft_sim.c` backing) that fails before the harness logic exists -and passes after. All tests are `XCE_TESTING`-gated and run via `/xce-build lib`. - -### Unit tests (checkers in isolation) - -1. Each checker flags a hand-constructed violating cluster and passes a valid one: - two leaders same term (Election Safety); same `(idx, term)` with different prefix - (Log Matching); a committed entry missing from a higher-term leader (Leader - Completeness); divergent `applied[k]` (State Machine Safety). (R3.1–3.4) -2. `raft_sim_check_all` returns the first failing invariant and halts (R3.5). - -### Scenario tests (targeted, R4) - -1. Partition/heal: isolate a minority during appends, heal, drain; assert all four - invariants hold throughout and after convergence (R4.1). -2. Restart churn: restart nodes mid-replication preserving persisted state; assert - re-convergence with no violation (R4.2). -3. Drop/dup/reorder: perturb bag delivery; assert idempotent convergence + invariants - (R4.3). -4. Figure-8: a previous-term entry on a quorum, leader change; assert it is **not** - reported committed until a current-term entry commits (R4.4). -5. Stale-rejoin (bug-851 G5): a term-lagging node rejoins under a higher-term leader; - assert it adopts the higher term from the AE/heartbeat path and does not re-elect at a - reused term (R4.5). - -### Randomized + CI (R6) - -1. M-seed sweep: run `raft_sim_run` for a fixed `max_steps` across M seeds; assert zero - violations. Bounded so the CI target fits the library test-suite budget (R6.4). -2. Runs clean under ASAN/UBSAN (R6.3). - -### Negative test (R7) - -1. With `RAFT_SIM_BREAK_TERM_STAMP` (or `RAFT_SIM_BREAK_TRUNCATE`) set, the Figure-8 / - divergent-suffix scenario **fails** an invariant and the trace is dumped; with the - macro off, the same scenario **passes** — both directions asserted in one test, proving - the oracle's sensitivity (R7.1, R7.2, R7.4). The macro is confined to the test build - (R7.3). - -### Replay test (R5) - -1. A run that fails for seed `S` reproduces the identical failing action and invariant - when re-run with seed `S` and the same `max_steps` (R5.3). diff --git a/.kiro/specs/raft-sim-conformance/requirements.md b/.kiro/specs/raft-sim-conformance/requirements.md deleted file mode 100644 index 4da21135..00000000 --- a/.kiro/specs/raft-sim-conformance/requirements.md +++ /dev/null @@ -1,163 +0,0 @@ -# Requirements Document - -## Introduction - -XCE's Raft consensus core lives in `src/lib/raft/raft.{c,h}` as a pure, callback-driven -state machine. With `raft-log-replication` (the `(term, index)` log, `AppendEntries`, -`HandleAppendEntriesRequest`, `UpdateTerm`-on-any-RPC) and `raft-commit-safety` -(`AdvanceCommitIndex` with the Figure-8 current-term rule, `(lastLogTerm, lastLogIndex)` -vote freshness) the library owns the full election + log + commit halves of -`raft/raft.tla` (Ongaro canonical). What is still missing is an **independent oracle**: -nothing continuously proves the implementation actually conforms to the four Raft safety -invariants. Today `src/lib/raft/raft_sim.c` exercises only the election RPC path; it has -no log, no commit, and no invariant checkers. - -This feature extends `raft_sim.c` into an **in-process conformance/verification -harness** that drives N library instances purely synchronously (no threads, no sockets, -no live cluster) through transitions equivalent to the `raft.tla` `Next` actions -(`raft.tla:454`) and asserts the four TLA+ safety invariants on every step. The harness -is the "use the verified spec to help" payoff: a deterministic regression oracle for the -consensus core. Its acceptance bar is concrete — it must **reproduce a bug-851-class -invariant divergence on a deliberately-broken build** (e.g. term stamping disabled or -`(term, index)` conflict truncation removed) and **pass on the correct build**, and it -must run M randomized seeds in CI with zero invariant violations. - -## Boundary Context - -- **In scope**: a seedable/deterministic action driver mirroring `raft.tla`'s message - bag + per-node variables; randomized and targeted scenario generators (network - partitions, node restarts, message drop/duplicate/reorder, the Figure-8 commit - scenario); per-step assertion of all four safety invariants over the N library - instances; failure-trace capture and reproduction (print the exact action sequence - that broke an invariant, replayable from the seed); a CI-runnable test target - alongside the existing library tests; an acceptance/negative test that a - deliberately-broken protocol build is caught and the correct build passes. - Optionally, an exported machine-readable trace format consumable by TLC for - cross-checking against `raft.tla`. -- **Out of scope**: the protocol implementation itself — the log model, `AppendEntries`, - `AdvanceCommitIndex`, vote-freshness, and term-adoption semantics are owned by - `raft-log-replication` and `raft-commit-safety`; this harness only *drives and checks* - them. Also out: the live controller / `/xce-deploy-ha` chaotic-failover regression - (owned by `ha-term-safety`); the controller WAL/repl/recovery/commit-barrier - internals; config-change markers and the upgrade snapshot (`ha-term-safety`); any - change to `raft.tla` itself or a TLC model-checking harness for the spec. -- **Adjacent expectations**: the harness consumes the `raft-log-replication` shared - callback contract verbatim (`on_persist_entry`, `on_get_entry_term`, - `on_send_append_entries`, `on_truncate_from`; 1-based index = WAL LSN; `term 0` means - absent) and the `raft-commit-safety` additions (`on_commit_advanced`, - `(lastLogTerm, lastLogIndex)` freshness). It re-defines none of these; if any of those - signatures change, this spec re-validates. The simulator backing log + commit state - lives in-memory per node (the harness is the adapter), exactly as the upstream design - docs anticipate (`raft-log-replication` design: "bytes live in the adapter (WAL) / - simulator (in-memory array)"). - -## Requirements - -### Requirement 1: Deterministic seedable action driver - -**Objective:** As a Raft library maintainer, I want a fully deterministic, seed-driven -in-process driver over N library instances, so that any invariant violation is exactly -reproducible from its seed without threads, sockets, or a live cluster. - -#### Acceptance Criteria - -1. When the harness is initialized with a node count N (1 ≤ N ≤ `RAFT_SIM_MAX_NODES`) and a 64-bit seed, the Raft Conformance Harness shall construct N library instances wired to in-memory adapter callbacks and a single shared message bag, and shall produce byte-identical execution for the same (N, seed) inputs across runs. -2. The Raft Conformance Harness shall derive every nondeterministic choice (which action to take, which node, which message to deliver/drop/duplicate, partition toggles, restart targets, client values) from a single seeded pseudo-random source, so that no wall-clock time, address values, or uninitialized memory influence the run. -3. While a run is in progress, the Raft Conformance Harness shall maintain per-node state mirroring the `raft.tla` variables it owns (in-memory log of `(index, term, value)` entries, `commitIndex`, applied/state-machine view) and a message bag of in-flight RPC requests and responses. -4. When a step is executed, the Raft Conformance Harness shall record the chosen action (action kind, actor node, target/message identity) into a replayable trace buffer before applying it. -5. The Raft Conformance Harness shall execute purely synchronously: it shall not spawn threads, open sockets, perform real I/O, or read wall-clock time during a run. -6. Where the seed and node count are fixed, when the same number of steps is executed twice, the Raft Conformance Harness shall reach an identical final per-node and message-bag state (deterministic replay). - -### Requirement 2: Drive transitions equivalent to the raft.tla Next actions - -**Objective:** As a Raft library maintainer, I want the harness to drive transitions -equivalent to every `raft.tla` `Next` action through the actual library entry points, so -that the implementation — not a re-modeled protocol — is what is being exercised. - -#### Acceptance Criteria - -1. When the driver selects a node-local action, the Raft Conformance Harness shall be able to apply `Restart` (re-init a node preserving its persisted `term`/`votedFor`/log per `raft.tla` `Restart`), `Timeout`/`RequestVote` (begin an election via the election entry points), `BecomeLeader` (on a quorum of granted votes), and `ClientRequest` (append a value on a leader via `raft_log_append`). -2. When a leader is selected for replication, the Raft Conformance Harness shall emit `AppendEntries` via the library's `raft_send_append_entries` entry point (computing `prevLogIndex`/`prevLogTerm` from the library's per-peer progress), placing the resulting RPC into the message bag. -3. When the driver advances commit, the Raft Conformance Harness shall invoke the library's `AdvanceCommitIndex` entry point (`raft-commit-safety`) and apply the resulting committed entries to that node's in-memory state machine. -4. When a message is delivered from the bag, the Raft Conformance Harness shall route it through the matching library handler (`raft_on_vote_request`/`raft_on_vote_reply`, `raft_on_append_entries_request`/`raft_on_append_entries_reply`), including the `UpdateTerm` higher-term adoption and the stale-response drop (`DropStaleResponse`) that those handlers implement. -5. When a message is selected for a network fault, the Raft Conformance Harness shall apply `DuplicateMessage` (re-enqueue an in-flight message) or `DropMessage` (discard it) without otherwise mutating node state, mirroring `raft.tla:443`/`raft.tla:448`. -6. The Raft Conformance Harness shall drive these transitions **only** through the published library entry points and callbacks; it shall not reach into or re-implement the protocol decision logic (logOk, conflict truncation, current-term commit rule, freshness) that lives in `raft.c`. - -### Requirement 3: Per-step assertion of the four Raft safety invariants - -**Objective:** As a Raft library maintainer, I want all four TLA+ safety invariants -checked after every applied action, so that the first divergence is caught at the step -that causes it, not many steps later. - -#### Acceptance Criteria - -1. After every applied action, the Raft Conformance Harness shall assert **Election Safety**: no two distinct nodes are in the `LEADER` role for the same term. -2. After every applied action, the Raft Conformance Harness shall assert **Log Matching**: for any two nodes whose logs contain an entry at the same index with the same term, all entries at and before that index are identical (same term and value) across those two logs. -3. After every applied action, the Raft Conformance Harness shall assert **Leader Completeness**: every entry that has been committed by any node is present, at the same index and term, in the log of every node that is currently or subsequently a leader of a higher term. -4. After every applied action, the Raft Conformance Harness shall assert **State Machine Safety**: no two nodes have applied different values at the same log index. -5. If any of the four invariant checks fails, then the Raft Conformance Harness shall halt the run immediately and report the failing invariant. -6. The Raft Conformance Harness shall compute the invariant checks only from observable per-node state (roles, terms, in-memory logs, commit indices, applied values), so that the checkers are independent of the library's internal protocol logic. - -### Requirement 4: Targeted adversarial scenarios - -**Objective:** As a Raft library maintainer, I want named targeted scenarios that exercise -the exact conditions behind bug-851 and the canonical Raft hazards, so that known-hard -cases are covered deterministically rather than left to chance. - -#### Acceptance Criteria - -1. The Raft Conformance Harness shall provide a network-partition scenario that isolates a subset of nodes for a number of steps and then heals it, and shall assert all four invariants throughout and after convergence. -2. The Raft Conformance Harness shall provide a restart scenario that restarts nodes (preserving persisted `term`/`votedFor`/log) during replication, and shall assert that restarted nodes re-converge without invariant violation. -3. The Raft Conformance Harness shall provide a drop/duplicate/reorder scenario that perturbs the message-bag delivery order and multiplicity, and shall assert idempotent convergence with all four invariants held. -4. The Raft Conformance Harness shall provide a Figure-8 scenario (a previous-term entry replicated to a quorum but not yet committed under the current leader) and shall assert that no node reports the previous-term entry as committed until a current-term entry is committed (Leader Completeness / State Machine Safety preserved). -5. When a stale-term node rejoins under a higher-term leader (the bug-851 G5 mechanism), the Raft Conformance Harness shall assert that the rejoining node adopts the higher term from the AppendEntries/heartbeat path and does not re-elect at a reused term (Election Safety preserved). - -### Requirement 5: Failure-trace capture and reproduction - -**Objective:** As a Raft library maintainer, I want a precise, replayable failure trace -whenever an invariant breaks, so that I can reproduce and diagnose the exact divergence. - -#### Acceptance Criteria - -1. If an invariant check fails, then the Raft Conformance Harness shall emit the seed, the node count, the failing invariant name, and the full ordered action trace leading to the failure. -2. The Raft Conformance Harness shall format the emitted trace so that each action is human-readable (action kind, actor node, message/target identity, relevant terms/indices). -3. When the harness is re-run with a previously failing seed and node count, the Raft Conformance Harness shall deterministically reproduce the same failing action and invariant (replay reproducibility). -4. When a run completes without any invariant failure, the Raft Conformance Harness shall report success quietly (no trace dump) unless verbose tracing is explicitly requested. - -### Requirement 6: CI-runnable conformance target - -**Objective:** As a Raft library maintainer, I want the conformance harness to run as a -first-class test target in CI, so that consensus regressions are caught automatically -before they ship. - -#### Acceptance Criteria - -1. The Raft Conformance Harness shall expose a test entry point in the existing library test suite (`src/lib/testsuite/`) that is built and run under the existing `XCE_TESTING` build (`/xce-build lib`). -2. When the conformance test runs in CI, the Raft Conformance Harness shall execute a fixed batch of M randomized seeds (M ≥ a configured minimum) plus all targeted scenarios, and shall fail the test run if any seed or scenario violates any invariant. -3. While running under sanitizers (ASAN/UBSAN as used by the library tests), the Raft Conformance Harness shall complete the CI batch without memory errors, leaks, or undefined behavior. -4. The Raft Conformance Harness shall bound each run (a maximum step count and a fixed seed batch) so that the CI target completes within the existing library test-suite time budget. - -### Requirement 7: Deliberately-broken-build acceptance (negative test) - -**Objective:** As a Raft library maintainer, I want proof the oracle actually has teeth, -so that a passing conformance run is meaningful — the harness must catch a real -bug-851-class defect when one is present. - -#### Acceptance Criteria - -1. Where a deliberately-broken protocol build is selected by a build-time fault-injection switch (for example, term stamping disabled so entries carry term 0, or `(term, index)` conflict truncation removed), the Raft Conformance Harness shall, under that switch, reproduce a bug-851-class invariant divergence (a Log Matching, Leader Completeness, or State Machine Safety violation) within the bounded run. -2. When the fault-injection switch is off (the correct build), the Raft Conformance Harness shall run the same scenarios and seeds with zero invariant violations. -3. The Raft Conformance Harness shall confine the fault-injection switch to the harness/test build configuration so that it neither alters the shipped protocol implementation nor remains enabled in the default CI conformance run. -4. The negative test shall assert both directions: the broken configuration fails (an invariant violation is detected and reported) and the correct configuration passes (no violation), so that the oracle's sensitivity is itself verified. - -### Requirement 8: Optional TLC-consumable trace export - -**Objective:** As a Raft library maintainer, I want the harness to optionally emit a -machine-readable action/state trace, so that a harness run can be cross-checked against -TLC's exploration of `raft.tla`. - -#### Acceptance Criteria - -1. Where trace export is enabled, the Raft Conformance Harness shall emit each step as a machine-readable record (action and the resulting values of the `raft.tla` variables the harness tracks) in a stable, documented format. -2. Where trace export is enabled, the Raft Conformance Harness shall name actions and variables consistently with `raft/raft.tla` so the export can be mapped to spec actions without re-interpretation. -3. The Raft Conformance Harness shall keep trace export off by default so that the standard CI conformance run incurs no export overhead. diff --git a/.kiro/specs/raft-sim-conformance/research.md b/.kiro/specs/raft-sim-conformance/research.md deleted file mode 100644 index 1c9f919c..00000000 --- a/.kiro/specs/raft-sim-conformance/research.md +++ /dev/null @@ -1,165 +0,0 @@ -# Gap Analysis — raft-sim-conformance (2026-06-24) - -Analyzed per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Scope: a single -feature — an in-process, deterministic, seed-driven harness that drives N `src/lib/raft` -library instances through `raft.tla`-`Next`-equivalent transitions and asserts the four -Raft safety invariants every step (the "use the verified spec as an oracle" payoff; -acceptance bar = catch a bug-851-class divergence on a deliberately-broken build, pass on -the correct build). - -## 1. Current State Investigation - -### Authoritative spec -- `raft/raft.tla` — Diego Ongaro's canonical Raft (abstract spec; ends `raft.tla:471`). - All `Next` actions present and located: `Restart` (167), `Timeout` (178), - `RequestVote` (190), `AppendEntries` (204), `BecomeLeader` (229), `ClientRequest` (246), - `AdvanceCommitIndex` (259), `Receive` (421) with `UpdateTerm` (406) and - `DropStaleResponse` (415), `DuplicateMessage` (443), `DropMessage` (448), - `Next` disjunction (454). The `.tla` file carries NO embedded invariant `THEOREM` block; - the four safety invariants (Election Safety, Log Matching, Leader Completeness, State - Machine Safety) are tracked prose-side in `src/controller/CLAUDE.md`. **No TLC config - (`*.cfg`), no `tla2tools`/TLC tooling exists in-repo** — the spec is a reference - document, not a model-checked artifact today. - -### How the C library is checked against raft.tla today -- **It is not, automatically.** Conformance is enforced by hand: `src/controller/CLAUDE.md` - holds the TLA+ variable→code and action→code mapping plus a manual "Protocol - Verification Checklist." There is no trace generation, no invariant checker that runs - per-step, and no randomized simulation that asserts the four invariants. -- `src/controller/raft/` does **not** exist; the library lives at `src/lib/raft/` - (`raft.c` 33 KB, `raft.h` 30 KB, `raft_sim.c` 18 KB). Per `src/controller/CLAUDE.md`, - the controller's `raft.c` was election-only; the full election+log+commit protocol now - lives in the **library** `src/lib/raft` (deps `raft-log-replication` + - `raft-commit-safety` landed — callbacks present and exercised in `test_raft.c`). - -### Existing harness (`src/lib/raft/raft_sim.c`, `XCE_TESTING`-gated) -Already present (the election + partial log layer it was grown to for the dependency -specs), all consuming published library entry points verbatim: -- `raft_sim_init`, `raft_sim_deliver_pending` (vote RPC routing via - `raft_on_vote_request`/`_reply`), `raft_sim_deliver_append_entries` (AE routing via - `raft_on_append_entries_request`/`_reply`), `raft_sim_start_all_elections`, - `raft_sim_fire_timeout`, `raft_sim_partition`, `raft_sim_advance_commit` - (`raft_advance_commit`), `raft_sim_restart_node` (models Restart, preserves persisted - term/votedFor/log/commit_index), and **one** invariant checker: - `raft_sim_check_log_matching` (`raft_sim.c:424`). -- In-memory adapter callbacks already back the four log callbacks + `on_commit_advanced` - into per-node `raft_sim_node_t` (`log[]`, `log_count`, `commit_index`, - `commit_advanced_count`, `trace_buf`). The library holds no entry bytes — the simulator - owns storage, exactly as the dependency design anticipated. -- `raft.h` `XCE_TESTING` section (`raft.h:432+`) already defines `raft_sim_rpc_t`, - `raft_sim_entry_t`, `raft_sim_ae_t`, `raft_sim_node_t`, `raft_sim_cluster_t`, and bounds - `RAFT_SIM_MAX_NODES=8`, `RAFT_SIM_MAX_LOG=64`, `RAFT_SIM_ENTRY_LEN=16`, - `RAFT_SIM_MAX_AE=64`. -- `test_raft.c` (7011 lines) already exercises election + AE + commit + a - hand-built partition/catch-up convergence path and both directions of - `raft_sim_check_log_matching` (pass-on-converged / fail-on-divergence). - -### All library entry points the harness needs already exist in `raft.h` -`raft_log_append`, `raft_send_append_entries`, `raft_advance_commit`, -`raft_on_append_entries_request`/`_reply`, `raft_on_vote_request`/`_reply`, `raft_init`, -`raft_start_election`, `raft_on_election_timeout`, `raft_snapshot` — all declared. -**Constraint:** `raft_snapshot_t` (`raft.h:69-76`) exposes term/role/voted_for/votes only — -NOT log or commit_index. So Election Safety can read from `raft_snapshot`, but Log -Matching / Leader Completeness / State Machine Safety MUST read the harness-owned -in-memory `log[]`/`commit_index`/`applied[]` (Req 3.6 "observable per-node state only" is -satisfied by the in-memory mirror, not by reaching into `raft.c`). - -### Spec phase state -`spec.json` = `tasks-generated`, all 3 gates auto-approved (`-y`), `ready_for_implementation`. -`tasks.md` has 8 task groups (1.1–8.1); **every task box is unchecked `[ ]`** — none of -the conformance machinery is implemented. The brief/requirements/design are complete and -internally consistent. - -## 2. Requirement-to-Asset Map (gaps tagged Missing / Unknown / Constraint) - -| Req | Need | Existing asset | Gap | -|-----|------|----------------|-----| -| 1.1–1.6 Deterministic seeded driver | Seeded PRNG (SplitMix64), `(N,seed)` byte-identical replay, message bag, action trace buffer, no threads/sockets/wall-clock | `raft_sim_init` (no seed/PRNG); synchronous already | **Missing**: PRNG, `seed`, `raft_sim_conformance_init`, bag (`bag[]`/`bag_count`), trace buffer, `raft_sim_run` replay. Constraint: must extend the existing `raft_sim_cluster_t`/`_node_t`, not rewrite. | -| 2.1–2.6 Drive `raft.tla` Next actions via library entry points | `Restart`/`Timeout`/`RequestVote`/`BecomeLeader`/`ClientRequest`/`AppendEntries`/`AdvanceCommitIndex`/`Receive`(UpdateTerm+DropStaleResponse)/`Duplicate`/`Drop` through published API | Vote+AE delivery, restart, advance-commit, partition all wired verbatim to library handlers | **Missing**: unified `raft_sim_step` scheduler selecting only *enabled* actions; `ClientRequest` via `raft_log_append`; bag-based `Duplicate`/`Drop` (current delivery drains a queue, no per-message drop/dup/reorder). UpdateTerm/DropStaleResponse are already inside the library handlers (consumed, not re-modeled). | -| 3.1–3.6 Per-step assertion of all four invariants | Election Safety, Log Matching, Leader Completeness, State Machine Safety; first-failure halt; observable-state-only | Only `raft_sim_check_log_matching` exists | **Missing**: `raft_sim_check_election_safety`, `_leader_completeness`, `_state_machine_safety`, `_check_all` (first-failure composer); `applied[]` per-node state machine view. | -| 4.1–4.5 Targeted adversarial scenarios | partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin (bug-851 G5) | A hand-built partition/catch-up test exists in `test_raft.c` (not a reusable named scenario) | **Missing**: 5 named `raft_sim_scenario_*` entry points asserting all four invariants throughout. | -| 5.1–5.4 Failure-trace capture/replay | seed+N+failing-invariant+ordered action trace dump; quiet-on-success; replay reproducibility | per-node `trace_buf` (string events only, not structured actions) | **Missing**: structured `raft_sim_action_t` trace + `raft_sim_dump_trace`; replay tie-in to `raft_sim_run`. | -| 6.1–6.4 CI-runnable target | M-seed sweep + all scenarios in `src/lib/testsuite`, ASAN/UBSAN-clean, bounded runtime | `test_raft.c` already builds/runs under `XCE_TESTING` + sanitizers | **Missing**: the sweep harness + bounds; mechanics (test entry point, sanitizer build) already exist. Low-risk. | -| 7.1–7.4 Deliberately-broken-build negative test | compile-time fault switches (`RAFT_SIM_BREAK_TERM_STAMP`, `RAFT_SIM_BREAK_TRUNCATE`) confined to adapter; both-directions assert | none | **Missing**. **Constraint**: switches must live in the *adapter* (`raft_sim.c` callbacks), never `raft.c` — the boundary forbids touching protocol code. This is the acceptance teeth. | -| 8.1–8.3 Optional TLC-consumable trace export | spec-named machine-readable record stream, off by default | none | **Missing** (optional `[ ]*`, deferred). **Unknown/Research**: no TLC tooling in-repo to actually consume it — export format can be written but cross-checking against `raft.tla` via TLC is unproven. | - -### Cross-cutting findings -- **Currently CHECKED against raft.tla (in code):** Log Matching only (one checker, - `raft_sim.c:424`), and only on hand-driven clusters — not per-step, not over randomized - runs. Election Safety / Leader Completeness / State Machine Safety are **UNCHECKED in - code** (asserted only by prose checklist). Of the `Next` actions, Restart / Timeout / - RequestVote / AppendEntries / AdvanceCommitIndex / Receive are *driven* by the existing - sim; `ClientRequest`, per-message `Duplicate`/`Drop`/reorder, and a unified - enabled-action scheduler are **not yet driven**. -- **raft.tla does NOT model durability** (restores `log` intact on Restart, - `raft.tla:165-174`) — so the harness, like the spec, cannot and should not assert the - commit-durability invariant (that rests on `ha-commit-durability` tests). This correctly - scopes the harness to the four safety invariants only. Confirmed against memory. - -## 3. Implementation Approach Options - -### Option A — Extend `raft_sim.c` / `test_raft.c` in place (spec's chosen path) -Add PRNG, bag, action trace, three new checkers + composer, `raft_sim_step`/`_run`, five -scenarios, fault switches — all inside the existing `XCE_TESTING` sim section and test -file. Reuse `raft_sim_node_t`/`_cluster_t` (extend with `bag[]`, `applied[]`, `seed`, -`trace[]`). -- ✅ Matches the spec boundary exactly (owns `raft_sim.c` + `raft.h` XCE_TESTING + `test_raft.c`); reuses the verbatim-library-entry-point pattern already proven by the dependency specs; no protocol change; sanitizer/CI plumbing already in place. -- ❌ `raft_sim.c` grows well past the existing 18 KB (toward the 800-line guideline ceiling); `test_raft.c` is already 7011 lines. - -### Option B — New `raft_conformance.c` / `test_raft_conformance.c` modules -Put the harness (PRNG, bag, scheduler, checkers, scenarios, fault switches) in dedicated -files; keep `raft_sim.c` as the thin election/AE adapter. -- ✅ Honors "many small files"; isolates the oracle from the dependency-era sim; easier to read/maintain; keeps `test_raft.c` from ballooning further. -- ❌ Deviates from the spec's stated file boundary ("owns `raft_sim.c` … ONLY"); needs a meson build wiring change and a boundary-note update; minor duplication of the in-memory adapter. - -### Option C — Hybrid (recommended) -Keep the in-memory adapter callbacks, bag, PRNG, and the **four checkers** in -`raft_sim.c`/`raft.h` (they are genuinely "simulator" primitives and reuse existing node -state). Lift the **scheduler + five scenarios + M-seed sweep + negative test** into a -new `test_raft_conformance.c` (or a clearly-fenced section) so `test_raft.c` does not grow -unbounded. Fault switches stay in `raft_sim.c` (adapter) per boundary. -- ✅ Best file-size/cohesion balance; preserves the "harness drives only published entry points" boundary; checkers stay reusable; the heavy scenario/sweep code is isolated. -- ❌ Splits ownership across two test files — needs a one-line boundary amendment in `tasks.md`/`design.md` (Allowed Dependencies already permit a new test file under `src/lib/testsuite`). - -## 4. Effort & Risk - -- **Effort: L (1–2 weeks).** Eight task groups; the hard parts are the seeded scheduler - with enabled-action selection (4.2), deterministic byte-identical replay (4.4/1.6), the - three new invariant checkers (3.1/3.3/3.4) and the bug-851 Figure-8 / stale-rejoin - scenarios (5.4/5.5). Substantial scaffolding already exists (adapter callbacks, - partition, restart, Log Matching, AE/vote delivery), which pulls it down from XL. -- **Risk: Medium.** Mitigants: all library entry points exist and are stable; deps landed; - no protocol change; synchronous/no-I/O design removes concurrency risk; the negative - test self-validates checker sensitivity. Residual risk: (a) determinism leaks (pointer - values / iteration order in action selection) — must be audited (Req 1.2); (b) the - Figure-8 and stale-rejoin scenarios must be precise enough to actually exercise the - bug-851 mechanism (else the oracle's "teeth" are blunt); (c) keeping fixed-size bounds - (`RAFT_SIM_MAX_*`) large enough for randomized runs without blowing the CI time/ASAN - budget. The optional TLC export (Req 8) is **Unknown** — no in-repo TLC to validate it. - -## 5. Recommendations for Design Phase - -- **Preferred approach: Option C (hybrid)** — checkers + bag + PRNG + fault switches in - `raft_sim.c`/`raft.h`; scheduler + scenarios + sweep + negative test in a new - `src/lib/testsuite/test_raft_conformance.c`. Amend the `tasks.md`/`design.md` file - boundary with that one new test file (design "Allowed Dependencies" already permits it). - If strict adherence to the stated single-file boundary is required, fall back to - Option A and accept the file-size growth (flag a `clang-format`/size exception). -- **Key decisions to lock:** (1) PRNG = SplitMix64 inlined, single `cluster.seed` source — - forbid any wall-clock/address input (Req 1.2); (2) invariant checkers read ONLY - harness-owned in-memory state (`log[]`/`commit_index`/`applied[]`) + `raft_snapshot` - (role/term) — never `raft.c` internals (Req 3.6); (3) `ClientRequest` drives - `raft_log_append` on a forced leader; `Duplicate`/`Drop`/reorder mutate ONLY the bag - (Req 2.5); (4) fault switches confined to adapter callbacks, default-off, never set in - the default CI run (Req 7.3). -- **Research items to carry forward:** (a) **TLC consumability (Req 8)** — no - `tla2tools`/`.cfg` exists in-repo; decide whether to add TLC tooling or keep Req 8 as a - documented-format-only deferral. (b) confirm `RAFT_SIM_MAX_LOG=64` / step bounds are - large enough for randomized convergence yet inside the ASAN/UBSAN CI budget (Req 6.4). - (c) verify the existing per-node `applied[]`/state-machine view needs adding (not present - in current `raft_sim_node_t`) for State Machine Safety (Req 3.4). -- **Dependency watch:** harness consumes `raft-log-replication` + `raft-commit-safety` - callback signatures verbatim; per design "Revalidation Triggers," any change to - `on_persist_entry`/`on_get_entry_term`/`on_send_append_entries`/`on_truncate_from`/ - `on_commit_advanced` or the `(lastLogTerm,lastLogIndex)` freshness contract re-validates - this spec. diff --git a/.kiro/specs/raft-sim-conformance/spec.json b/.kiro/specs/raft-sim-conformance/spec.json deleted file mode 100644 index 83fa1b2e..00000000 --- a/.kiro/specs/raft-sim-conformance/spec.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "feature_name": "raft-sim-conformance", - "created_at": "2026-06-21T13:00:00Z", - "updated_at": "2026-06-21T13:00:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety.", - "updated_at_tasks": "2026-06-24T16:43:39.427Z", - "validate_impl": { - "decision": "GO", - "validated_at": "2026-07-17T18:00:00Z", - "note": "First post-implementation validation run -- all 23 tasks (1.1-8.1) were already [x] with no prior validate_impl record and spec.json's phase field had never advanced past tasks-generated despite implementation being complete. Full `/xce-build lib` ASAN run: clean compile, 293/293 Unity tests PASS (0 Fail, 0 Ignored), meson 'libxce test' target 1/1 OK in 36.11s, zero AddressSanitizer/LeakSanitizer/UBSan findings anywhere in the log. Every raft-sim conformance test observed by name and PASS: all 4 invariant-checker unit tests (election safety, log matching, leader completeness, state machine safety) both directions, raft_sim_check_all first-failure composition, the 5 targeted scenarios (partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin/G5), the M-seed randomized sweep (test_sim_6_1_seed_sweep), the deliberately-broken-build negative test asserting both directions (test_sim_7_2_negative_both_directions), and the optional TLC trace-export test (task 8.1, gated off by default). Boundary audit: grepped every commit subject-tagged feat|test|fix|docs(raft-sim-conformance) across full git history -- touched files are exactly raft.h, raft_sim.c, test_raft.c (+ this spec's own docs); zero raft.c protocol edits, zero controller files. Harness confirmed allocation-free (no malloc/calloc/realloc/free in raft_sim.c), matching the design's ASAN-clean-by-construction claim. Requirements coverage: all 8 requirement sections (1-8, 34 acceptance criteria) map to committed tasks and passing tests; no gaps. Design alignment: File Structure Plan matches actual (raft.h XCE_TESTING-section-only, raft_sim.c, test_raft.c); component graph (message bag, in-memory adapter, seeded scheduler, 4 checkers, trace recorder, scenario drivers, fault switch) all present and named as designed. Cross-task integration: task 4.2's Implementation Notes record an UPSTREAM raft.c defect (bug-1095/1096, Election Safety violation from a non-idempotent vote-reply tally + missing stale-term drop) surfaced by the oracle itself -- correctly reported out-of-boundary rather than masked, and subsequently fixed upstream in raft.c (commit b5b7e01, verified live at raft_on_vote_reply: per-peer vote_replies[] dedup + term<s->term drop), after which downstream tasks 5.x/6.1/7.2 assert zero violations on the correct build as required. This validation pass also corrected buglog.json's bug-1095 entry, which was stale (said 'NOT FIXED' although the fix has been live since 2026-06-24 per bug-1096 and tasks.md's own notes).", - "known_limitations": [ - "tasks.md task 7.1 describes the negative-test mechanism as a 'compile-time fault switch,' but the shipped implementation (and design.md, corrected in commit ffeb7c2) is a runtime bitflag (cluster.fault_inject / RAFT_SIM_BREAK_TERM_STAMP|TRUNCATE set via raft_sim_set_fault_inject) so both the broken and correct configurations can be asserted within a single test binary per R7.4. Wording drift only in tasks.md's task header; behavior is correct, intentional, and matches design.md.", - "design.md's invariant-checker narrative does not reflect the leader-only commit model correction recorded in tasks.md's own Implementation Notes for tasks 5.1/5.2: a FOLLOWER never advances commit_index/applied[] from AppendEntries (raft_on_append_entries_request does (void)leader_commit -- intended XCE design, follower commit visibility is a controller/heartbeat concern owned by ha-heartbeat-committed-propagation, not this library). Scenario tests correctly assert leader-side commit_index and log convergence rather than follower commit_index equality; design.md's prose was not back-ported to state this explicitly. Documentation-drift note only.", - "raft.h's XCE_TESTING sim section is also extended by a later, adjacent spec (raft-apply-decoupling, commit 5f98d1b, test_sim_apply_decoupling_* tests) building on this harness's scheduler/checkers. This is expected downstream consumption, not a boundary violation of this spec, but a future change to raft-sim-conformance's scheduler/action-record shape should check that spec's tests too." - ] - } -} diff --git a/.kiro/specs/raft-sim-conformance/tasks.md b/.kiro/specs/raft-sim-conformance/tasks.md deleted file mode 100644 index bc76d044..00000000 --- a/.kiro/specs/raft-sim-conformance/tasks.md +++ /dev/null @@ -1,275 +0,0 @@ -# Implementation Plan - -> **TDD-mandatory** (`testing.md`): every checker, scheduler action, and scenario lands as -> a RED test in `src/lib/testsuite/test_raft.c` (with the `raft_sim.c` backing) that fails -> before the harness logic exists and passes after. Build/run library tests via -> `/xce-build lib`. All harness code is `XCE_TESTING`-gated. -> -> **Spec boundary** (applies to ALL tasks): this spec owns `src/lib/raft/raft_sim.c`, the -> `XCE_TESTING` sim section of `src/lib/raft/raft.h`, and the conformance tests in -> `src/lib/testsuite/test_raft.c` ONLY. It does **NOT** modify the protocol -> implementation: no task changes `raft.c` protocol logic or any protocol declaration in -> `raft.h`, and no task edits a controller file. The harness drives the published library -> entry points and consumes the `raft-log-replication` + `raft-commit-safety` callback -> contract **verbatim**. -> -> **Depends** (whole spec): `raft-log-replication` (log model + AppendEntries entry points -> + the four callbacks), `raft-commit-safety` (`raft_advance_commit`, `on_commit_advanced`, -> `(lastLogTerm, lastLogIndex)` freshness). If those signatures change, re-validate per the -> design's Revalidation Triggers. - -## 1. Harness types and in-memory adapter backing - -- [x] 1.1 Extend the `XCE_TESTING` sim section of `raft.h` with the harness data types - - Add `raft_sim_msg_kind_t`, `raft_sim_msg_t` (vote/AE req+rsp union fields), `raft_sim_log_entry_t {term, value}`, `raft_sim_action_t {kind, actor, msg_index, term, index}`, and the `RAFT_SIM_MAX_LOG/MSGS/STEPS` bounds - - Extend `raft_sim_node_t` with `log[]`, `log_len`, `commit_index`, `applied[]`, and a persisted-`(term, votedFor)` shadow for Restart; extend `raft_sim_cluster_t` with `bag[]`+`bag_count`, PRNG state, `seed`, and `trace[]`+`trace_len` - - Observable completion: `raft.h` compiles under `XCE_TESTING`; no protocol declaration changed; `sizeof(raft_sim_cluster_t)` grows by the new fixed arrays - - _Requirements: 1.1, 1.3, 2.5_ - - _Boundary: raft.h (XCE_TESTING section only)_ - -- [x] 1.2 Implement the seeded PRNG and the in-memory adapter callbacks - - Add a small inlined deterministic generator (e.g. SplitMix64) seeded by `cluster.seed`; add `on_persist_entry` → `node.log[]`, `on_get_entry_term` → term|0, `on_truncate_from` → shrink `log_len`, `on_commit_advanced` → set `commit_index` + apply into `applied[]`; route `on_send_vote_request`/`on_send_append_entries` to a bag push - - Add `raft_sim_conformance_init(c, n, timeout, seed)` wiring all six callbacks + the bag - - Observable completion: a test initializes a cluster, drives one `raft_log_append` on a forced leader, and reads the entry back via the in-memory log and `on_get_entry_term` - - _Requirements: 1.1, 1.2, 2.1_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.1_ - -## 2. Message bag (mirrors raft.tla `messages`) - -- [x] 2.1 RED+GREEN: bag push/deliver routes vote and AppendEntries RPCs through the library handlers - - Test: pushing a vote-req and an AE-req then delivering them invokes `raft_on_vote_request` / `raft_on_append_entries_request`, and the handler reply is pushed back as a `*_RSP` message - - Implement `raft_sim_bag_push` and `raft_sim_bag_deliver(i)` (route by `kind`, honor `node.partition`, push replies) - - Observable completion: delivered requests produce matching responses in the bag; partitioned endpoints are not delivered - - _Requirements: 2.2, 2.4_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.2_ - -- [x] 2.2 RED+GREEN: Drop / Duplicate / Reorder mutate only the bag - - Test: `raft_sim_bag_drop`/`_duplicate` change bag multiplicity without mutating any node's term/role/log; reorder delivers in a PRNG-chosen order - - Implement the three bag operations (mirroring `raft.tla:443`/`448` `UNCHANGED` of non-message vars) - - Observable completion: node snapshots are identical before/after a drop or duplicate of an undelivered message - - _Requirements: 2.5_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 2.1_ - -## 3. The four invariant checkers (TDD against hand-built clusters) - -- [x] 3.1 RED+GREEN: Election Safety checker (P) - - Test: a cluster with two `LEADER` nodes at the same term fails; a single-leader-per-term cluster passes - - Implement `raft_sim_check_election_safety` over roles+terms read from `raft_snapshot` - - Observable completion: both directions assert correctly - - _Requirements: 3.1, 3.6_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.1_ - -- [x] 3.2 RED+GREEN: Log Matching checker (P) - - Test: two logs sharing `(idx, term)` but differing earlier fail; identical-prefix logs pass - - Implement `raft_sim_check_log_matching` (pairwise; shared `(idx, term)` ⇒ identical `1..idx`) - - Observable completion: both directions assert correctly - - _Requirements: 3.2, 3.6_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.1_ - -- [x] 3.3 RED+GREEN: Leader Completeness checker (P) - - Test: a committed `(index, term)` absent from a higher-term leader's log fails; present passes - - Implement `raft_sim_check_leader_completeness` using `commit_index` + per-node logs, tracking the highest-term leader to date - - Observable completion: both directions assert correctly - - _Requirements: 3.3, 3.6_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.1_ - -- [x] 3.4 RED+GREEN: State Machine Safety checker (P) - - Test: divergent `applied[k]` across two nodes fails; identical applied prefixes pass - - Implement `raft_sim_check_state_machine_safety` over `applied[]` - - Observable completion: both directions assert correctly - - _Requirements: 3.4, 3.6_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.1_ - -- [x] 3.5 RED+GREEN: `raft_sim_check_all` returns the first failing invariant and halts - - Test: a cluster violating two invariants reports the first; a clean cluster returns OK - - Implement `raft_sim_check_all` composing 3.1–3.4 in a fixed order - - Observable completion: first-failure semantics verified - - _Requirements: 3.5_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 3.1, 3.2, 3.3, 3.4_ - -## 4. Action scheduler and trace recorder - -- [x] 4.1 RED+GREEN: trace recorder records every action before it is applied - - Test: after a fixed step sequence, `cluster.trace[]` holds the actions in order; `raft_sim_dump_trace` prints seed, n, and the ordered actions - - Implement `raft_sim_action_t` recording (R1.4) + `raft_sim_dump_trace` (R5.1, R5.2); quiet on success unless `RAFT_SIM_VERBOSE` (R5.4) - - Observable completion: a recorded trace round-trips to a readable dump - - _Requirements: 1.4, 5.1, 5.2, 5.4_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 1.1_ - -- [x] 4.2 RED+GREEN: `raft_sim_step` drives one Next-equivalent action through the library and checks invariants - - Test: stepping reaches each enabled action kind over a run; every step calls `raft_sim_check_all`; a forced violation halts with a trace dump and a `<0` return - - Implement the seeded scheduler dispatch (Restart, Timeout, RequestVote/BecomeLeader via vote delivery, ClientRequest, AppendEntries via `raft_send_append_entries`, AdvanceCommitIndex via `raft_advance_commit`, Receive via bag deliver, Duplicate/Drop) selecting only **enabled** actions, driving **only** library entry points - - Observable completion: a multi-step run touches log append, AE replication, commit advance, and message faults; invariants checked each step - - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.6, 3.5_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 2.2, 3.5, 4.1_ - -- [x] 4.3 RED+GREEN: Restart preserves persisted `term`/`votedFor`/log - - Test: a Restart action re-inits a node while keeping `node.log[]`, recovered `log_len`, and last-persisted `(term, votedFor)`; volatile state resets per `raft.tla` `Restart` - - Implement the Restart branch (feed `initial_log_len` + restored `(term, votedFor)` into `raft_init`) - - Observable completion: a restarted node retains its log and term; no invariant violation results from the restart alone - - _Requirements: 2.1_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ - -- [x] 4.4 RED+GREEN: deterministic replay from `(n, seed, max_steps)` - - Test: two runs with the same `(n, seed, max_steps)` reach identical final per-node + bag state and an identical recorded trace - - Implement `raft_sim_run(c, max_steps)` and confirm no wall-clock/address/uninitialized input leaks into choices - - Observable completion: byte-identical replay verified across two runs - - _Requirements: 1.5, 1.6, 5.3_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ - -## 5. Targeted adversarial scenarios - -- [x] 5.1 Partition/heal scenario (P) - - Implement `raft_sim_scenario_partition_heal`: isolate a minority during appends, heal, drain the bag; assert all four invariants throughout and after convergence - - Observable completion: scenario passes with zero violations on the correct build - - _Requirements: 4.1_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ - -- [x] 5.2 Restart-churn scenario (P) - - Implement `raft_sim_scenario_restart_churn`: restart nodes mid-replication preserving persisted state; assert re-convergence with no violation - - Observable completion: scenario passes; restarted nodes converge to the leader's prefix - - _Requirements: 4.2_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.3_ - -- [x] 5.3 Drop/duplicate/reorder scenario (P) - - Implement `raft_sim_scenario_drop_dup_reorder`: perturb bag delivery order + multiplicity; assert idempotent convergence with all four invariants held - - Observable completion: scenario passes; duplicated AE is a no-op (already-have branch) - - _Requirements: 4.3_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ - -- [x] 5.4 Figure-8 scenario - - Implement `raft_sim_scenario_figure8`: a previous-term entry on a quorum, a leader change; assert the entry is NOT reported committed until a current-term entry commits (Leader Completeness / State Machine Safety preserved) - - Observable completion: the previous-term entry's `commit_index` does not cover it until a current-term commit - - _Requirements: 4.4_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ - -- [x] 5.5 Stale-rejoin scenario (bug-851 G5) - - Implement `raft_sim_scenario_stale_rejoin`: a term-lagging node rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does not re-elect at a reused term (Election Safety preserved) - - Observable completion: the rejoining node's term advances via AE; no duplicate-term leader appears - - _Requirements: 4.5_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ - -## 6. CI batch and randomized seed sweep - -- [x] 6.1 RED+GREEN: randomized M-seed sweep with zero violations - - Test: run `raft_sim_run` for a fixed `max_steps` across M seeds (M ≥ configured minimum) plus all targeted scenarios; fail if any seed/scenario violates any invariant - - Implement the sweep harness in `test_raft.c`; bound `max_steps`, M, and array sizes so the target fits the library test-suite budget - - Observable completion: the full batch runs clean and within budget - - _Requirements: 6.1, 6.2, 6.4_ - - _Boundary: test_raft.c_ - - _Depends: 5.1, 5.2, 5.3, 5.4, 5.5_ - -- [x] 6.2 Verify the conformance batch is ASAN/UBSAN-clean - - Run the conformance batch under the library test sanitizer build; fix any leak/overflow/UB in the harness (fixed-size arrays, no allocation) - - Observable completion: `/xce-build lib` sanitizer run reports no memory errors, leaks, or UB for the conformance target - - _Requirements: 6.3_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 6.1_ - -## 7. Deliberately-broken-build negative test (the oracle's teeth) - -- [x] 7.1 RED+GREEN: compile-time fault switch emulates the bug-851 defects in the adapter - - Implement `RAFT_SIM_BREAK_TERM_STAMP` (adapter `on_persist_entry` writes term 0) and `RAFT_SIM_BREAK_TRUNCATE` (adapter `on_truncate_from` no-op), confined to the test/harness build configuration; default off - - Observable completion: with a macro off, scenarios are clean; the macro is not set in the default CI run and does not touch `raft.c` - - _Requirements: 7.1, 7.3_ - - _Boundary: raft_sim.c_ - - _Depends: 5.4, 5.5_ - -- [x] 7.2 RED+GREEN: negative test asserts both directions - - Test: under the broken configuration the Figure-8 / divergent-suffix scenario triggers a Log Matching / Leader Completeness / State Machine Safety violation (with trace dump); under the correct configuration the same scenario passes — both asserted in one test - - Observable completion: the broken build is caught and the correct build passes, proving checker sensitivity - - _Requirements: 7.1, 7.2, 7.4_ - - _Boundary: test_raft.c_ - - _Depends: 7.1_ - -## 8. Optional TLC-consumable trace export - -- [x]* 8.1 TLC-consumable trace export (deferred; off by default) - - Implement `raft_sim_export_trace` gated by `RAFT_SIM_EXPORT`: per step, emit the action + tracked `raft.tla` variables (`messages` summary, per-node `currentTerm`/`state`/`log`/`commitIndex`) named per `raft/raft.tla`, in a stable line-oriented format; off by default - - Observable completion: with `RAFT_SIM_EXPORT` set, a run emits a stable, spec-named record stream; default runs emit nothing extra - - _Requirements: 8.1, 8.2, 8.3_ - - _Boundary: raft_sim.c, test_raft.c_ - - _Depends: 4.2_ -``` - -## Implementation Notes - -- **Task 4.2 — UPSTREAM raft.c BUG SURFACED (bug-1095/1096, bug-851 family).** While building - the scheduler, the conformance oracle detected a real split-brain defect in `raft.c` - `raft_on_vote_reply`: (1) vote grants are tallied with a plain counter that ignores - `from_idx`, so a legitimate `DuplicateMessage` of a vote-response double-counts toward - quorum; (2) there is no stale-term (`mterm < currentTerm`) reply drop. In `raft.tla` - `votesGranted` is a SET (idempotent under the legal Duplicate action) and - `DropStaleResponse` discards stale replies — so a correct implementation must tolerate - both. Result: two leaders at the same term (Election Safety violation) on some seeds. - This is OUT OF THIS SPEC'S BOUNDARY (raft.c protocol logic, owned by raft-core / - ha-term-safety). The harness must NOT mask it (R2.6). **Consequence for downstream - tasks:** any task asserting "zero invariant violations on the correct build" (5.x targeted - scenarios, 6.1 M-seed sweep, 7.2 correct-direction) may surface this defect. Implementers - of those tasks MUST NOT weaken the oracle or hand-pick seeds to hide the bug; if a - legitimate scenario surfaces it, report BLOCKED with the upstream cause rather than - working around it. Seed 3 is clean over 600 steps and is used only to prove the 4.2 - scheduler mechanism reaches all action kinds. -- **UPDATE (post-4.2): bug-1095/1096 FIXED in raft.c** (commit `fix(raft): vote-reply - per-peer dedup + stale-term drop`). `raft_on_vote_reply` now tracks grants in a per-peer - `vote_replies[]` set (idempotent under DuplicateMessage) and drops stale-term replies - (DropStaleResponse), mirroring the existing pre-vote idiom; 47 raft / 264 lib tests pass. - The library is now correct, so downstream tasks (5.x scenarios, 6.1 M-seed sweep, 7.2 - correct-direction) CAN and MUST assert zero invariant violations on the correct build. - The user authorized this cross-boundary fix. The deliberately-broken-build fault switch - (task 7) remains the way to deliberately re-introduce a divergence for the negative test. -- **Task 5.1 — scenario commit-propagation pattern (applies to 5.2–5.5).** A follower's - `commit_index`/`applied[]` only advance when it RECEIVES an AppendEntries carrying a - `leader_commit` greater than its own (the library applies it). A scenario that replicates - entries and calls `raft_advance_commit` on the LEADER, but never sends a follow-up - AE/heartbeat round afterward, leaves followers at `commit_index=0` — making any per-follower - commit-prefix / State-Machine-Safety / Leader-Completeness assertion VACUOUS on the follower - side (5.1 has this weakness; Log Matching still validates log convergence there). For - commit-critical scenarios — ESPECIALLY 5.4 (Figure-8) where follower commit advancement is - the whole point — after `raft_advance_commit` on the leader, drive ANOTHER AppendEntries - round so followers observe the new `leader_commit` and advance, then assert follower - `commit_index`/`applied[]` actually moved. Do NOT assert convergence via a loop bounded by - `min(commit_index)` when followers are at 0 (it silently passes). 5.1 was accepted as-is - (meets R4.1 via throughout-checking + log matching); 5.2–5.5 should propagate commit so - their commit assertions are non-vacuous. -- **CORRECTION (task 5.2, confirmed): this library uses a LEADER-ONLY commit model.** - `raft_on_append_entries_request` does `(void)leader_commit;` (raft.c:646-655) — a FOLLOWER - never advances its `commit_index`/`applied[]` from AppendEntries; `on_commit_advanced` - fires ONLY on the leader via `raft_advance_commit` (raft.c:816). This is intended XCE - design (follower commit visibility is a controller/heartbeat concern — see - ha-heartbeat-committed-propagation — not the raft library). CONSEQUENCE for scenarios: a - follow-up AE round does NOT advance follower commit; do NOT assert follower - `commit_index == leader commit_index`. Assert re-convergence over the LOG instead - (every node's `log_count` reaches the leader's tip + per-entry term match / Log Matching), - and assert leader-side commit via `leader.commit_index`. For 5.4 (Figure-8), the property - under test is LEADER-side (the leader must not mark a previous-term entry committed until a - current-term entry commits) — driven by `raft_advance_commit` on the leader, which works. - applied[]/SM-Safety is exercised across leader CHANGES (each node accumulates applied[] - while it is leader). This SUPERSEDES the follower-commit-propagation guidance above. -- **PROCESS FIX (task 5.3 commit e445404 was BROKEN).** That commit captured the - drop_dup_reorder decl (raft.h) + test (test_raft.c) but NOT its implementation in - raft_sim.c — a stray `git checkout` (a reviewer cleaning up the `src/build` symlink) - reverted the uncommitted impl between review and commit, so the lib build was broken - (undefined reference) from e445404 onward. Detected during task 5.4. The exact - implementation was recovered (byte-identical) and re-inserted into raft_sim.c; it is - committed together with task 5.4. LESSON: after every task commit, verify `git show - --stat` includes every expected source file (raft_sim.c was silently absent from - e445404's stat). Reviewers must use targeted `git checkout <path>` (never `git checkout .` - / `git stash`) so uncommitted impl is never reverted. diff --git a/.kiro/specs/resource-distribution-mapping/requirements.md b/.kiro/specs/resource-distribution-mapping/requirements.md deleted file mode 100644 index ccf63374..00000000 --- a/.kiro/specs/resource-distribution-mapping/requirements.md +++ /dev/null @@ -1,7 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Map resources.yaml's `resourcemap` field (per-host static/dynamic resource value assignments) through to the controller. Currently xcecommon.mapping._build_resource cannot map it because controller.capnp's Resource struct has its distributions/hostBased fields commented out (controller.capnp:208-209, corresponding to xce_resource_t's distributions/hbased_defined in include/xce/types.h). Scope: (1) design and un-comment/wire the capnp Resource.distributions (List(ResourceDistribution)) and Resource.hostBased fields, (2) regenerate generated capnp bindings, (3) implement xcecommon.mapping's YAML resourcemap -> capnp distributions/hostBased translation (YAML shape: list of {value, location: [hostnames]} entries -- see src/configs/resources.yaml for real examples covering dynamic hostbased, static shared, and static per-host-distinct resource patterns), (4) remove resourcemap from build.py's _UNMAPPED_BUT_VALID_ITEM_KEYS once mapped, (5) verify round-trip against the C side (res_distribution_t / xce_resource_t consumers, e.g. scheduler.c/worker.c resource distribution logic) to confirm the encoding actually matches what the C code expects at runtime, not just that it compiles. This is a real capnp schema change (adds wire-format fields) affecting the ClusterConfig contract other components already depend on -- needs design review, not a blind field un-comment. - -## Requirements -<!-- BLOCKED: scope decision pending, see research.md -- do not generate requirements until one of the three candidate scopes there is chosen. Initial framing (a simple capnp-field-uncomment) understated the real scope once the C-side consumers were traced: the fields are dead end-to-end (not just uncommented capnp), a separate live per-host mechanism already exists via hosts.yaml, and two of the three plausible resourcemap semantics (shared pool, dynamic per-host reporting) require genuinely new controller/scheduler/worker runtime capability, not just a mapping fix. --> diff --git a/.kiro/specs/resource-distribution-mapping/research.md b/.kiro/specs/resource-distribution-mapping/research.md deleted file mode 100644 index 560e2016..00000000 --- a/.kiro/specs/resource-distribution-mapping/research.md +++ /dev/null @@ -1,30 +0,0 @@ -# Research: resourcemap / distributions / hostBased - -## Status: scope decision pending — do not generate requirements.md until a scope option below is chosen - -## What triggered this spec - -`xceadmin config build` warns `resources.yaml: field 'resourcemap' on '<name>' is schema-valid but has no ClusterConfig destination -- ignored` for every resource entry that uses `resourcemap`. The initial framing (a quick capnp-field-uncomment-and-wire task) turned out to understate the real scope once the C-side consumers were traced — see findings below. - -## Findings (verified against code, file:line cited) - -- `res_distribution_t` (`include/xce/types.h:127-131`): `{value: xce_val_t*, nhosts: int, hosts: char**}` — maps 1:1 to one `resourcemap` YAML entry `{value, location: [...]}`. Commented-out capnp counterpart: `src/schema/controller.capnp:195-198,208-209`. -- `xce_resource_t.hbased_defined` is typed `xce_val_t*` (a value, not a bool flag) — plausibly a fallback/default value for hosts not covered by any explicit `distributions` entry (matches the `location: [default]` sentinel used by `dynres1`/`strres1`/`staticres3`/`host_quota`). Inference only — nothing in the codebase reads this field to confirm. -- **The fields are dead on the C side too, not just uncommented in capnp.** `resobj_init()` (`src/lib/resobj.c:37-51`) explicitly zeroes `dst->distributions`/`dst->hbased_defined` with `/* FIXME: */` comments, even under a hypothetical populated input. `scheduler.c:787-813`'s resource update handler copies only `valtype`/`ordtype`/`flags`/`interval`, with a comment explaicitly excluding these two as "deep-structs." No other file in scheduler/, worker/, controller/, lib/ reads `res_distribution_t`, `->distributions`, or `->hbased_defined` at all (repo-wide grep, zero other hits). -- **A separate, fully-live per-host resource-value pipeline already exists**, independent of `resourcemap`, via `hosts.yaml`'s host-level `resource: [{name, value}]` list. Full trace: `mapping.py:_build_host_config` → `HostConfig.resources` (capnp) → controller `apply_join_state()` → `mark_host_res()` (`src/controller/worker.c:163-221`) → `update_val_res()` writes `w->host.resvals[idx]` → pushed to scheduler via `collect_resval()`/`Worker.resvals` → `schedobj.c:286-304` `zarray_put(&(worker->resvals), ...)` → read by `sched_check_host_satisfy_()` (`src/scheduler/src/resource.c:330,357-379`) during job placement. -- `mark_host_res()` (`worker.c:190-193`) **actively rejects** a resource flagged `XCE_RESF_SHARED`: `if (r->flags & XCE_RESF_SHARED) { ERROR(...); return -EINVAL; }`. So "shared pool" and "per-host static value" are mutually exclusive today, by construction. -- `XCE_RESF_SHARED`/`GLOBAL` pooling machinery (`sched->sharedInst`, `sched_check_shared_satisfy_()`, `src/scheduler/src/resource.c:232-282`) is real and live, but exclusively serves job-limit virtual resources (`src/lib/xcelimit.c:399-404`'s `add_vres()`), never a physical `resources.yaml` resource. The scheduler plumbing for "N consumers draw from one pooled value" is architecturally reusable, just not reachable from `resourcemap` today. -- `interval`/dynamic-resource semantics have **zero enforcement logic anywhere** — `last_report` is never read; `interval` is only copied, never acted on. The only real dynamic-reporting mechanism is `src/worker/src/resource_sync.c`'s 11 hardcoded built-in load indices (`init_builtin_load_indices()`, lines 59-94). A plugin hook exists to register a *custom* dynamic resource (`resource_sync_register_load_index()`, line 104) but nothing calls it from any YAML-driven path — a user-defined dynamic resource (`dynres1`/`dynres2`-style) has no live way to ever acquire a value today. -- `src/configs/resources.yaml`'s own example comments are internally inconsistent about "share" semantics: `staticres1` (one `resourcemap` entry, two hosts in `location:`) says "host1 and host2 **share** the 100 res"; `staticres2` (two separate entries, one host each) says "each of host1 and host2 **shares** the 100 res" — opposite shapes, same word. Not a reliable semantic source as-is. - -## Three candidate scopes (sizes verified against the above, not guessed) - -1. **Static replication to N hosts** (`staticres3`-style: one `resourcemap` declaration → the same static value written into N named hosts' per-host `resvals`, reusing the existing `mark_host_res()`/`update_val_res()` path). Pure DRY/ergonomics win — functionally redundant with just repeating `{name, value}` across N `hosts.yaml` host entries, since `hosts.yaml` already does the "one host, one static value" case live end-to-end. Small: capnp field un-comment + mapping.py translation + a controller-side loop over the host list at config-load/push time. No scheduler changes, no new runtime concept. - -2. **Static replication + shared pool** (adds `staticres1`/`staticres4`-style: N hosts collectively draw down ONE pooled value). Requires: allowing `XCE_RESF_SHARED` on a physical resource (currently `mark_host_res` refuses this), wiring `sched->sharedInst` for resources.yaml-sourced resources (today limits-only), and defining job-placement semantics for a shared physical resource (distinct from the existing shared *limit* semantics — a resource being consumed to zero is a scheduling constraint, not a rejection). Real new capability, touches job-placement code. - -3. **All three, plus dynamic per-host custom resource reporting** (`dynres1`/`dynres2`-style). Needs a new worker-side mechanism: something must call `resource_sync_register_load_index()` (or an equivalent) per YAML-declared dynamic resource, decide how/when it's collected per host, and thread it into the same `resvals` pipewith proper interval-based refresh — none of which exists today even as a partial scaffold. Largest scope. - -## Decision - -Not yet made — user chose to record the options and decide later rather than commit now. **Do not proceed to requirements.md generation until one of the three scopes (or a variant) is chosen.** When resumed, re-run `/kiro-spec-requirements resource-distribution-mapping` with the chosen scope stated explicitly, since the EARS requirements (and everything downstream) differ substantially by option. diff --git a/.kiro/specs/resource-distribution-mapping/spec.json b/.kiro/specs/resource-distribution-mapping/spec.json deleted file mode 100644 index 6f1890e7..00000000 --- a/.kiro/specs/resource-distribution-mapping/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "resource-distribution-mapping", - "created_at": "2026-07-23T08:25:47Z", - "updated_at": "2026-07-23T08:25:47Z", - "language": "en", - "phase": "initialized", - "approvals": { - "requirements": { - "generated": false, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false -} diff --git a/.kiro/specs/resource-value-verification/design.md b/.kiro/specs/resource-value-verification/design.md deleted file mode 100644 index 7ae1aa3f..00000000 --- a/.kiro/specs/resource-value-verification/design.md +++ /dev/null @@ -1,360 +0,0 @@ -# Design Document - -## Overview -This feature closes a coverage gap in `xce_test`'s worker-context `resupdate` command (its resource-name table is missing 7 real dynamic resources) and adds a repeatable, self-contained skill that injects a synthetic resource value through `resupdate` and confirms it is correctly received and stored at the controller, the live scheduler, and the Go apiserver — the exact three-layer propagation path that took a multi-hour live-deployment investigation to verify manually during bug-1569. - -**Purpose**: Turn a class of bug (resource-value routing/propagation defects) that previously required deploying real worker/controller/scheduler/apiserver daemons and waiting on periodic reporting cycles into a command that reproduces and confirms the fix in seconds. -**Users**: XCE engineers debugging or regression-testing resource-value propagation across the controller/scheduler/apiserver boundary. -**Impact**: Extends `xce_test` (test.c) with a small, backward-compatible change; extends `xce-deploy-ha`'s `deploy-ha.sh`/`SKILL.md` with an opt-in, conditionally-generated apiserver deployment step (see Design Decisions below — no change to existing callers that don't need an apiserver); adds two single-line trace-log additions (scheduler, apiserver); adds one new skill. No production runtime *behavior* changes — the trace-log additions are inert log-only output at debug/trace level, and the `xce-deploy-ha` extension only activates when the apiserver binary is present, mirroring its existing worker/scheduler conditional-generation pattern. - -### Goals -- `resupdate` accepts any resource name the cluster has registered (Requirement 1). -- Confirm the existing `join` + `resupdate` worker-context flow needs no real worker daemon (Requirement 2). -- A single command deploys controller + scheduler + apiserver, injects one or more resource values, and reports PASS/FAIL per layer (Requirements 3–6). - -### Non-Goals -- Renaming `xce_test`, per-command help, reference documentation, JSON output mode, non-interactive batch mode, argument-convention unification — explicitly deferred to a future "enhance xce_test" project (see requirements.md Future Considerations). -- A new live scheduler admin query RPC — rejected in favor of trace-log observation, which needs no new RPC at all (see research.md Design Decisions). -- The record+replay+REPL flow and the apiserver REST/`xcecli` query path — both considered and superseded during design validation in favor of live trace-log observation at each layer's point of application; see research.md Design Decisions for why (this feature's own originating bugs, bug-1568 and bug-1569A, were both storage-layer defects found via exactly this technique, and the REPL's step/continue requirement had already produced one false negative live). -- General-purpose apiserver production deployment/HA — the new `xce-deploy-ha` apiserver step is a minimal single-node instance for local verification only, and no longer needs functional auth/RBAC since no REST query is issued against it. -- A new, standalone apiserver deploy script — considered and superseded in favor of extending `xce-deploy-ha` itself, reusing its already-precedented conditional (binary-present-only) generation pattern instead of re-implementing the same conventions in parallel (see research.md Design Decisions). -- CI/automated-pipeline integration. -- Fixing unrelated bugs the verification run happens to surface — those are reported, not silently patched, by this feature. - -## Boundary Commitments - -### This Spec Owns -- `set_resval()`/`cmd_resupdate` in `src/controller/testsuite/test.c`: accepting any registered resource name, with an explicit type-override escape hatch. -- A single trace/debug log line in `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), emitted at the point the live scheduler daemon applies a resource update — the only scheduler-side change this spec makes. -- A single trace/debug log line in `OnResourceLocationUpdated` (`src/frontend/apiserver/backend/handler.go:92-95`), emitted at the point the apiserver stores a received resource update — the only apiserver-side change this spec makes. -- An opt-in, conditionally-generated apiserver deployment step added to `xce-deploy-ha`'s `src/tools/deploy-ha.sh` and documented in its `SKILL.md` (Step 4d, alongside the existing Step 4b worker/scheduler steps) — generates `start-apiserver.sh` only when a pre-built apiserver binary already exists at its own Go-build output path (same "caller opts in by having already built it" spirit as the worker/scheduler conditional, though checked at a different path since apiserver is a separate build system — see Components for the precise distinction). Existing `xce-deploy-ha` callers that never build an apiserver (e.g. `xce-test-ha`) see zero behavior change. -- A new skill (`.claude/skills/xce-test-resources/`) that orchestrates deploy + inject + three-layer verify + PASS/FAIL report. - -### Out of Boundary -- `resource_sync.c`'s real, production periodic load-index collection — unchanged. -- The controller's resval-forwarding hooks (`do_update_resvals`, `sched_resource_hook_fun`, `apiserver_resource_hook_fun`) — unchanged; this design only consumes them. -- The scheduler's REPL (`repl.c`/`repl_info.c`), recording admin RPCs, and the apiserver's REST API/`xcecli` query path — none of these are touched or driven by this design (superseded by the trace-log decision; see research.md Design Decisions). They remain available as a future follow-up if a query-layer-specific check is ever needed. -- All other scheduler/apiserver logic beyond the two single-line trace additions listed under "This Spec Owns" above — no behavior change, no new code paths, purely additive log output. -- `xce-deploy-ha`'s existing controller/worker/scheduler steps (1–4c) — unmodified; only a new, conditionally-triggered step is added alongside them. -- The broader "enhance xce_test" ideas listed in requirements.md's Future Considerations. - -### Allowed Dependencies -- `xce-deploy-ha`'s controller deployment conventions and its existing worker/scheduler conditional-generation pattern (directory layout, TLS cert generation, `cluster.bin` build via `xceadmin`, Step 3's "generate the extra start script only if the binary exists" rule) — reused directly, by extending the same script, not duplicated in a parallel one (see research.md Design Decisions). -- `xce_test`'s existing worker context (`join`, `resupdate`) and client context (`resval`) — unmodified except for Requirement 1's table extension. -- `make build-server` (`src/frontend/Makefile`) to build the apiserver binary; `make test-integration`'s `XCE_BIN`/`SRV_URL` convention as the closest existing precedent for pointing it at a controller. -- Each daemon's own existing logging infrastructure (`tinylog` in C for the scheduler, zap `logger.Debug` in Go for the apiserver — per `error-handling.md`) as the channel the new trace lines are written through; no new logging mechanism is introduced. - -### Revalidation Triggers -- The wire shape of `xce_resval_t` / `resupdate`'s CLI flags changes. -- `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` handling, or `OnResourceLocationUpdated`'s call site, is refactored such that the trace log's format/location moves. -- `xce-deploy-ha`'s Step 3 conditional-generation pattern (or its directory layout) changes shape — this feature's apiserver step is now part of that same script, so a shape change there is a direct, first-party revalidation trigger rather than a drift risk between two separate scripts. -- If a future bug turns out to live specifically in the query/REST/serialization path rather than storage, revisit adding a real query-based check for that layer (not precluded by this design, just not built here). - -## Architecture - -### Existing Architecture Analysis -A resource value reaches all three layers through **one** controller-side write, not three independent paths (see research.md): `do_update_resvals()` applies it to the controller's own `w->host.resvals` (queried via `resval`) and fires `HOOK_DETAIL_RESOURCE_UPDATE`, which fans out to both `sched_resource_hook_fun` (scheduler) and `apiserver_resource_hook_fun` (apiserver). This design's job is purely to exercise this existing path with a synthetic worker and observe the result at each of the three existing points of application — it introduces no new production data path, and (per the trace-log decision) reads state at the storage point rather than through a query surface. - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - Skill[xce-test-resources skill] - WorkerCtx[xce_test worker context] - ClientCtx[xce_test client context] - Controller[xce_controller] - Scheduler[xce_scheduler daemon] - SchedLog[scheduler log] - Apiserver[apiserver] - ApiLog[apiserver log] - - Skill --> WorkerCtx - WorkerCtx -->|join and resupdate| Controller - Skill --> ClientCtx - ClientCtx -->|resval query| Controller - Controller -->|resource update notify| Scheduler - Controller -->|resource update push| Apiserver - Scheduler -->|trace log on apply| SchedLog - Apiserver -->|trace log on apply| ApiLog - Skill -->|tail and grep| SchedLog - Skill -->|tail and grep| ApiLog -``` - -**Architecture Integration**: -- Selected pattern: black-box orchestration over existing interfaces plus two minimal, inert trace-log additions at each layer's point of application — this feature adds a coordinating layer (the skill), one small compatibility fix (`resupdate`'s resource table), and the two log lines; it does not add a new data path or query surface. -- Domain boundaries: `resupdate` table fix (C, test.c) is independent of the skill (bash orchestration); the `xce-deploy-ha` apiserver extension is independent of both and reusable by any `xce-deploy-ha` caller, not just this skill; the two trace-log additions are independent single-line changes in their respective daemons. -- Existing patterns preserved: `xce-deploy-ha`'s scratch-directory deploy convention and worker/scheduler conditional-generation pattern (extended, not duplicated, for apiserver), `xce-test-ha`'s VERDICT-line reporting convention, each daemon's own existing logging infrastructure (no new logging mechanism). -- New components rationale: the `xce-test-resources` skill is new because no existing tool exercises all three layers together; the `resupdate` fix is new because the current table is demonstrably incomplete; the `xce-deploy-ha` apiserver step and `start-all.sh` extend an existing tool rather than introduce a new one, following its own established conditional-generation precedent; the two trace-log lines are new because neither layer currently logs the value it applies (confirmed by direct inspection — see research.md). -- Steering compliance: production code changes are two single-line, debug/trace-level log additions with no behavior change, plus `test.c`'s test-tooling scope; TDD applies to the `set_resval()` change (see Testing Strategy). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Test client (C, gnu17) | `xce_test` (existing binary, `src/controller/testsuite/test.c`) | Inject via worker context; query controller via client context | Requirements 1, 2, 3 | -| Scheduler (C, gnu17) | Live `xce_scheduler` daemon (existing, one new trace-log line) | Applies the resource update and logs it at the point of application | Requirement 4; see research.md Design Decisions | -| API server (Go) | `apiserver` binary via `make build-server` (one new trace-log line in `backend/handler.go`) | Receives and stores the resource update, logs it at the point of application | Requirement 5; see research.md Design Decisions | -| Orchestration | Bash skill script, `xce-deploy-ha`/`xce-test-ha` conventions | Deploy, inject, tail/grep each layer's log, verify, report, clean up | Requirement 6 | - -## File Structure Plan - -### Directory Structure -``` -src/ -└── controller/testsuite/ - ├── resval_typeresolve.c # NEW: set_resval()'s name→Value_u_which resolution, extracted for isolated unit testing - ├── resval_typeresolve.h # NEW: its declaration - ├── test_resval_typeresolve.c # NEW: Unity unit tests for resval_typeresolve.c (4 cases, see Testing Strategy) - └── test.c # MODIFIED: set_resval() delegates to resval_typeresolve.c; cmd_resupdate() gains -t flag + post-update accept/reject verification -src/controller/meson.build # MODIFIED: add resval_typeresolve.c to xce_test_src (both xce_test and test_resval_typeresolve.c link against it); add test_resval_typeresolve.c + a new controller_unit_test-style test() target — these are hand-maintained explicit file lists, not globs -src/scheduler/src/core_events.c # MODIFIED: one new trace/debug log line in xce_core_push_events()'s XCE_CORE_EVENT_RESVALS_DELTA case -src/frontend/apiserver/backend/handler.go # MODIFIED: one new trace/debug log line in OnResourceLocationUpdated -src/tools/ -└── deploy-ha.sh # MODIFIED: Step 3 gains (a) a conditional apiserver-config-generation branch mirroring the existing worker/scheduler conditional, generating start-apiserver.sh (minimal boot-only config + start command — no functional auth/RBAC required, see research.md Design Decisions), and (b) an always-generated start-all.sh that sequences start.sh + health-check + whichever of start-worker.sh/start-scheduler.sh/start-apiserver.sh exist for this deploy — automates the manual recipe SKILL.md's Steps 4/4b/4c already document -.claude/skills/ -├── xce-deploy-ha/ -│ └── SKILL.md # MODIFIED: new Step 4d documenting the opt-in apiserver step, alongside existing Step 4b (worker/scheduler) -└── xce-test-resources/ - ├── SKILL.md # NEW: skill description, usage, VERDICT format (xce-test-ha convention) - └── run-resource-verify.sh # NEW: orchestrates deploy + inject + 3-layer verify (log-tail/grep for scheduler+apiserver) + verdict + cleanup -``` - -### Modified Files -- `src/controller/testsuite/test.c` — `set_resval()` becomes a thin wrapper around the new `resval_typeresolve.c` module; `cmd_resupdate()` gains an optional `-t <numeric|string>` flag per `-n`/`-v` pair, and, after sending the update, a follow-up per-name verification query (see Components below — Requirement 1.2/1.3's mechanism). -- `src/controller/meson.build` — `xce_test_src` (line 57) gains `testsuite/resval_typeresolve.c` so both `xce_test` and the new unit test binary can link it; a new Unity test source list/target (following the `controller_unit_test`/`sched_push_unit_test` pattern already in this file, e.g. lines 86, 215, 222) gains `testsuite/test_resval_typeresolve.c`. Both lists are hand-maintained and explicit, not glob-based — this file must be edited or the new module/test never builds. -- `src/scheduler/src/core_events.c` — `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case gains one trace/debug-level log line (via this component's existing `tinylog`-based logging, e.g. `TRACE`) emitting the resolved worker host, resource name, and applied value, immediately after the value is written into the worker's live state. Debug/trace level only — never fires at WARN/ERROR, never changes control flow. This is the exact site bug-1568 was root-caused at via ad hoc instrumentation; this spec makes that instrumentation permanent and machine-greppable. -- `src/frontend/apiserver/backend/handler.go` — `OnResourceLocationUpdated` (line 92) gains one `logger.Debug(...)` call (zap, per this component's existing convention) logging the resource name, host/location, and stored value, immediately after `store.ResourceLocations.Put()`. -- `src/tools/deploy-ha.sh` — Step 3 gains two additions: - 1. A new conditional branch, same *opt-in-by-presence* spirit as the existing worker/scheduler branch (but checking a pre-built binary at the Go build's own output path, e.g. `/tmp/apiserver`, rather than alongside `$BINARY` — a different build system, see Components below): if present, generate `$TOP/scripts/start-apiserver.sh` (minimal boot-only config) and a `logger.yaml` with `logging.level: debug`; absent, generates nothing, so existing callers are unaffected. - 2. An always-generated `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skips any that weren't) — the same backgrounding pattern (`&`, or `setsid nohup … & disown` per SKILL.md's own noted improvement) Steps 4/4b/4c already document doing by hand. -- `.claude/skills/xce-deploy-ha/SKILL.md` — documents the new opt-in apiserver step as "Step 4d" (following the existing Step 4b worker/scheduler pattern) and `start-all.sh` as a documented shortcut for Steps 4/4b/4c combined. - -### New Files -- `src/controller/testsuite/resval_typeresolve.c`/`.h` — `set_resval()`'s existing `resinfo[]` table lookup plus the new numeric-by-default fallback, moved out of `test.c` into a small, dependency-light module (no networking, no REPL state) so it is independently unit-testable. `test.c` (4298 lines, its own `main()`, `libedit`/network/REPL dependencies) cannot itself be white-box `#include`d into a Unity test the way smaller single-purpose files in this codebase are — extraction is required, not optional, to satisfy Requirement 1.1/1.4's testability. -- `src/controller/testsuite/test_resval_typeresolve.c` — Unity test program for `resval_typeresolve.c`, registered as its own `meson.build` `test()` target (see Modified Files above); this is the file task 1.1's RED tests and task 2.1's GREEN tests live in. - -## System Flows - -```mermaid -sequenceDiagram - participant Skill as xce-test-resources - participant Worker as xce_test (worker ctx, ONE session) - participant Nested as xce_test (nested ! client ctx) - participant Ctrl as controller - participant Sched as scheduler daemon - participant Api as apiserver - - Skill->>Worker: join synthetic-worker - Skill->>Worker: resupdate -n <res> -v <value> - Worker->>Ctrl: xce_update_resvals (wire) - Ctrl->>Ctrl: apply to w.host.resvals - Ctrl->>Sched: notify schedUpdateResvals - Ctrl->>Api: notify schedUpdateResvals - Sched->>Sched: apply + trace-log resource, host, value - Api->>Api: store.ResourceLocations.Put + debug-log resource, host, value - Worker->>Nested: !<shell-cmd> --internal-controller-check (worker STILL connected) - Nested->>Ctrl: resval -n <res> -m synthetic-worker (query) - Ctrl-->>Nested: controller-observed value - Nested-->>Worker: result written to a marker file (nested exit code ignored) - Worker->>Ctrl: quit (worker disconnects HERE -- not before) - Skill->>Skill: parse marker file for controller-observed value - Skill->>Sched: tail/grep scheduler log for expected line (bounded wait) - Sched-->>Skill: scheduler-observed value (or timeout) - Skill->>Api: tail/grep apiserver log for expected line (bounded wait) - Api-->>Skill: apiserver-observed value (or timeout) - Skill->>Skill: compare all three vs injected value, emit verdict -``` - -**Load-bearing constraint the diagram makes explicit (as-built, bug-1579)**: the controller-layer check is NOT an independently-launched client session run after the worker disconnects. Any host-attrs-derived resource (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` — `worker.c`'s `N_HOST_ATTRS`) is reset to baseline by the controller the instant the injecting worker disconnects — so a controller-layer check run as a separate, later session against any host-attrs name would ALWAYS false-MISMATCH, independent of whether the underlying propagation is actually correct (discovered as bug-1579, task 6.2, round-1 REJECTED). The fix nests the controller-layer `resval` query INSIDE the injecting worker's own still-open session, via `xce_test`'s `!<shell-command>` escape (`preprocess_and_run()` — works in any REPL context, including `worker`, with no `leave_ctx()`/disconnect) — the query genuinely runs before `quit`, not merely before some unrelated timeout. The scheduler and apiserver legs have no such constraint: both are pure log-tail/grep against a file, entirely independent of whether the worker connection is still open, so they run after the worker session ends like any other step. - -Key simplification from the superseded design: no `record`/`replay`/REPL round-trip and no REST/`xcecli` query are needed — the scheduler and apiserver legs both poll their own live daemon's log file for the trace line the injection is expected to produce, within a bounded wait, exactly like the controller leg's own bounded-wait posture. This removes the REPL's `step`/`continue` pitfall entirely (there is no replay step in this flow) and the apiserver's auth/RBAC config from the critical path (see research.md Design Decisions). - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1–1.4 | `resupdate` accepts any registered resource name | `set_resval()`/`cmd_resupdate` (test.c) | CLI flags `-n`/`-v`/`-t` | — | -| 2.1–2.2 | Inject without a real worker daemon | `xce_test` worker context (`join`, `resupdate`, existing) | CLI | Sequence diagram: join → resupdate → wire update | -| 3.1–3.2 | Controller-side visibility check | `xce-test-resources` skill, `xce_test` client context (`resval`, existing) | CLI | Sequence diagram: `resval` query → controller-observed value | -| 4.1–4.3 | Scheduler-side visibility check | `xce-test-resources` skill, scheduler trace log (`core_events.c`, new) | Log line, bounded-wait grep | Sequence diagram: scheduler applies + trace-logs → skill tails/greps scheduler log | -| 5.1, 5.2, 5.4 | Apiserver-side visibility check | `xce-test-resources` skill, `xce-deploy-ha` apiserver step, apiserver trace log (`handler.go`, new) | Log line, bounded-wait grep | Sequence diagram: apiserver stores + debug-logs → skill tails/greps apiserver log | -| 5.3 | Apiserver local deployment capability | `xce-deploy-ha` apiserver step + `start-all.sh` | Bash script | — | -| 6.1–6.5 | Repeatable run, PASS/FAIL verdict, cleanup | `xce-test-resources` skill (`run-resource-verify.sh`) | CLI, VERDICT line | Sequence diagram: final compare + verdict step | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|--------------|--------|---------------|---------------------------|-----------| -| `set_resval()` / `cmd_resupdate` | Test Tooling / C | Accept any registered resource name for injection | 1.1–1.4 | `xcecli_update_resvals` (P0), `xcecli_resval` (P0) | Batch | -| Scheduler resval trace log | Scheduler Core / C | Log the resolved host/resource/value at the point the live daemon applies an update | 4.1–4.3 | `xce_core_push_events()` (P0), `tinylog` (P0) | Event | -| Apiserver resval trace log | API Server / Go | Log the resource/host/value at the point the apiserver stores a received update | 5.1, 5.2, 5.4 | `OnResourceLocationUpdated` (P0), zap `logger` (P0) | Event | -| `xce-deploy-ha` apiserver step | Deployment Tooling / Bash | Extend `deploy-ha.sh` to conditionally stand up a minimal, boot-only single-node apiserver for local verification | 5.3 | `make build-server` (P0), a running controller (P0), `deploy-ha.sh`'s existing Step 3 (P0) | Batch | -| `xce-test-resources` skill | Test Tooling / Bash | Orchestrate deploy, inject, three-layer verify (log-tail/grep for scheduler+apiserver), report, clean up | 3.1–3.2, 4.1–4.3, 5.1, 5.2, 5.4, 6.1–6.5 | `xce-deploy-ha` (P0, extended), `xce_test` (P0), scheduler/apiserver trace logs (P0) | Batch | - -### Test Tooling / C - -#### `set_resval()` / `cmd_resupdate` - -| Field | Detail | -|-------|--------| -| Intent | Resolve a `-n <name> -v <value>` pair into a wire `xce_resval_t`, for any resource the cluster has registered | -| Requirements | 1.1, 1.2, 1.3, 1.4 | - -**Responsibilities & Constraints** -- Resolve a resource name to a `Value_u_which` tag (`resval_typeresolve.c`): known non-numeric names (`type`, `model`) resolve to `Value_u_svalue`; every other name — including ones not seen before — defaults to `Value_u_fvalue`, overridable via a new optional `-t <numeric|string>` flag. -- Preserve existing behavior byte-for-byte for every currently-supported resource name (Requirement 1.4) — this is a strictly additive change to the fallback path, not a rewrite of the existing table lookup. -- Report per-name accept/reject outcome when a single invocation carries multiple `-n`/`-v` pairs (Requirement 1.3). **Confirmed via direct source inspection**: the controller's synchronous RPC reply cannot carry this signal — `update_resval_list()` (`src/controller/worker.c:185-198`) unconditionally `return 0` regardless of any per-item failure, so `do_update_resvals()`'s `reply.status` is always success even when every submitted name is unregistered. `cmd_resupdate()` therefore verifies acceptance itself: after the `xcecli_update_resvals()` call, it issues one follow-up `xcecli_resval()` query per submitted `(resource, worker)` pair — the same client-library call `cmd_resval`/`resval` already uses — and compares the returned value against what was just sent. A match is "accepted"; a not-found result or a mismatched value is "rejected", satisfying Requirement 1.2's clear per-name error without any controller-side change. - -**Dependencies** -- Outbound: `xcecli_update_resvals` (`src/lib/client.c`) — unchanged, existing wire call (P0). `xcecli_resval` (`src/lib/client.c`) — unchanged, existing wire call, reused here as the accept/reject verification signal (P0). - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ] - -##### Batch / Job Contract -- Trigger: operator (or the `xce-test-resources` skill) runs `resupdate -n <name> -v <value> [-t <numeric|string>] [-n <name2> -v <value2> ...]` in the `xce_test` worker context. -- Input / validation: resource name (string, required per pair), value (string, parsed per resolved type), optional type override. -- Output / destination: one `xce_update_resvals` RPC to the controller carrying every resolved pair; then one `xcecli_resval` query per pair to confirm it landed; per-pair accept/reject reported to stdout based on that follow-up query, not the update RPC's own reply (which cannot distinguish per-item outcomes — see Responsibilities above). -- Idempotency & recovery: re-running with the same name/value is safe (matches existing `resupdate` semantics — a plain overwrite, no accumulation). - -**Implementation Notes** -- Integration: the type-resolution table lookup moves into `resval_typeresolve.c` unchanged in behavior; `cmd_resupdate()` gains the new post-update verification loop described above. -- Validation: the follow-up `xcecli_resval()` query adds one RPC round-trip per submitted resource name — acceptable for a test/verification tool, not a production hot path. -- Risks: a resource that is genuinely registered but whose value legitimately hasn't propagated yet (a race, not a rejection) could be misreported as "rejected" if queried immediately; mitigate with a short bounded retry on the follow-up query before concluding rejection (this codebase's existing resval-propagation timing, confirmed live during the bug-1569 investigation this spec originated from, is sub-second for a controller-local query). - -### Scheduler Core / C - -#### Scheduler resval trace log - -| Field | Detail | -|-------|--------| -| Intent | Make the live scheduler daemon's application of a resource update observable via its own log, without a query RPC or replay | -| Requirements | 4.1, 4.2, 4.3 | - -**Responsibilities & Constraints** -- Emit one trace/debug-level log line inside `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), immediately after the resolved worker's live resource value is written — including the resolved host name, resource name, and applied value, in a fixed, greppable format. -- Debug/trace level only (`TRACE`, which expands to `tlog_debug3` — `src/include/utils.h:93`) — never WARN/ERROR, never on any decision path, so normal operation is unaffected; this is purely additive observability. -- Fires on every real resource update in production too (not just this feature's synthetic injections) — acceptable, since it is the same log-volume category as this component's other existing `TRACE` calls (e.g. `apiserver_worker_change`'s `TRACE` in `src/controller/apiserver.c:446`). -- **Load-bearing operational requirement**: `tlog_debug3`/`TRACE` output is gated by the scheduler's own runtime log level (`src/scheduler/src/main.c`'s `-v` flag, cumulative — confirmed by reading `main.c`: `arguments->verbose++` per `-v`, then `verbose_to_loglevel(arguments.verbose)`). At the daemon's default (unset `-v`) log level, this line is silently never written, and the verification skill's log-grep would time out on every run — a false FAIL, not a bug in the feature under test. The skill's scheduler-deploy step MUST start the scheduler daemon with `-vvv` (verbosity 3, matching `tlog_debug3`) — the same convention `xce-deploy-ha`'s generated `start.sh` already uses for the controller (`start.sh ctrl-1 -vvv`) — see task 5.1. - -**Dependencies** -- Inbound: fires as part of the existing `XCE_CORE_EVENT_RESVALS_DELTA` handling path (P0) — no new caller, no new event type. -- Outbound: `tinylog` (P0) — existing logging infrastructure, unchanged. - -**Contracts**: Service [ ] / API [ ] / Event [x] / Batch [ ] / State [ ] - -##### Event Contract -- Trigger: the live scheduler daemon processes a `RESVALS_DELTA` event carrying a `resupdate`-injected (or real) value. -- Output / destination: one line to the scheduler daemon's own log file. -- Idempotency & recovery: log-only; no state changes, safe to fire on every event including retries/duplicates. - -### API Server / Go - -#### Apiserver resval trace log - -| Field | Detail | -|-------|--------| -| Intent | Make the apiserver's storage of a received resource update observable via its own log, without a REST/CLI query | -| Requirements | 5.1, 5.2, 5.4 | - -**Responsibilities & Constraints** -- Add one `logger.Debug(...)` call (zap, per `error-handling.md`'s existing convention) inside `OnResourceLocationUpdated` (`src/frontend/apiserver/backend/handler.go:92-95`), immediately after `store.ResourceLocations.Put(key, *res)`, logging the resource name, location/host, and stored value. -- Debug level only — never `Error`/`Warn`, no behavior change to the store write itself. -- **Load-bearing operational requirement**: the apiserver's log level is config-driven (`config/logger.yaml`'s `logging.level`, loaded by `loadLoggerConfig()` in `src/frontend/apiserver/utils/logger.go`). Zap's default level is not Debug, so unless this config explicitly sets `level: debug`, `logger.Debug(...)` output is silently suppressed and the skill's log-grep would time out on every run — a false FAIL, not a bug in the feature under test. `xce-deploy-ha`'s apiserver step (task 3.1) MUST generate a `logger.yaml` with `logging.level: debug`. - -**Dependencies** -- Inbound: fires as part of the existing `OnResourceLocationUpdated` notification-handling path (P0) — no new caller. -- Outbound: zap `logger` (P0) — existing logging infrastructure, unchanged. - -**Contracts**: Service [ ] / API [ ] / Event [x] / Batch [ ] / State [ ] - -##### Event Contract -- Trigger: the live apiserver process receives and stores a resource-location update pushed from the controller. -- Output / destination: one line to the apiserver's own log file. -- Idempotency & recovery: log-only; no state changes. - -### Deployment Tooling / Bash - -#### `xce-deploy-ha` apiserver step + `start-all.sh` - -| Field | Detail | -|-------|--------| -| Intent | Extend `deploy-ha.sh` to (a) conditionally stand up a minimal, boot-only single-node apiserver against an already-deployed controller, and (b) generate a single `start-all.sh` that sequences controller → (worker/scheduler/apiserver, whichever were generated) so callers need no manual multi-script sequencing | -| Requirements | 5.3, 6.1 | - -**Responsibilities & Constraints** -- Apiserver step: generate the minimal config needed for the process to **boot and connect to the controller**, plus a `logger.yaml` with `logging.level: debug` (required for task 4.2's trace line to actually be emitted — see that component's Load-bearing operational requirement), modeled on `make test-integration`'s existing `XCE_BIN`/`SRV_URL` convention. Because verification is trace-log-based (Requirement 5, see research.md Design Decisions), this config no longer needs to produce *functional* auth/RBAC — no REST query is ever issued against this instance — only enough to start the process and complete the apiserver↔controller connection that `apiserver_resource_hook_fun`/`OnResourceLocationUpdated` depends on. -- Generated **only if** a pre-built apiserver binary is already present at its conventional output path (e.g. `/tmp/apiserver`, `make build-server`'s default). **Not an exact mirror of the worker/scheduler conditional**: those binaries live in the same meson build directory as the controller and are simple presence-checked; the apiserver is a separate Go build (`make build-server`, `src/frontend/Makefile`) with its own output path, so `deploy-ha.sh` does **not** invoke `make build-server` itself — building the apiserver remains the caller's prerequisite, exactly like the existing "run `/xce-build controller` first if stale" convention in SKILL.md's Prerequisites section, just naming a different build command. This preserves the same "caller opts in by having already built the optional component" semantics the worker/scheduler conditional relies on for backward compatibility — a caller who never runs `make build-server` sees no apiserver step generated, same as one who never builds worker/scheduler today. -- `start-all.sh`: always generated (independent of which optional components exist for a given deploy). Runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated — skipping any that weren't — with the scheduler started at `-vvv` (verbosity 3) specifically so task 4.1's `TRACE`/`tlog_debug3` line is actually emitted (see that component's Load-bearing operational requirement). -- Does not attempt HA/multi-instance apiserver deployment — single instance only, sufficient for this feature's verification needs. - -**Dependencies** -- Inbound: `xce-test-resources` skill (P0) — the primary consumer motivating this extension; also available to any other `xce-deploy-ha` caller by construction (same reusability posture as the existing worker/scheduler steps). -- Outbound: `make build-server` (P0), a running controller (P0), `deploy-ha.sh`'s existing Step 3/Step 5 (P0). - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ] - -##### Batch / Job Contract -- Trigger: `deploy-ha.sh` invoked normally (apiserver/`start-all.sh` generation happens automatically as part of Step 3, not a separate opt-in flag the caller must pass); `start-all.sh` then invoked directly by the `xce-test-resources` skill, or by an operator. -- Input / validation: target controller host:port, a deploy/scratch directory (same as existing `xce-deploy-ha` inputs — no new required parameters). -- Output / destination: a running apiserver process connected to the controller, logs under the scratch directory (same convention as `xce-deploy-ha`'s `logs/`) — this is the log task 6.4 greps for Requirement 5's trace line. -- Idempotency & recovery: each invocation deploys fresh into a caller-specified scratch directory, matching `xce-deploy-ha`'s existing "every invocation is a fresh, self-contained deploy" convention — no in-place upgrade semantics. - -**Implementation Notes** -- Integration: exact minimal-boot config shape is confirmed during implementation, not fully pinned here — see research.md Risks; scope is narrower than originally planned since functional auth/RBAC is no longer required. -- Validation: `start-all.sh`'s own controller health-check gate (reusing Step 5) must complete before it proceeds to start worker/scheduler/apiserver — the skill must additionally confirm the apiserver process has registered with the controller (connection-level, not HTTP-listener) before injecting and grepping for Requirement 5's trace line. -- Risks: apiserver startup/config requirements were previously this design's least-verified part; narrowed by dropping the REST/auth requirement, but boot-time config shape is still unconfirmed until implementation. - -### Test Tooling / Bash - -#### `xce-test-resources` skill - -| Field | Detail | -|-------|--------| -| Intent | Deploy controller+scheduler+apiserver, inject one or more resource values via `resupdate`, verify all three layers, report PASS/FAIL, clean up | -| Requirements | 3.1, 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.4, 6.1, 6.2, 6.3, 6.4, 6.5 | - -**Responsibilities & Constraints** -- Deploy controller, scheduler, and apiserver into a fresh scratch directory by invoking `xce-deploy-ha`'s `deploy-ha.sh` (Step 3) then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already handles controller→health-check→scheduler/apiserver ordering (see the `xce-deploy-ha` apiserver step + `start-all.sh` component above). -- Join a synthetic worker and inject one or more resource values via `resupdate` (default set: the 7 names Requirement 1 adds, plus a couple of already-supported names for regression coverage). -- Query the controller (`resval`), and tail/grep the scheduler daemon's and apiserver's own log files (bounded wait each) for the trace lines their respective resval-update handlers now emit; compare all three against the injected value and attribute any mismatch or timeout to a specific layer (Requirement 4.3's distinct-reporting requirement generalizes to all three layers). -- Emit a single machine-parseable VERDICT line, following `xce-test-ha`'s existing convention (e.g. `[xce-test-resources] VERDICT resources=7 layers_checked=3 mismatches=0 result=PASS`), plus the usual terminal PASS/FAIL oracle line. -- Clean up all deployed **processes** and runtime/data state (PIDs, sockets, controller/scheduler data directories) unconditionally — on PASS, on FAIL, and on a mid-deployment abort alike — unless the operator passes a "keep running" flag (Requirement 6.5; see Error Handling's Deployment failure entry for why this must cover the abort path too, not just a completed run). **Log files under the scratch directory are always preserved**, independent of the "keep running" flag — this is a deliberate divergence from `xce-deploy-ha`'s own `cleanup.sh` (which wipes `logs/*.log`), because the Monitoring section below promises logs survive "for post-run inspection," and this feature's own trace-log verdict evidence (the exact scheduler/apiserver lines a FAIL was based on) would otherwise be unrecoverable after a completed run. - -**Dependencies** -- Inbound: none (top-level entry point, invoked directly by an operator). -- Outbound: `xce-deploy-ha` (P0, extended with the apiserver step + `start-all.sh`), `xce_test` worker/client contexts (P0), the scheduler daemon's and apiserver's own log files (P0, via the two new trace-log additions). - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ] - -##### Batch / Job Contract -- Trigger: `/xce-test-resources [resource=value ...]` (defaults to Requirement 1's gap-resource set if none given). -- Input / validation: optional list of `resource=value` pairs; optional scratch directory; optional "keep running" flag. -- Output / destination: per-layer pass/fail detail (injected vs. observed value, per resource, per layer) plus one VERDICT line and the standard terminal PASS/FAIL exit code, matching `xce-test-ha`'s reporting contract closely enough to be immediately familiar to this codebase's engineers. -- Idempotency & recovery: safe to re-run — fresh scratch directory per run (`xce-deploy-ha` convention), no shared mutable state across runs. - -**Implementation Notes** -- Integration: invokes `deploy-ha.sh` then its generated `start-all.sh`, which already gates scheduler/apiserver startup on the controller health-check internally — this skill does not re-implement that sequencing. -- Integration: registers cleanup via a `trap` (or equivalent) as soon as `start-all.sh` is invoked, not just at the end of a successful run — deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`), because Requirement 6.5 requires cleanup-by-default on every path including a mid-deployment abort (see Error Handling). -- Validation: the scheduler and apiserver checks are both a bounded-wait poll of a live log file (no replay, no step/continue) — poll with a short interval up to a generous timeout, consistent with `xce-test-ha`'s existing soak-test conventions; a match found before timeout short-circuits the wait. -- Risks: timing-sensitive (bounded waits for controller→scheduler and controller→apiserver propagation); use generous, explicit timeouts, not tight polling that could flake. - -## Error Handling - -### Error Strategy -This feature is a verification tool, not a production service — "errors" are primarily reportable verification failures (a layer did not observe the injected value) rather than exceptions to recover from mid-run. - -### Error Categories and Responses -- **Unrecognized resource name** (Requirement 1.2): `resupdate` reports a clear per-name error, sourced from its own follow-up `xcecli_resval()` verification query (see Components — the controller's update RPC reply cannot carry this signal); the skill treats this as a run-level failure with the specific rejected name(s) named in its report. -- **Layer mismatch** (Requirements 3.2, 4.2, 5.2): the skill reports which layer(s) failed, with both the injected and observed value, per Requirement 6.3. -- **Layer unreachable** (e.g. apiserver never came up, or the expected trace line never appears in the scheduler/apiserver log before the bounded wait elapses): treated the same as a mismatch for verdict purposes (FAIL, layer identified, "timeout" distinguished from "mismatch" in the report), not a silent skip. -- **Deployment failure** (controller/scheduler/apiserver fails to start): the skill aborts before attempting injection, with a clear "deployment failed at step X" message. **Cleanup still runs on this path** — this is a deliberate strengthening of `xce-deploy-ha`'s precedent, not a mirror of it: `xce-deploy-ha`'s `deploy-ha.sh` has no trap-based cleanup and leaves already-started processes running on a mid-deploy failure, requiring the caller to run its separate `cleanup.sh` manually (confirmed against `src/tools/deploy-ha.sh`). Requirement 6.5 promises cleanup by default regardless of outcome, unless the operator explicitly requested the deployment be left running — a deployment failure is not that request, so the skill installs a trap (or equivalent) covering every deployment step from the first process start onward, not just the post-injection verification path. -- **Custom host-attrs resource name false-MISMATCH (known residual risk, not fully closed)**: the default resource set avoids every host-attrs-derived name (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` — `worker.c`'s `N_HOST_ATTRS`) specifically to route around this; the System Flows section above documents the nested-`!`-escape fix and why it closes the controller-layer check for that default set. If an operator passes a CUSTOM `resource=value` argument naming a host-attrs resource, the controller-layer check will still report a false MISMATCH — not because propagation is wrong, but because that resource is reset to baseline the instant the worker session ends, and no code path today re-checks it before that reset for an arbitrary caller-supplied name. This is a known, documented gap (tasks.md task 6.2), not silently swallowed: a genuine fix needs either a C-level `xce_test` change (no REPL path from `worker` context to `client` context without an equivalent disconnect) or a bash coprocess restructuring to keep the worker connected across every layer's check — out of scope for the current implementation. The skill should warn (not silently mismatch) when a caller-supplied resource name matches a known host-attrs name. - -### Monitoring -Not applicable — this is an on-demand, operator-invoked CLI tool, not a running service. Logs are written under the run's scratch directory, following `xce-deploy-ha`/`xce-test-ha`'s existing `logs/` convention, for post-run inspection. - -## Testing Strategy - -- **Unit Tests**: `resval_typeresolve.c`'s type-resolution fallback — 4 cases: a currently-known numeric name (e.g. `mem`) unchanged; a currently-known string name (`type`) unchanged; an unrecognized name defaults to numeric; an unrecognized name with `-t string` override resolves to string — directly compilable and testable in isolation since the extracted module has no networking, REPL, or `test.c`-global dependency (unlike `test.c` itself, which cannot be white-box included — see File Structure Plan). The skill's verdict-aggregation/comparison step (Requirement 6) is implemented as a self-contained function over (resource, injected value, per-layer observed value) records specifically so its negative path — a genuine per-layer mismatch correctly producing an overall FAIL naming the right layer — can be verified the same way, with synthetic records, without requiring a live deployment. -- **Integration Tests**: the skill's own live run against a real controller+scheduler+apiserver deployment *is* this feature's primary integration/e2e coverage, following this codebase's established precedent for `xce-test-ha`-class tooling (soak/verification tools in this project are validated live, not via `meson test()`). At minimum, one live run per implementation change verifying: (1) a previously-missing resource name (e.g. `r15m`) is now accepted and correctly injected, (2) all three layers' trace lines report the correct value within the bounded wait, (3) a genuinely-unregistered resource name included in the same batch is reported as a specific, clearly-identified rejection (not silently dropped) and correctly drives the run's overall verdict to FAIL, per this design's Error Handling. -- **E2E Path**: the full flow in System Flows' sequence diagram, run end-to-end at least once with the default resource set before this feature is considered done. diff --git a/.kiro/specs/resource-value-verification/requirements.md b/.kiro/specs/resource-value-verification/requirements.md deleted file mode 100644 index 94a1b884..00000000 --- a/.kiro/specs/resource-value-verification/requirements.md +++ /dev/null @@ -1,73 +0,0 @@ -# Requirements Document - -## Introduction -XCE currently has no fast, self-contained way to verify that a worker's resource value change propagates correctly through the full pipeline: controller → scheduler → API server. During a recent investigation (bug-1569), engineers had to deploy full worker daemons and wait on periodic reporting cycles to reproduce and verify resource-value routing bugs, when `xce_test`'s existing `worker` context already offers a `resupdate` command capable of injecting synthetic resource values directly — but its resource-name whitelist is incomplete, and no tooling exists to confirm those values are correctly visible downstream at the scheduler or the Go API server. This feature closes the `resupdate` gap and adds an end-to-end resource-value verification capability so this class of bug can be reproduced and confirmed fixed in seconds instead of through multi-minute live-deployment investigations. - -## Boundary Context (Optional) -- **In scope**: extending `xce_test`'s worker-context `resupdate` command to support any resource the cluster has registered (not just a fixed subset); a repeatable verification flow/tool that injects a resource value and confirms it is visible at the controller, scheduler, and API server layers; any deployment scripting needed to stand up a Go API server instance for this verification (none currently exists for local testing). -- **Out of scope**: changing production resource-collection behavior (`resource_sync.c`'s real load-index collection); building CI/automated-pipeline integration; fixing unrelated bugs surfaced incidentally during verification (they should be reported, not silently fixed as part of this feature); broader xce_test tooling improvements (per-command help, reference documentation, structured output, batch mode, argument-convention unification, renaming) — deliberately deferred to a separate future project, see Future Considerations below. -- **Adjacent expectations**: this feature depends on a working controller/scheduler/API-server deployment mechanism (the existing `xce-deploy-ha` skill covers controller deployment only; API-server deployment is a new capability this feature must provide or explicitly hand off). How scheduler-side (and apiserver-side) visibility is confirmed is left to the design phase, not fixed here — design.md/research.md settle on live trace-log observation at each layer's point of application, a third option not enumerated at requirements time (neither "extend the REPL flow" nor "add a new live query RPC"). - -## Requirements - -### Requirement 1: Complete resource-name coverage in `resupdate` -**Objective:** As a developer testing XCE's resource-value pipeline, I want `xce_test`'s `resupdate` command to accept any resource name the cluster has registered, so that I can inject a synthetic value for any resource without being blocked by an incomplete hardcoded list. - -#### Acceptance Criteria -1. When an operator runs `resupdate -n <resource-name> -v <value>` for a resource name currently missing from the command's supported list (including `r15m`, `pg`, `ls`, `it`, `swp`, `io`, `tmp`), the worker test context shall accept the update and report success. -2. When an operator runs `resupdate -n <resource-name> -v <value>` for a resource name that does not exist in the cluster's registered resource set, the worker test context shall report a clear error identifying the unrecognized name. -3. The worker test context shall report which resource names were accepted and which were rejected when a single `resupdate` invocation includes multiple `-n`/`-v` pairs. -4. Where a resource name is recognized, the worker test context shall preserve the resource's existing value-type behavior (numeric vs. string) exactly as before this change, for every currently-supported resource. - -### Requirement 2: Inject a resource value without a real worker daemon -**Objective:** As a developer reproducing a resource-value bug, I want to set a worker's resource value through `xce_test`'s worker context alone, so that I don't need to deploy and wait on a real worker daemon's periodic reporting cycle. - -#### Acceptance Criteria -1. When an operator joins a synthetic worker through `xce_test`'s worker context and then runs `resupdate`, the system shall apply the update as if it came from a real, currently-connected worker daemon, with no other worker process required. -2. The system shall tag every `resupdate`-injected value with the joined worker's own identity, consistently with how a real worker daemon's periodic report is identified. - -### Requirement 3: Confirm the value is visible at the controller -**Objective:** As a developer verifying a resource-value fix, I want confirmation that an injected value is visible through the controller's own resource query, so that I know the controller-side portion of the pipeline is correct. - -#### Acceptance Criteria -1. When a resource value has been injected via `resupdate`, the verification capability shall query the controller for that resource/worker pair and shall report whether the returned value matches the injected value. -2. If the controller's returned value does not match the injected value, the verification capability shall report this as a failure and shall include both the injected and observed values in its report. - -### Requirement 4: Confirm the value is visible at the scheduler -**Objective:** As a developer verifying a resource-value fix, I want confirmation that an injected value is visible in the scheduler's own live resource state, so that I know the scheduler-side portion of the pipeline is correct. - -#### Acceptance Criteria -1. When a resource value has been injected via `resupdate`, the verification capability shall determine, within a bounded wait time, whether that value is visible in the scheduler's own resource state for the corresponding worker. -2. If the scheduler's resource state does not reflect the injected value within the bounded wait time, the verification capability shall report this as a failure and shall include both the injected and observed values in its report. -3. The verification capability shall report the scheduler-side observed value distinctly from the controller-side observed value, so a mismatch can be attributed to a specific layer. - -### Requirement 5: Confirm the value is visible at the API server -**Objective:** As a developer verifying a resource-value fix, I want confirmation that an injected value is correctly received and stored by the Go API server, so that I know the apiserver-side portion of the pipeline is correct. - -#### Acceptance Criteria -1. When a resource value has been injected via `resupdate` and a Go API server instance is running against the same cluster, the verification capability shall determine, within a bounded wait time, whether that value is visible in the API server's own resource state for the corresponding worker. -2. If the API server's observed value does not match the injected value within the bounded wait time, the verification capability shall report this as a failure and shall include both the injected and observed values in its report. -3. Where no Go API server deployment mechanism currently exists for local testing, the verification capability shall provide one as part of standing up the test environment. -4. The verification capability shall report the apiserver-side observed value distinctly from the controller-side and scheduler-side observed values, so a mismatch can be attributed to a specific layer. - -**Design note**: this requirement is intentionally worded like Requirement 4 (implementation-agnostic — "visible in the apiserver's own resource state," not "returned by a query") because the chosen mechanism observes the apiserver's storage layer directly (a trace log at the point of ingestion) rather than exercising its REST/CLI query path — see design.md/research.md for the rationale and the accepted trade-off (a query/serialization-layer-only regression would not be caught by this design). - -### Requirement 6: Repeatable, reportable verification run -**Objective:** As a developer diagnosing or regression-testing resource-value propagation, I want a single repeatable command that runs the full injection-and-verification flow and reports a clear pass/fail result, so that I can confirm a fix (or catch a regression) without manual multi-step investigation. - -#### Acceptance Criteria -1. When an operator invokes the verification capability, the system shall stand up (or reuse) a controller + scheduler + API server test deployment, inject one or more resource values, and check all three layers (controller, scheduler, API server) without further manual steps. -2. When every checked layer's observed value matches the injected value, the verification capability shall report an overall PASS. -3. When any checked layer's observed value does not match the injected value (or a layer is unreachable), the verification capability shall report an overall FAIL and shall identify which layer(s) failed. -4. The verification capability shall support checking multiple resource names (including the ones named in Requirement 1) in a single run. -5. The system shall clean up any deployment or process state it created for the verification run, unless the operator explicitly requests it be left running for further inspection. - -## Future Considerations (explicitly out of scope for this project) -Raised while gathering requirements. These are real, worth doing, but the user has explicitly deferred all of them to a separate future "enhance xce_test" project — none are implemented as part of this spec: - -- **Renaming `xce_test`**: as its role grows from an ad-hoc test client into a serious tool for verifying XCE controller/scheduler behavior and core architecture correctness, a more descriptive name could better communicate that purpose. Carries real blast radius (invoked by name in scripts, the `xce-test-ha`/`xce-deploy-ha` skills, and developer muscle memory) — needs its own explicit decision (candidate name, rename-vs-alias) when picked up. -- **Discoverable per-command help**: today's `help` only lists command names with no per-command usage/arguments/examples. A future project should add detailed help for every command, in every context (client/worker/scheduler). -- **Reference documentation**: a maintained written reference describing every xce_test context and command, its purpose, arguments, and an example invocation. -- **Structured (JSON) output mode** for query commands (`resval`, `joblist`, `workerlist`, etc.) so scripts/skills can parse results reliably instead of scraping human-readable tables — this session's own investigation relied entirely on fragile table-text `grep`. -- **Non-interactive batch/script mode** with clear per-command success/failure exit codes, for more robust use from skills and CI (this session drove xce_test via heredoc-piped stdin, which works but gives no clean per-command pass/fail signal). -- **Consistent argument conventions** across contexts (e.g. `resupdate`'s `-n`/`-v` flags vs. `jobsub`'s positional arguments) — a larger, more invasive cleanup with real backward-compat risk for existing scripts/skills. diff --git a/.kiro/specs/resource-value-verification/research.md b/.kiro/specs/resource-value-verification/research.md deleted file mode 100644 index cc0d6612..00000000 --- a/.kiro/specs/resource-value-verification/research.md +++ /dev/null @@ -1,102 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `resource-value-verification` -- **Discovery Scope**: Extension (existing systems: `xce_test`, controller, scheduler, apiserver) -- **Key Findings**: - - `xce_test`'s worker-context `resupdate` command already exists, already sends updates through the identical wire path a real worker daemon's periodic report uses (`xcecli_update_resvals` → `xce_update_resvals` → controller's `do_update_resvals`), and already tags `.host` correctly via `get_setting("myname")`. It is the right injection primitive for this feature — no new wire path is needed. - - `resupdate`'s resource-name coverage is gated by a hardcoded table (`set_resval()`, `src/controller/testsuite/test.c:3267`) that is missing 7 real dynamic resources (`r15m`, `pg`, `ls`, `it`, `swp`, `io`, `tmp`) present in `resource_sync.c`'s load-index set. - - The scheduler has no live (non-replay) query mechanism for its own resource state today. The only existing *query* path is `xceadmin scheduler record start/stop` + `xce_scheduler -i -s <file> --interactive` (the REPL). The REPL starts **paused** at the initial snapshot and requires an explicit `step`/`continue` before any subsequently-recorded event (including a resource update) is reflected in `info worker` output — discovered directly during this session's bug-1569 investigation, where querying immediately after load repeatedly and misleadingly looked like a bug. **This path was the original design choice and is now superseded — see Design Decisions below**: the scheduler applies the update to its own live state regardless of whether anything ever queries it, and a trace log at that point of application is both simpler and a closer match to how this feature's own originating bugs were actually found. - - No apiserver deployment mechanism exists in this repo for local/skill-driven testing today. `xce-deploy-ha` only deploys the C controller. The Go apiserver is built via `make build-server` (`src/frontend/Makefile`) → `/tmp/apiserver`, and `make test-integration` already establishes a `XCE_BIN`/`SRV_URL` convention for pointing Go-side tooling at a running controller — the closest existing precedent for this feature's apiserver deploy step. - - The apiserver already exposes a query this feature could use: `xcecli resources distribution list` → `GET /xce/v1/resources/distribution`, backed by Go's `store.ResourceLocations`. **Superseded — see Design Decisions**: the final design observes `store.ResourceLocations`'s write directly via a trace log in `OnResourceLocationUpdated`, not through this REST/CLI path, so the apiserver deploy step no longer needs functional auth/RBAC. - - On the controller, `apiserver_resource_hook_fun` (`src/controller/apiserver.c`) forwards the same raw wire resval payload to the apiserver that `sched_resource_hook_fun` forwards to the scheduler — both are fed by the same `HOOK_DETAIL_RESOURCE_UPDATE` hook `do_update_resvals()` fires. This confirms a `resupdate` injection is visible to all three layers through one single controller-side write, not three independent paths. - - **Final approach (see Design Decisions)**: both the scheduler and apiserver checks observe each layer's own point of application via a new one-line trace/debug log (`core_events.c` for the scheduler, `handler.go`'s `OnResourceLocationUpdated` for the apiserver), greppable from the skill within a bounded wait — not the REPL and not a REST/CLI query. This directly matches how bug-1568 and bug-1569(A), the two real bugs that motivated this feature, were actually root-caused. - -## Research Log - -### How does a resource value actually reach the scheduler and apiserver today? -- **Context**: Needed to confirm `resupdate` is a faithful stand-in for a real worker's periodic report, and to identify what each verification layer should query. -- **Sources Consulted**: `src/controller/testsuite/test.c` (`cmd_resupdate`, `set_resval`), `src/lib/client.c` (`xcecli_update_resvals`), `src/controller/opupd.c` (`do_update_resvals`), `src/controller/sched.c` (`sched_resource_hook_fun`), `src/controller/apiserver.c` (`apiserver_worker_change`, `apiserver_resource_hook_fun`), `src/scheduler/src/core_events.c` (`XCE_CORE_EVENT_RESVALS_DELTA` handling) — all read directly (not summarized) during this session's bug-1569 investigation and the follow-up research for this spec. -- **Findings**: One controller-side apply (`do_update_resvals` → `update_resval_list`) plus one hook firing (`HOOK_DETAIL_RESOURCE_UPDATE`) fans out to both the scheduler (`sched_resource_hook_fun` → `notify()` → `XCE_CORE_EVENT_RESVALS_DELTA`) and the apiserver (`apiserver_resource_hook_fun` → `notify()` to each connected apiserver instance). The controller's own `w->host.resvals` (queried via `resval` in the client context) is updated synchronously in the same call, independent of either forward. -- **Implications**: All three verification layers (Requirements 3–5) are downstream of the exact same single injection call — the design does not need three separate injection paths, only three separate read/verify steps. - -### Does a live scheduler resource query already exist? -- **Context**: Requirements left "how to confirm scheduler-side visibility" as an open design decision (extend the replay/REPL path vs. add a new live admin query). -- **Sources Consulted**: `src/scheduler/src/handler.c` (`sched_handle_admin_request`, `SchedAdminAction_*`), `src/scheduler/src/repl.c`/`repl_info.c` (the `--interactive` REPL), `xceadmin`'s `scheduler record start/stop`/`scheduler dump` commands. -- **Findings**: The only scheduler-admin RPC surface today (`OPCode_schedAdmin`) covers recording control (`recordStart`/`recordStop`/`recordStatus`) — nothing exposes live resource/worker state over the wire. The only way to inspect scheduler-side state is to record a `.replay` file and load it into the offline `--interactive` REPL, which requires explicit `step`/`continue` before it reflects anything beyond the initial snapshot. -- **Implications**: See Design Decision below — adopting the existing REPL path (documented pitfall and all) instead of building a new live-query RPC. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Trace log at point of application (scheduler + apiserver) | One debug/trace log line each in `core_events.c`'s resval-delta handling and `handler.go`'s `OnResourceLocationUpdated`; skill greps each live process's log | Matches how bug-1568/1569A were actually found; no record/replay round-trip; no REPL step/continue pitfall; no apiserver auth/RBAC functional requirement | Small, deliberate production-code touch (one log line each) in scheduler and apiserver; does not exercise the real query/REST/CLI path, so a query-layer-only regression wouldn't be caught | **Selected** | -| Record+replay+REPL for scheduler; deploy+REST-query for apiserver | Record a short window, replay it, `step`/`continue`, `info worker`; separately, deploy apiserver with working auth and query via `xcecli` | Exercises the real client-facing query path; zero scheduler/apiserver code changes | REPL step/continue pitfall already produced one false negative live; apiserver auth/RBAC config previously flagged as this design's least-verified part | Original choice; superseded — see Design Decisions | -| New live scheduler admin query RPC | Add a new `SchedAdminAction` (e.g. `infoWorker`) returning live resource state | Instant, no record/replay round-trip, exercises a real query path | New wire schema (capnp) + new scheduler handler + new `xceadmin` verb — real production-code surface for a test-tooling spec | Rejected; possible future follow-up, not built here | - -## Design Decisions - -### Decision (SUPERSEDED — see below): Adopt the existing record+replay+REPL flow for scheduler-side verification -- **Context**: Requirement 4 needs to confirm scheduler-side visibility; requirements.md deliberately left the mechanism to design. -- **Alternatives Considered**: - 1. Add a new live scheduler admin query (e.g. `SchedAdminAction_infoWorker`) that returns current resource state directly, without recording. - 2. Reuse the existing `xceadmin scheduler record start/stop` + `xce_scheduler -i -s <file>` REPL flow, with explicit `step`/`continue` before querying. -- **Selected Approach (at the time)**: (2) — reuse the existing REPL flow. -- **Why superseded**: re-examined during design validation against this spec's own originating bug history (bug-1568, bug-1569A) and replaced by the trace-log decision immediately below — see that entry for the full rationale. This entry is kept for the record of what was considered and rejected the first time; do not implement against it. - -### Decision: Verify scheduler- and apiserver-side visibility via live trace-log observation, not REPL replay or a REST/CLI query -- **Context**: Requirement 4 (scheduler) and Requirement 5 (apiserver) need a mechanism to confirm an injected value is visible at each layer; the record+replay+REPL approach above was the original design-phase choice, revisited after the user asked (during design validation) whether scheduler-sim-mode dependence could be avoided. -- **Key finding that changed the decision**: this feature's own originating bugs were re-examined for what kind of defect they actually were. **Bug-1568** (`.wolf/buglog.json`) — "`info worker` shows 0.00 for every resource" — was root-caused as a genuine *storage*-layer bug: `sched_add_worker()` (`src/scheduler/src/core.c:270`) is a plain `key_add()` (ADD-ONLY; fails `-EEXIST` on a worker that already exists), so a second `XCE_CORE_EVENT_WORKERS_PUSH` carrying the real values silently loses to the first, zero-valued push — the scheduler's actual in-memory state was wrong, not merely mis-displayed. It was root-caused by adding **live TRACE instrumentation directly at the point of application** (`core_events.c`), not by querying through the REPL. **Bug-1569(A)** was likewise a storage-layer bug (`resource_sync.c` tagging periodic reports with the wrong `.host` key) — and the REPL path's step/continue requirement itself produced a *false negative* during that investigation (looked broken, wasn't). Neither of the two real bugs this feature exists to catch was a query/display-layer defect; both were "the stored value is wrong," and both were actually found via a trace log at the storage site, not via REPL/REST querying. -- **Alternatives Considered**: - 1. (original) Record+replay+REPL for scheduler; deploy apiserver + `xcecli resources distribution list` REST query for apiserver. - 2. Add a debug/trace log line at each layer's actual point of application — scheduler: `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case in `src/scheduler/src/core_events.c` (adjacent to `sched_get_worker`); apiserver: `OnResourceLocationUpdated` in `src/frontend/apiserver/backend/handler.go:92-95` (where `store.ResourceLocations.Put()` is called). The skill greps each layer's own live log file for the expected line within a bounded wait, instead of recording/replaying (scheduler) or deploying-and-querying (apiserver). -- **Selected Approach**: (2), for both layers. -- **Rationale**: Directly matches how the actual defects in this bug class were found and is simpler on every axis — no `record`/`replay` round-trip, no REPL `step`/`continue` pitfall (already proven capable of a false negative once), no apiserver auth/RBAC config to get right (previously flagged as this design's least-verified, riskiest part), no dependency on the REST/JSON stack. The apiserver still needs to be deployed and connected to the controller (Requirement 5.3 — `OnResourceLocationUpdated` only fires on a live, connected instance), but no longer needs *functional* auth/RBAC — only enough config to boot and connect, which substantially de-risks `deploy-apiserver.sh`. -- **Trade-offs (explicitly accepted by the user)**: this checks that the value was correctly received and stored at each layer, not that a real client can retrieve it through the actual query/REST/CLI path. A future bug in the query/serialization layer specifically (as opposed to the storage layer) would not be caught by this design. Requirement 5's acceptance criteria are reworded in requirements.md to stop mandating "query the API server" as the specific mechanism, so this is a requirements-level acceptance of the trade-off, not just a design-level shortcut. -- **Follow-up**: if a future bug turns out to be in the query/serialization path rather than storage, re-add a real query-based check for that layer specifically — this decision does not preclude it. - -### Decision: Default unrecognized `resupdate` resource names to numeric, keep a small explicit list for non-numeric ones -- **Context**: Requirement 1 needs `resupdate` to accept any registered resource name, not just the current hardcoded 14-entry table. -- **Alternatives Considered**: - 1. Query the controller/resmgr live for each resource's registered value type before sending the update. - 2. Extend the hardcoded table to enumerate every resource name across every possible cluster config. - 3. Default any unrecognized name to numeric (`Value_u_fvalue`), keep only the currently-known non-numeric names (`type`, `model`) in a small explicit list, and add an optional override flag for a caller that needs a non-numeric type. -- **Selected Approach**: (3). -- **Rationale**: (1) adds a network round-trip and new query dependency to a simple CLI command for a case (wrong type guess) that is rare and already recoverable via the override flag. (2) does not scale — resource sets are cluster-config-defined, not fixed, so any static enumeration will eventually be incomplete again (as demonstrated by the 7 already-missing names). (3) is the minimal change that satisfies "accept any resource name" without new runtime dependencies, is backward compatible (existing behavior for `type`/`model` and all currently-numeric names is unchanged), and gives an explicit escape hatch instead of guessing silently wrong. -- **Trade-offs**: A brand-new non-numeric resource requires the caller to pass the override flag once; this is judged acceptable given non-numeric resources are rare in this codebase (currently only 2 of ~30+ registered resources). -- **Follow-up**: None required before implementation. - -### Decision: Verify per-name accept/reject via a follow-up `xcecli_resval()` query, not the update RPC's own reply status -- **Context**: Requirement 1.2 (clear error for a genuinely unregistered resource name) and 1.3 (per-name accept/reject reporting for multi-pair invocations) need a real signal distinguishing per-item success from failure. The original design draft assumed the controller's synchronous RPC reply (`do_update_resvals`'s `reply.status`) could carry this, flagged as unconfirmed in this file's Risks — that assumption was checked directly against source during a final spec review and found to be false. -- **Confirmed finding**: `update_resval_list()` (`src/controller/worker.c:185-198`) loops over the whole batch, logs each per-item failure via `log_update_resval_failure()`, but unconditionally `return 0`. `do_update_resvals()` (`src/controller/opupd.c`) sets `reply.status` from that single return value, so the RPC reply is always "success" for the whole batch regardless of how many individual names were rejected — there is no per-name signal on this path at all, and no batch-level failure signal either. -- **Alternatives Considered**: - 1. Modify `update_resval_list()`/`do_update_resvals()` to return/report per-item status on the wire. - 2. Have `cmd_resupdate()` verify each submitted name itself, via a follow-up `xcecli_resval()` query (the same client-library call `resval`/`cmd_resval` already uses) per `(resource, worker)` pair, comparing the returned value against what was just sent. -- **Selected Approach**: (2). -- **Rationale**: (1) requires modifying production controller code (`worker.c`, `opupd.c`, and the RPC reply wire shape) — explicitly out of this spec's boundary, and disproportionate scope for a test-tooling fix. (2) uses only existing, unmodified client-library functions, stays entirely inside `test.c` (this spec's boundary), and gives a genuinely observable per-name result (a live query confirming — or failing to confirm — the value actually landed), which is a stronger correctness signal than a wire-level status flag would have been anyway. -- **Trade-offs**: Adds one RPC round-trip per submitted resource name (acceptable for a test tool, not a production hot path). A resource that is genuinely registered but whose value hasn't propagated to the query path yet could theoretically be misreported as rejected — mitigated with a short bounded retry before concluding rejection. -- **Follow-up**: None required before implementation; this fully resolves what was previously an open risk (see below). - -### Decision: Extend `xce-deploy-ha`'s `deploy-ha.sh` with an opt-in apiserver step, instead of a new standalone `deploy-apiserver.sh` -- **Context**: this feature needs an apiserver instance connected to the controller for local verification; the original design (see design.md history) planned a brand-new, standalone `src/tools/deploy-apiserver.sh` that separately re-implemented `xce-deploy-ha`'s directory/scratch/health-check conventions "modeled on" the original rather than reusing it directly. Revisited during design validation after the user asked whether the deploy skill itself should be enhanced to support apiserver. -- **Existing precedent found**: `deploy-ha.sh` (Step 3, `.claude/skills/xce-deploy-ha/SKILL.md`) already has exactly this shape of extension for worker and scheduler — it generates `start-worker.sh`/`start-scheduler.sh` **conditionally**, only if the `xcecd_virtual`/`xce_scheduler` binaries exist alongside the controller binary at deploy time. Callers that never build worker/scheduler (e.g. `xce-test-ha`) get no such scripts and are completely unaffected. This is an established, low-risk, additive pattern for exactly this kind of "optional extra daemon" need. -- **Alternatives Considered**: - 1. (original) A new, standalone `src/tools/deploy-apiserver.sh`, independently modeling `xce-deploy-ha`'s conventions. - 2. Extend `deploy-ha.sh` itself: generate `start-apiserver.sh` under the same "only if the binary exists" rule already used for worker/scheduler, and document it as Step 4d in `xce-deploy-ha`'s own `SKILL.md`. -- **Selected Approach**: (2). -- **Rationale**: avoids re-implementing the same directory/cert/health-check conventions in parallel (a reuse/duplication smell the original design already half-acknowledged via "modeled on"/"mirrors" language instead of actual reuse); removes the Revalidation Trigger risk of the two scripts' conventions drifting apart over time, since there is now only one script; matches an already-precedented conditional-generation pattern (worker/scheduler), so this is idiomatic to the codebase rather than novel; and requirements.md's own "Adjacent expectations" note already floated "hand off" to `xce-deploy-ha` as a legitimate option at requirements time. -- **Trade-offs**: this feature's implementation now touches a shared, actively-used skill (`xce-deploy-ha`) instead of staying fully isolated in a new file — real blast radius, mitigated by making the apiserver step strictly opt-in/conditional (existing callers that don't build/need an apiserver see zero behavior change, exactly like the worker/scheduler precedent) and additive-only (no changes to existing controller/worker/scheduler steps). -- **Follow-up**: None required before implementation. -- **Extension (raised by the user during the same design-validation pass)**: also generate a combined `start-all.sh` in Step 3, alongside the existing per-component scripts (`start.sh`, `start-worker.sh`, `start-scheduler.sh`, and the new `start-apiserver.sh`) — it starts the controller node(s) via `start.sh`, health-checks (Step 5's existing check), then starts whichever of worker/scheduler/apiserver scripts were actually generated for this deploy, in the same order and with the same backgrounding SKILL.md's Step 4b already documents doing by hand. This is not new sequencing logic — it automates the exact manual recipe `xce-deploy-ha`'s own SKILL.md Steps 4/4b/4c already instruct a human to run — and it directly serves Requirement 6.1 ("stand up... without further manual steps") by letting the `xce-test-resources` skill call one generated script instead of re-implementing controller→scheduler/apiserver sequencing itself. Always generated (regardless of which optional components are present for a given deploy), so it has one consistent name and behavior across all callers. - -## Risks & Mitigations -- No apiserver deployment mechanism exists yet, and this session never exercised the Go apiserver live. Now that verification no longer depends on a functional REST query (trace-log decision above), the remaining unknown is narrower: just enough config for the binary to boot and register with the controller (no working auth/RBAC required). Mitigation: model the new `xce-deploy-ha` apiserver step on `make test-integration`'s existing `XCE_BIN`/`SRV_URL` convention, reusing `deploy-ha.sh`'s own directory/config-generation pattern directly (see the "extend xce-deploy-ha" decision above) rather than a separate script; confirm the minimal-boot config shape during implementation. -- The scheduler's `TRACE`/`tlog_debug3` output and the apiserver's `logger.Debug` output are both gated by runtime log-level configuration (`-vvv` for the scheduler, `logger.yaml`'s `logging.level: debug` for the apiserver) — if `start-all.sh` fails to set either, the corresponding verification check times out on every run with no indication the trace line was simply never written (looks identical to a genuine propagation failure). Mitigation: this is called out explicitly as a load-bearing requirement in design.md's two trace-log component sections and in task 3.1/4.1/4.2, not left implicit. -- A trace log line added to production code (`core_events.c`, `handler.go`) must be genuinely inert in normal operation — gated at debug/trace level (never WARN/ERROR, never on the hot path for every event), consistent with each component's existing logging conventions (`tinylog` in C, zap `logger.Debug` in Go per `error-handling.md`). -- The chosen approach verifies storage-layer correctness, not the real query/REST/CLI path — accepted trade-off (see Design Decisions); a future query-layer bug would need a separate check to catch. -- Live-deployment verification (this feature's core value) is inherently timing-sensitive (bounded waits per Requirements 4/6). Mitigation: use generous, explicit timeouts consistent with this codebase's existing soak-test conventions (`xce-test-ha`), not tight polling that could flake. -- `test.c` (4298 lines, its own `main()`, heavy `libedit`/network/REPL dependencies) cannot be white-box `#include`d into a Unity test the way smaller single-purpose files in this codebase are — confirmed by direct inspection during final spec review. Mitigation: extract `set_resval()`'s type-resolution logic into a new, small, dependency-light `resval_typeresolve.c`/`.h` module that both `test.c` and a new unit test can compile against independently (see design.md File Structure Plan). - -## References -- `.wolf/buglog.json` bug-1569 — the investigation that surfaced every finding in this research log, including the REPL step/continue pitfall and the incomplete `resupdate` resource table. -- `.claude/skills/xce-test-ha/SKILL.md`, `.claude/skills/xce-deploy-ha/SKILL.md` — existing skill conventions this design follows (VERDICT line format, scratch-directory deploy pattern, `setsid nohup … & disown` process backgrounding). diff --git a/.kiro/specs/resource-value-verification/spec.json b/.kiro/specs/resource-value-verification/spec.json deleted file mode 100644 index 55f25ab1..00000000 --- a/.kiro/specs/resource-value-verification/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "resource-value-verification", - "created_at": "2026-08-17T14:28:10Z", - "updated_at": "2026-08-18T14:35:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/resource-value-verification/tasks.md b/.kiro/specs/resource-value-verification/tasks.md deleted file mode 100644 index b7c46a9e..00000000 --- a/.kiro/specs/resource-value-verification/tasks.md +++ /dev/null @@ -1,138 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: test scaffolding and skill directory setup -- [x] 1.1 Extract resupdate's type-resolution logic and write failing unit tests for its fallback behavior - - Move `set_resval()`'s resource-name → `Value_u_which` table lookup into a new, small, dependency-light `resval_typeresolve.c`/`.h` module with no networking/REPL/global-state dependency — `test.c` (4298 lines, its own `main()`, heavy `libedit`/network dependencies) cannot itself be white-box included into a unit test, so this extraction is a hard prerequisite for testing this logic in isolation at all - - Create `test_resval_typeresolve.c` (Unity) and wire both new files into `src/controller/meson.build`: add `testsuite/resval_typeresolve.c` to `xce_test_src` (line 57) so `xce_test` still links, and add a new `test()` target for `test_resval_typeresolve.c` following the existing `controller_unit_test`/`sched_push_unit_test` pattern (lines 86, 215, 222) — these are hand-maintained explicit lists, not globs, so the module and test will not build without this step - - Write tests covering: a currently-known name (e.g. `mem`) still resolves numeric exactly as today; a currently-known string name (`type`) still resolves string exactly as today; an unrecognized name (e.g. `r15m`) defaults to numeric; an unrecognized name with an explicit type-override flag resolves to the requested type - - Observable completion: the new module and test target both compile and link via `meson`, and the new test target fails (RED) because the fallback-to-numeric and type-override behavior don't exist yet - - _Requirements: 1.1, 1.4_ - -- [x] 1.2 (P) Scaffold the xce-test-resources skill directory - - Create the skill directory with a `SKILL.md` stub describing the intended usage (`/xce-test-resources [resource=value ...]`) and the VERDICT line format, following the `xce-test-ha`/`xce-deploy-ha` SKILL.md convention - - Observable completion: the skill is discoverable/invocable (even though its underlying script is not yet functional) and its documented contract matches design.md's Batch/Job Contract for the skill - - _Requirements: 6.1_ - - _Boundary: xce-test-resources skill_ - -- [x] 2. Core: resupdate resource-name coverage fix -- [x] 2.1 Wire the extracted resolver into resupdate and add the type-override flag - - Extend `resval_typeresolve.c`'s fallback path so any resource name not in the existing table defaults to numeric (`Value_u_fvalue`), while `type`/`model` and all currently-supported names keep their exact existing behavior - - Make `test.c`'s `set_resval()` a thin wrapper delegating to `resval_typeresolve.c` (for type resolution) plus its own existing `.host` tagging (unchanged, still needs REPL state) - - Add an optional `-t <numeric|string>` flag to `cmd_resupdate()`, applied per `-n`/`-v` pair - - Observable completion: the unit tests from 1.1 pass (GREEN); running `resupdate -n r15m -v 1.5` in the worker context (against a live controller) now succeeds where it previously silently failed - - _Requirements: 1.1, 1.4_ - - _Depends: 1.1_ - -- [x] 2.2 Implement per-name accept/reject verification in resupdate - - After `cmd_resupdate()` sends the update RPC, issue one follow-up `xcecli_resval()` query per submitted `(resource, worker)` pair (the same client-library call `resval`/`cmd_resval` already uses) and compare the returned value against what was just sent - - Report "accepted" for a matching value, "rejected: unknown resource" for a not-found result, with a short bounded retry before concluding rejection (guards against a genuinely-registered resource whose value hasn't propagated to the query path yet) - - Report every submitted name's individual accept/reject outcome when a single invocation carries multiple `-n`/`-v` pairs - - Observable completion: against a live controller, `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports `mem` as accepted and `not-a-real-resource` as rejected, in the same invocation - - _Requirements: 1.2, 1.3_ - - _Depends: 2.1_ - -- [x] 3. Core: apiserver local deployment capability -- [x] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh` - - In `deploy-ha.sh` Step 3, add a new conditional branch in the same *opt-in-by-presence* spirit as the existing worker/scheduler branch: check for a pre-built apiserver binary at its own Go-build output path (e.g. `/tmp/apiserver`, `make build-server`'s default) — NOT alongside `$BINARY`, since apiserver is a separate build system, not the same meson build dir worker/scheduler share; `deploy-ha.sh` does not invoke `make build-server` itself, building it remains a caller prerequisite (document in SKILL.md's Prerequisites section, same pattern as the existing `/xce-build controller` note). If present, generate `$TOP/scripts/start-apiserver.sh` plus a minimal `logger.yaml` (modeled on `make test-integration`'s `XCE_BIN`/`SRV_URL` convention) — functional auth/RBAC is NOT required (verification is trace-log-based, task 6.4; no REST query is ever issued against this instance), but `logger.yaml` MUST set `logging.level: debug` or task 4.2's trace line is silently never written; absent the binary, generate nothing, so existing callers are unaffected - - Also in Step 3, always generate `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skipping any that weren't) — start the scheduler with `-vvv` specifically (verbosity 3, matching `tlog_debug3`/`TRACE`), overriding SKILL.md's existing `-v`-only example for this feature's needs, or task 4.1's trace line is silently never written - - Update `.claude/skills/xce-deploy-ha/SKILL.md` with a new "Step 4d" documenting the apiserver step (following the existing Step 4b pattern) and documenting `start-all.sh` as a combined shortcut for Steps 4/4b/4c - - Write apiserver/scheduler logs under the caller's scratch directory, matching the existing `logs/` convention — these are the logs tasks 6.3/6.4 grep - - Observable completion: running `deploy-ha.sh` against a build that includes an apiserver binary, then `start-all.sh`, leaves a running controller, scheduler (at `-vvv`), and apiserver (with debug-level logging) with no further manual steps; running `deploy-ha.sh` against a build without an apiserver binary (e.g. `xce-test-ha`'s existing usage) is unaffected — no `start-apiserver.sh` is generated, and `start-all.sh` skips that step - - _Requirements: 5.3, 6.1_ - - _Boundary: xce-deploy-ha (deploy-ha.sh, SKILL.md)_ - -- [x] 4. Core: scheduler and apiserver trace-log instrumentation -- [x] 4.1 (P) Add a trace log line to the scheduler's live resource-update handling - - In `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), add one `TRACE`-level log line (this component's existing `tinylog` convention) immediately after the resolved worker's value is written, logging the resolved host, resource name, and applied value in a fixed, greppable format - - Confirm the existing scheduler testsuite (`test_core_events.c` and friends) still passes unchanged — this is a pure log addition with no control-flow change, so no new unit test is required (consistent with this codebase's existing precedent: other `TRACE`/`WARN` calls in this area, e.g. `apiserver_worker_change`, have no dedicated unit test coverage either); the log line's actual content/format is confirmed live in task 8.1 - - Observable completion: a live scheduler daemon's log file contains one line per applied resource update, in the format task 6.3 will grep for; existing scheduler tests are unaffected - - _Requirements: 4.1, 4.2, 4.3_ - -- [x] 4.2 (P) Add a trace log line to the apiserver's resource-update storage handling - - In `OnResourceLocationUpdated` (`src/frontend/apiserver/backend/handler.go:92-95`), add one `logger.Debug(...)` call (zap, per this component's existing convention) immediately after `store.ResourceLocations.Put(key, *res)`, logging the resource name, location/host, and stored value - - Confirm existing apiserver backend tests still pass unchanged — same rationale as 4.1, no new unit test required for a pure log addition; content/format confirmed live in task 8.1 - - Observable completion: a live apiserver's log file contains one line per stored resource-location update, in the format task 6.4 will grep for; existing backend tests are unaffected - - _Requirements: 5.1, 5.2, 5.4_ - -- [x] 5. Integration: full-stack deployment orchestration -- [x] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh` - - Invoke `xce-deploy-ha`'s `deploy-ha.sh` (task 3.1) into a fresh scratch directory, then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already gates scheduler/apiserver startup on the controller health-check and handles the `-vvv`/debug-level logging needed for tasks 4.1/4.2's trace lines - - Register a `trap`-based cleanup (or equivalent) before invoking `start-all.sh`, not just at the end of the script — this must cover a mid-deployment abort (e.g. apiserver fails to start after controller+scheduler are already up), not only a completed PASS/FAIL run; deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`) because Requirement 6.5 requires cleanup-by-default on every path unless the operator passed the "keep running" flag - - Observable completion: a single invocation of the orchestration script leaves a running controller, live scheduler daemon (built with task 4.1's trace log, started at `-vvv`), and apiserver (built with task 4.2's trace log, debug-level logging enabled), all logging under the same scratch directory, with no manual steps; killing the script mid-deployment (e.g. before the apiserver step completes) still removes every process it had already started, unless "keep running" was passed - - _Requirements: 5.3, 6.1, 6.5_ - - _Depends: 1.2, 3.1, 4.1, 4.2_ - -- [x] 6. Core: resource injection and per-layer verification checks -- [x] 6.1 Implement worker join and resource injection - - Join a synthetic worker through the `xce_test` worker context (no real worker daemon process), then run `resupdate` for one or more resource names/values - - Observable completion: after this step runs, `resupdate` has been accepted for every requested resource name with zero real worker daemon processes running - - _Requirements: 2.1, 2.2, 6.1, 6.4_ - - _Depends: 2.1, 2.2, 5.1_ - -- [x] 6.2 Implement the controller-side verification check - - Query the controller via `resval` for each injected resource/worker pair and compare against the injected value - - Observable completion: the check reports, per resource, whether the controller's observed value matches the injected value, including both values when it does not - - _Requirements: 3.1, 3.2_ - - _Depends: 6.1_ - - _Boundary: xce-test-resources skill (controller check) — same file as 6.3/6.4 (`run-resource-verify.sh`), sequential, not parallel-safe_ - -- [x] 6.3 Implement the scheduler-side verification check via trace-log tail - - Tail/grep the live scheduler daemon's log file (task 4.1's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no record, no replay, no REPL - - Compare the scheduler's observed value against the injected value, reported distinctly from the controller-side result; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch - - Observable completion: the check reports, per resource, whether the scheduler's observed value matches the injected value within the bounded wait, attributable specifically to the scheduler layer - - _Requirements: 4.1, 4.2, 4.3_ - - _Depends: 6.1, 6.2_ - - _Boundary: xce-test-resources skill (scheduler check) — same file as 6.2/6.4, sequential_ - -- [x] 6.4 Implement the apiserver-side verification check via trace-log tail - - Tail/grep the live apiserver's log file (task 4.2's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no `xcecli` query, no REST call - - Compare the apiserver's observed value against the injected value, reported distinctly from the controller-side and scheduler-side results; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch - - Observable completion: the check reports, per resource, whether the apiserver's observed value matches the injected value within the bounded wait, attributable specifically to the apiserver layer - - _Requirements: 5.1, 5.2, 5.4_ - - _Depends: 6.1, 6.3_ - - _Boundary: xce-test-resources skill (apiserver check) — same file as 6.2/6.3, sequential_ - -- [x] 7. Integration: verdict aggregation, reporting, and cleanup -- [x] 7.1 Aggregate layer results, emit the verdict, and clean up - - Combine the controller/scheduler/apiserver check results into a single per-resource, per-layer comparison, implemented as a self-contained comparison step that accepts a set of (resource, injected value, per-layer observed value) records — so it can be invoked directly with synthetic records for testing (task 8.2), not only reachable from a live run - - Emit a machine-parseable VERDICT line (following `xce-test-ha`'s convention) plus the standard terminal PASS/FAIL oracle line, naming every layer that mismatched or was unreachable (timed out) - - Support checking multiple resource names in one run, defaulting to the resource names added in task 2.1 when none are supplied - - Clean up every process and runtime/data-directory artifact this run created, unless the operator passed a flag requesting the deployment be left running — this is the normal-completion firing of the same trap-based cleanup task 5.1 registered for the abort path, not a separate mechanism. **Log files under the scratch directory are always preserved**, regardless of the keep-running flag, so a FAIL verdict's underlying scheduler/apiserver trace-log evidence remains inspectable after the run - - Observable completion: a full run against the default resource set produces exactly one VERDICT line and a matching process exit code, no controller/scheduler/apiserver processes remain afterward (unless the keep-running flag was used), and the run's log files are still present on disk regardless - - _Requirements: 6.2, 6.3, 6.4, 6.5_ - - _Depends: 6.2, 6.3, 6.4_ - -- [x] 8. Validation: end-to-end live verification runs -- [x] 8.1 Run the full flow live, including one deliberately-unregistered resource name - - Execute the skill end-to-end against a freshly deployed controller/scheduler/apiserver, using the default resource set (task 2.1's previously-missing names) plus one resource name that does not exist in the cluster's registered set - - Observable completion: every genuinely-registered resource individually reports matching values at all three layers (confirming tasks 4.1/4.2's trace-log lines are correctly formatted and greppable, not just present), AND the unregistered name is reported as a clear, specifically-identified rejection (not a silent omission) with correct per-name accept/reject accounting across the whole batch; per design.md's Error Handling (an unrecognized name is a run-level failure), the run's single overall VERDICT is FAIL, attributable specifically to the unregistered name — not a false overall PASS - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.3, 5.4, 6.1, 6.2, 6.4_ - - _Depends: 7.1_ - -- [x] 8.2 Verify the verdict logic correctly reports FAIL on a genuine mismatch - - Invoke task 7.1's comparison step directly with a synthetic set of per-layer observed-value records where one layer's value deliberately does not match the corresponding injected value (no live deployment required for this check, since the comparison step is self-contained per task 7.1) - - Observable completion: the comparison step reports an overall FAIL verdict that correctly names the specific layer whose synthetic record was mismatched, and reports PASS for every other, matching layer/resource in the same input set — proving the verdict logic distinguishes pass from fail rather than always reporting PASS - - _Requirements: 6.3_ - - _Depends: 7.1_ - -## Implementation Notes -- Task 8.1 (validation-only, no code changes) confirmed the full spec end-to-end live: `VERDICT resources=10 layers_checked=3 mismatches=0 unreachable=0 rejected=1 result=FAIL` (default 9 + 1 fake name) and `VERDICT resources=9 ... rejected=0 result=PASS` (defaults only, positive control). Found (but correctly did not fix, out of boundary) **bug-1583**: the apiserver's `OnResourceLocationUpdated` performs zero name validation and stores ANY resource-location update pushed to it — including a genuinely-unregistered name the controller/scheduler both reject — because `do_update_resvals()` fires the `HOOK_DETAIL_RESOURCE_UPDATE` hook with the raw, unfiltered submitted list; the scheduler happens to independently re-validate and reject, the apiserver does not. Confirmed this does NOT affect this feature's own correctness (accept/reject is determined entirely by `cmd_resupdate()`'s own controller-side follow-up query, before any layer is checked) — but it's a real, previously-undocumented data-integrity gap in production code, worth a dedicated follow-up outside this feature. -- Task 7.1 (round 1 REJECTED, round 2 fixed): `aggregate_verdict()`/`build_verdict_record()` (VERDICT_FIELD_SEP=`$'\x1f'`, ASCII Unit Separator — bash's IFS tab-splitting silently collapses adjacent empty fields even with IFS set to only a tab, bug-1581) are the self-contained, synthetic-testable comparison step task 8.2 needs. `main()`'s live-data wiring is `build_verdict_records_from_tsv()` — derive a layer's "ok" status from `SCHEDULER_CHECK_RESULTS[$name]`/`APISERVER_CHECK_RESULTS[$name]`'s STRING CONTENT (`UNREACHABLE:*` prefix check), never from array-key existence — tasks 6.3/6.4 always set the key, even on a genuine timeout (just with an empty value), so key-existence can never distinguish UNREACHABLE from MISMATCH (bug-1582). `CONTROLLER_OBSERVED`'s key-existence check IS correct for the controller layer specifically (task 6.2 has no separate UNREACHABLE concept — a query failure there aborts the whole run). VERDICT line format: `[xce-test-resources] VERDICT resources=<N> layers_checked=3 mismatches=<M> unreachable=<U> rejected=<R> result=PASS|FAIL` — `rejected=` is an intentional addition beyond design.md's illustrative template (Requirement 6.3 needs rejected names attributable in the machine-parseable line itself); `layers_checked=3` is a fixed constant, not a per-run computed count. SKILL.md fully rewritten to describe the now-complete, functional tool. -- Task 6.4: `poll_apiserver_log()`/`query_apiserver_resvals()` mirror task 6.3's scheduler-check pattern exactly (same bounded-wait/batched-polling shape). The apiserver's zap `console` encoder renders structured fields as a trailing JSON object (`resource-location stored {"resource": "r15m", "host": "...", "value": 1.5}`), not classic `key=value` text — grep for the message string, then parse the trailing JSON block. All three layers (controller/scheduler/apiserver) confirmed simultaneously MATCH on the full default set including `cpuf`, proving the bug-1579 fix holds end-to-end. -- **bug-1579 is now FULLY fixed** (retrofit on top of tasks 6.1/6.2, not a numbered task — user-requested follow-up). `inject_resources()`'s worker-context `xce_test` script now inserts a `!<shell-command>` line between `resupdate` and `quit`: `! bash <script> --internal-controller-check <top> <peers> <worker-host> <names...>` — this runs the controller-check query via `xce_test`'s `!` shell-escape (`preprocess_and_run()`, handled in the single top-level REPL loop before context dispatch, so it works in ANY context including `worker` mode without disconnecting). The controller check now runs WHILE the injecting worker's connection is still open, so host-attrs resources are never reset before being checked. `DEFAULT_RESOURCE_ARGS` now includes `cpuf` again (proof the fix works: `cpuf` now MATCHes at the controller layer, not the old false MISMATCH). `query_controller_resvals()`'s comparison logic (`controller_values_match()`) is confirmed byte-for-byte unchanged — this was a pure sequencing fix. Any future task touching `inject_resources()`/the controller-check flow should be aware of this nested-process structure (`--internal-controller-check` is a hidden internal dispatch flag, gated before `main()`/`parse_args()` runs — do not remove it thinking it's dead code). -- Non-blocking notes from the retrofit's review: `internal_controller_check()` queries ALL requested resource names (including ones that will later be rejected), not just accepted ones like the original design — forced by sequencing (accept/reject isn't known until the whole injecting session's output is parsed after it returns); confirmed benign, the final `controller-check-results.tsv` still correctly omits rejected names. Also: invoking `--internal-controller-check` directly with fewer than 3 args hits `set -u` and dies with a raw "unbound variable" message instead of a clean `fail()` — cosmetic only, this path is unreachable from any normal invocation. -- Minor non-blocking follow-up from task 6.3's review: `poll_scheduler_log()`'s output isn't filtered to the caller's `pending` list (harmless today since the only caller already filters, but worth tightening). -- Task 6.2 (round 1 REJECTED, round 2 fixed) discovered **bug-1579**: any host-attrs-derived resource (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` — `worker.c`'s `N_HOST_ATTRS`) gets reset to baseline by the controller the instant the synthetic worker disconnects, and `inject_resources()` (6.1) always disconnects (`quit`) before each layer's separate check session runs — so a controller-layer check on any host-attrs name will ALWAYS false-MISMATCH under the current two-session structure. Fixed for the default set by swapping `DEFAULT_RESOURCE_ARGS`'s `cpuf` for `r1m` (periodic/dynamic, confirmed safe). **This risk is NOT fully closed** — a user-supplied custom `resource=value` argument using any host-attrs name will still hit this. A real fix requires either a C-level `xce_test` change (its REPL has no path from the `worker` context to the `client` context without a disconnect-equivalent `leave_ctx()`) or a bash coprocess restructuring to keep the worker connected across all layer checks — out of scope for tasks 6.1-6.4, worth a dedicated follow-up task if host-attrs resources ever need default-set coverage. -- Task 6.1 uncovered and got fixed a critical, unrelated, pre-existing controller bug (**bug-1578**, commit `b406dd7`, standalone — a dangling-pointer use-after-free in `sched_worker_change()`/`sched.c` that crashed the controller on worker disconnect once a scheduler had ever joined, 5 months old, unrelated to this feature). It's fixed now with regression coverage (`test_sched_worker_change.c`). If any later task sees a live controller die unexpectedly during a worker-disconnect sequence, confirm you're building from a commit that includes `b406dd7` before assuming it's a new bug. `inject_resources()` also fixed **bug-1577** (in-boundary): `XCE_WORKER_BIN=/bin/true` is exported before invoking `start-all.sh` to suppress the REAL `xcecd_virtual` worker daemon that `deploy-ha.sh`'s pre-existing worker branch would otherwise silently start (Requirement 2.1 violation) whenever the build dir happens to contain one. -- Task 6.1: injection results are stored in `INJECTED_VALUES`/`INJECTED_RESULTS` associative arrays plus `$TOP/logs/injection-results.tsv` (`name\tvalue\tresult`) for tasks 6.2-6.4's extension points, which are appended directly after `inject_resources()` in `main()` (explicit "Extension point" comments mark exactly where). `SKILL.md` is now stale (still says injection "not yet implemented", still illustrates `type` in the default set) — deliberately left unedited since task 6.1's CLI parsing shape didn't change; fix this in task 7.1 when SKILL.md is next touched. -- Task 5.1: `run-resource-verify.sh` exists now (`.claude/skills/xce-test-resources/`), structured with explicit "Extension point" comments marking where task 6.1 (injection), 6.2-6.4 (per-layer checks), and 7.1 (verdict/cleanup-on-every-path) hook in — read those before starting 6.1. Trap is registered as the first statement in `main()`, before any process starts; `cleanup()` uses a single combined `pkill -9 -f` (alternation pattern) plus retry-until-`pgrep`-clean, not sequential per-process kills (a sequential version was tried first and found non-deterministically flaky under an async SIGTERM trap — bug-1575). Default deployment is single-node (not 3-node HA) — deliberate, since this skill verifies propagation not failover; override via `$XCE_RESVERIFY_PEERS`. Known non-blocking gap: `$XCE_RESVERIFY_SCHED_BIN`/`$XCE_RESVERIFY_APISERVER_BIN` only gate this script's own pre-flight check, NOT which binary `deploy-ha.sh` actually deploys (it hardcodes `/tmp/apiserver` and derives the scheduler path from `$BINARY`'s build dir) — don't rely on these two env vars to test against an alternate binary. -- Task 4.2: the apiserver's trace log line (task 6.4 must grep for exactly this) is zap message `"resource-location stored"` with fields `resource` (string), `host` (string), `value` (float64 for numeric resources — the common case; string for `type`/`model`; bool for boolean-typed resources), e.g. `{"resource": "r15m", "host": "review-worker", "value": 1.75}`. Grep the log for `resource-location stored`, then parse `"resource": "<name>"` / `"host": "<host>"` / `"value": <val>` from the trailing field block. -- **Task 4.2 discovered bug-1574, which will affect task 8.1 and possibly task 7.1's default resource set — read before implementing either**: `xce_test` can SEGFAULT on `resupdate -n type -v ...` / `resupdate -n model -v ...` (the two `Value_u_svalue`/string-typed resources). Root cause: `set_resval()`'s switch has no `Value_u_svalue` case, so `vals[idx].name` is left pointing at whatever garbage was already on `cmd_resupdate()`'s never-zero-initialized stack array — reading genuinely uninitialized stack memory, invisible to ASan (would need MemorySanitizer), and NON-DETERMINISTIC (task 2.2's reviewer saw `-n type -v foo` NOT crash under full ASAN; task 4.2's implementer saw `-n model -v CustomModelX` crash immediately — different argv/stack layout, same underlying UB). Task 1.2's `SKILL.md` scaffold already documents `type` as part of an illustrative default resource set "for string-path regression coverage" — **before task 7.1 finalizes the actual default resource set, either exclude `type`/`model` from any default until bug-1574 is fixed, or fix bug-1574 first** (it's a small, well-understood fix: zero-initialize `vals[]` in `cmd_resupdate()` and/or add the missing `Value_u_svalue` case to `set_resval()`'s value-population switch — out of boundary for tasks 4.1-4.2, in boundary for whichever task next touches `test.c`'s `cmd_resupdate()`/`set_resval()`, most likely as a prerequisite fix before task 8.1's live e2e run). A crash mid-run would make task 8.1's or the skill's own live verification flaky/non-deterministic, undermining the whole feature's purpose. -- Consolidated list of pre-existing (not-fixed-here) bugs surfaced by this feature's own live testing so far: bug-1571 (apiserver `RegisterToBackend` gate checks `Address` before `Peers` — worked around in generated config, not source, by task 3.1), bug-1572 (scheduler double-free/crash on a resource's 2nd periodic report — task 4.1), bug-1573 (see buglog.json, task 4.1), bug-1574 (`xce_test` segfault on string-typed `resupdate` — task 4.2, described above). -- Task 4.1: the scheduler's trace log line format (task 6.3 must grep for exactly this) is `resval applied host=<H> resource=<R> value=<V>` at `DEBUG3` level (requires `-vvv`, per task 3.1's `start-all.sh`). Numeric resources use `%f` (fixed 6-decimal, e.g. `12345.500000` — parse-and-compare as float, not exact string match); string-typed resources (`type`/`model`) use `%s`. Any other `Value_u_which` tag (e.g. `Value_u_bvalue`, boolean resources) is silently NOT logged by this line (matches this file's own sibling-function convention) — not relevant to this feature's resource set, which is entirely fvalue/svalue. -- Task 4.1 (round 2) discovered and logged, but deliberately did NOT fix (out of boundary, confirmed pre-existing since 2026-08-03/04, unrelated to this feature): **bug-1572** — the live scheduler daemon double-frees and silently crashes on a worker's SECOND periodic resource report for the same resource (`core_events.c`'s RESVALS_DELTA case stores `rv->val` into `w->resvals` without `MOVE_PTR`, while `driver.c`'s `release_translated_payload()` unconditionally frees the same list). This feature's own resupdate-based injection flow (tasks 5.1+) should avoid resupdate-ing the exact same resource name twice against the same live scheduler within one run until bug-1572 is fixed separately, or the live e2e task (8.1) could intermittently crash the scheduler mid-run. -- Task 1.1: `resval_typeresolve.c`'s new function is `int resval_resolve_type(const char* name, int type_override)`, returning a `Value_u_which` tag or `RESVAL_TYPE_UNKNOWN` (-1, doesn't collide with the real 0-6 enum range). Task 2.1 should build on this exact signature rather than inventing a new one. `test.c`'s `set_resval()`/`cmd_resupdate()` were deliberately left untouched by task 1.1 — task 2.1 is the one that makes `set_resval()` a thin wrapper. -- `controller_unit_test` (the big legacy Unity binary) has a pre-existing, unrelated flaky failure in `test_config_repl.c` (bug-1561, overlayfs `fdatasync errno=5` sandbox artifact) that shows up on every `/xce-build controller` run in this environment. It is NOT caused by this feature's changes — do not chase it in later tasks; cross-check `.wolf/buglog.json`'s bug-1561 entry before treating a `controller_unit_test` failure as real, and confirm via `git status` that `test_config_repl.c`/`config_repl.c` are untouched by the current task's diff. -- Task 3.1: `src/frontend/Makefile`'s `build-server` target does not work standalone in this environment — it never sets `CGO_CFLAGS`/`CGO_LDFLAGS` the way `test-backend` does, failing with `fatal error: client.h: No such file or directory`. Pre-existing, out-of-boundary for this feature (confirmed unrelated to any of this feature's changes). To build a real apiserver binary for a live smoke test, export the same CGO flags `make test-backend` derives before running `go build`. Worth a separate bug-log entry / follow-up fix to the Makefile itself, not fixed here. Also found and worked around (config-only, no source change): `main.go`'s `RegisterToBackend` gate checks `BackendYamlConfig.Address` before consulting `Peers`, so a generated config needs `address`/`port` populated from the first `--peers` candidate, not just `peers` itself (bug-1571). -- Task 2.2: `src/controller/meson.build` compiles `test.c` only into the `xce_test` binary (`xce_test_src`), never into `controller_unit_test` (`controller_unit_test_src`) — a `test.c`-only diff structurally cannot regress `controller_unit_test`'s results. Also: `controller_unit_test` is built with ASAN `detect_leaks=0` as pre-existing, repo-wide meson policy (not something any task in this feature turned off) — to actually check a `test.c` change for new leaks, run `xce_test` directly with leak detection left ON, don't rely on the meson suite's leak-detection status. -- Task 2.1 (round 1 REJECTED, round 2 fixed): refactoring a getopt-loop's flush/finalization timing (needed to let a trailing flag like `-t` apply to its preceding `-n`/`-v` pair) can silently drop an unrelated existing guard (here: duplicate-`-v` rejection) that depended on the old immediate-finalization timing. When touching `cmd_resupdate()`-shaped CLI parsers in this codebase again, explicitly re-trace EVERY branch of the old getopt loop against the new one, not just the branches directly related to the new flag — a "traced every path" claim is only as good as the list of paths actually enumerated. `xce_test` also needs `ASAN_OPTIONS=detect_leaks=0` for reliable non-interactive stdin-piped smoke testing (its default leak-detection exit path can swallow buffered stdout). diff --git a/.kiro/specs/resval-id-decoupling/design.md b/.kiro/specs/resval-id-decoupling/design.md deleted file mode 100644 index 56620a14..00000000 --- a/.kiro/specs/resval-id-decoupling/design.md +++ /dev/null @@ -1,435 +0,0 @@ -# Design: resval-id-decoupling - -> Feature: resval-id-decoupling -> Created: 2026-05-12 -> Status: Design Generated - -## Architecture Pattern & Boundary Map - -**Pattern:** Targeted replacement of an index-keyed cross-subsystem data carrier with a name-keyed one, following the precedent set by `Data_data_resvals` (Path C). No new architectural layers are introduced. - -**Affected boundary:** Controller → Scheduler (initial config push) and Controller → Scheduler/API-server (per-worker data push). The wire protocol (`controller.capnp`) at these two field positions changes from `List(ValueWithIdx)` to `List(NameValue)`. - -**Scope:** - -``` -src/schema/controller.capnp ← 2 field type edits -src/include/xce/types.h ← 1 struct field type change -src/include/typemap.h ← 1 struct field type change -src/include/utils.h ← swap free function declaration -src/controller/include/cluster.h ← remove collect_resval_with_idx declaration -src/controller/dataex.c ← replace collect_resval_with_idx with collect_resval -src/controller/sched.c ← 3 call-site updates -src/controller/apiserver.c ← 2 call-site updates -src/controller/reply.c ← 1 free call update -src/controller/opget.c ← 1 free call update -src/lib/utils.c ← swap free function implementation -src/scheduler/src/scheduler.c ← sched_init_sharedInst update -src/scheduler/src/schedobj.c ← make_sched_worker update -``` - -**Not in scope:** `src/scheduler/src/handler.c` (Path C untouched), `src/frontend/apiserver/` Go code (no direct access to `resvals_to_sched` in Go layer), capnp generated files (auto-regenerated by `ninja -C build` after schema edit). - -## Technology Stack & Alignment - -| Aspect | Choice | Rationale | -|--------|--------|-----------| -| Language | C99 | Matches existing codebase convention | -| Wire protocol | Cap'n Proto — `List(NameValue)` replacing `List(ValueWithIdx)` | `NameValue`/`xce_name_val_t` already defined in schema and types; codec already generated | -| Build | Meson + capnpc-c (existing) | Schema edit triggers automatic regeneration of `controller.capnp.[ch]` | -| Logging | Existing `WARN`/`ERROR` macros | Consistent with codebase; unknown resource → `WARN` + skip | -| Iterator pattern | `iter_make` / `iter_next` / `iter_destroy` with read lock | Matches `collect_resources()` pattern already in `dataex.c` | -| Backward compatibility | Not required | No public release; breaking wire change is acceptable | - -## Components & Interface Contracts - ---- - -### Component 1: `controller.capnp` — Wire type changes (Req 1, 2, 3) - -Two field type replacements in `src/schema/controller.capnp`. Both field ordinal numbers are preserved (no renumbering); only the element type changes. - -``` -Before: - struct Worker { resvals @5: List(ValueWithIdx) ... } - struct Conf4Sched { resvals @6: List(ValueWithIdx) ... } - -After: - struct Worker { resvals @5: List(NameValue) ... } - struct Conf4Sched { resvals @6: List(NameValue) ... } -``` - -After this edit, `ninja -C build` regenerates `controller.capnp.[ch]`. The generated -`encode_Conf4Sched` and `decode_Conf4Sched` functions will automatically call -`encode_NameValue_list` / `decode_NameValue_list` (already implemented in the generated -code) instead of their `ValueWithIdx` counterparts. - -**Requirement traceability:** 1.1, 2.1, 3.1, 4.2 - ---- - -### Component 2: Type changes in shared headers (Req 1, 2, 3, 4) - -#### `src/include/xce/types.h` — `xce_worker_t.resvals_to_sched` - -``` -Before: - int nresvals_to_sched; - xce_val_with_idx_t** resvals_to_sched; - -After: - int nresvals_to_sched; - xce_name_val_t** resvals_to_sched; -``` - -#### `src/include/typemap.h` — `xce_conf4sched_t.resvals` - -``` -Before: - int nresvals; - xce_val_with_idx_t** resvals; - -After: - int nresvals; - xce_name_val_t** resvals; -``` - -**Requirement traceability:** 1.1, 2.1, 4.2 - ---- - -### Component 3: `dataex.c` — New collector; remove old (Req 1, 2, 5) - -#### Remove: `collect_resval_with_idx_` and `collect_resval_with_idx` - -Both the static inner function and its public wrapper are deleted. The declaration in -`src/controller/include/cluster.h` is also removed. - -#### Add: `collect_resval` - -Iterates `resmgr` (not the zarray directly) so resource names are always available. -For each named resource, looks up its value by `r->idx` in the provided `zarray_t`. - -``` -Signature: - int collect_resval(var resmgr, zarray_t* array, xlist_t* l) - -Parameters: - resmgr — read-locked resource manager; provides name→idx mapping - array — zarray_t holding xce_val_t* values indexed by r->idx (lock held by caller) - l — output xlist_t; receives xce_name_val_t* entries (caller owns memory) - -Behavior: - 1. lock(resmgr, RTTI_LOCK_READ) - 2. iter_make(&it, resmgr) - 3. For each xce_resource_t* r from iter_next: - a. If r == NULL, continue - b. src_val = (xce_val_t*)zarray_get(array, r->idx) [array already caller-locked] - c. If src_val == NULL, continue [resource exists but has no value yet] - d. Allocate xce_name_val_t* nv = calloc(1, sizeof(*nv)) - e. nv->name = strdup(r->name) - f. nv->val = calloc(1, sizeof(xce_val_t)) - g. Copy value: if svalue, strdup; else memcpy - h. xlist_store(l, nv) - 4. iter_destroy(&it) - 5. unlock(resmgr) - 6. Return l->count - -Return value: - >= 0 number of name-val entries collected - -ENOMEM allocation failure (partial list may exist; caller responsible for cleanup) - -Note: array lock must be acquired by the caller before calling this function, -consistent with how collect_resval_with_idx_ was called from collect_resval_with_idx -(which held array->lock). This keeps locking responsibility explicit. -``` - -#### Update: `collect_shared_resvals` - -``` -Before: - void collect_shared_resvals(cluster_t* c, collect_check_fn_t fn, xlist_t* l) { - collect_resval_with_idx(&(c->sharedinst), l); - } - -After: - void collect_shared_resvals(cluster_t* c, collect_check_fn_t fn, xlist_t* l) { - mylock_lock(&(c->sharedinst.lock)); - collect_resval(c->resmgr, &(c->sharedinst), l); - mylock_unlock(&(c->sharedinst.lock)); - } -``` - -`fn` parameter is unused (was unused before); kept for signature compatibility with -other `collect_*` functions. The `cluster_t*` provides access to both `resmgr` and -`sharedinst`. - -#### Update: `collect_resvals_workers` (inlined into `collect_workers`) - -``` -Before (in collect_resvals_workers): - collect_resval_with_idx(&(w->resvals), &lst); - -After: - mylock_lock(&(w->resvals.lock)); - collect_resval(resmgr, &(w->resvals), &lst); - mylock_unlock(&(w->resvals.lock)); -``` - -`collect_resvals_workers` requires access to `resmgr`. Pass it as a new parameter, or -access it from the containing `cluster_t*` context (the function is only called from -`collect_workers` which receives `cluster_t* c`). Preferred: pass `c->resmgr` from -`collect_workers`. - -Updated `collect_resvals_workers` signature: - -``` - static void collect_resvals_workers(var resmgr, xlist_t* l) -``` - -**Requirement traceability:** 1.1, 2.1, 5.1, 5.2 - ---- - -### Component 4: `src/lib/utils.c` and `src/include/utils.h` — Free function swap (Req 5) - -#### Remove: `free_resval_with_idx_list` - -The existing implementation and declaration are deleted. No remaining callers after -the controller-side changes in Component 5. - -#### Add: `free_name_val_list` - -Mirrors the removed function's structure; frees each `xce_name_val_t*` entry. - -``` -Signature: - void free_name_val_list(xce_name_val_t*** vals, int count) - -Behavior: - 1. resvals = *vals - 2. If resvals == NULL or count <= 0, return - 3. For i in [0, count): - a. nv = resvals[i]; if NULL, continue - b. FREEUP(nv->name) - c. If nv->val != NULL: - if nv->val->which == Value_u_svalue: FREEUP(nv->val->u.svalue) - FREEUP(nv->val) - d. FREEUP(nv) - 4. FREEUP(*vals) - -Header declaration (utils.h): - extern void free_name_val_list(xce_name_val_t***, int); -``` - -**Requirement traceability:** 5.1 - ---- - -### Component 5: Controller call sites — `sched.c` and `apiserver.c` (Req 1, 2, 3) - -All call sites that previously called `collect_resval_with_idx` and -`free_resval_with_idx_list` are updated to use the new functions and types. - -#### `sched.c` — `xcesched_push_configs` (shared resvals, Req 1) - -``` -Before: - collect_shared_resvals(c, NULL, &xreslist); - configdata.data.conf4sched->nresvals = xreslist.count; - configdata.data.conf4sched->resvals = (xce_val_with_idx_t**)xreslist.data; - ... - free_resval_with_idx_list(&(configdata.data.conf4sched->resvals), - configdata.data.conf4sched->nresvals); - -After: - collect_shared_resvals(c, NULL, &xreslist); - configdata.data.conf4sched->nresvals = xreslist.count; - configdata.data.conf4sched->resvals = (xce_name_val_t**)xreslist.data; - ... - free_name_val_list(&(configdata.data.conf4sched->resvals), - configdata.data.conf4sched->nresvals); -``` - -#### `sched.c` — `xcesched_push_workerlist` (worker resvals, Req 2) - -``` -Before: - collect_resval_with_idx(&(w->host.resvals), &resvalist); - w->host.nresvals_to_sched = resvalist.count; - w->host.resvals_to_sched = (xce_val_with_idx_t**)resvalist.data; - ... - free_resval_with_idx_list(&(w->host.resvals_to_sched), w->host.nresvals_to_sched); - -After: - mylock_lock(&(w->host.resvals.lock)); - collect_resval(c->resmgr, &(w->host.resvals), &resvalist); - mylock_unlock(&(w->host.resvals.lock)); - w->host.nresvals_to_sched = resvalist.count; - w->host.resvals_to_sched = (xce_name_val_t**)resvalist.data; - ... - free_name_val_list(&(w->host.resvals_to_sched), w->host.nresvals_to_sched); -``` - -#### `apiserver.c` — `apiserver_worker_change` (Req 3) - -Same pattern as the `sched.c` worker case above. Replace collect and free calls; -change cast to `(xce_name_val_t**)`. - -#### `sched.c:226`, `reply.c:66`, `opget.c:128` — cleanup-only sites - -Replace `free_resval_with_idx_list` with `free_name_val_list` at each cleanup site -where `w->resvals_to_sched` is freed without being re-populated. - -**Requirement traceability:** 1.1, 2.1, 3.1, 5.1 - ---- - -### Component 6: Scheduler — `sched_init_sharedInst` (Req 1, 6) - -Receives `xce_conf4sched_t.resvals` which is now `xce_name_val_t**`. -For each entry, look up the resource by name in `sched->resmgr` to get the local idx, -then store into `sched->sharedInst`. - -**Critical ordering constraint:** `sched_init_resources()` must be called before -`sched_init_sharedInst()`. This is already guaranteed by `sched_init_config()` -(`scheduler.c:603-609`). This call order must not change. - -``` -Before: - static int sched_init_sharedInst(xce_scheduler_t* sched) { - for (i = 0; i < sched->config->nresvals; i++) { - int idx = sched->config->resvals[i]->idx; - xce_val_t* val = NULL; - MOVE_PTR(val, sched->config->resvals[i]->val); - zarray_put(&(sched->sharedInst), idx, val); - } - return 0; - } - -After: - static int sched_init_sharedInst(xce_scheduler_t* sched) { - for (i = 0; i < sched->config->nresvals; i++) { - xce_name_val_t* nv = sched->config->resvals[i]; - xce_resource_t* r = NULL; - xce_val_t* val = NULL; - - if (nv == NULL || nv->name == NULL) continue; - - if (key_find(sched->resmgr, (void*)nv->name, (void**)&r) < 0 || r == NULL) { - WARN("%s unknown resource '%s' in resvals, skipping", - __func__, nv->name); - continue; - } - - MOVE_PTR(val, nv->val); - zarray_put(&(sched->sharedInst), r->idx, val); - } - return 0; - } -``` - -**Requirement traceability:** 1.2, 1.3, 6.1, 6.2, 6.3 - ---- - -### Component 7: Scheduler — `make_sched_worker` in `schedobj.c` (Req 2, 6) - -Receives `xce_worker_t.resvals_to_sched` which is now `xce_name_val_t**`. -For each entry, look up by name in `sched->resmgr` to get local `r->idx`. - -``` -Before: - for (i = 0; i < w->nresvals_to_sched; i++) { - int idx = w->resvals_to_sched[i]->idx; - xce_val_t* val = NULL; - MOVE_PTR(val, w->resvals_to_sched[i]->val); - zarray_put(&(worker->resvals), idx, val); - } - -After: - for (i = 0; i < w->nresvals_to_sched; i++) { - xce_name_val_t* nv = w->resvals_to_sched[i]; - xce_resource_t* r = NULL; - xce_val_t* val = NULL; - - if (nv == NULL || nv->name == NULL) continue; - - if (key_find(sched->resmgr, (void*)nv->name, (void**)&r) < 0 || r == NULL) { - WARN("%s unknown resource '%s' in worker resvals, skipping", - __func__, nv->name); - continue; - } - - MOVE_PTR(val, nv->val); - zarray_put(&(worker->resvals), r->idx, val); - } -``` - -`make_sched_worker` already receives `xce_scheduler_t* sched` as its first argument -(`void* arg`), cast at top of function — `sched->resmgr` is available. - -**Requirement traceability:** 2.2, 2.3, 2.4, 6.3 - ---- - -## Data Flow - -### Before (fragile index-keyed path) - -``` -Controller Scheduler ----------- --------- -c->sharedinst[r->idx] = val - │ -collect_resval_with_idx() - emit {idx=i, val} ──────────────► sched_init_sharedInst() - zarray_put(&sharedInst, idx, val) - ↑ - ASSUMES controller idx == scheduler idx - (NO ENFORCEMENT — silent corruption if false) -``` - -### After (name-keyed path, matches Path C) - -``` -Controller Scheduler ----------- --------- -c->sharedinst[r->idx] = val - │ -collect_resval(resmgr, array) - iter resmgr → r->name, r->idx - val = array->data[r->idx] - emit {name=r->name, val} ────────► sched_init_sharedInst() - key_find(sched->resmgr, name) → r - zarray_put(&sharedInst, r->idx, val) - ↑ - Uses scheduler's OWN r->idx - No cross-subsystem index assumption -``` - -## Requirements Traceability - -| Requirement | Component(s) | -|------------|--------------| -| 1.1 — Name-keyed shared resvals in config push | Comp 1, 2, 3, 5 | -| 1.2 — Scheduler resolves by name in resmgr | Comp 6 | -| 1.3 — Warn and skip unknown resource names | Comp 6 | -| 1.4 — Config path produces same sharedInst state as Path C | Comp 6 (same key_find → r->idx logic as Path C) | -| 2.1 — Name-keyed worker resvals | Comp 1, 2, 3, 5 | -| 2.2 — Scheduler resolves worker resvals by name | Comp 7 | -| 2.3 — Warn and skip unknown worker resource names | Comp 7 | -| 2.4 — Dynamic resource catalog changes handled correctly | Comp 7 (no cached idx; lookup on each receive) | -| 3.1 — Name-keyed worker resvals to API server | Comp 1, 2, 5 (`apiserver.c`) | -| 3.2 — API server match by name without numeric index | Comp 2 (type change), Comp 5 | -| 4.1 — Path C (`Data_data_resvals`) unchanged | No changes to `handler.c` or `scheduler.c:770-788` | -| 4.2 — All struct consumers updated consistently | Comp 1, 2, 5, 6, 7 | -| 5.1 — No cross-subsystem slot-indexed resval path remains | Comp 3 (remove old function), Comp 4, 5, 6, 7 | -| 5.2 — `collect_resval_with_idx` removed | Comp 3 | -| 6.1 — Correct values when idx assignments differ | Comp 6, 7 | -| 6.2 — Identical behavior when idx assignments match | Comp 6, 7 | -| 6.3 — Scheduling lookups use scheduler's own indices | Comp 6, 7 | - -## Open Questions - -None — all design decisions resolved in gap analysis and user feedback. diff --git a/.kiro/specs/resval-id-decoupling/requirements.md b/.kiro/specs/resval-id-decoupling/requirements.md deleted file mode 100644 index 81dc8abb..00000000 --- a/.kiro/specs/resval-id-decoupling/requirements.md +++ /dev/null @@ -1,95 +0,0 @@ -# Requirements: resval-id-decoupling - -> Generated: 2026-05-12T06:52:00-07:00 -> Updated: 2026-05-12T07:10:00-07:00 -> Feature: resval-id-decoupling -> Status: Requirements Generated - -## Overview - -The controller passes resource values to the scheduler (initial config push) and to the API server (worker data) using `xce_val_with_idx_t`, a struct carrying a numeric array-slot index. The scheduler and API server then store values at that same index in their own `zarray_t` stores. This works only when both components assign the same slot number to each resource — an assumption that holds when resource registration order is strictly identical but is not enforced by any contract and will silently produce wrong values if it breaks. - -The ongoing update path (`Data_data_resvals` dataset, `xce_resval_t` with `name` field) already resolves this correctly via name-based lookup in the receiver's own resmgr. The goal of this feature is to apply the same name-keyed approach uniformly across all resource-value transfer paths, eliminating the hidden index-identity assumption. - -## Codebase Analysis Summary - -Three distinct paths currently transfer resource values between components: - -**Path A — Shared resvals in initial config push** (`src/controller/sched.c:118-120`) -- `collect_shared_resvals()` → `collect_resval_with_idx(&c->sharedinst, ...)` emits `{idx, val}` pairs where `idx` is the controller's `sharedinst` zarray slot (= controller resource `idx`). -- Scheduler receives via `xce_conf4sched_t.resvals` (`xce_val_with_idx_t**`) and stores directly at the same index: `zarray_put(&sched->sharedInst, idx, val)` (`src/scheduler/src/scheduler.c:583`). -- **Bug**: If `resource->idx` differs between controller and scheduler (different insertion order into each component's independent resmgr), values land in the wrong slot. - -**Path B — Worker resvals to scheduler** (`src/controller/sched.c:442-444`, `src/controller/apiserver.c:367-369`) -- `collect_resval_with_idx(&w->host.resvals, ...)` emits `{idx, val}` from the controller's per-worker resvals zarray (slot = controller resource `idx`). -- Scheduler stores at the same slot: `zarray_put(&worker->resvals, idx, val)` (`src/scheduler/src/schedobj.c:279`). -- Later, `resource.c` indexes `worker->resvals` using the scheduler's own `residx` values — a mismatch if indices differ. - -**Path C — `Data_data_resvals` dataset update** (already correct) -- Uses `xce_resval_t` (name + host + val). Scheduler looks up `r = key_find(sched->resmgr, rv->name)` then uses `r->idx` (its own index): `zarray_put(&sched->sharedInst, r->idx, rv->val)` (`src/scheduler/src/scheduler.c:784`). -- No index-identity assumption. This is the target pattern. - -The fix must align Paths A and B with Path C's name-based resolution approach. The existing `xce_name_val_t` type (`src/include/xce/types.h:43`) already provides the right shape for shared resvals (name + val). The `xce_resval_t` type already provides the shape for per-host/worker resvals (name + host + val). - -## Requirements - -### Requirement 1: Name-keyed shared resource value transfer in initial config - -The scheduler SHALL receive shared resource values from the controller as name-keyed records (resource name + value) rather than as array-slot-indexed records. - -Upon receiving the initial config push, the scheduler SHALL resolve each resource value to its own slot index by looking up the resource name in its own resource manager (`sched->resmgr`), then store the value at the resolved index in `sched->sharedInst`. - -The scheduler SHALL log a warning and skip (not crash or silently corrupt) any resource value for which the resource name is not found in its resource manager. - -**Acceptance criteria:** -- EARS-R1-1: WHEN the controller sends an initial config push (`ConfigData_data_schedconfig`) containing shared resource values, THEN each resource value SHALL be identified by resource name, not by a numeric array slot index. -- EARS-R1-2: WHEN the scheduler processes a shared resource value record whose name matches a resource in `sched->resmgr`, THEN the scheduler SHALL store the value at the index assigned by its own resmgr (`r->idx`). -- EARS-R1-3: WHEN the scheduler processes a shared resource value record whose name does not match any resource in `sched->resmgr`, THEN the scheduler SHALL emit a warning log and skip that record without modifying `sched->sharedInst`. -- EARS-R1-4: WHEN an existing system running Path C updates (`Data_data_resvals`) works correctly, THEN after this change the initial-config path SHALL produce the same `sched->sharedInst` state as Path C would for the same resource values. - -### Requirement 2: Name-keyed per-worker resource value transfer to scheduler - -The scheduler SHALL receive per-worker resource values from the controller as name-keyed records (resource name + value) rather than as array-slot-indexed records. - -Upon receiving worker data, the scheduler SHALL resolve each worker resource value to its own slot index by looking up the resource name in its resource manager, then store the value at the resolved index in the worker's `resvals` zarray. - -The scheduler SHALL log a warning and skip any worker resource value whose resource name is not found. - -**Acceptance criteria:** -- EARS-R2-1: WHEN the controller sends worker data containing per-worker resource values, THEN each resource value SHALL be identified by resource name, not by a numeric array slot index. -- EARS-R2-2: WHEN the scheduler builds a `sched_worker` from received worker data, THEN it SHALL resolve each resource value name to the scheduler's own `r->idx` before calling `zarray_put(&worker->resvals, ...)`. -- EARS-R2-3: WHEN a resource name in a worker's resource values is unknown to the scheduler's resmgr, THEN the scheduler SHALL log a warning and skip that entry without aborting worker construction. -- EARS-R2-4: WHILE the scheduler is running and a resource is added or removed from the resource catalog, THEN subsequently received worker data SHALL correctly resolve resource values using the updated resmgr, with no stale index cached from before the change. - -### Requirement 3: Name-keyed per-worker resource value transfer to API server - -The API server SHALL receive per-worker resource values from the controller as name-keyed records (resource name + value) so that its own resource catalog lookup can resolve values independently of the controller's internal index assignment. - -**Acceptance criteria:** -- EARS-R3-1: WHEN the controller sends worker data to an API server connection, THEN per-worker resource values SHALL be transmitted using resource names as keys. -- EARS-R3-2: WHEN the API server (`src/frontend/apiserver`) converts worker resource values for its models (e.g., `cResvalToDistribution`), THEN it SHALL be able to match values to resources by name without assuming any particular numeric index. - -### Requirement 4: Backward compatibility with the `Data_data_resvals` update path - -The name-keyed changes to the initial config and worker data paths SHALL NOT alter the behavior of the existing `Data_data_resvals` dataset update path (Path C), which is already name-keyed. - -**Acceptance criteria:** -- EARS-R4-1: WHEN the scheduler processes a `Data_data_resvals` dataset message, THEN the behavior SHALL be identical before and after this change. -- EARS-R4-2: WHEN the `xce_conf4sched_t` struct is updated to use name-keyed resval records, THEN all callers that populate or consume this struct SHALL be updated consistently (controller, scheduler, and any serialization/deserialization layer in `libxce`). - -### Requirement 5: Removal of the index-keyed `collect_resval_with_idx` call sites - -The `collect_resval_with_idx` function (or its callers) SHALL be updated or replaced so that no code path transmits resource values indexed solely by a numeric array slot without an accompanying resource name. - -**Acceptance criteria:** -- EARS-R5-1: WHEN this feature is complete, THEN no code path that crosses a subsystem boundary (controller → scheduler, controller → API server) SHALL rely on the receiving subsystem's `zarray_t` slot indices matching the sending subsystem's slot indices. -- EARS-R5-2: WHEN `collect_resval_with_idx` is no longer needed for cross-subsystem transfer, THEN it SHOULD be removed or restricted to intra-subsystem use only, with a comment explaining its scope. - -### Requirement 6: No regression in resource value correctness - -The scheduler's scheduling decisions SHALL produce the same resource value lookups after this change as they did when indices happened to be consistent, and SHALL produce correct results when indices are inconsistent (where the old code would silently corrupt scheduling state). - -**Acceptance criteria:** -- EARS-R6-1: WHEN the controller and scheduler register resources in a different order (causing their resmgr index assignments to differ), THEN `sched->sharedInst` SHALL still contain the correct value for each resource after receiving the initial config push. -- EARS-R6-2: WHEN the controller and scheduler register resources in the same order, THEN the behavior of the scheduler SHALL be identical to the current behavior. -- EARS-R6-3: WHEN worker resource values are applied during a scheduling cycle, THEN `calc_wanted_global_resvals` and `calc_wanted_host_resvals` SHALL retrieve values from `sharedInst` and `worker->resvals` using the scheduler's own resource indices, unaffected by the controller's index ordering. diff --git a/.kiro/specs/resval-id-decoupling/research.md b/.kiro/specs/resval-id-decoupling/research.md deleted file mode 100644 index f497b822..00000000 --- a/.kiro/specs/resval-id-decoupling/research.md +++ /dev/null @@ -1,303 +0,0 @@ -# Gap Analysis: resval-id-decoupling - -> Generated: 2026-05-12T07:25:00-07:00 -> Feature: resval-id-decoupling - -## Executive Summary - -- **Scope**: Three call sites in the controller use `collect_resval_with_idx()` to produce index-keyed resource values sent to two consumers (scheduler and API server). Fixing all three brings every cross-subsystem resval path into alignment with the already-correct `Data_data_resvals` dataset path. -- **Existing Infrastructure**: `xce_name_val_t` (`NameValue` in capnp) already exists in `types.h` and has a **complete capnp codec** (`encode_NameValue_list` / `decode_NameValue_list`) in the generated code — it was defined for this purpose but never wired in at the resval-transfer boundary. -- **Wire format impact**: `Conf4Sched.resvals` (field @6) and `Worker.resvals` (field @5) in `controller.capnp` currently declare `List(ValueWithIdx)`. Changing these to `List(NameValue)` is a **breaking wire change** — no intermediate backward-compatibility window unless two field slots are used. -- **Build chain**: The capnp schema in `src/schema/controller.capnp` drives generation of `src/build/lib/controller.capnp.[ch]` via `meson` + `capnpc-c`. A schema edit automatically regenerates the C codecs on the next build. -- **Key insight**: `makeResource_()` in `resobj.c:137` always overwrites `resobj->idx = rc` (from `key_add`) regardless of the `idx` transmitted in the `Resource` wire struct — so the controller's idx is lost at the scheduler's resmgr and the assumption that they match is never enforced. - ---- - -## Existing Components Inventory - -### Types and Structs - -| Symbol | Location | Role | Status | -|--------|----------|------|--------| -| `xce_val_with_idx_t` | `src/include/xce/types.h:38` | `{int idx, xce_val_t* val}` — current cross-subsystem carrier | **Replace at boundaries** | -| `xce_name_val_t` | `src/include/xce/types.h:43` | `{char* name, xce_val_t* val}` — name-keyed carrier | **Ready to use** | -| `xce_resval_t` | `src/include/xce/types.h:48` | `{char* name, char* host, xce_val_t* val}` — used by Path C | Already correct | -| `xce_conf4sched_t.resvals` | `src/include/typemap.h:50` | `xce_val_with_idx_t**` — shared resvals in initial config | **Change to `xce_name_val_t**`** | -| `xce_worker_t.resvals_to_sched` | `src/include/xce/types.h:217` | `xce_val_with_idx_t**` — per-worker resvals | **Change to `xce_name_val_t**`** | - -### Controller — Call Sites (Sender Side) - -| File | Line | Function | Purpose | Change needed | -|------|------|----------|---------|---------------| -| `src/controller/dataex.c` | 62 | `collect_resval_with_idx()` | Iterates `zarray_t`, emits `{idx, val}` | Replace/augment with name-keyed variant | -| `src/controller/dataex.c` | 363 | `collect_shared_resvals()` | Calls `collect_resval_with_idx(&c->sharedinst)` | Change to name-keyed collection | -| `src/controller/dataex.c` | 141–163 | `collect_resvals_workers()` | Calls `collect_resval_with_idx(&w->resvals)` per worker | Change to name-keyed collection | -| `src/controller/sched.c` | 118–120 | `xcesched_push_configs()` | Puts shared resvals into `conf4sched->resvals` | Change field type; update free | -| `src/controller/sched.c` | 442–444 | `xcesched_push_workerlist()` | Puts worker resvals into `w->host.resvals_to_sched` | Change field type; update free | -| `src/controller/apiserver.c` | 367–369 | `apiserver_worker_change()` | Same pattern as sched for API server path | Change field type; update free | - -### Free/cleanup Sites - -| File | Line | Call | Must be updated | -|------|------|------|-----------------| -| `src/controller/sched.c` | 145 | `free_resval_with_idx_list(&conf4sched->resvals, ...)` | → `free_NameValue_list` or equivalent | -| `src/controller/sched.c` | 226, 463 | `free_resval_with_idx_list(&w->resvals_to_sched, ...)` | Same | -| `src/controller/apiserver.c` | 51, 397–399 | `free_resval_with_idx_list(...)` | Same | -| `src/controller/dataex.c` | 404 | `free_resval_with_idx_list(...)` | Same | -| `src/controller/reply.c` | 66 | `free_resval_with_idx_list(...)` | Same | -| `src/controller/opget.c` | 128 | `free_resval_with_idx_list(...)` | Same | -| `src/include/utils.h` | 130 | `free_resval_with_idx_list` declaration | Update or keep alongside new | - -### Scheduler — Consumer Side - -| File | Line | Code | Change needed | -|------|------|------|---------------| -| `src/scheduler/src/scheduler.c` | 574–586 | `sched_init_sharedInst()` — uses `idx` directly from `config->resvals[i]->idx` | Change to name-lookup in `sched->resmgr` | -| `src/scheduler/src/schedobj.c` | 274–280 | `make_sched_worker()` — uses `idx` directly from `w->resvals_to_sched[i]->idx` | Change to name-lookup | - -### Capnp Schema and Generated Code - -| Asset | Location | Current | Change | -|-------|----------|---------|--------| -| `Conf4Sched.resvals` | `src/schema/controller.capnp:230` | `List(ValueWithIdx)` | → `List(NameValue)` | -| `Worker.resvals` | `src/schema/controller.capnp:129` | `List(ValueWithIdx)` | → `List(NameValue)` | -| `controller.capnp.c/h` | `src/build/lib/` | Auto-generated — regenerated by `ninja -C build` | Automatically updated after schema edit | -| `encode_NameValue_list` | `src/build/lib/controller.capnp.c:3402` | Already exists, unused at these boundaries | Wire in to replace `encode_ValueWithIdx_list` | -| `decode_NameValue_list` | `src/build/lib/controller.capnp.c:3426` | Already exists, unused at these boundaries | Wire in to replace `decode_ValueWithIdx_list` | -| `free_NameValue_list` | `src/build/lib/controller.capnp.c:3458` | Already exists | Replace `free_ValueWithIdx_list` at these call sites | - -### API Server (Go frontend) — Impact - -The Go API server (`src/frontend/apiserver/backend/clib.go`) includes `controller.capnp.h` and works with `xce_worker_t*` via CGo. It accesses `resvals_to_sched` in `apiserver_worker_change()` (controller side) which populates `xce_worker_t.resvals_to_sched`. The Go frontend's `cResvalToDistribution` in `convert.go:273` already processes `xce_resval_t` (name-keyed), not `xce_val_with_idx_t`. The API server impact is limited to updating any code that reads `resvals_to_sched` field — if it accesses it at all on the Go side (currently no direct access found). - ---- - -## Root Cause Deep Dive - -The zarray-based stores (`c->sharedinst`, `w->host.resvals`) use `r->idx` as the slot key. `r->idx` is assigned by `key_add()` in `makeResource_` (`resobj.c:137`) — it is the slot returned by the internal container, which depends on insertion order. The controller and scheduler each call `makeResource()` independently (controller at cluster init, scheduler at `sched_init_resources()`), with no guarantee of identical ordering. - -Crucially, the `Resource` capnp struct **does transmit `r->idx` on the wire** (`encode_Resource` at `build/lib/controller.capnp.c:3889`), but `decode_Resource` writes it into a temporary `xce_resource_t`, which is then passed to `makeResource()`, which **immediately overwrites** the decoded `idx` with the local `key_add` return value (`resobj->idx = rc`). The transmitted idx is therefore discarded at the scheduler. - ---- - -## Implementation Options - -### Option A — Replace `ValueWithIdx` with `NameValue` at both capnp fields (Recommended) - -**What changes:** -1. `controller.capnp`: change `Conf4Sched.resvals @6` and `Worker.resvals @5` from `List(ValueWithIdx)` to `List(NameValue)` -2. Rebuild capnp — generated `encode_Conf4Sched`/`decode_Conf4Sched` and worker codec automatically use `NameValue` functions -3. `typemap.h`: `xce_conf4sched_t.resvals` → `xce_name_val_t**` -4. `types.h`: `xce_worker_t.resvals_to_sched` → `xce_name_val_t**` -5. Controller: new `collect_resval_by_name(zarray_t* vals, var resmgr, xlist_t* l)` that walks `resmgr` entries, reads the value by `r->idx` from the array, and emits `{name, val}` records -6. Scheduler `sched_init_sharedInst()`: iterate name-val list, `key_find(sched->resmgr, name)` → `r->idx` → `zarray_put` -7. Scheduler `make_sched_worker()`: same pattern -8. Update all free/cleanup call sites to use `free_NameValue_list` - -**Pros:** Cleanest — removes the misleading `idx` field entirely; `NameValue` codec already exists; aligns with Path C's design; no backward compatibility concerns given no public release -**Cons:** None significant — wire-format break is a non-issue as there is no released version requiring compatibility - -**Affected files (estimate):** 8 C files, 1 capnp schema, 0 Go files - ---- - -### Option B — Add `name` field to `xce_val_with_idx_t` / `ValueWithIdx` ~~(Additive wire change)~~ [REJECTED] - -Ruled out: backward compatibility is not a requirement for this project at its current stage. Option B's only advantage over Option A is a softer wire transition — without that constraint, it is strictly worse: it keeps a semantically dead `idx` field, requires more code changes, and leaves misleading structure in the API. - ---- - -### Option C — Use Controller's idx as canonical; stop overwriting in `makeResource_` [REJECTED] - -Ruled out: does not fix the design — just moves the source of truth without eliminating the hidden identity assumption. A resource deleted and re-added in the controller gets a new idx, and the scheduler has no way to know its cached idx is stale. - ---- - -## Recommendation - -**Option A** is the selected approach. Backward compatibility is not a constraint (no public release), so the wire-format change to `List(NameValue)` is straightforward. The `NameValue` codec already exists in the generated capnp code and is fully functional — this is primarily a wiring exercise. The semantic change aligns the initial-config and worker-data paths with the already-correct `Data_data_resvals` update path (Path C). - ---- - -## Files to Change (Option A) - -| File | Change Type | -|------|-------------| -| `src/schema/controller.capnp` | Schema edit: 2 field type changes | -| `src/include/xce/types.h` | Struct field type change: `resvals_to_sched` | -| `src/include/typemap.h` | Struct field type change: `resvals` | -| `src/controller/dataex.c` | New `collect_resval_by_name()` function; update `collect_shared_resvals()` and `collect_resvals_workers()`; **remove** `collect_resval_with_idx` and `collect_resval_with_idx_` | -| `src/controller/sched.c` | Update 3 call sites (populate and free) | -| `src/controller/apiserver.c` | Update 2 call sites (populate and free) | -| `src/controller/reply.c` | Update free call | -| `src/controller/opget.c` | Update free call | -| `src/scheduler/src/scheduler.c` | Update `sched_init_sharedInst()` | -| `src/scheduler/src/schedobj.c` | Update `make_sched_worker()` | -| `src/include/utils.h` | Remove `free_resval_with_idx_list` declaration; add `free_name_val_list` declaration | -| `src/lib/utils.c` | Remove `free_resval_with_idx_list`; add `free_name_val_list(xce_name_val_t***, int)` | -| `src/controller/include/cluster.h` | Remove `collect_resval_with_idx` declaration | - -**Not changed:** `src/scheduler/src/handler.c` (Path C untouched), `src/frontend/apiserver/` (Go API server does not directly access `resvals_to_sched` in Go code), capnp generated files (auto-regenerated by `ninja -C build`). - ---- - -## Decisions (Closed Questions) - -1. **New collector function signature**: `collect_resval_by_name` is controller-internal logic — add it to `dataex.c` alongside `collect_resval_with_idx`. It needs both the `zarray_t` (values indexed by `r->idx`) and the `resmgr` (name→idx mapping) as parameters. - -2. **Handling unknown resources in scheduler**: **Warn and skip.** When `sched_init_sharedInst` or `make_sched_worker` encounters a name not in `sched->resmgr`, emit a `WARN` log and continue. This is safe because the resource list always arrives in the same config push before resvals are applied (see decision 3). - -3. **Ordering guarantee**: **Confirmed — current behavior, must be preserved.** `sched_init_resources()` is called before `sched_init_sharedInst()` in `sched_init_config()` (`scheduler.c:603-609`). The design must not reorder these calls. - -4. **Removal of `collect_resval_with_idx`**: **Remove entirely.** No cross-subsystem callers will remain after this change, and intra-subsystem callers (e.g., `worker.c` reading from `sharedinst` for API get-resval) access the zarray directly by `r->idx` without going through this function. Delete `collect_resval_with_idx`, `collect_resval_with_idx_`, and remove the declaration from `src/controller/include/cluster.h`. - -5. **`free_name_val_list` utility**: **Add to `utils.c`/`utils.h`.** A new `free_name_val_list(xce_name_val_t***, int)` function mirroring `free_resval_with_idx_list` is needed. The capnp-generated `free_NameValue_list` is an option but couples cleanup to the serialization layer — a standalone utility in `utils.c` is cleaner and consistent with the existing pattern. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **resval-id-decoupling** is **~100% complete**. All 8 task groups, 22 subtasks (1.1–8.3) are `[x]` in tasks.md. -- **No remaining gaps**: `collect_resval_with_idx` and `free_resval_with_idx_list` are removed (verified by grep: zero matches). `collect_resval` (name-keyed, iterates resmgr not zarray) replaces it. The capnp schema (`Conf4Sched.resvals @6`, `Worker.resvals @5`) now use `List(NameValue)`. The scheduler resolves by name via `key_find(sched->resmgr, name)` → `r->idx`. The Go API server has `cNameValToWorkerResval` in `convert.go` for the new `xce_name_val_t**` type. -- **Task approval gap**: spec.json shows `tasks.approved: false`. All tasks are implemented but the tasks document was never formally approved. This is a process artifact — the implementation matches the approved design. -- **Cross-spec impact**: `scheduler-controller-msg-gap` extends the same scheduler handlers (`sched_init_sharedInst`, `make_sched_worker`) that were modified by this spec. The two specs are consistent — both use name-based resource resolution. -- **Recommendation**: Mark tasks as approved in spec.json, or run `/kiro-spec-status resval-id-decoupling` to sync. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -## Method - -Independent re-verification of every requirement (requirements.md R1–R6) against the current code on branch `persistent_ha`, via grep + direct read of the controller, scheduler, libxce, capnp schema, and Go API-server sources. Prior audit (2026-06-08) claimed ~100% completion; this pass confirms that claim and adds two residual findings it missed. - -## Current State Summary - -- The feature is **implemented end-to-end**. `collect_resval_with_idx` and `free_resval_with_idx_list` no longer exist anywhere in `src/` (zero grep matches). The replacement `collect_resval(var resmgr, zarray_t*, xlist_t*)` lives at `src/controller/dataex.c:23` and iterates the resmgr (names) rather than the zarray (slots), exactly as designed. -- Wire format: `src/schema/controller.capnp` — `Worker.resvals @5` (line 138) and `Conf4Sched.resvals @6` (line 239) are both `List(NameValue)`. Field ordinals preserved. -- All tasks in tasks.md are `[x]` (1.1–8.3), but `spec.json` still shows `tasks.approved: false` (phase: `tasks-generated`). Implementation landed despite the unapproved tasks document — a process artifact, not a code gap. - -## Requirement-to-Asset Map - -| Req | EARS | Asset(s) | Status | -|-----|------|----------|--------| -| R1 — name-keyed shared resvals in initial config | R1-1 | `controller.capnp:239` `List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c:120` casts to `xce_name_val_t**` | **Implemented** | -| | R1-2 | `src/scheduler/src/scheduler.c:574-599` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | **Implemented** | -| | R1-3 | `scheduler.c:587-592`: WARN + `continue` on unknown name; NULL name/entry also skipped | **Implemented** | -| | R1-4 | Same name→`r->idx` resolution as Path C handler (`scheduler.c:782-797`) | **Implemented** (by inspection; no equivalence test exists — see R6) | -| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:138` `List(NameValue)`; `dataex.c:151-166` `collect_resvals_workers(resmgr,...)`; `sched.c:442` | **Implemented** | -| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` | **Implemented** | -| | R2-3 | `schedobj.c:283-288`: WARN + skip, worker construction continues | **Implemented** | -| | R2-4 | Resolution happens at each receipt against the live `sched->resmgr`; no index is cached across messages | **Implemented** (by design; no automated test for resource add/remove mid-run) | -| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c:367-369`: same `collect_resval` + `xce_name_val_t**` path as the scheduler push | **Implemented** | -| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `resvals_to_sched`, `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210`; `models/hosts.go:94` `ResourceValues` | **Implemented** | -| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c:82-103` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:782-800` untouched, still `xce_resval_t` name-keyed | **Implemented** | -| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h:62`) and `xce_worker_t.resvals_to_sched` (`types.h:219`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` in `src/lib/utils.c:388` / `utils.h:130`; all controller free sites converted | **Implemented** | -| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep-verified); both receivers resolve by name | **Implemented** | -| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:176`) still exist with **zero users and no scoping comment** | **Partial** (SHOULD-level: dead type lingers, contrary to "removed or restricted ... with a comment") | -| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent); **no automated test** registers resources in different orders and asserts `sharedInst` contents | **Partial** (code Implemented, test Missing) | -| | R6-2 | Existing controller/scheduler suites passed at task 8.2; same-order behavior preserved | **Implemented** | -| | R6-3 | `calc_wanted_*` paths read `sharedInst`/`worker->resvals` via scheduler-side `r->idx` (scheduler testsuite exercises these at zarray level) | **Implemented** | - -### Adjacent finding (Constraint — out of spec scope, same defect class) - -`XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update, `handler.c:231-251` → `scheduler.c:801-816`) receives name-keyed `xce_resval_t` records but stores them with a **hardcoded slot**: `zarray_put(&w->resvals, 0, rv->val)` — it finds the worker by `rv->host` but never resolves `rv->name` to `r->idx`. Any per-host update for a resource whose scheduler idx ≠ 0 lands in the wrong slot (or clobbers slot 0). This path was not enumerated in this spec (Paths A/B/C only) and belongs to the `scheduler-controller-msg-gap` surface, but it violates the spirit of EARS-R5-1. Flagged for follow-up, not a regression of this feature. - -## Implementation Approach Options (for residual gaps) - -### Option A — Minimal cleanup within this spec (Recommended) -Delete the dead `xce_val_with_idx_t` typedef and capnp `ValueWithIdx` struct (or add the scoping comment EARS-R5-2 asks for); add one scheduler testsuite case that feeds `sched_init_sharedInst`/`make_sched_worker` a name-val list against a resmgr populated in a different order and asserts slot contents (closes R6-1 test gap). -- ✅ Closes both Partial items with ~2 files + 1 test; no wire change (struct ordinals of NameValue untouched) -- ❌ Touches a shared header (`types.h`) — full rebuild needed to confirm nothing else references the type - -### Option B — Fold residuals into `scheduler-controller-msg-gap` -Fix the `zarray_put(..., 0, ...)` slot bug together with the R5-2/R6-1 residuals in the adjacent spec, since it already owns the per-host update handlers. -- ✅ Single coherent change set for all per-host resval handling; this spec can be closed as-is -- ❌ R5-2/R6-1 residue stays attached to the wrong spec's history; risk of being dropped if that spec stalls - -### Option C — Accept and close -Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known accepted debt. -- ✅ Zero effort; all MUST-level acceptance criteria are demonstrably met -- ❌ Dead type invites accidental reuse; the divergent-order scenario — the entire point of the feature — remains untested by automation - -## Effort & Risk - -- **Effort: S** — residual work is a dead-type deletion plus one table-driven C test; the feature itself requires no further code. -- **Risk: Low** — all changes verified in place and grep-confirmed; the only Medium-risk item is the adjacent slot-0 bug, which is pre-existing and outside this spec. - -## Recommendations - -1. Approve tasks in `spec.json` (`tasks.approved: true`) to close the process gap — implementation matches the approved design. -2. Take Option A: remove (or comment-scope) `xce_val_with_idx_t` / `ValueWithIdx`, and add the divergent-registration-order test for `sched_init_sharedInst` / `make_sched_worker` (TDD: the test also documents EARS-R6-1 permanently). -3. File the `XCE_SCHED_OP_UPDATE_RESVALS` slot-0 defect against `scheduler-controller-msg-gap` (or a standalone fix) — it is the last index-style hazard in the resval surface. - -## Research Needed - -- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code. -- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this. -- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller. - ---- - -## Gap Re-Audit (2026-06-18) - -> POST-IMPLEMENTATION GAP AUDIT. Tasks 24/24 `[x]`, `spec.json.phase: implemented`. -> Independent re-verification of R1–R6 against current `persistent_ha` code via codegraph + grep + verbatim reads. Supersedes the 2026-06-09 validation pass; records two status changes since then. - -### Changes since the 2026-06-09 pass - -1. **Process gap CLOSED.** `spec.json` now shows `phase: implemented` and `approvals.tasks.approved: true` (was `false`/`tasks-generated`). The prior "tasks unapproved" finding no longer applies. -2. **Adjacent slot-0 defect FIXED.** `XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update) at `src/scheduler/src/scheduler.c:816-834` now resolves `rv->name` via `key_find(sched->resmgr, ...)` and stores with `zarray_put(&w->resvals, r->idx, rv->val)` (line 831) — the hardcoded `zarray_put(..., 0, ...)` flagged on 2026-06-09 is gone. The last index-style hazard in the resval surface is closed. (Belongs to `scheduler-controller-msg-gap` scope, but worth recording: the defect class this spec targets is now eliminated across ALL resval paths A/B/C and both running-update handlers.) - -### Requirement-to-Asset Map - -| Req | EARS | Asset(s) — file:line | Tag | -|-----|------|----------------------|-----| -| R1 — name-keyed shared resvals in initial config | R1-1 | `src/schema/controller.capnp:241` `resvals @6: List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c` cast to `xce_name_val_t**` | MET | -| | R1-2 | `src/scheduler/src/scheduler.c:587-595` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | MET | -| | R1-3 | `scheduler.c:583-592`: NULL nv/name skipped; unknown name → `WARN` + `continue`, `sharedInst` untouched | MET | -| | R1-4 | Same name→`r->idx` resolution as Path C (`scheduler.c:797-812`, `UPDATE_RESVALS_FULL`) | MET (by inspection; no equivalence test — see R6-1) | -| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:140` `resvals @5: List(NameValue)`; `dataex.c:374` `collect_resvals_workers(c->resmgr, l)`; `sched.c:477` `collect_resval(c->resmgr, &w->host.resvals, ...)` | MET | -| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` (line 291) | MET | -| | R2-3 | `schedobj.c:279-288`: NULL skip; unknown name → `WARN` + skip; worker construction continues | MET | -| | R2-4 | Resolution runs against live `sched->resmgr` on every receipt; no idx cached across messages (`schedobj.c:274-292`) | MET (by design; no mid-run add/remove test — RN-3) | -| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c` `apiserver_worker_change`: same `collect_resval` + `xce_name_val_t**` path as scheduler push | MET | -| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `worker.resvals_to_sched` (`unsafe.Slice`), `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210` `ResourceValues: r.resvals`; `models/hosts.go:94` `ResourceValues []ResourceDistributionInfo` | MET | -| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:797-814` untouched, still `xce_resval_t` name-keyed | MET | -| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h`) and `xce_worker_t.resvals_to_sched` (`types.h`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` (`src/lib/utils.c:388`); all 7 controller free sites converted (codegraph: 7 callers across apiserver.c, dataex.c, opget.c, reply.c, sched.c) | MET | -| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep: `collect_resval_with_idx`/`free_resval_with_idx_list` = 0 matches in src/ excl. build/); both receivers resolve by name | MET | -| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:178`) still present — zero non-generated users, **no scoping comment** | Missing (SHOULD-level) | -| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent at `scheduler.c:587` and `schedobj.c:283`); scheduler testsuite (`src/scheduler/testsuite/test.c`) has **zero** references to `sched_init_sharedInst` / `make_sched_worker` / divergent resval order | Missing (test only; code MET) | -| | R6-2 | Same-order behavior preserved; controller/scheduler suites passed at task 8.2 | MET | -| | R6-3 | `calc_wanted_*` read `sharedInst` / `worker->resvals` via scheduler-side `r->idx`; all writers (`sched_init_sharedInst`, `make_sched_worker`, both UPDATE_RESVALS handlers) now key by `r->idx` | MET | - -**Score: 21/23 EARS MET. 2 Missing — both non-MUST (R5-2 SHOULD; R6-1 test-only, code present).** All MUST-level acceptance criteria are demonstrably satisfied. - -### Residual Gaps - -- **R5-2 (SHOULD):** Dead `xce_val_with_idx_t` / `struct ValueWithIdx` linger without the "removed or restricted to intra-subsystem use, with a comment explaining its scope" that EARS-R5-2 requests. Invites accidental reuse of the very index-keyed shape this feature eliminated. -- **R6-1 (test gap):** The feature's entire raison d'être — correctness when controller and scheduler register resources in *different* orders — has no automated test. Code is order-independent by construction, but the invariant is unguarded against regression. -- **Carry-forward note:** RN-3 (EARS-R2-4 mid-run resource add/remove) still has no harness. Lower priority than R6-1. - -### Options for Open Work - -**Option A — Close residuals in-spec (Recommended).** -Delete `xce_val_with_idx_t` (types.h:38-41) + `struct ValueWithIdx` (controller.capnp:178) [or add the EARS-R5-2 scoping comment if retained for tooling], and add one table-driven scheduler testsuite case feeding `sched_init_sharedInst` / `make_sched_worker` a name-val list against a resmgr populated in reversed order, asserting `sharedInst` / `worker->resvals` slot contents. -- Effort: **S** (~2 files + 1 test). Risk: **Low**. -- Caveat: deleting from `types.h` is a shared-header edit — requires a clean `nix develop --command ninja -C src/build` to confirm no generated/out-of-tree consumer references `ValueWithIdx` (RN-2). - -**Option B — Defer to a cleanup/test sweep across HA specs.** -Bundle R5-2 + R6-1 + RN-3 with similar dead-type/test debt in adjacent specs. -- Effort: **S-M**. Risk: **Low**. Cost: residue stays detached from this spec's history; may be dropped if the sweep stalls. - -**Option C — Accept and close as-is.** -Record R5-2 and the missing R6-1 test as known accepted debt; all MUST criteria met, slot-0 hazard already fixed, spec.json fully approved. -- Effort: **none**. Risk: **Low-Medium** — dead type invites reuse; the feature's core scenario remains unverified by automation. - -### Verdict - -Feature is **functionally complete and CLOSED for all MUST requirements** (R1-1..R1-3, R2-1..R2-4, R3-1/R3-2, R4-1/R4-2, R5-1, R6-2/R6-3). Two non-blocking residuals remain (R5-2 dead-type cleanup; R6-1 divergent-order test). Recommend **Option A** to fully satisfy the SHOULD clause and lock the core invariant under test. Effort **S**, Risk **Low**. diff --git a/.kiro/specs/resval-id-decoupling/spec.json b/.kiro/specs/resval-id-decoupling/spec.json deleted file mode 100644 index c4160055..00000000 --- a/.kiro/specs/resval-id-decoupling/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "resval-id-decoupling", - "description": "Investigated the controller-to-scheduler/apiserver resource-value transport, which previously assumed idx/id parity between controller and scheduler resource arrays (collect_resval_with_idx(), src/controller/dataex.c) -- a silent-corruption risk on divergent registration order. Recommended and shipped a name-keyed replacement.", - "phase": "implemented", - "created_at": "2026-05-12T06:52:00-07:00", - "updated_at": "2026-08-16T11:35:00Z", - "language": "en", - "status": "closed", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "note": "CLOSED 2026-08-16. Recommended fix was implemented, not just proposed -- collect_resval_with_idx() replaced by name-keyed collect_resval() (commit d47b3375, 2026-05-13, landed one day after this spec was created). Verified in current code: zero remaining callers of the old idx-keyed function; controller.capnp's Worker.resvals/Conf4Sched.resvals are List(NameValue); scheduler resolves via key_find() by name (scheduler.c, schedobj.c). Related hazard bug-335 (UPDATE_RESVALS hardcoded slot 0) closed 2026-06-10. research.md's own Gap Re-Audit (2026-06-18): 21/23 EARS requirements met, functionally complete. 2 SHOULD-level residuals left as separate small cleanup, not a spec: delete the now-dead ValueWithIdx/xce_val_with_idx_t type (zero real callers); add one regression test for divergent controller/scheduler registration order (EARS-R6-1)." -} diff --git a/.kiro/specs/resval-id-decoupling/tasks.md b/.kiro/specs/resval-id-decoupling/tasks.md deleted file mode 100644 index dc55c90f..00000000 --- a/.kiro/specs/resval-id-decoupling/tasks.md +++ /dev/null @@ -1,224 +0,0 @@ -# Implementation Tasks: resval-id-decoupling - -> Generated: 2026-05-12T07:45:00-07:00 -> Feature: resval-id-decoupling -> Status: Tasks Generated - -## Task Overview - -7 major tasks, 19 sub-tasks. All 6 requirements covered. - -Dependencies: Task 1 must complete first (headers + schema unblock everything). -Tasks 2 and 3 can proceed in parallel after Task 1. -Tasks 4, 5, 6 can proceed in parallel after Tasks 2 and 3. -Task 7 last. - ---- - -## Tasks - -### 1. Update shared headers and capnp schema - -Change the wire type and C struct types that carry resource values across subsystem -boundaries. This is the structural foundation — all downstream tasks depend on these -types compiling correctly. - -- [x] 1.1 In `src/schema/controller.capnp`, change `Worker.resvals @5` and - `Conf4Sched.resvals @6` from `List(ValueWithIdx)` to `List(NameValue)`. Preserve - field ordinal numbers; only the element type changes. - _Requirements: 1, 2, 3, 4_ - -- [x] 1.2 In `src/include/xce/types.h`, change `xce_worker_t.resvals_to_sched` from - `xce_val_with_idx_t**` to `xce_name_val_t**`. - _Requirements: 2, 3_ - -- [x] 1.3 In `src/include/typemap.h`, change `xce_conf4sched_t.resvals` from - `xce_val_with_idx_t**` to `xce_name_val_t**`. - _Requirements: 1, 4_ - -- [x] 1.4 Run `nix develop --command ninja -C src/build` to regenerate - `controller.capnp.[ch]`. Confirm the generated `encode_Conf4Sched` and - `decode_Conf4Sched` now call `encode_NameValue_list` / `decode_NameValue_list` - instead of their `ValueWithIdx` counterparts. Expect compiler errors at this stage - from old call sites — that is expected and resolved in subsequent tasks. - ---- - -### 2. (P) Replace `free_resval_with_idx_list` with `free_name_val_list` - -Swap the heap-cleanup utility function to match the new element type. Old callers in -the controller will be updated in Task 4. - -- [x] 2.1 In `src/lib/utils.c`, add `free_name_val_list(xce_name_val_t*** vals, int count)`: - iterate the array, free each entry's `name`, free `val->u.svalue` if string-typed, - free `val`, free the entry, then null and free the array pointer. Mirror the structure - of the existing `free_resval_with_idx_list`. - -- [x] 2.2 In `src/include/utils.h`, remove the `free_resval_with_idx_list` declaration - and add `extern void free_name_val_list(xce_name_val_t***, int);`. - -- [x] 2.3 In `src/lib/utils.c`, delete the `free_resval_with_idx_list` implementation. - _Requirements: 5_ - ---- - -### 3. (P) Add `collect_resval` and remove `collect_resval_with_idx` - -Replace the index-keyed collector in the controller with a name-keyed one. The key -design inversion: iterate `resmgr` (names) rather than the `zarray_t` (slots), using -`r->idx` to look up values. This guarantees every emitted record carries a valid name. - -- [x] 3.1 In `src/controller/dataex.c`, add `collect_resval(var resmgr, zarray_t* array, xlist_t* l)`: - acquire a read lock on `resmgr`, iterate with `iter_make`/`iter_next`/`iter_destroy`, - for each `xce_resource_t* r` call `zarray_get(array, r->idx)` to get the value (skip - if NULL), allocate an `xce_name_val_t`, copy name (strdup) and value (strdup svalue or - memcpy), store in the list. Release the resmgr lock. Return `l->count`. The array lock - is the caller's responsibility (consistent with old pattern). - _Requirements: 1, 2, 5_ - -- [x] 3.2 In `src/controller/dataex.c`, rewrite `collect_shared_resvals` to lock - `c->sharedinst.lock`, call `collect_resval(c->resmgr, &c->sharedinst, l)`, then - unlock. The unused `fn` parameter is retained for signature consistency. - _Requirements: 1_ - -- [x] 3.3 In `src/controller/dataex.c`, update `collect_resvals_workers` to accept - `var resmgr` as a new first parameter. Replace the `collect_resval_with_idx` call with - `mylock_lock(&w->resvals.lock)` / `collect_resval(resmgr, &w->resvals, &lst)` / - `mylock_unlock`. Update the call site in `collect_workers` to pass `c->resmgr`. - _Requirements: 2_ - -- [x] 3.4 In `src/controller/dataex.c`, delete both `collect_resval_with_idx_` (static) - and `collect_resval_with_idx`. In `src/controller/include/cluster.h`, remove the - `collect_resval_with_idx` declaration. - _Requirements: 5_ - ---- - -### 4. (P) Update controller call sites - -Update all controller sites that populate or clean up `resvals_to_sched` / -`conf4sched->resvals` to use the new type and free function. Tasks 2 and 3 must be -complete before this task. - -- [x] 4.1 In `src/controller/sched.c` — `xcesched_push_configs`: change the cast from - `(xce_val_with_idx_t**)` to `(xce_name_val_t**)` when assigning `xreslist.data` to - `conf4sched->resvals`, and replace `free_resval_with_idx_list` with - `free_name_val_list`. - _Requirements: 1_ - -- [x] 4.2 In `src/controller/sched.c` — `xcesched_push_workerlist` and its worker-loop - cleanup: add `mylock_lock`/`mylock_unlock` around `collect_resval`, change cast to - `(xce_name_val_t**)`, replace all `free_resval_with_idx_list` calls with - `free_name_val_list`. - _Requirements: 2_ - -- [x] 4.3 In `src/controller/sched.c:226`, `src/controller/reply.c:66`, and - `src/controller/opget.c:128`, replace each `free_resval_with_idx_list` call with - `free_name_val_list`. - _Requirements: 5_ - -- [x] 4.4 In `src/controller/apiserver.c` — `apiserver_worker_change`: add - `mylock_lock`/`mylock_unlock` around `collect_resval`, change cast to - `(xce_name_val_t**)`, replace `free_resval_with_idx_list` with `free_name_val_list` - at both cleanup sites (lines 51 and 397–399). - _Requirements: 3_ - ---- - -### 5. (P) Update scheduler — shared resource value initialisation - -Change `sched_init_sharedInst` to resolve resource values by name via the scheduler's -own resmgr instead of using the raw index received from the controller. Task 1 must be -complete before this task. - -- [x] 5.1 In `src/scheduler/src/scheduler.c` — `sched_init_sharedInst`: replace the - `idx`-direct loop with a name-lookup loop. For each `xce_name_val_t* nv` in - `sched->config->resvals`: skip if `nv` or `nv->name` is NULL; call - `key_find(sched->resmgr, nv->name, &r)`; if not found emit `WARN` and continue; then - `MOVE_PTR(val, nv->val)` and `zarray_put(&sched->sharedInst, r->idx, val)`. Do not - change the call order in `sched_init_config` — `sched_init_resources` must remain - before `sched_init_sharedInst`. - _Requirements: 1, 6_ - ---- - -### 6. (P) Update scheduler — per-worker resource value initialisation - -Change `make_sched_worker` to resolve worker resource values by name. Task 1 must be -complete before this task. - -- [x] 6.1 In `src/scheduler/src/schedobj.c` — `make_sched_worker`: replace the - `idx`-direct loop with a name-lookup loop. For each `xce_name_val_t* nv` in - `w->resvals_to_sched`: skip if `nv` or `nv->name` is NULL; call - `key_find(sched->resmgr, nv->name, &r)`; if not found emit `WARN` and continue; then - `MOVE_PTR(val, nv->val)` and `zarray_put(&worker->resvals, r->idx, val)`. The - existing `sched` pointer (cast from `void* arg`) is already available at the top of - the function. - _Requirements: 2, 6_ - ---- - -### 7. Update Go API server to consume worker resource values - -Update the Go API server frontend to read `resvals_to_sched` (now `xce_name_val_t**`) -from received worker data and expose resource values in the `HostInfo` model. Previously -the Go layer ignored this field entirely. - -- [x] 7.1 In `src/frontend/apiserver/models/hosts.go`, add - `ResourceValues []models.ResourceDistributionInfo` (or - `ResourceValues []ResourceDistributionInfo` if within the same package) to `HostInfo`. - _Requirements: 3_ - -- [x] 7.2 In `src/frontend/apiserver/backend/convert_pure.go`, add a `resvals` field - (`[]models.ResourceDistributionInfo`) to `rawWorkerData`, and update `rawWorkerToModel` - to assign `info.ResourceValues = r.resvals`. - _Requirements: 3_ - -- [x] 7.3 In `src/frontend/apiserver/backend/convert.go`, add - `cNameValToWorkerResval(workerName string, nv *C.xce_name_val_t) *models.ResourceDistributionInfo`: - return nil if `nv` or `nv.val` is nil (with a WARN log); set `ResourceName` from - `nv.name`, `Location` from `workerName`, `IsDynamic: true`; decode the value type - using the same `xce_val_which` switch as `cResvalToDistribution`. - _Requirements: 3_ - -- [x] 7.4 In `src/frontend/apiserver/backend/convert.go`, update `cWorkerToModel` to - iterate `worker.nresvals_to_sched` entries in `worker.resvals_to_sched`, call - `cNameValToWorkerResval` for each, collect non-nil results, and store them in - `rawWorkerData.resvals`. - _Requirements: 3_ - ---- - -### 8. Build, verify, and clean up - -Confirm the full system compiles and passes tests with no regressions. All prior tasks -must be complete. - -- [x] 8.1 Run a full clean build: `nix develop --command bash -c "cd src && ninja -C build"`. - Resolve any remaining compiler errors (expected: none if tasks 1–7 complete). Confirm - `controller.capnp.[ch]` uses `NameValue` codecs for both `Conf4Sched.resvals` and - `Worker.resvals`. - -- [x] 8.2 Run the scheduler and controller test suites: - `nix develop --command bash -c "cd src/frontend && make test"` and any C-level tests - under `src/lib/testsuite/`. Confirm all tests pass. Note: `xcecli/common/output` - failure (`undefined: parseJobTime`) is pre-existing and unrelated to this feature. - -- [x] 8.3 Verify no remaining references to `collect_resval_with_idx` or - `free_resval_with_idx_list` in `src/` (excluding comments or git history): - `grep -r "collect_resval_with_idx\|free_resval_with_idx_list" src/`. Expected: zero - matches. - _Requirements: 5_ - ---- - -## Requirements Coverage - -| Requirement | Tasks | -|------------|-------| -| 1 | 1.1, 1.3, 3.2, 4.1, 5.1 | -| 2 | 1.1, 1.2, 3.3, 4.2, 4.3, 6.1 | -| 3 | 1.1, 1.2, 4.4 | -| 4 | 1.1, 1.3 | -| 5 | 2.1, 2.2, 2.3, 3.4, 4.3, 7.3 | -| 6 | 5.1, 6.1 | diff --git a/.kiro/specs/scheduler-allocation/brief.md b/.kiro/specs/scheduler-allocation/brief.md deleted file mode 100644 index 08b01d5d..00000000 --- a/.kiro/specs/scheduler-allocation/brief.md +++ /dev/null @@ -1,210 +0,0 @@ -# Brief: scheduler-allocation - -## Problem - -The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: `sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: `parjob_allocator.c`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` -- any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()`) produces no allocation at all. Third, allocation is eager and one-shot: the sibling `seqjob_allocator.c`'s `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails. - -The first gap (hardcoded picker) is addressed in the sibling **scheduler-core** spec. This spec, **scheduler-allocation**, closes the other two: the parjob stub and one-shot allocation. - -## Current State - -### What exists (allocation surface) - -- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources, collects workers via `sched_collect_workers()`, tries one host, returns first match. No retry with alternative plans. -- **Parjob allocator stub** (`parjob_allocator.c`): registered as a plugin but `my_alloc()` is empty (`return 0`). Any job with `maxTasks > 1` silently fails allocation. -- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds). The type hierarchy is well-defined but allocation at the PRIMARY leaf lacks iteration abstractions. -- **Select expression evaluator** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC. Already reusable for host filtering. -- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session. -- **Allocation dispatch** (`scheduler.c:sched_make_allocation()`, `sched_make_primary_alloc_()`): routes to seqjob or parjob allocator based on `maxTasks`. - -### What's missing - -- **Lazy plan generation**: no `xce_alloc_plan_iter_t` -- plans are not generated at all, let alone lazily -- **Lazy host iteration**: no `xce_host_iter_t` -- host collection is eager via `sched_collect_workers()`, materializing the full list before trying any -- **Parallel job allocation**: `parjob_allocator` is a stub. No combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1), no multi-slot host matching -- **Unsatisfiable resreq caching**: a batch of similarly-shaped unsatisfiable jobs each re-pays the full allocation cost within a session - -## Desired Outcome - -### Iterator-Driven Allocation Architecture - -A lazy `xce_alloc_plan_iter_t` (candidate `(nhosts, slots_per_host)` plans, most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session. - -### Three-Layer Iterator Composition - -``` -ALTER -> OR-of-compounds: first compound that fully allocates wins (unchanged) - └─ COMPOUND -> AND-of-primaries: ALL primaries must allocate (unchanged) - └─ PRIMARY -> plan_iter -> host_iter -> try_alloc (NEW iterator layer) -``` - -The iterator composition operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today. - -### Iterator Pipeline - -``` -xce_alloc_plan_iter_t -- init(resreq) -> next() -> plan -> next() -> ... -> NULL -> fini() - │ plan - ▼ -xce_host_iter_t -- init(plan, workers) -> next() -> worker -> ... -> NULL -> fini() - │ worker - ▼ -try_alloc(worker, plan) -- check slots, shared res, commit or advance -``` - -### Key Properties - -- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization. -- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points. -- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. -- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf are unchanged from today. -- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host x 1 slot). This validates the abstraction before applying it to the complex parjob case. - -## Alternatives Considered - -### Approach A: Incremental Enhancement (rejected) - -| Aspect | Detail | -|--------|--------| -| **Plan generation** | Pre-computed plan array. Eager generation for parjob. | -| **Host iteration** | No new abstraction. Simple for-loop over `sched_collect_workers()` result. | -| **Unsatisfiable resreq** | `xlist_t unsatisfiable_resreqs` in session state | - -**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost. - -### Approach B: Iterator-Driven Allocation (chosen) - -| Aspect | Detail | -|--------|--------| -| **Plan iterator** | `xce_alloc_plan_iter_t` — lazy, cancel-safe, generates one plan per `next()` call | -| **Host iterator** | `xce_host_iter_t` — lazy per-plan host filtering via existing select-expression evaluator | -| **Parallel jobs** | Combinatorial plan generation: 1host x N -> 2hosts x ceil(N/2) -> ... -> Nhhosts x 1 | -| **Unsatisfiable resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` | -| **Seqjob path** | Trivial rewrap — single plan (1x1) through the same iterator interfaces | - -**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the resreq type hierarchy naturally. Seqjob rewrap validates the abstraction on the simple case before applying it to the complex parjob case. The cache eliminates redundant work for batches of unsatisfiable jobs. - -### Decision Rationale - -Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The two-iterator composition (plan -> host -> try_alloc) is a proven pattern: the plan iterator answers "how many hosts, how many slots each?", the host iterator answers "which specific hosts can fulfill this plan?", and the allocator commits or advances. This separation of concerns makes each layer independently testable. - -## Approach - -**Iterator-Driven Allocation**: - -### PRIMARY leaf allocation pseudocode - -```c -int allocate_primary(sched, primary_resreq, ctx) { - plan_iter = plan_iter_init(primary_resreq, sched); - while ((plan = plan_iter->next(plan_iter)) != NULL) { - host_iter = host_iter_init(plan, sched->workers); - while ((worker = host_iter->next(host_iter)) != NULL) { - if (try_alloc_on_host(worker, plan, ctx) > 0) { - host_iter->fini(host_iter); - plan_iter->fini(plan_iter); - return ALLOC_SUCCESS; - } - } - host_iter->fini(host_iter); - } - plan_iter->fini(plan_iter); - return ALLOC_FAIL; -} -``` - -### Who uses this composition - -| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior | -|-----------|------------|---------------|---------------|----------| -| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host x 1 slot) | Full: filters hosts by select expression | Unchanged from today — first matching host wins | -| `parjob_allocator` | `> 1` | Full combinatorial: 1hostxN, 2hosts x ceil(N/2), ..., Nhosts x 1 | Full: filters hosts with enough available slots | New — walks plans until one succeeds | -| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` | - -### Unsatisfiable resreq cache - -```c -// Per-session cache on xce_scheduler_t -// Key: fingerprint of {select, rusage, user, queue, project} -// Cleared at begin_session() -// Populated when a job's plan iterator exhausts without success -// Checked before starting allocation for each job — hit = immediate pend -``` - -### What stays unchanged - -- `seqjob_allocator` externally observable behavior — rewrap is internal -- COMPOUND and ALTER resreq allocation paths — dispatch is unchanged -- Select expression system — no changes -- Shared resource tracking — no changes -- Existing plugin framework — no changes - -### What's new - -- `xce_alloc_plan_iter_t` — lazy plan iterator interface -- `xce_host_iter_t` — lazy host iterator interface -- `parjob_allocator` combinatorial plan generation -- Session-level unsatisfiable resreq fingerprint cache -- Seqjob rewrap onto iterator interfaces (trivial case, behavior unchanged) - -## Scope - -### In - -- `xce_alloc_plan_iter_t` interface: `init(resreq, sched)`, `next() -> plan`, `fini()` -- `xce_host_iter_t` interface: `init(plan, workers)`, `next() -> worker`, `fini()` -- `parjob_allocator` combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) via plan iterator -- `seqjob_allocator` rewrap onto iterator interfaces (trivial 1-host/1-slot case, behavior unchanged) -- Session-level unsatisfiable resreq fingerprint cache (keyed on `{select, rusage, user, queue, project}`) -- COMPOUND and ALTER resreq allocation paths unchanged — they compose over the PRIMARY leaf allocator -- Unit tests: plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence -- Integration test: end-to-end multi-slot parallel job allocation - -### Out - -- Pluggable job picker interface (`xce_job_picker_iface_t`) — in **scheduler-core** -- `builtin_fcfs` picker — in **scheduler-core** -- Configurable picker plugin with per-queue override — in **scheduler-core** -- Pickers in general (Req 1, 5, 8 of scheduler-framework) — in **scheduler-core** -- Preemption (Req 6 of scheduler-framework) — in **scheduler-preemption** -- Core/driver separation (Req 10 of scheduler-framework) — in **scheduler-core** -- Queue vs Project section — in **scheduler-core** -- New job selection policies beyond what the picker interface enables — deferred -- Resreq template registry (named/shared resreq objects) — deferred -- Select expression language or evaluator changes -- GPU/topology/affinity-aware allocation — left pluggable via future allocator plugins -- Controller-side scheduling-state replication / commit-barrier wiring - -## Upstream / Downstream - -### Upstream (dependencies) - -- **scheduler-core** — the plan and host iterators consume the core event interface (`xce_core_event_t`) for worker/resource state updates, but are independently unit-testable with constructed state -- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed -- Existing select expression evaluator (`resreq.h`) — NO changes needed -- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed -- Existing resreq type hierarchy (PRIMARY/COMPOUND/ALTER) — NO changes needed - -### Downstream (future consumers) - -- **Affinity/GPU allocators** — register as allocator plugins, implement custom plan iterators for topology-aware placement, reuse `xce_host_iter_t` -- **Bin-packing allocator** — custom plan iterator strategy -- **Resreq template registry** — the fingerprint cache's key can incorporate template identity when sharing is added - -## Existing Spec Touchpoints - -### Depends on -- **scheduler-core** — the iterators operate within the core's scheduling session, consuming worker and resource state pushed through the core event interface. The iterators do NOT depend on the core's driver or I/O layer — they are purely computational, operating on in-memory state. - -### Adjacent -- **scheduler-preemption** — preemption fires after allocation failure. The unsatisfiable resreq cache is populated after both allocation and preemption fail (or preemption is inapplicable). The iterator interfaces are not preemption-aware. -- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume. - -## Constraints - -- **C (gnu17)** — no C++, no external libraries beyond existing (pthreads) -- **Cancel-safe** — `plan_iter->next()` and `host_iter->next()` must have `pthread_cancel` cancellation points -- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session` -- **Backward compatible** — existing seqjob allocation behavior unchanged. COMPOUND/ALTER dispatch unchanged. New functionality is additive. -- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M x N candidate evaluations (acceptably bounded). -- **No controller dependency** — iterators are purely computational, operating on in-memory worker/resource state. They do not perform I/O or call into the controller. diff --git a/.kiro/specs/scheduler-allocation/requirements.md b/.kiro/specs/scheduler-allocation/requirements.md deleted file mode 100644 index b60505ae..00000000 --- a/.kiro/specs/scheduler-allocation/requirements.md +++ /dev/null @@ -1,619 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. The first gap -- job selection policy is hardcoded (`sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal) -- is addressed in the sibling **scheduler-core** spec. - -This spec closes the remaining two allocation gaps: - -1. **Parallel job allocation is a stub**: `parjob_allocator.c`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` -- any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()` at `scheduler.c:93-103`) produces no allocation at all. - -2. **Allocation is eager and one-shot**: the sibling `seqjob_allocator.c`'s `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails. - -This spec introduces an **iterator-driven allocation architecture** with these components: - -- A lazy `xce_alloc_plan_iter_t` -- candidate `(nhosts, slots_per_host)` plans, most-preferred first -- A lazy `xce_host_iter_t` -- candidate hosts for a given plan, filtered by the existing select-expression evaluator -- A `parjob_allocator` that walks the plan iterator to do real combinatorial multi-slot allocation -- A session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session -- A seqjob_allocator rewrap onto the iterator interfaces as a trivial 1-plan/1-slot case, validating the abstraction before applying it to the complex parjob case - -> Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in `.kiro/specs/scheduler-allocation/brief.md`. - -## Boundary Context - -- **In scope**: `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) for `maxTasks > 1`; `xce_alloc_mode_t` on `xce_sched_ctx_t` for allocation mode (NORMAL/PREEMPT); session-level unsatisfiable-resreq fingerprint cache keyed by `xce_cache_key_t` (O(1) content hash from parsed resreq + identity); unit tests for plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence; integration test for end-to-end multi-slot parallel job allocation. - -- **Out of scope**: Pluggable job picker interface (`xce_job_picker_iface_t`) -- in **scheduler-core**; `builtin_fcfs` picker -- in **scheduler-core**; configurable picker plugin with per-queue override -- in **scheduler-core**; preemption (Req 6 of scheduler-framework) -- in **scheduler-preemption**; core/driver separation (Req 10 of scheduler-framework) -- in **scheduler-core**; Queue vs Project roles -- in **scheduler-core**; new job selection policies beyond the builtin FCFS picker; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins). - -- **Adjacent expectations**: depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking (`xce_scheduler_t.sharedInst`/`sharedInUse`) -- none of these change. Depends on scheduler-core for the core event interface that pushes worker and resource state; the iterators are independently unit-testable with constructed state. `ha-consistency-invariants`' commit barrier is a future consumer of scheduling decisions; this spec does not wire it. - -## Allocation Architecture (Reqs 2-4) - -Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER): - -### Resreq Type Hierarchy - -``` -ALTER -> OR-of-compounds: first compound that fully allocates wins - └─ COMPOUND -> AND-of-primaries: ALL primaries must allocate for success - └─ PRIMARY -> plan_iter -> host_iter -> try_alloc (leaf level) -``` - -The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators -- they call into the same primary allocator, unchanged from today's `sched_make_compound_allocation()` and `sched_make_alter_allocation()` dispatch. - -### Iterator Composition (PRIMARY leaf) - -``` - ┌──────────────────────────┐ - │ xce_alloc_plan_iter_t │ Req 2 - │ │ - │ init(primary_resreq, sched) │ - │ next() -> plan │ generates (nhosts, slots_per_host) - │ fini() │ candidates, most-preferred first - └────────────┬───────────────┘ - │ plan - ▼ - ┌──────────────────────────┐ - │ xce_host_iter_t │ Req 3 - │ │ - │ init(plan, workers) │ for a given plan, iterate - │ next() -> worker │ candidate hosts matching - │ fini() │ select expression + availability - └────────────┬───────────────┘ - │ worker - ▼ - ┌──────────────────────────┐ - │ try_alloc(worker, plan) │ Req 4 (parjob) or - │ │ existing seqjob logic - │ check slots, shared res, │ - │ commit or advance │ - └────────────────────────────┘ -``` - -### Composition Algorithm (All Three Types) - -```c -// === PRIMARY leaf: iterator composition === -// Returns >0 on success, 0 on failure -int allocate_primary(sched, primary_resreq, ctx) { - plan_iter = plan_iter_init(primary_resreq, sched); - while ((plan = plan_iter->next(plan_iter)) != NULL) { - host_iter = host_iter_init(plan, sched->workers); - while ((worker = host_iter->next(host_iter)) != NULL) { - if (try_alloc_on_host(worker, plan, ctx) > 0) { - host_iter->fini(host_iter); - plan_iter->fini(plan_iter); - return ALLOC_SUCCESS; - } - } - host_iter->fini(host_iter); - } - plan_iter->fini(plan_iter); - return ALLOC_FAIL; -} - -// === COMPOUND: AND of primaries === -// ALL primaries must allocate. On any failure, roll back prior allocations -// and return failure. Same logic as today's sched_make_compound_allocation(). -int allocate_compound(sched, compound_resreq, ctx) { - for (int i = 0; i < compound_resreq->primaries.count; i++) { - primary = compound_resreq->primaries[i]; - if (allocate_primary(sched, primary, ctx) <= 0) { - // Roll back all primaries allocated so far in this compound - rollback_compound_allocations(ctx, i); - return ALLOC_FAIL; - } - } - return ALLOC_SUCCESS; -} - -// === ALTER: OR of compounds === -// First compound that fully allocates wins. Same logic as today's -// sched_make_alter_allocation(). On failure, no rollback needed -// (prior compounds were never committed -- they failed their own checks). -int allocate_alter(sched, alter_resreq, ctx) { - for (int i = 0; i < alter_resreq->compounds.count; i++) { - compound = alter_resreq->compounds[i]; - if (allocate_compound(sched, compound, ctx) > 0) { - return ALLOC_SUCCESS; - } - // This compound failed -- advance to next alternative - } - return ALLOC_FAIL; -} - -// === Top-level dispatch === -// Unchanged from today's sched_make_allocation() dispatch on obj->tag. -int allocate_job(sched, job, ctx) { - switch (job->resreq_type) { - case PRIMARY: - return allocate_primary(sched, job->primary_resreq, ctx); - case COMPOUND: - return allocate_compound(sched, job->compound_resreq, ctx); - case ALTER: - return allocate_alter(sched, job->alter_resreq, ctx); - } -} -``` - -**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back -- the compound is atomic. Today this is handled by `sched_make_compound_allocation()` tracking allocations in `ctx` and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract. - -### Who Uses This Composition - -| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior | -|-----------|------------|---------------|---------------|----------| -| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host x 1 slot) | Full: filters hosts by select expression | Unchanged from today -- first matching host wins | -| `parjob_allocator` | `> 1` | Full combinatorial: 1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1 | Full: filters hosts with enough available slots | New -- walks plans until one succeeds | -| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` | - -### Key Design Properties - -- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization. -- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points. -- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. `fini()` on one does not affect the other. -- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside `allocate_primary()`. -- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case -- its plan iterator always returns the single plan (1 host x 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case. -- **Pluggable plan generation, shared host iteration**: plan generation is allocator-specific — each `job_alloc_fn` brings its own strategy (trivial, combinatorial, topology-aware, bin-packing) and creates its own `xce_alloc_plan_iter_t` internally. Host iteration is shared infrastructure — `xce_host_iter_t` has one implementation reused by all allocators, parameterized by sort order (`HOST_ORDER_ANY`, `HOST_ORDER_MOST_FREE`, etc.). Swapping allocation strategy = swapping the plugin; no core loop change. - -## Requirements - -### Requirement 2: Lazy allocation plan iterator - -**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front. - -#### Acceptance Criteria - -1. The Scheduler shall define an `xce_alloc_plan_iter_t` interface with `init(resreq, sched)`, `next() -> plan`, and `fini()` operations. The `init()` signature passes the full `xce_primary_resreq_t` (including `span`, `same`, `affinity`) and scheduler state — sufficient for any allocator to derive its plan space without future interface changes. - - **Plan-shaping fields** (on `xce_primary_resreq_t`, consumed by plan iterator internally): - - `span`: bounds host count (`nhosts_min`/`nhosts_max`) and per-host task cap (`max_per_host`, analogous to LSF `ptile`). v1 combinatorial generation ignores these (empty structs); fill when ptile/nhosts support is needed. - - `same`: co-location constraints ("all tasks on same rack"). v1 ignores; future. - - `affinity`: restrict to named host group. v1 ignores; future. - - `minTasks`/`maxTasks`: task count bounds — the primary driver of plan space. Used in v1. - - `select`: host filter expression — consumed by host iterator (Req 3), not plan iterator. - - `rusages`: resource requests — consumed by `try_alloc_on_host`, not plan iterator. - - v1 implements simple combinatorial generation: `1host×N, 2hosts×ceil(N/2), ..., Nhosts×1`. Future allocators read `span`/`same`/`affinity` to constrain the plan space without touching the `init()` signature. - -2. Each call to `next()` shall generate exactly one candidate `(nhosts, slots_per_host, host_filter)` plan without generating or materializing any other plan in the same call. - -3. The Scheduler shall order plans returned by successive `next()` calls from most-preferred to least-preferred, per the owning allocator's preference policy. - -4. `next()` shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion. - -5. Calling `fini()` shall release all resources held by the iterator, and the interface shall define no other required cleanup step. - -### Requirement 3: Lazy host candidate iterator - -**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering. - -#### Acceptance Criteria - -1. The Scheduler shall define an `xce_host_iter_t` interface with `init(plan, workers)`, `next() -> worker`, and `fini()` operations. - -2. Each call to `next()` shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator. - -3. `next()` shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call `next()` again after receiving NULL for that plan. - -4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()`. - -5. Calling `fini()` shall release all resources held by the host iterator, independent of the plan iterator's lifecycle. - -### Requirement 4: Parallel job combinatorial allocation - -**Objective:** As a cluster operator submitting a job that needs multiple task slots, I want `parjob_allocator` to actually allocate across hosts, so that jobs with `maxTasks > 1` are no longer silently unallocatable. - -#### Acceptance Criteria - -1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host x N slots, 2 hosts x ceil(N/2) slots, ..., N hosts x 1 slot, using Requirement 2's plan iterator. - -2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan. - -3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job. - -4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected. - -5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents -- the current stub behavior -- for any job with `maxTasks > 1`. - -### Requirement 7: Session-level unsatisfiable resreq cache - -**Objective:** As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost. - -#### Acceptance Criteria - -1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by `xce_cache_key_t` — a value type composing `xce_resreq_fingerprint_t` (pre-computed `{minTasks, maxTasks, select_node_id, rusage_hash}`) and identity (`{user, queue, project}` as xstring pointers). The resreq fingerprint is stored on the scheduling object (`xce_primary_schedobj_t`) and computed once on job ingest. The full cache key is constructed on the stack at lookup time — a single `kvtab_lookup()` with `cache_key_hash()` and `cache_key_eq()`. Fully O(1): no list walks, no string traversals, no allocations. This captures everything the allocator and limit plugin consume. - -2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`). - -3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. In v1, cache population is preemption-agnostic: it fires on allocation failure alone. When the scheduler-preemption spec ships, preemption fires between allocation failure and cache population — if preemption succeeds, no cache entry is created; if preemption fails or is inapplicable, the cache entry is populated as normal. The allocation layer does not call into preemption; the caller (scheduling loop) orchestrates the sequence. - -4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption. - -5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped. - -### Requirement 9: No regression to existing allocators or HA soak - -**Objective:** As a maintainer, I want the iterator-driven rework scoped so that existing single-slot allocation and COMPOUND/ALTER dispatch are unaffected, so that this framework change is safe to ship independently of any new allocator. - -#### Acceptance Criteria - -1. **Seqjob regression**: `seqjob_allocator`'s externally observable allocation behavior (which host a `maxTasks == 1` job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the `xce_host_iter_t` interface. - -2. **COMPOUND/ALTER unchanged**: The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today. - -3. **Unit tests**: Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the eager-one-shot behavior or stub return value this feature replaces. - -4. **HA soak**: A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. - -## Data Structure Changes Summary - -### `xce_alloc_plan_iter_t` (new, Req 2) - -```c -typedef struct xce_alloc_plan_iter_s xce_alloc_plan_iter_t; - -struct xce_alloc_plan_iter_s { - // Initialize the iterator for a given primary resreq and scheduler state. - // Returns 0 on success, nonzero on error. - int (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq, - xce_scheduler_t* sched); - - // Return the next candidate plan, or NULL when exhausted. - // The returned plan pointer is valid until the next next() or fini() call. - xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter); - - // Release all resources held by the iterator. - void (*fini)(xce_alloc_plan_iter_t* iter); - - // Private state: current position, strategy-specific data - void* private; -}; - -// A plan describes how many hosts and slots to attempt -typedef struct { - int nhosts; // number of hosts needed - int slots_per_host; // slots required on each host - // host_filter: derived from primary_resreq->select, plus slot availability -} xce_alloc_plan_t; -``` - -### `xce_host_iter_t` (new, Req 3) - -```c -typedef struct xce_host_iter_s xce_host_iter_t; - -struct xce_host_iter_s { - // Initialize the iterator for a given plan and worker set. - // Returns 0 on success, nonzero on error. - int (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan, - var workers); - - // Return the next candidate worker matching the plan's host filter, - // or NULL when exhausted. The returned worker pointer is valid until - // the next next() or fini() call. - xce_worker_t* (*next)(xce_host_iter_t* iter); - - // Release all resources held by the iterator. - void (*fini)(xce_host_iter_t* iter); - - // Private state: current position, filtering state - void* private; -}; -``` - -### New types (Req 7) - -```c -// Resreq portion of the cache key — pre-computed, shareable across jobs -// with identical resreqs. Fits in a single cache line (24 bytes). -typedef struct { - int min_tasks; - int max_tasks; - uint32_t select_node_id; // CSE-interned root; 0 if select is NULL - uint64_t rusage_hash; // computed at parse time, stable for life of resreq -} xce_resreq_fingerprint_t; - -// Full cache key — resreq fingerprint + identity. -// Identity fields are xstring pointers (not owned, not freed). -// Pointer equality = content equality for xstring-interned strings. -typedef struct { - xce_resreq_fingerprint_t resreq; - const char* user; // xstring pointer - const char* queue; // xstring pointer - const char* project; // xstring pointer -} xce_cache_key_t; - -// Hash — single call site, no ad-hoc field assembly -static inline uint64_t cache_key_hash(const xce_cache_key_t* k) { - uint64_t h = FNV64_INIT; - hash_update(&h, &k->resreq, sizeof(k->resreq)); - hash_update_ptr(&h, k->user); - hash_update_ptr(&h, k->queue); - hash_update_ptr(&h, k->project); - return h; -} - -// Equality — memcmp on fixed-size resreq portion + pointer equality on identity -static inline bool cache_key_eq(const xce_cache_key_t* a, const xce_cache_key_t* b) { - return memcmp(&a->resreq, &b->resreq, sizeof(a->resreq)) == 0 - && a->user == b->user - && a->queue == b->queue - && a->project == b->project; -} -``` - -### `xce_primary_resreq_t` addition (Req 7) - -One new field, set once at parse time: - -```c -// In xce_primary_resreq_t (resreq.h): -uint64_t rusage_hash; // set by parse_resreq_primary(), stable for life of resreq -``` - -### `xce_sched_obj_t` / `xce_primary_schedobj_t` addition (Req 7) - -Store the resreq portion of the fingerprint on the scheduling object so it is computed once and reused for every cache lookup of that job: - -```c -// On xce_primary_schedobj_t (or xce_sched_obj_t): -xce_resreq_fingerprint_t resreq_fp; // populated on job ingest, read by cache lookup -``` - -### Lookup site (pseudocode) - -```c -// Before allocation: construct key on stack, single hash + lookup -xce_cache_key_t key = { - .resreq = obj->u.primary->resreq_fp, - .user = obj->user, - .queue = obj->queue, - .project = obj->project, -}; -if (kvtab_lookup(sched->unsatisfiable_cache, &key, cache_key_hash, cache_key_eq)) { - pend_job(obj, original_reason); - return; -} -``` - -### Key design decisions - -- **struct-based**: hash and equality live next to the type. Single call site, no scattered field assembly. -- **`memcmp` on resreq portion**: all fields are fixed-size integers/IDs — no pointers, no deep compare. -- **pointer equality on identity**: xstring guarantees identical content → identical pointer. -- **select via `node_id`**: CSE intern table deduplicates identical select expressions. O(1). -- **rusages via pre-computed hash**: set once at parse time, stable for life of resreq. O(1). -- **span/same/affinity deferred**: currently empty structs; add to `xce_resreq_fingerprint_t` when they gain fields. - -### `xce_sched_ctx_t` addition (allocation mode) - -A mode field that controls which resource pool the allocator draws from. Defaults to `XCE_ALLOC_NORMAL`. Consumed by `try_alloc_on_host()` and any resource-availability check — applies to all resource types (slots, shared resources, limit instances, licenses, etc.), not just CPU slots. - -```c -// In xce_sched_ctx_t (sched/types.h): -typedef enum { - XCE_ALLOC_NORMAL = 0, // only truly free resources - XCE_ALLOC_PREEMPT, // count preemptible job resources as available - // Future: - // XCE_ALLOC_RESERVE, // count reserved-but-idle resources as available - // XCE_ALLOC_BACKFILL, // count reserved + preemptible resources -} xce_alloc_mode_t; - -xce_alloc_mode_t alloc_mode; // XCE_ALLOC_NORMAL by default -``` - -When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the full resource footprint of victim-candidate jobs (not just slots) as borrowable. The allocator logic is otherwise identical — no fork in `try_alloc_on_host()`. - -### `xce_preempt_victim_iter_t` (new, scheduler-preemption spec) - -Same lazy-iterator pattern as plan iterator, applied to victim selection: - -```c -typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t; - -struct xce_preempt_victim_iter_s { - int (*init)(xce_preempt_victim_iter_t*, xce_sched_job_t* job, xce_scheduler_t*); - xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t*); - void (*fini)(xce_preempt_victim_iter_t*); - void* private; -}; -``` - -The default policy (`lowest_priority_first`) generates victim sets lazily: smallest sufficient set of lowest-priority running jobs from `preemptible_queues` that would free enough resources. Listed here because the preemption flow wraps `allocate_primary()` and consumes `xce_alloc_mode_t` — both defined in this spec. - -### `parjob_allocator` changes (Req 4) - -- `my_alloc()` is replaced from a stub (`return 0`) to a real implementation that creates a plan iterator, walks candidate plans, and commits on first success. -- The allocator's registration in the plugin framework is unchanged -- only the `job_alloc_fn` implementation changes. - -### `seqjob_allocator` changes (Req 9 AC 1) - -- Internal rewrap: `my_alloc()` is restructured to use the plan and host iterator interfaces internally as a trivial case (1 host x 1 slot plan, then host iteration matching today's `sched_collect_workers()` + `xlist_find()` logic). -- Externally observable behavior is unchanged -- same host selection given same state. - -## Architecture Diagram - -### Allocation Flow (PRIMARY leaf) - -``` -┌──────────────────────────────────────────────────────────┐ -│ ALLOCATE PRIMARY (Req 2+3+4) │ -│ │ -│ plan_iter_init(primary_resreq) │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────┐ │ -│ │ plan_iter->next() │ ←─ generates (nhosts, │ -│ │ returns plan or NULL │ slots_per_host) candidates │ -│ └──────────┬───────────┘ most-preferred first │ -│ │ │ -│ plan │ NULL (exhausted) │ -│ ▼ ▼ │ -│ ┌──────────────────┐ return ALLOC_FAIL │ -│ │ host_iter_init() │ │ -│ └────────┬─────────┘ │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────┐ │ -│ │ host_iter->next() │ ←─ filters workers by │ -│ │ returns worker or NULL│ select expression + │ -│ └──────────┬───────────┘ slot availability │ -│ │ │ -│ worker │ NULL (no host for plan) │ -│ ▼ ▼ │ -│ ┌──────────────────┐ host_iter->fini() │ -│ │ try_alloc(worker, │ plan_iter->next() (next plan) │ -│ │ plan, ctx) │ │ -│ └────────┬─────────┘ │ -│ │ │ -│ success │ fail │ -│ ▼ ▼ │ -│ ┌────────────┐ host_iter->next() (next worker for plan) │ -│ │ commit │ │ -│ │ return │ │ -│ │ ALLOC_SUCCESS│ │ -│ └────────────┘ │ -└──────────────────────────────────────────────────────────┘ -``` - -### Unsatisfiable Resreq Cache (Req 7) - -``` -┌──────────────────────────────────────────────────────────┐ -│ SESSION START (begin_session) │ -│ │ -│ Clear unsatisfiable resreq fingerprint cache │ -│ │ -│ For each job from picker: │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────────────┐ │ -│ │ Lookup fingerprint in cache │ │ -│ │ key = hash(minTasks, │ │ -│ │ maxTasks, select_node_id, │ │ -│ │ rusage_hash, user, queue, │ │ -│ │ project) │ │ -│ └──────────┬───────────────────┘ │ -│ │ │ -│ hit │ miss │ -│ ▼ │ ▼ │ -│ ┌────────┐ │ ┌──────────────────┐ │ -│ │ PEND │ │ │ Run full │ │ -│ │ immedi- │ │ │ allocation │ │ -│ │ ately │ │ │ (Req 2+3+4) │ │ -│ └────────┘ │ └────────┬─────────┘ │ -│ │ │ │ -│ │ success │ fail (exhausted) │ -│ │ ▼ ▼ │ -│ │ ┌────────────┐ ┌──────────────────────┐ │ -│ │ │ DISPATCH │ │ Populate cache with │ │ -│ │ │ (no cache │ │ resreq fingerprint │ │ -│ │ │ entry) │ │ then PEND job │ │ -│ │ └────────────┘ └──────────────────────┘ │ -└──────────────────────────────────────────────────────────┘ -``` - -## Integration Note: Preemption Interaction - -Preemption (scheduler-preemption spec) fires when allocation fails for a preemptible job. It follows the same lazy-iterator pattern as allocation, applied to victim selection instead of host selection. - -### Preemption Iterator - -Same shape as plan iterator, different domain: - -```c -// Preemption victim iterator — lazy generation of victim sets -typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t; - -struct xce_preempt_victim_iter_s { - int (*init)(xce_preempt_victim_iter_t* iter, xce_sched_job_t* job, - xce_scheduler_t* sched); - xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t* iter); - void (*fini)(xce_preempt_victim_iter_t* iter); - void* private; -}; -``` - -The default policy (`lowest_priority_first`) generates victim sets in order: smallest sufficient set of lowest-priority running jobs from `preemptible_queues` that would free enough resources for the preempting job. - -### Allocation Mode - -A `xce_alloc_mode_t` enum on `xce_sched_ctx_t` controls which resource pool the allocator draws from — applies to all resource types (slots, shared resources, limit instances, licenses), not just CPU slots: - -```c -typedef enum { - XCE_ALLOC_NORMAL = 0, // only truly free resources - XCE_ALLOC_PREEMPT, // count preemptible job resources as available -} xce_alloc_mode_t; -``` - -When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the full resource footprint of victim-candidate jobs as borrowable. The allocator logic is otherwise identical — no fork in `try_alloc_on_host()`. - -### Full Preemption Flow - -```c -int allocate_job(sched, job) { - // 1. Cache check - if (cache_lookup(job)) return pend(job); - - // 2. Normal allocation - if (allocate_primary(sched, job->resreq, ctx) > 0) return dispatch(job); - - // 3. Preemption (if applicable) - if (job->queue->preemptible_queues == NULL) { - cache_insert(job); - return pend(job); - } - - victim_iter_init(&viter, job, sched); - while ((victims = viter.next(&viter)) != NULL) { - ctx.alloc_mode = XCE_ALLOC_PREEMPT; - if (allocate_primary(sched, job->resreq, &ctx) > 0) { - viter.fini(&viter); - return commit_preempt(victims, job); - } - ctx.alloc_mode = XCE_ALLOC_NORMAL; - } - viter.fini(&viter); - - cache_insert(job); - return pend(job); -} -``` - -Key properties: -- v1 cache is preemption-agnostic (populated on allocation failure alone). -- When preemption ships, it fires between allocation failure and cache population. -- If preemption succeeds → no cache entry, preempting job is re-pended for next round. -- If preemption fails/exhausted → cache entry populated, job pended normally. -- Plan and host iterators are not preemption-aware — they simply report success/failure. -- `xce_alloc_mode_t alloc_mode` is the only new allocator surface needed. - -## Implementation Order - -Scheduler sub-specs are designed for sequential delivery: - -``` -Phase 1: scheduler-allocation (this spec) - ├── plan_iter + host_iter foundation iterators - ├── seqjob rewrap validates abstraction on simple case - ├── parjob combinatorial real parallel allocation - └── resreq cache performance optimization - -Phase 2: scheduler-core - ├── core/driver split extracts xce_sched_core_t from xce_scheduler_t - ├── picker interface xce_job_picker_iface_t + builtin_fcfs - └── session loop wires core_go() calls allocate_job() - -Phase 3: scheduler-preemption - ├── victim_iter lazy victim-set generator - ├── XCE_ALLOC_PREEMPT mode tells allocator to count borrowed resources - └── preempt_policy plugin lowest_priority_first default -``` - -| Phase | Why there | -|---|---| -| Allocation first | Self-contained. Unit-testable with fake workers. No core, no preemption needed. Iterators are the building block everything else sits on. | -| Core second | Wires allocation into a real scheduling loop. Picker feeds jobs → allocator places them → decisions flow. Without allocation, core has nothing to call. | -| Preemption last | Wraps allocation. Needs `allocate_primary()` to exist. Needs core session loop to re-pend preempted job. Is a "retry with borrowing" layer on top of normal allocation. | diff --git a/.kiro/specs/scheduler-allocation/spec.json b/.kiro/specs/scheduler-allocation/spec.json deleted file mode 100644 index 67be233b..00000000 --- a/.kiro/specs/scheduler-allocation/spec.json +++ /dev/null @@ -1,27 +0,0 @@ -{ - "feature_name": "scheduler-allocation", - "created_at": "2026-07-27T00:00:00Z", - "updated_at": "2026-07-27T00:00:00Z", - "language": "en", - "phase": "requirements-generated", - "parent_spec": "scheduler-framework", - "description": "Lazy iterator-driven allocation architecture with parallel job support and resreq caching for the XCE scheduler", - "approvals": { - "requirements": { - "generated": true, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "blocked_on": ["scheduler-core", "scheduler-dispatch-commit"], - "sibling_specs": ["scheduler-core", "scheduler-preemption", "scheduler-dispatch-commit"], - "sequencing_note": "Added 2026-08-23 (user directive): blocked on scheduler-dispatch-commit landing first -- this spec's try_alloc_on_host() reads the same resInUse/sharedInUse totals bug-1624 (fixed by scheduler-dispatch-commit) corrupts, and the planned XCE_ALLOC_PREEMPT borrowing mode interacts with what counts as committed/borrowable under that spec's redefined RUN-state timing." -} diff --git a/.kiro/specs/scheduler-config-version-tag/design.md b/.kiro/specs/scheduler-config-version-tag/design.md deleted file mode 100644 index 1e088cc3..00000000 --- a/.kiro/specs/scheduler-config-version-tag/design.md +++ /dev/null @@ -1,283 +0,0 @@ -# Design Document - -## Overview -**Purpose**: This feature lets the scheduler recognize when a controller's configuration push carries content it already has, so it can skip redundant resource/shared-value/queue re-registration — eliminating the burst of false error-level log entries that currently fires on every HA-failover reconnect (bug-1727/1728/1729). - -**Users**: Operators monitoring scheduler logs during HA failover, who currently see 20-33 misleading error entries per failover cycle with no way to tell them apart from a genuine resource-registration failure. - -**Impact**: Extends the controller's existing config-push payload (`Conf4Sched`) with one new field, and adds one comparison + one stored value to the scheduler's core state. No change to what configuration data is sent, how HA election/failover works, or how explicit administrator resource/queue update operations behave. - -### Goals -- A scheduler reconnect that delivers an unchanged configuration produces zero error-level log entries and does not re-run resource/shared-value/queue registration. -- A scheduler reconnect (or live config push) that delivers a genuinely changed configuration is still fully applied, exactly as today. -- The skip decision reuses the controller's existing, already-replicated config-identity value — no new versioning or consensus mechanism. -- Both existing config-application code paths (join/reconnect and live config upgrade) share one consistent skip decision. - -### Non-Goals -- Changing how the controller computes, stores, or replicates its config-identity value. -- Changing the add-or-update behavior of explicit administrator resource/queue update operations (`XCE_SCHED_OP_UPDATE_RESOURCES` and equivalents) — unaffected, per Requirement 4. -- Changing what configuration data is sent in a push, or the wire shape of anything other than the one new field. -- Any change to HA election, promotion, or failover mechanics. - -## Boundary Commitments - -### This Spec Owns -- The `configVersionTag` field on the `Conf4Sched` wire message and its population by the controller. -- The scheduler's stored "last-applied configuration identity" state and the comparison that decides skip vs. full apply. -- The skip/apply branching in both existing config-application code paths (join/reconnect via `sched_init_config()`, and live config upgrade via `core_events.c`'s `XCE_CORE_EVENT_CONFIG_PUSH` handling), and the diagnostic-level log entry recorded on skip. -- The memory-ownership responsibility for the incoming config payload when it is not adopted (skip branch). - -### Out of Boundary -- Computation or replication of the controller's underlying config-identity value (`config_version_tag` in `controller/config_repl.c`) — an existing, unmodified mechanism this feature only reads. -- The add-or-update logic for `XCE_SCHED_OP_UPDATE_RESOURCES` and any other explicit administrator update operation — untouched, verified against Requirement 4. -- HA election, promotion, heartbeat, or failover logic in `controller/ha.c`. -- Any change to the scheduler's `.replay` snapshot save/load format (`snapshot.c`/`snapshot_load.c`) — see Data Models' explicit scope decision below. - -### Allowed Dependencies -- The controller's existing `myconf->config_version_tag` (`controller/config_repl.c`), read-only. -- The existing `Conf4Sched`/`xce_conf4sched_t` capnp-generated type and its existing `free_Conf4Sched_ptr()` deallocator. -- The existing `xce_sched_core_t` struct (`scheduler/include/core.h`) as the storage location for the new comparison state. -- The existing two config-application call sites (`scheduler/src/scheduler.c`'s `sched_init_config()`, `scheduler/src/core_events.c`'s `XCE_CORE_EVENT_CONFIG_PUSH` case) as the two integration points — not new call sites. - -### Revalidation Triggers -- `config_version_tag`'s computation in `controller/config_repl.c` becomes dependent on which node is primary/leadership state (would break the cross-node stability this feature's safety rests on). -- `Conf4Sched`'s field list or generated-binding mechanism changes shape. -- A third live config-application code path is introduced (would need the same skip-decision wiring to stay consistent with Path A/Path B). -- `xce_sched_core_t`'s zero-initialization contract changes (this feature's "no prior config" sentinel relies on `config_version_tag` starting at `0`). - -### Project Invariant Cross-Check -- **Async I/O (delaymgr trigger-only rule)**: not applicable — this feature adds one `uint64_t` comparison to code that already runs synchronously on the existing request-handling/event-translation path (`handler.c`'s `scheduler_req_handler`, `driver.c`'s event-translation loop), neither of which is delaymgr. No new blocking or I/O is introduced; the skip path does strictly less work than today. -- **Controller self-termination rule**: not applicable — no controller-side control-flow or exit path is touched; the controller-side change is a single field population in an existing send path. -- **Save/restore field-completeness**: not applicable to the `.replay` snapshot format — see Data Models section for the explicit scope decision. - -## Architecture - -### Existing Architecture Analysis -- The controller already computes a stable, content-derived config identity (`config_version_tag`) and keeps it consistent across HA nodes via the existing config-replication mechanism (`controller/config_repl.c`) — this feature is a pure consumer, not a new producer. -- Two independent code paths apply a pushed config to the scheduler's core state, both unconditionally re-registering every resource/shared-value/queue on every invocation (see `research.md`'s "Which code path(s)" finding): Path A (join/reconnect, `scheduler.c`'s `sched_init_config()`) and Path B (live config upgrade, `core_events.c`'s `XCE_CORE_EVENT_CONFIG_PUSH` case). Both read/write the same underlying `xce_sched_core_t`. -- `xce_sched_core_t` already carries a self-incrementing local counter, `configver`, bumped by both paths on every apply — this feature adds a second, independent field next to it (the controller's own content identity, not a local counter). - -### Architecture Pattern & Boundary Map -One new wire field, one new stored value + one shared comparison function on the core, consumed identically by both existing application paths. - -```mermaid -graph TB - ConfigRepl[controller config_repl.c config_version_tag] - PushConfigs[controller sched.c xcesched_push_configs] - Wire[Conf4Sched configVersionTag] - PathA[scheduler.c sched_init_config Path A reconnect] - PathB[core_events.c CONFIG_PUSH handling Path B live upgrade] - Compare[core.c config_changed comparison] - CoreState[xce_sched_core_t config_version_tag] - - ConfigRepl --> PushConfigs - PushConfigs --> Wire - Wire --> PathA - Wire --> PathB - PathA --> Compare - PathB --> Compare - Compare --> CoreState - CoreState --> Compare -``` - -**Architecture Integration**: -- Selected pattern: shared core-level comparison function consumed by both existing application paths (see `research.md`'s Architecture Pattern Evaluation for the rejected alternatives). -- Domain/feature boundaries: the controller owns computing/replicating the identity value (unchanged); the scheduler core owns storing the last-applied value and deciding skip vs. apply; each application path (A/B) owns only invoking that decision and branching its own registration calls accordingly. -- Existing patterns preserved: capnp schema → generated C struct → existing per-path apply logic, unchanged in shape; only one new field and one new conditional branch per path. -- New components rationale: one new pure function (`xce_sched_core_config_changed`) is the only new component — everything else extends an existing struct field list or existing conditional logic. -- Steering compliance: TDD required for every new behavior (`testing.md`); no new blocking work on any async-I/O-constrained thread (see Project Invariant Cross-Check above). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Wire schema | Cap'n Proto (`schema/controller.capnp`, existing toolchain) | Carries `configVersionTag` from controller to scheduler | One new `UInt64` field on the existing `Conf4Sched` struct; generated bindings regenerate automatically via the existing meson `CUSTOM_COMMAND` | -| Controller (C) | `controller/sched.c` | Populates the new field from the existing `myconf->config_version_tag` | No new dependency | -| Scheduler (C) | `scheduler/src/core.c`/`core.h`, `scheduler/src/scheduler.c`, `scheduler/src/core_events.c` | Stores the last-applied identity, compares, branches skip/apply, frees the redundant payload on skip | No new dependency | - -## File Structure Plan - -### Modified Files -- `schema/controller.capnp` — add `configVersionTag @9: UInt64;` to the `Conf4Sched` struct (next available field number after `limits @8`). -- `controller/sched.c` — in `xcesched_push_configs()`, populate `configdata.data.conf4sched->configVersionTag = c->myconf->config_version_tag;`. -- `scheduler/include/core.h` — add `uint64_t config_version_tag;` to `xce_sched_core_t` (alongside the existing `configver` field); declare `bool xce_sched_core_config_changed(xce_sched_core_t* core, const xce_conf4sched_t* config);`. -- `scheduler/src/core.c` — define `xce_sched_core_config_changed()`: returns `true` when `config == NULL`, or `config->configVersionTag == 0`, or `config->configVersionTag != core->config_version_tag`; returns `false` (unchanged) only when `config->configVersionTag != 0 && config->configVersionTag == core->config_version_tag`. -- `scheduler/src/scheduler.c` — `sched_init_config()` (Path A): call `xce_sched_core_config_changed()` before the existing `sched_init_resources()`/`sched_init_sharedInst()`/`sched_init_queues()` calls; on unchanged, skip those three calls, free the incoming `config` via `free_Conf4Sched_ptr()`, log at `XCE_LOG_DEBUG`, and return without adopting it as `sched->core.config`; on changed, keep today's behavior and additionally set `sched->core.config_version_tag = config->configVersionTag;`. -- `scheduler/src/core_events.c` — the `XCE_CORE_EVENT_CONFIG_PUSH` case (Path B): same skip/apply branching as Path A, applied to `core_apply_config_resources()`/`core_apply_config_resvals()`/`core_apply_config_queues()`; the existing `picker->picker_iface->on_config()` hook still fires unconditionally in both branches (unaffected by this feature — it is not a resource/shared-value/queue registration call). - -### New Test Files -- `scheduler/testsuite/test_core_config_version.c` — unit tests for `xce_sched_core_config_changed()` in isolation (pure function, no I/O). -- Extends existing `scheduler/testsuite/test_handlers.c` (or the file covering `sched_init_config()`/Path A today) with skip/apply/first-push/zero-tag cases. -- Extends existing `scheduler/testsuite/test_core_events.c` (Path B's `CONFIG_PUSH` coverage) with the equivalent skip/apply cases. -- `controller/testsuite/` — extends the existing test covering `xcesched_push_configs()` (or equivalent) to assert `configVersionTag` is populated from `myconf->config_version_tag`. - -> No changes to `scheduler/src/snapshot.c`/`snapshot_load.c` or their test files — see Data Models' explicit scope decision. - -## System Flows - -```mermaid -sequenceDiagram - participant Ctrl as Controller - participant Sched as Scheduler Path A/B - participant Core as xce_sched_core_t - - Ctrl->>Sched: Config push (configVersionTag = T) - Sched->>Core: xce_sched_core_config_changed(core, config) - alt T != 0 and T == core.config_version_tag - Core-->>Sched: unchanged - Sched->>Sched: free incoming config payload - Sched->>Sched: log DEBUG config unchanged - else T == 0 or T != core.config_version_tag - Core-->>Sched: changed - Sched->>Core: register resources, shared values, queues - Sched->>Core: core.config_version_tag = T - Sched->>Core: core.config = incoming config payload - end -``` - -**Key decisions visible in the diagram**: the comparison happens once, before any registration work; the incoming payload is either adopted (changed branch) or freed (unchanged branch) — it is never both, and never leaked. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | Config push carries an identity indicator | `Conf4Sched` schema, `xcesched_push_configs()` | `configVersionTag` field | Sequence diagram, "Config push" step | -| 1.2 | Same content → same indicator across nodes | `config_version_tag` (existing, controller) | — | research.md "Is config_version_tag safe to trust" | -| 1.3 | Different content → different indicator | `config_version_tag` (existing, controller) | — | research.md, same | -| 1.4 | Indicator derived from existing replicated identity | `xcesched_push_configs()` | — | Architecture diagram | -| 2.1 | Skip re-registration on match | `xce_sched_core_config_changed()`, Path A, Path B | `xce_sched_core_config_changed()` | Sequence diagram, "unchanged" branch | -| 2.2 | No error-level log on skip | Path A, Path B | — | Sequence diagram, "unchanged" branch | -| 2.3 | Existing state retained on skip | Path A, Path B (no-op on skip) | — | Sequence diagram, "unchanged" branch | -| 2.4 | Diagnostic-level log on skip | Path A, Path B | — | Sequence diagram, "unchanged" branch | -| 3.1 | Full apply on mismatch | Path A, Path B | Existing `sched_init_resources()`/etc., `core_apply_config_resources()`/etc. | Sequence diagram, "changed" branch | -| 3.2 | Adopt new indicator after apply | Path A, Path B | `core.config_version_tag` assignment | Sequence diagram, "changed" branch | -| 3.3 | First-ever push always applied | `xce_sched_core_config_changed()` (zero-initialized `config_version_tag`) | — | Design Decisions: "Zero as sentinel" | -| 3.4 | Controller "no config" indicator always applied | `xce_sched_core_config_changed()` (`configVersionTag == 0` check) | — | Design Decisions: "Zero as sentinel" | -| 4.1 | Admin update operations unaffected | Out of Boundary (untouched) | — | — | -| 4.2 | Admin update on a skipped-push resource still works | Out of Boundary (untouched) | — | — | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|--------------|--------|--------------|--------------------------|-----------| -| `Conf4Sched.configVersionTag` | Wire schema | Carries config identity to the scheduler | 1.1, 1.4 | `config_repl.c`'s `config_version_tag` (P0) | State | -| `xcesched_push_configs()` (modified) | Controller | Populates the new field | 1.1, 1.4 | `myconf->config_version_tag` (P0) | State | -| `xce_sched_core_config_changed()` (new) | Scheduler core | Pure skip/apply decision | 2.1, 3.1, 3.3, 3.4 | `xce_sched_core_t.config_version_tag` (P0) | Service | -| `sched_init_config()` (modified, Path A) | Scheduler | Join/reconnect config application, branches on the decision | 2.1-2.4, 3.1-3.2 | `xce_sched_core_config_changed()` (P0), `free_Conf4Sched_ptr()` (P0) | Service | -| `core_events.c` `CONFIG_PUSH` case (modified, Path B) | Scheduler core | Live config-upgrade application, branches on the decision | 2.1-2.4, 3.1-3.2 | `xce_sched_core_config_changed()` (P0), `free_Conf4Sched_ptr()` (P0) | Event | - -### Scheduler Core - -#### `xce_sched_core_config_changed` - -| Field | Detail | -|-------|--------| -| Intent | Decide whether a pushed configuration differs from the one currently held, using the controller's content-identity indicator | -| Requirements | 2.1, 3.1, 3.3, 3.4 | - -**Responsibilities & Constraints** -- Pure function: no side effects, no I/O, no locking (reads two `uint64_t` values already available to the caller). -- Treats `configVersionTag == 0` (controller reports no established config) as always "changed," per 3.4. -- Treats `core->config_version_tag == 0` (no configuration applied since process start, or never yet applied) as always "changed," per 3.3 — this falls out of the same comparison without a separate flag, since `xce_sched_core_t` is zero-initialized and a real content-derived tag is practically never exactly `0`. - -**Dependencies** -- Inbound: `sched_init_config()` (Path A, P0), `core_events.c`'s `CONFIG_PUSH` case (Path B, P0) -- Outbound: none - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -/* Returns true when `config` should be fully applied (registered), false - * when it is identical to what `core` already holds and may be skipped. */ -bool xce_sched_core_config_changed(xce_sched_core_t* core, - const xce_conf4sched_t* config); -``` -- Preconditions: `core != NULL`. -- Postconditions: no mutation of `core` or `config`; return value is deterministic given the two inputs. -- Invariants: `xce_sched_core_config_changed(core, config) == false` implies `config->configVersionTag == core->config_version_tag` and both are nonzero. - -**Implementation Notes** -- Integration: called once, before any resource/shared-value/queue registration, in both Path A and Path B. -- Validation: unit-tested in isolation (`test_core_config_version.c`) — no fixture beyond two `uint64_t` values and a minimal `xce_conf4sched_t`. -- Risks: none — pure function, fully covered by unit tests. - -### Scheduler Application Paths (Path A / Path B) - -Both paths get the identical shape of change; documented once, applied at two call sites (`scheduler.c`'s `sched_init_config()` and `core_events.c`'s `CONFIG_PUSH` case). - -| Field | Detail | -|-------|--------| -| Intent | Apply or skip a pushed configuration based on the shared decision, owning the incoming payload's lifetime either way | -| Requirements | 2.1, 2.2, 2.3, 2.4, 3.1, 3.2 | - -**Responsibilities & Constraints** -- On "changed": unchanged from today's behavior (full registration), plus one new line adopting `configVersionTag` into `core->config_version_tag`. -- On "unchanged": skip the resource/shared-value/queue registration calls entirely; do not touch existing `core` state; free the incoming `config` payload via `free_Conf4Sched_ptr()` (it is not adopted as `core->config`, so it has no other owner); log one `XCE_LOG_DEBUG` entry identifying the skip. -- Neither branch changes the existing unconditional `picker->picker_iface->on_config()` call (Path B) or `plugin_config_update()` call (Path A) — out of this feature's boundary (Non-Goals). - -**Dependencies** -- Inbound: `handler.c`'s `scheduler_req_handler()` (Path A, P0), `driver.c`'s event-translation loop via `xce_core_push_events()` (Path B, P0) -- Outbound: `xce_sched_core_config_changed()` (P0), `free_Conf4Sched_ptr()` (P0) - -**Contracts**: Service [ ] / API [ ] / Event [x] / Batch [ ] / State [x] - -##### Event Contract -- Consumed events: Path A's `Request_body_configpushmsg`; Path B's `XCE_CORE_EVENT_CONFIG_PUSH`. -- Ordering / delivery guarantees: unchanged from today — each path already processes pushes strictly in arrival order; this feature adds no new ordering requirement. - -##### State Management -- State model: `xce_sched_core_t.config_version_tag`, a single `uint64_t`, updated only on the "changed" branch. -- Persistence & consistency: in-memory only, process-lifetime; resets to `0` on scheduler restart, which correctly forces a full apply on the next push (Requirement 3.3) — see Data Models. -- Concurrency strategy: unchanged from today's existing single-threaded-per-path access pattern for `core->config`/`core->configver` (both paths already mutate these without new locking; this feature's field follows the same existing access discipline). - -**Implementation Notes** -- Integration: two call sites, one shared decision function — see File Structure Plan. -- Validation: TDD — extend each path's existing test file with skip/apply/first-push/zero-tag cases before changing the production code (`testing.md`). -- Risks: memory leak if the free-on-skip step is omitted at either call site — mitigated by an explicit test asserting no leak under ASAN (the project's default debug build) for the skip path at both call sites. - -## Data Models - -### Domain Model -- No new domain entity. `config_version_tag` is a scalar attribute added to two existing structures: `Conf4Sched` (wire) and `xce_sched_core_t` (in-memory scheduler state). - -### Logical Data Model -- `Conf4Sched.configVersionTag: UInt64` — new wire field, always present (defaults to `0` if a sender predates this feature, which this feature's own comparison already treats as "always changed," so an old controller talking to a new scheduler degrades safely to today's unconditional-apply behavior). -- `xce_sched_core_t.config_version_tag: uint64_t` — new in-memory field, zero-initialized, updated only on a full config apply. - -### Explicit Scope Decision — Not Part of Snapshot Save/Restore -`config_version_tag` is **not** added to `sched_snapshot_data_t` or the `.replay` snapshot format (`snapshot.c`/`snapshot_load.c`), and is **not** restored on snapshot load. Rationale: -- It is live-session reconnect-optimization state, not part of the scheduling-decision-relevant entity the snapshot format exists to reconstruct (jobs/workers/queues/resources/allocations) — its absence does not change any scheduling decision or resource accounting. -- The simulation/replay driver (`sim.c`) never calls `sched_init_config()` or pushes an `XCE_CORE_EVENT_CONFIG_PUSH` event at all (`sim.c`'s own documented scope boundary) — Path A and Path B, and therefore this feature, are entirely inapplicable to snapshot/replay loading. -- On a real scheduler process restart, `config_version_tag` naturally zero-initializes, which this feature's own comparison already treats as "no prior config" — forcing exactly one full re-apply on the next live push after restart. This is the same safe, existing behavior a restart already produces today (no regression, no special-cased recovery logic needed). - -### Data Contracts & Integration -**Wire field**: `Conf4Sched.configVersionTag: UInt64` (Cap'n Proto, `schema/controller.capnp`), populated by `xcesched_push_configs()` from `myconf->config_version_tag`, consumed identically by Path A and Path B on the scheduler side. No versioning strategy needed beyond the existing capnp schema-evolution convention already used for every other field in this struct (new fields get the next unused field number; older readers/writers default absent fields to `0`, which this feature's comparison already treats as "always apply"). - -## Error Handling - -### Error Strategy -This feature introduces no new error path. It removes a class of misleading error-level log entries (Requirement 2.2) by recognizing them as a no-op case, not by suppressing or downgrading a real failure — a genuine resource-registration failure (e.g., a malformed resource definition) still occurs, and still logs, on the "changed" branch exactly as today, since that branch is otherwise unmodified. - -### Error Categories and Responses -- **Redundant, unchanged push** (not an error): recognized and skipped per Requirement 2; logged at `XCE_LOG_DEBUG` only. -- **Genuinely changed push, registration failure** (existing behavior, unmodified): still logs at whatever level the existing `sched_init_resources()`/`core_apply_config_resources()` etc. already use today — out of this feature's boundary. -- **Malformed/absent config payload** (`config == NULL`): `xce_sched_core_config_changed()` returns `true` (changed) for a `NULL` config, matching each path's existing `NULL`-tolerant behavior (Path B already null-checks before calling the apply trio; Path A's caller already guarantees non-`NULL` per the existing capnp decode contract) — no new failure mode introduced. - -### Monitoring -No new metrics. The new `XCE_LOG_DEBUG` entry (Requirement 2.4) is the only new observability surface, deliberately below the operating log level so it does not itself become new noise. - -## Testing Strategy - -### Unit Tests -- `xce_sched_core_config_changed()`: matching nonzero tags → `false`; differing nonzero tags → `true`; incoming tag `0` → `true`; stored tag `0` (fresh core) → `true`; `NULL` config → `true`. -- Path A (`sched_init_config()`): unchanged push does not call `makeResource()`/queue-add functions (assert via existing call-count/spy conventions this test suite already uses) and does not emit an error-level log line; changed push behaves exactly as today plus adopts the new tag. -- Path B (`core_events.c`'s `CONFIG_PUSH` case): same two cases as Path A, exercised via `xce_core_push_events()`. - -### Integration Tests -- Controller `xcesched_push_configs()` populates `configVersionTag` from `myconf->config_version_tag` (assert the wire struct field, not just that the function returns success). -- End-to-end: two consecutive config pushes with identical controller-side config content produce two different observable outcomes (first: full registration; second: skip, zero error-level log lines) — this is the direct regression test for bug-1727/1728/1729's originally-reported symptom. -- Leak check (ASAN): the skip branch, exercised at both Path A and Path B, produces no leaked `xce_conf4sched_t`. - -### E2E / Live HA Tests -- Live 3-node HA soak (`xce-test-ha` conventions): trigger a failover and confirm the reconnecting scheduler's log contains zero error-level `sched_init_resources`/`core_apply_config_resources` entries for the reconnect, while a genuinely changed config pushed mid-soak is still fully applied and observable (e.g., via `info resource`/`info worker` in the scheduler REPL, matching this session's earlier live-verification technique). diff --git a/.kiro/specs/scheduler-config-version-tag/requirements.md b/.kiro/specs/scheduler-config-version-tag/requirements.md deleted file mode 100644 index 8833dff4..00000000 --- a/.kiro/specs/scheduler-config-version-tag/requirements.md +++ /dev/null @@ -1,55 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Thread the controller's existing config_version_tag (SHA-256-derived content fingerprint, controller/config_repl.c) through the scheduler CONFIG_PUSH wire message (Conf4Sched, schema/controller.capnp) so the scheduler can detect an unchanged config on HA-failover reconnect and skip redundant re-initialization. Currently sched_init_resources() (scheduler.c:828) and its mirror core_apply_config_resources() (core_events.c:363) unconditionally call makeResource() for every configured resource on every CONFIG_PUSH -- which fires on every scheduler reconnect/re-registration during HA failover, not just on genuine config changes -- and log ERROR for every already-registered resource (makeResource returns NULL/EEXIST), producing a burst of 20-33 false ERROR log lines per failover cycle even though nothing is actually broken (this is bug-1727/1728/1729's session-sibling finding, logged in .wolf/buglog.json). - -Design direction already agreed with the user: -1. Schema: add a configVersionTag (UInt64) field to Conf4Sched in schema/controller.capnp. -2. Controller: in xcesched_push_configs() (controller/sched.c:138-199), populate the new field from c->myconf->config_version_tag (already computed, already stable across HA failover/leadership changes -- confirmed by reading config_repl.c:686 and config_repl.c:906, both config-load/config-replication events only, never touched by ha.c's election code). -3. Scheduler: store the last-applied tag (new field on xce_sched_core_t, alongside the existing self-incrementing core->configver in scheduler.c/core.h); in sched_init_config() (scheduler.c:934) and/or its core_events.c CONFIG_PUSH-handling mirror, compare the incoming tag against the stored one -- if equal, skip sched_init_resources()/sched_init_sharedInst()/sched_init_queues() entirely (log at DEBUG that the config was unchanged); if different, run the full init and update the stored tag. - -This must follow this project's TDD discipline (test in testsuite/ before the code change) and needs to preserve today's existing behavior for a genuinely new/changed config (full re-init still happens, no regression). Needs to be verified against a live 3-node HA soak per this project's HA testing conventions (xce-test-ha and/or the existing scheduler reconnect scenario) since it touches the controller<->scheduler wire protocol and HA reconnect path. - -## Introduction -On every scheduler reconnect to a controller — including the routine re-registration that happens on every HA failover, not just when an operator actually changes the cluster configuration — the controller re-sends the full configuration to the scheduler, and the scheduler unconditionally re-runs its resource/queue/shared-value initialization. Because that initialization currently cannot tell "this is the exact same configuration I already have" apart from "this is a genuinely new configuration," every reconnect with an unchanged configuration produces a burst of false error log entries (one per already-registered resource) even though nothing is actually broken. This feature gives the scheduler a reliable way to recognize an unchanged configuration on reconnect and skip the redundant re-initialization, while still fully re-initializing whenever the configuration has genuinely changed. - -## Boundary Context (Optional) -- **In scope**: carrying a configuration-identity indicator from the controller to the scheduler on every configuration push; the scheduler comparing that indicator against the configuration it already holds; skipping resource/queue/shared-value re-initialization only when the indicator matches; always falling back to full re-initialization when it does not match or has never been seen before. -- **Out of scope**: changing how the controller computes or replicates its own configuration-identity indicator (an existing mechanism this feature reuses, not redefines); changing the add-or-update behavior of explicit administrator-initiated resource/queue update operations (a separate, already-existing code path); changing what configuration data is sent on a push (only that an identity indicator accompanies it); any change to how HA election, failover, or primary promotion itself works. -- **Adjacent expectations**: this feature depends on the controller's existing configuration-identity indicator being identical across controller nodes for the same configuration content, and unaffected by which node is currently primary — that guarantee is provided by the existing configuration replication mechanism, not by this feature. If a newly-promoted primary ever reports a different indicator for what is actually the same configuration, that is a defect in the underlying replication guarantee, not something this feature is responsible for detecting or correcting. - -## Requirements - -### Requirement 1: Configuration Push Carries a Configuration-Identity Indicator -**Objective:** As a scheduler, I want every configuration push from the controller to include an indicator of the configuration's identity, so that I can tell an unchanged configuration apart from a genuinely new one without re-inspecting every individual setting. - -#### Acceptance Criteria -1. When the Controller sends a configuration push to the Scheduler, the Controller shall include a configuration-identity indicator that represents the exact configuration content being sent. -2. While two configuration pushes carry byte-for-byte identical configuration content, the Controller shall present the same configuration-identity indicator for both pushes, regardless of which controller node sent them. -3. While two configuration pushes carry different configuration content, the Controller shall present different configuration-identity indicators for the two pushes. -4. The Controller shall derive the configuration-identity indicator from its own existing, already-replicated notion of configuration identity, not from a separately maintained value. - -### Requirement 2: Skip Redundant Re-Initialization for an Unchanged Configuration -**Objective:** As an operator monitoring scheduler logs, I want the scheduler to recognize when a reconnect delivers a configuration it already has, so that routine HA failover reconnects do not produce misleading error log entries or repeat unnecessary work. - -#### Acceptance Criteria -1. When the Scheduler receives a configuration push whose configuration-identity indicator matches the indicator of the configuration it currently holds, the Scheduler shall skip re-registering its resources, shared resource values, and queues from that push. -2. When the Scheduler skips re-registration under this condition, the Scheduler shall not emit an error-level log entry for any resource, shared resource value, or queue from that push. -3. While the Scheduler skips re-registration under this condition, the Scheduler shall retain the resource, shared resource value, and queue state it already holds unchanged. -4. When the Scheduler skips re-registration under this condition, the Scheduler shall record a diagnostic-level log entry noting that the pushed configuration was recognized as unchanged, so the behavior remains observable to someone investigating scheduler startup or reconnect behavior without being visible at normal operating log levels. - -### Requirement 3: Fully Apply a Genuinely Changed Configuration -**Objective:** As an operator who updates the cluster configuration, I want the scheduler to always fully apply a configuration that has actually changed, so that skipping redundant work on reconnect never causes a real configuration change to be silently ignored. - -#### Acceptance Criteria -1. When the Scheduler receives a configuration push whose configuration-identity indicator differs from the indicator of the configuration it currently holds, the Scheduler shall fully register the resources, shared resource values, and queues from that push, the same as it does today. -2. When the Scheduler completes registering a configuration push under this condition, the Scheduler shall adopt the newly pushed configuration's identity indicator as the configuration it currently holds. -3. When the Scheduler receives its first configuration push since starting, the Scheduler shall treat that push as a changed configuration and fully register it, regardless of any indicator value it carries. -4. If a configuration push carries a configuration-identity indicator that signals the controller's configuration is not yet established, the Scheduler shall treat that push as a changed configuration and fully register it rather than skipping registration. - -### Requirement 4: No Regression to Explicit Configuration Update Operations -**Objective:** As an operator, I want administrator-initiated resource and queue update operations to keep working exactly as they do today, so that this feature's reconnect-skip behavior does not interfere with intentional, explicit configuration changes. - -#### Acceptance Criteria -1. The Scheduler shall apply an administrator-initiated resource or queue update operation the same way it does today, independent of any configuration-identity indicator comparison introduced by this feature. -2. While an administrator-initiated resource or queue update changes a resource or queue that was previously registered via a configuration push recognized as unchanged, the Scheduler shall apply that update without requiring a subsequent configuration push to take effect. diff --git a/.kiro/specs/scheduler-config-version-tag/research.md b/.kiro/specs/scheduler-config-version-tag/research.md deleted file mode 100644 index fcec1692..00000000 --- a/.kiro/specs/scheduler-config-version-tag/research.md +++ /dev/null @@ -1,73 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `scheduler-config-version-tag` -- **Discovery Scope**: Extension (existing controller<->scheduler wire protocol and existing HA config-replication mechanism) -- **Key Findings**: - - The controller already computes and replicates a content-derived config identity (`config_version_tag`, first 8 bytes of the config file's SHA-256) that is stable across HA failover/leadership changes — this feature reuses it rather than inventing a new versioning scheme. - - There are **two independent, structurally-mirrored code paths** that apply a pushed config to the live scheduler core, both exhibiting the same unconditional-re-registration behavior and needing the same fix. - - `xce_conf4sched_t`'s generated C fields keep their capnp camelCase names as-is (no snake_case mapname), confirmed against `CtrlStatus.configVersion` (`controller/ctrlstatus.c:119`, referenced directly as `st->configVersion` in C). - - A memory-ownership responsibility is newly introduced by the "skip" branch: the incoming, now-redundant `xce_conf4sched_t` payload must be explicitly freed (`free_Conf4Sched_ptr()`) rather than silently dropped, because both existing call sites currently treat "config push handled" as unconditional ownership transfer. - -## Research Log - -### Which code path(s) actually apply a config push to the live scheduler -- **Context**: bug-1727/1728/1729 (session-sibling findings, `.wolf/buglog.json`) diagnosed `sched_init_resources()` (`scheduler/src/scheduler.c:828`) as the ERROR-spam source, reached via the scheduler's join/reconnect handshake. Before designing a fix, confirmed whether this is the only live application path. -- **Sources Consulted**: `scheduler/src/handler.c`, `scheduler/src/scheduler.c`, `scheduler/src/core_events.c`, `scheduler/src/driver.c`, `scheduler/src/main.c`-adjacent comments, `scheduler/CLAUDE.md`'s architecture description. -- **Findings**: - - **Path A (join/reconnect — the reported bug's trigger)**: `handler.c`'s `scheduler_req_handler()`, on `Request_body_configpushmsg`, calls `sched_init_config()` (`scheduler.c:934`) synchronously, which unconditionally calls `sched_init_resources()`/`sched_init_sharedInst()`/`sched_init_queues()`. The controller sends `ConfigPushMsg` on every scheduler (re)join, including every HA-failover reconnect — matching the observed ERROR bursts correlated with `xcecli failover: attempt N` log lines. - - **Path B (live config upgrade while already connected)**: an admin-initiated `XCE_SCHED_OP_UPDATE_CONFIG` request is queued as a pend_update, later drained and translated by the normal-mode driver (`driver.c:202`, `driver.c:555`) into an `XCE_CORE_EVENT_CONFIG_PUSH` event applied via `xce_core_push_events()` → `core_events.c`'s `case XCE_CORE_EVENT_CONFIG_PUSH` (`core_events.c:702-717`), which unconditionally calls `core_apply_config_resources()`/`core_apply_config_resvals()`/`core_apply_config_queues()` — a structural mirror of Path A's trio, confirmed by `core_events.c`'s own header comment stating it mirrors `scheduler.c`'s functions "exactly." - - Both trios ultimately read/write the same `xce_sched_core_t` (`sched->core` in Path A is the same struct type `core` in Path B) — `resmgr`/`sharedInst`/`queues` are core-level state, not duplicated per path. -- **Implications**: the fix must live at a shared, core-level seam so both paths stay consistent (Path B fixed for free, not left to silently diverge from Path A) rather than duplicating comparison logic once per path. A single new field on `xce_sched_core_t` plus one shared comparison function, called from both `sched_init_config()` and `core_events.c`'s CONFIG_PUSH case, satisfies this. - -### Is `config_version_tag` safe to trust as a reconnect-stability signal -- **Context**: Requirement 1.2 depends on the controller's config-identity indicator being identical across nodes for identical content and unaffected by leadership changes. -- **Sources Consulted**: `controller/config_repl.c` (both `config_version_tag` assignment sites), `controller/ha.c` (grepped for any assignment — none found). -- **Findings**: `config_version_tag = be64toh(first 8 bytes of SHA256(config file bytes))`, set only in `config_repl.c:686` (primary config load) and `config_repl.c:906` (post-replication commit of a new config version) — both are config-load/config-replication events. No assignment exists anywhere in `ha.c` (election/promotion/heartbeat code), confirming it does not change on failover/leadership transitions absent an actual config change. `ha.c:8733`'s own comment independently confirms `config_version_tag == 0` is already an established, meaningful sentinel ("a config-less node ... must NEVER start ...") elsewhere in this codebase — reused here for Requirement 3.4 rather than inventing a new sentinel convention. -- **Implications**: safe to treat as an equality oracle for "did the config actually change." No new coordination/consistency mechanism needed — this feature is purely a consumer of an existing, already-trusted value. - -### Ownership of the incoming config payload on a skipped push -- **Context**: today, both Path A (`handler.c`, unconditional `request->body.configpushmsg->data.conf4sched = NULL;` after any `rc>=0`) and Path B assume the callee took ownership of the pushed `xce_conf4sched_t*` once "handled," regardless of whether it was actually adopted as the live config. -- **Sources Consulted**: `scheduler/src/handler.c:687-692`, `lib/controller.capnp.h`/`.c` (`free_Conf4Sched_ptr` existence, already used at `driver.c:114` and `scheduler.c:406` for an unrelated discard path). -- **Findings**: `free_Conf4Sched_ptr()` is a real, already-used deallocator for this exact type. When this feature's skip branch chooses not to adopt the incoming payload as `core->config`, that payload has no other owner and must be freed there, or every skipped reconnect leaks one `xce_conf4sched_t` (plus its nested resource/queue/resval/group arrays). -- **Implications**: the skip branch's owning component (see design's Components section) is explicitly responsible for calling `free_Conf4Sched_ptr()` on the incoming payload — captured as a Data Ownership decision in `design.md`, not left implicit. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Shared core-level comparison function | One `xce_sched_core_t`-level field + one pure comparison function, called from both Path A and Path B | Single source of truth; both paths stay consistent by construction; minimal surface area | None significant | **Selected** | -| Per-path duplicated comparison | Inline the same check independently in `scheduler.c` and `core_events.c` | Slightly less coupling between files | Duplicates logic the codebase already flags as "mirrors ... exactly"; two places to keep in sync | Rejected — repeats the exact maintenance hazard the existing mirrored trios already represent | -| Field-by-field resource/queue diffing (no version tag) | Compare each resource/queue's individual fields on `EEXIST` instead of a whole-config identity check | Doesn't require a schema change | Only fixes resources, not queues/resvals/groups; re-does the same comparison work on every single reconnect instead of one O(1) check; doesn't reuse an already-trusted value | Rejected in favor of the user-directed version-tag approach (see conversation record) — kept here for traceability | - -## Design Decisions - -### Decision: Store the comparison state on `xce_sched_core_t`, not `xce_scheduler_t` -- **Context**: Path A's caller has an `xce_scheduler_t* sched` (which embeds `xce_sched_core_t core`); Path B's caller has a bare `xce_sched_core_t* core`. The new stored tag must be reachable from both. -- **Alternatives Considered**: - 1. Store on `xce_scheduler_t` — unreachable from Path B without threading an extra pointer through `core_events.c`. - 2. Store on `xce_sched_core_t` — reachable from both as `sched->core.config_version_tag` and `core->config_version_tag` (same field). -- **Selected Approach**: Option 2. -- **Rationale**: `xce_sched_core_t` is already the shared state both paths mutate (`resmgr`, `sharedInst`, `queues`, the existing self-incrementing `configver`); this field belongs alongside `configver` for the same reason. -- **Trade-offs**: None material. -- **Follow-up**: none. - -### Decision: Zero as "no config applied yet" sentinel, no separate boolean flag -- **Context**: Requirement 3.3 (first push after start) and 3.4 (controller reports no established config) both must force a full apply. -- **Alternatives Considered**: - 1. Add a separate `bool config_version_tag_valid` flag alongside the stored tag. - 2. Reuse `0` as "unset"/"no config" on both sides of the comparison (matching `ha.c`'s existing `config_version_tag == 0` convention). -- **Selected Approach**: Option 2 — `xce_sched_core_t.config_version_tag` starts at `0` (already the case via existing zero-initialization of `xce_sched_core_t`); the comparison treats a match as "skip" only when the incoming tag is nonzero AND equal to the stored tag. -- **Rationale**: reuses an already-established codebase convention instead of introducing a second, redundant state variable; a SHA-256-derived tag colliding with `0` is not a practical concern (same assumption `ha.c` already relies on elsewhere). -- **Trade-offs**: none material. -- **Follow-up**: none. - -## Risks & Mitigations -- **Risk**: skip branch leaks the incoming `xce_conf4sched_t` payload if the free responsibility is missed. — **Mitigation**: explicit Data Ownership decision in `design.md`; ASAN is already the project's default debug build (per `build-deployment.md`), so a leak here is caught by the existing test/build pipeline if a leak-detecting test exercises the skip path. -- **Risk**: a future change to `config_version_tag`'s computation (e.g., made leadership-dependent) would silently break this feature's core safety assumption. — **Mitigation**: captured as a Revalidation Trigger in `design.md`. -- **Risk**: Path B (live config upgrade) skipping an "unchanged" push could surprise an admin who explicitly re-ran an upgrade expecting a visible re-apply. — **Mitigation**: the resulting state is identical whether skipped or reapplied (same content); Requirement 2.4's diagnostic-level log entry keeps the behavior inspectable without being a behavior change an admin would notice functionally. - -## References -- `.wolf/buglog.json` bug-1727, bug-1728, bug-1729 — session findings that motivated this feature. -- `controller/config_repl.c` — `config_version_tag` computation and replication (existing mechanism this feature reuses). -- `scheduler/CLAUDE.md` — Core/Drivers architecture description confirming Path A vs. Path B are genuinely separate, both-live code paths. diff --git a/.kiro/specs/scheduler-config-version-tag/spec.json b/.kiro/specs/scheduler-config-version-tag/spec.json deleted file mode 100644 index cf8b1067..00000000 --- a/.kiro/specs/scheduler-config-version-tag/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "scheduler-config-version-tag", - "created_at": "2026-08-28T10:41:46Z", - "updated_at": "2026-08-28T11:20:00Z", - "language": "en", - "phase": "tasks-generated", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/scheduler-config-version-tag/tasks.md b/.kiro/specs/scheduler-config-version-tag/tasks.md deleted file mode 100644 index 87912f8d..00000000 --- a/.kiro/specs/scheduler-config-version-tag/tasks.md +++ /dev/null @@ -1,50 +0,0 @@ -# Implementation Plan - -- [ ] 1. Add the configuration-identity wire field and the scheduler-core skip/apply decision - - Write failing unit tests first (TDD red) in a new `test_core_config_version.c`: matching nonzero identity values on both sides yields "unchanged"; differing nonzero values yields "changed"; an incoming value of zero yields "changed" regardless of what is stored; a freshly-initialized core (stored value still zero) yields "changed" regardless of the incoming value; a `NULL` incoming configuration yields "changed". - - Add the new configuration-identity field to the config-push wire message (`schema/controller.capnp`'s `Conf4Sched`) and let the existing build regenerate its bindings — no other field on that message changes shape. - - Add a zero-initialized configuration-identity field to the scheduler's core state (`xce_sched_core_t`), alongside its existing self-incrementing local config-version counter. - - Implement the pure comparison function against the five cases above; it must not mutate either input. - - Observable completion: the new test suite goes from failing to passing, and the comparison function returns "changed" for every input combination except two identical nonzero values. - - _Requirements: 1.1, 1.4, 3.3, 3.4_ - -- [ ] 2. Core: populate the identity on push, and skip redundant re-registration on both existing application paths -- [ ] 2.1 (P) Populate the configuration-identity field on every controller-side configuration push - - Extend the existing controller-side config-push test coverage (`controller/testsuite/test_job_array_sched_push.c`) with a failing case first (TDD red) asserting the pushed configuration carries the controller's own current configuration-identity value. - - Populate the field from the controller's existing, already-replicated configuration-identity value — do not compute or derive a new one. - - Observable completion: the new test goes from failing to passing, and every configuration push's wire payload carries a value equal to the controller's current configuration identity at push time. - - _Requirements: 1.1, 1.4_ - - _Depends: 1_ - - _Boundary: Controller configuration push (xcesched_push_configs)_ - -- [ ] 2.2 (P) Skip redundant re-registration on an unchanged join/reconnect configuration push - - Extend the existing join/reconnect configuration-application test coverage (`scheduler/testsuite/test_handlers.c`) with failing cases first (TDD red): a push whose identity matches the currently-held configuration registers no resources, shared resource values, or queues, and emits no error-level log entry; a push whose identity differs is fully registered exactly as today and the scheduler adopts the new identity afterward; the scheduler's first-ever push is always fully registered regardless of the identity value it carries. - - Implement the skip/apply branching at the join/reconnect configuration-application call site using the comparison function from 1: on "unchanged," skip resource/shared-value/queue registration entirely, free the now-redundant incoming configuration payload (it is not adopted), and record one diagnostic-level log entry noting the skip; on "changed," keep today's full registration behavior and additionally adopt the pushed configuration's identity value as currently held. - - Observable completion: the extended test suite goes from failing to passing, and a two-push sequence with identical content produces zero error-level log entries and zero resource/queue registration calls on the second push, with no leaked configuration payload under the project's memory-safety build. - - _Requirements: 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4_ - - _Depends: 1_ - - _Boundary: Scheduler join/reconnect configuration application (Path A)_ - -- [ ] 2.3 (P) Skip redundant re-registration on an unchanged live configuration-upgrade push - - Extend the existing live configuration-upgrade test coverage (`scheduler/testsuite/test_core_events.c`) with the same failing cases as 2.2 (unchanged skips with no error log, changed fully applies and adopts the new identity, first-ever push always applies), exercised through this path's own event-application entry point instead of the join/reconnect one. - - Implement the identical skip/apply branching at this second configuration-application call site, using the same comparison function from 1 and the same core-level stored identity field as 2.2 — both paths must observe and update the same value. - - Observable completion: the extended test suite goes from failing to passing, and a two-push sequence with identical content produces zero error-level log entries and zero resource/queue registration calls on the second push, with no leaked configuration payload under the project's memory-safety build. - - _Requirements: 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4_ - - _Depends: 1_ - - _Boundary: Scheduler live configuration-upgrade application (Path B)_ - -- [ ] 3. Validation: prove the end-to-end fix and confirm no regressions -- [ ] 3.1 Verify the full skip/apply lifecycle end to end and confirm administrator update operations are unaffected - - Write an integration test driving a scheduler instance through a first configuration push (full registration observed), a second, identical push (skip observed: no registration calls, no error-level log entries), and a third push with genuinely different content (full registration observed again, including the previously-skipped resources). - - Run the existing explicit administrator resource/queue update operation tests unmodified and confirm they still pass exactly as today, independent of any configuration-identity comparison. - - Observable completion: the integration test passes end to end without any error-level log entry appearing for the identical second push, and the full existing administrator-update test suite remains green with no changes to its own assertions. - - _Requirements: 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 4.1, 4.2_ - - _Depends: 2.1, 2.2, 2.3_ - -- [ ] 3.2 Close the originally-reported symptom on a live 3-node HA cluster - - Run a live HA soak that triggers at least one failover and confirms the reconnecting scheduler's log contains zero error-level resource/queue-registration entries for that reconnect. - - During the same soak, push a genuinely changed configuration and confirm it is still fully and correctly observed by the scheduler (for example, via its existing resource/worker inspection interface), proving the skip behavior never suppresses a real change. - - Confirm the configuration-identity value observed after a failover to a different, newly-primary controller node matches what the scheduler already held when the underlying configuration content is unchanged — the direct live proof that the identity value is stable across which node is primary. - - Observable completion: the soak log shows zero false error-level entries across every reconnect, and the mid-soak configuration change is confirmed applied. - - _Requirements: 1.2, 1.3, 2.1, 2.2, 2.4, 3.1_ - - _Depends: 3.1_ diff --git a/.kiro/specs/scheduler-controller-msg-gap/design.md b/.kiro/specs/scheduler-controller-msg-gap/design.md deleted file mode 100644 index 172f36cd..00000000 --- a/.kiro/specs/scheduler-controller-msg-gap/design.md +++ /dev/null @@ -1,471 +0,0 @@ -# Design: scheduler-controller-msg-gap - -> Feature: scheduler-controller-msg-gap -> Created: 2026-05-11 -> Status: Design Generated - -## Architecture Pattern & Boundary Map - -**Pattern:** Extension of existing message handler infrastructure. The scheduler uses a two-phase processing model: - -1. **Enqueue phase** (handler.c, runs on IO thread) — lightweight: validate, allocate request struct, enqueue into `pend_updates[0]` -2. **Dequeue phase** (scheduler.c `sched_update()`, runs on main thread) — heavyweight: mutate scheduler state, call plugin hooks, trigger scheduling sessions - -All new handlers follow this pattern. No architectural boundary changes needed. - -**Scope boundary:** Only `src/scheduler/` source files, `src/lib/helper.c`, and `src/include/helper.h` are modified. No controller-side changes. No Cap'n Proto schema changes. - -## Technology Stack & Alignment - -| Aspect | Choice | Rationale | -|--------|--------|-----------| -| Language | C99 | Matches existing codebase convention | -| IPC protocol | Cap'n Proto (existing) | No schema changes — only handler gaps filled | -| Build system | Meson (existing) | No build changes needed | -| Threading model | Same as existing — handler enqueues, main thread dequeues | Preserves thread safety guarantees | -| Logging | Existing `TRACE`/`INFO`/`WARN`/`ERROR` macros | Consistent with codebase | -| Plugin hooks | Follow existing `plugin_*_update()` pattern | Consistency with established plugin API | - -## Components & Interface Contracts - -### Component 1: handler.c — Notify handlers (Req 3, 4, 5, 6) - -Extend `sched_handle_notify()` to process the four currently-stubbed NotifyData events. - -#### `sched_handle_sched_update_job` - -``` -Signature: static int sched_handle_sched_update_job( - xce_scheduler_t* sched, - xce_job_t* job) - -Behavior: - - If job == NULL, return -EINVAL - - Enqueue XCE_SCHED_OP_UPDATE_JOB with job pointer - - On calloc failure, return -ENOMEM - - Return 0 on success -``` - -**Requirement traceability:** 3.1, 3.2, 3.3 - -#### `sched_handle_sched_update_queue` - -``` -Signature: static int sched_handle_sched_update_queue( - xce_scheduler_t* sched, - xce_queue_t* queue) - -Behavior: - - If queue == NULL, return -EINVAL - - Enqueue XCE_SCHED_OP_UPDATE_QUEUE with queue pointer - - On calloc failure, return -ENOMEM - - Return 0 on success -``` - -**Requirement traceability:** 4.1, 4.2, 4.3 - -#### `sched_handle_sched_update_resvals` - -``` -Signature: static int sched_handle_sched_update_resvals( - xce_scheduler_t* sched, - xce_resval_list_t* resvals) - -Behavior: - - If resvals == NULL or count == 0, return 0 - - Enqueue XCE_SCHED_OP_UPDATE_RESVALS with resvals pointer - - On calloc failure, return -ENOMEM - - Return 0 on success -``` - -**Requirement traceability:** 5.1, 5.2, 5.3 - -#### `sched_handle_sched_job_rusage` - -``` -Signature: static int sched_handle_sched_job_rusage( - xce_scheduler_t* sched, - xce_job_rusages_t* rusages) - -Behavior: - - If rusages == NULL, return -EINVAL - - Enqueue XCE_SCHED_OP_JOB_RUSAGE with rusages pointer - - On calloc failure, return -ENOMEM - - Return 0 on success -``` - -**Requirement traceability:** 6.1, 6.2, 6.3 - -#### `sched_handle_notify` switch changes - -The existing switch in `sched_handle_notify()` gains four new cases: - -| NotifyData event | Handler function | Previous behavior | -|---|---|---| -| `NotifyData_data_schedUpdateJob` | `sched_handle_sched_update_job(sched, notify->data.sched_update_job)` | TRACE log only | -| `NotifyData_data_schedUpdateQueue` | `sched_handle_sched_update_queue(sched, notify->data.sched_update_queue)` | Empty `break` | -| `NotifyData_data_schedUpdateResvals` | `sched_handle_sched_update_resvals(sched, notify->data.sched_update_resvals)` | Empty `break` | -| `NotifyData_data_schedJobRusages` | `sched_handle_sched_job_rusage(sched, notify->data.sched_job_rusages)` | Empty `break` | - -### Component 2: handler.c — Dataset handlers (Req 1) - -#### `sched_handle_resources` - -``` -Signature: static int sched_handle_resources( - xce_scheduler_t* sched, - xce_resource_data_t* resdata) - -Behavior: - - If resdata == NULL or resdata->count == 0, return 0 - - Enqueue XCE_SCHED_OP_UPDATE_RESOURCES with resdata pointer - - On calloc failure, return -ENOMEM - - Return 0 on success -``` - -**Requirement traceability:** 1.1, 1.3, 1.4 - -#### `sched_handle_resvals` - -``` -Signature: static int sched_handle_resvals( - xce_scheduler_t* sched, - xce_resval_data_t* resvaldata) - -Behavior: - - If resvaldata == NULL or resvaldata->count == 0, return 0 - - Enqueue XCE_SCHED_OP_UPDATE_RESVALS_FULL with resvaldata pointer - - On calloc failure, return -ENOMEM - - Return 0 on success -``` - -**Requirement traceability:** 1.2, 1.3, 1.4 - -#### `sched_handle_dataset` switch changes - -| Data table | Handler function | Previous behavior | -|---|---|---| -| `Data_data_resources` | `sched_handle_resources(sched, dataset->data.resources)` | Empty `break` | -| `Data_data_resvals` | `sched_handle_resvals(sched, dataset->data.resvals)` | Empty `break` | - -### Component 3: scheduler.h — New op types and update request fields - -Extend `xce_sched_op_t` and `xce_sched_update_request_t`: - -``` -typedef enum { - XCE_SCHED_OP_NONE = 0, - XCE_SCHED_OP_UPDATE_CONFIG, - XCE_SCHED_OP_UPDATE_JOB, - XCE_SCHED_OP_JSTATE_CHANGE, - XCE_SCHED_OP_UPDATE_WORKER, - XCE_SCHED_OP_UPDATE_QUEUE, // NEW - XCE_SCHED_OP_UPDATE_RESOURCES, // NEW - XCE_SCHED_OP_UPDATE_RESVALS_FULL, // NEW — from dataset bulk resvals - XCE_SCHED_OP_UPDATE_RESVALS, // NEW — from notify incremental resvals - XCE_SCHED_OP_JOB_RUSAGE // NEW -} xce_sched_op_t; -``` - -Extend `xce_sched_update_request_t` union: - -``` -struct { - xce_queue_t* queue; // for UPDATE_QUEUE -} queueupdate; -struct { - xce_resource_data_t* resdata; // for UPDATE_RESOURCES -} resourceupdate; -struct { - xce_resval_data_t* resvaldata; // for UPDATE_RESVALS_FULL -} resvalfullupdate; -struct { - xce_resval_list_t* resvals; // for UPDATE_RESVALS (notify) -} resvalupdate; -struct { - xce_job_rusages_t* rusages; // for JOB_RUSAGE -} job_rusage; -``` - -### Component 4: scheduler.c — Dequeue processing in `sched_update()` - -Extend the `switch (req->op)` in `sched_update()` with new cases: - -#### `XCE_SCHED_OP_UPDATE_QUEUE` - -- Find existing queue by name with `sched_get_queue()` -- If found: update mutable fields (status, priority, accept_interval, sched_delay, resreq) -- If not found: return `-ENOENT` — queues are created only via the initial `Conf4Sched` config push, never via notify -- Call `plugin_queue_update()` - -**Requirement traceability:** 4.1, 4.2 - -#### `XCE_SCHED_OP_UPDATE_RESOURCES` - -- Iterate `resdata->resources[]` -- For each: call `makeResource(sched->resmgr, r)` to add or update -- NULL entries skipped (1.3) - -**Requirement traceability:** 1.1 - -#### `XCE_SCHED_OP_UPDATE_RESVALS_FULL` - -- Iterate `resvaldata->vals[]` -- For each: find resource by name in `sched->resmgr`, update value -- Unknown resources skipped with WARN log - -**Requirement traceability:** 1.2 - -#### `XCE_SCHED_OP_UPDATE_RESVALS` - -- Iterate `resvals->vals[]` -- For each: find worker by host name via `sched_get_worker()` -- If found: update worker's `resvals` zarray at the resource index -- Unknown hosts skipped (5.2) - -**Requirement traceability:** 5.1, 5.2, 5.3 - -#### `XCE_SCHED_OP_JOB_RUSAGE` - -- Find job by jobid via `sched_get_job()` -- If not found: return `-ENOENT` (6.2) -- Store rusage data on job object -- Call `plugin_job_rusage_update()` - -**Requirement traceability:** 6.1, 6.2 - -### Component 5: scheduler.c — Job update-in-place (Req 3) - -#### New function: `sched_update_existing_job` - -``` -Signature: static int sched_update_existing_job( - xce_scheduler_t* sched, - xce_job_t* updated) - -Behavior: - - Find existing job by jobid via sched_get_job() - - If not found: return -ENOENT — jobs are created only via - the Data_data_schedjobs dataset push, never via notify - - If found: update mutable fields (state, priority, allocs) - - Call plugin_job_update() - - Return 0 on success, negative on error -``` - -**Requirement traceability:** 3.1, 3.2, 3.3 - -This is wired into the existing `XCE_SCHED_OP_UPDATE_JOB` case — no new op type needed since `sched_on_jobs()` already creates new jobs. The change is to make `sched_update_jobs()` also handle job updates for existing jobids by replacing rather than just adding. - -### Component 6: plugin.h / plugin.c — New rusage hook (Req 6) - -Add to `plugin_hooks_t`: - -``` -typedef int (*plugin_job_rusage_hook_t)(void*, xce_sched_job_t*, xce_job_rusages_t*); -``` - -Add to `plugin_hooks_t` struct: - -``` -plugin_job_rusage_hook_t rusage_hook; -``` - -Add global function: - -``` -int plugin_job_rusage_update(void* sched, xce_sched_job_t* job, xce_job_rusages_t* rusages); -``` - -This follows the exact pattern of `plugin_job_state_change()` — iterate registered plugins, call their hook if non-NULL. - -### Component 7: handler.c — Dead code removal (Req 2) - -Remove from `scheduler_req_handler()`: -- The entire `Request_body_pushmsg` case block (lines 185–198) -- The static functions `sched_handle_jobpush()` and `sched_handle_workerpush()` - -These are unreachable because `vtask_entry()` in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg`. - -**Requirement traceability:** 2.1, 2.2 - -### Component 8: Name-lookup functions (Req 8) - -> **Layering correction (bug-765 G-7, 2026-06-18):** these two helpers live in DIFFERENT layers. `xce_notify_event_name(int)` is lib-level and lives in `src/lib/helper.c` / `src/include/helper.h`. The scheduler-op name lookup operates on `xce_sched_op_t`, a **scheduler-only** type (`scheduler.h`); `src/lib` must not include scheduler headers (no layering inversion), so it is a **static `sched_op_name(xce_sched_op_t)` in `src/scheduler/src/scheduler.c`**, NOT `xce_sched_op_name` in lib/helper. The original "both in helper.c" text below is superseded for the scheduler-op helper. - -#### New function: `xce_notify_event_name` (lib — `src/lib/helper.c`) - -``` -Signature: char* xce_notify_event_name(int event) - -Behavior: Switch onNotifyData_data_* enum values, return symbolic string. -Fallback: "UNKNOWN(%d)" with numeric substitution. -``` - -Matches pattern of `xce_dataset_table()` — takes int, returns `char*`. - -#### Scheduler-op name lookup: static `sched_op_name` (scheduler — `src/scheduler/src/scheduler.c`) - -``` -Signature: static char* sched_op_name(xce_sched_op_t op) - -Behavior: Switch on XCE_SCHED_OP_* enum values, return symbolic string. -Fallback: "UNKNOWN(%d)" with numeric substitution. -``` - -Kept in the scheduler layer (NOT lib/helper as `xce_sched_op_name`) because `xce_sched_op_t` is a scheduler-only type and `src/lib` must not depend on scheduler headers — see the layering correction above. Required for Requirement 7's error logging in `sched_update()`. - -#### Changes to `handler.c` logging - -| Current | Changed to | -|---|---| -| `TRACE("%s table = %d", __func__, dataset->table)` | `TRACE("%s table = %s", __func__, xce_dataset_table(dataset->table))` | -| `TRACE("%s...", __func__)` (in stub handlers) | `INFO("%s event = %s jobid = %s", __func__, xce_notify_event_name(notify->event), jobidString)` | - -**Requirement traceability:** 8.1, 8.2, 8.3, 8.4 - -### Component 9: scheduler.c — Error logging in `sched_update()` (Req 7) - -In the `sched_update()` main loop, after the `switch (req->op)` block, when `rc < 0`: - -``` -WARN("%s: op %s failed: %s (%d)", __func__, - sched_op_name(req->op), strerror(-rc), rc); -``` - -**Requirement traceability:** 7.1 - -### Component 10: schedobj.c — Queue update function (Req 4) - -#### New function: `sched_update_existing_queue` - -``` -Signature: int sched_update_existing_queue( - xce_scheduler_t* sched, - xce_queue_t* updated) - -Behavior: - - Find existing queue by name via sched_get_queue() - - If not found: return -ENOENT — queues are created only via - the initial Conf4Sched config push, never via notify - - If found: update mutable fields: - sched_queue->status = updated->status - sched_queue->priority = updated->priority - sched_queue->accept_interval = updated->accept_interval - sched_queue->sched_delay = updated->sched_delay - // resreq is a string field — reallocate and replace - - Call plugin_queue_update() - - Return 0 on success, negative on error -``` - -**Requirement traceability:** 4.1, 4.2 - -### Component 11: `sched_pend_update_killer` updates (scheduler.c) - -Extend the `sched_pend_update_killer` destructor to handle new op types: - -``` -case XCE_SCHED_OP_UPDATE_QUEUE: - // queue pointer belongs to scheduler, no free needed - break; -case XCE_SCHED_OP_UPDATE_RESOURCES: - // resource_data rows belong to scheduler, no free needed - break; -case XCE_SCHED_OP_UPDATE_RESVALS_FULL: - // resval_data rows owned by capnp decode, freed by free_Request - break; -case XCE_SCHED_OP_UPDATE_RESVALS: - // resval_list owned by capnp decode, freed by free_Request - break; -case XCE_SCHED_OP_JOB_RUSAGE: - // rusage struct owned by capnp decode, freed by free_Request - break; -``` - -**Important:** Memory ownership must be traced carefully. For `XCE_SCHED_OP_UPDATE_JOB` and `XCE_SCHED_OP_UPDATE_WORKER`, the existing pattern nullifies `dataset->nrows` and `dataset->rows` after enqueue to transfer ownership. New queue/rusage/resvals handlers should follow the same pattern — the request struct owns the data, and `free_Request()` in `xcectx.c` handles cleanup. - -## Data Flow Diagrams - -### Notify message processing flow - -``` -Controller (sched.c) Scheduler (handler.c) Scheduler (scheduler.c) -───────────────────── ───────────────────── ───────────────────── -xce_notify_req() - │ - ├── schedStartJob ──────────► sched_job_state_change() ──► enq_pend_update(JSTATE_CHANGE) - │ │ - ├── schedExitJob ───────────► sched_job_state_change() ──► │ - │ │ - ├── schedFailJob ───────────► sched_job_state_change() ──► │ - │ │ - ├── schedUpdateJob ────────► sched_handle_sched_update_job() ─► enq_pend_update(UPDATE_JOB) [NEW] - │ │ - ├── schedUpdateQueue ──────► sched_handle_sched_update_queue() ─► enq_pend_update(UPDATE_QUEUE) [NEW] - │ │ - ├── schedUpdateResvals ────► sched_handle_sched_update_resvals() ─► enq_pend_update(UPDATE_RESVALS) [NEW] - │ │ - └── schedJobRusages ───────► sched_handle_sched_job_rusage() ──► enq_pend_update(JOB_RUSAGE) [NEW] - │ - ▼ - sched_update() main loop - ├── UPDATE_JOB ──► sched_update_existing_job() - ├── UPDATE_QUEUE ──► sched_update_existing_queue() - ├── UPDATE_RESVALS ──► update worker resvals - ├── UPDATE_RESOURCES ──► update resmgr [NEW] - ├── UPDATE_RESVALS_FULL ──► update sharedInst [NEW] - └── JOB_RUSAGE ──► store on job + plugin hook [NEW] -``` - -### Dataset message processing flow - -``` -Controller (dataex.c) Scheduler (handler.c) Scheduler (scheduler.c) -──────────────────── ───────────────────── ───────────────────── -xce_push_data_req() - │ - ├── Data_data_schedjobs ──► sched_on_jobs() ──────────► enq_pend_update(UPDATE_JOB) - │ │ - ├── Data_data_schedworkers ► sched_on_workers() ───────► enq_pend_update(UPDATE_WORKER) - │ │ - ├── Data_data_resources ──► sched_handle_resources() ─► enq_pend_update(UPDATE_RESOURCES) [NEW] - │ │ - └── Data_data_resvals ────► sched_handle_resvals() ───► enq_pend_update(UPDATE_RESVALS_FULL) [NEW] - │ - ▼ - sched_update() main loop -``` - -## File Change Summary - -| File | Change Type | Requirements | -|------|-------------|-------------| -| `src/scheduler/src/handler.c` | Modified — add handlers, remove dead code, update logging | 1, 2, 3, 4, 5, 6, 8 | -| `src/scheduler/include/scheduler.h` | Modified — extend enums and structs | 1, 3, 4, 5, 6 | -| `src/scheduler/src/scheduler.c` | Modified — add dequeue cases, error logging, update functions | 1, 3, 4, 5, 6, 7 | -| `src/scheduler/include/plugin.h` | Modified — add rusage hook typedef and global function | 6 | -| `src/scheduler/src/plugin.c` | Modified — add `plugin_job_rusage_update()` implementation | 6 | -| `src/scheduler/src/schedobj.c` | Modified — add `sched_update_existing_queue()` | 4 | -| `src/lib/helper.c` | Modified — add `xce_notify_event_name()` (lib-level, int arg); the scheduler-op name lookup is a static `sched_op_name()` in `scheduler.c` instead — see Component 8 layering note | 8 | -| `src/include/helper.h` | Modified — declare `xce_notify_event_name()` | 8 | -| `src/scheduler/testsuite/test.c` | Modified — add unit tests for all new handlers | All | - -## Build Order - -1. `src/include/helper.h` — add declarations (no dependencies) -2. `src/lib/helper.c` — add implementations (depends on step 1) -3. `src/scheduler/include/scheduler.h` — extend enums and structs (depends on typemap headers) -4. `src/scheduler/include/plugin.h` — add rusage hook typedef (depends on xce/types.h) -5. `src/scheduler/src/plugin.c` — add `plugin_job_rusage_update()` (depends on step 4) -6. `src/scheduler/src/schedobj.c` — add `sched_update_existing_queue()` (depends on step 3) -7. `src/scheduler/src/scheduler.c` — add dequeue processing, error logging, update functions (depends on steps 3–6) -8. `src/scheduler/src/handler.c` — add handlers, remove dead code, update logging (depends on steps 1–7) -9. `src/scheduler/testsuite/test.c` — add unit tests (depends on all above) - -## Risks & Mitigations - -| Risk | Severity | Mitigation | -|------|----------|-----------| -| Memory ownership of queued data — capnp-decoded structs freed before dequeue processes them | High | Follow existing pattern: for dataset paths, nullify `dataset->nrows`/`dataset->rows` after enqueue to transfer ownership. For notify paths, copy essential data into the request struct rather than holding pointers to capnp memory. | -| `sched_update_existing_job` must handle concurrent modification — handler thread enqueues while main thread processes | Medium | Already mitigated by existing double-buffer swap in `sched_update()`. The handler thread only touches `pend_updates[0]` while the main thread drains `pend_updates[1]`. | -| New `UPDATE_QUEUE` op requires queue name lookup — potential lock contention | Low | `sched_get_queue()` already uses read lock. Queue updates are infrequent. | -| `sched_op_name` (static, scheduler.c) must handle future enum additions | Low | Follow `xce_dataset_table` pattern — return `"UNKNOWN(%d)"` for unrecognized values. | \ No newline at end of file diff --git a/.kiro/specs/scheduler-controller-msg-gap/requirements.md b/.kiro/specs/scheduler-controller-msg-gap/requirements.md deleted file mode 100644 index b3096be1..00000000 --- a/.kiro/specs/scheduler-controller-msg-gap/requirements.md +++ /dev/null @@ -1,110 +0,0 @@ -# Requirements: scheduler-controller-msg-gap - -> Generated: 2026-05-11T01:34:00-07:00 -> Updated: 2026-05-11T02:00:00-07:00 -> Feature: scheduler-controller-msg-gap -> Status: Requirements Generated - -## Overview - -Analyze the codebase to find gaps in the scheduler's handling of controller messages — missing message types, unhandled edge cases, or incomplete message processing paths. - -## Codebase Analysis Summary - -The scheduler (`src/scheduler/src/handler.c`) receives controller messages through the `scheduler_req_handler` dispatcher, which handles two top-level opcodes: - -- **OPCode_push** — carries ConfigData or Dataset (`datamsg`) messages. Raw `pushmsg` packets are reassembled into `datamsg` by the `vtask_entry()` assembly layer in `src/lib/xcectx.c` (lines 93–106); the handler never sees individual `pushmsg` fragments. -- **OPCode_notify** — carries NotifyData event messages - -The controller (`src/controller/sched.c`) sends six NotifyData event types and two Data table types. The scheduler's handler has **6 gap locations** where the case is present but stubbed out or empty-bodied. The `Request_body_pushmsg` branch in `scheduler_req_handler` (which calls `sched_handle_jobpush`/`sched_handle_workerpush`) is dead code — it can never be reached because `vtask_entry` rewrites all `pushmsg` into `datamsg` before invoking the handler. - -## Requirements - -### Requirement 1: Resource data message handling - -The scheduler SHALL process `Data_data_resources` dataset messages received from the controller. Upon receipt, the system SHALL update the internal resource catalog (resource name, type, flags, ordering type, interval) in the scheduler's resource manager (`sched->resmgr`), reconciling additions, removals, and modifications against the existing set. - -The scheduler SHALL process `Data_data_resvals` dataset messages received from the controller. Upon receipt, the system SHALL update the shared resource value array (`sched->sharedInst`) with the new values indexed by position, and SHALL reset the `sharedInUse` tracking array for the next scheduling session. - -**Acceptance criteria:** -- EARS-UB-1.1: WHEN the scheduler receives a dataset with `table == Data_data_resources`, THEN the system shall add or update each resource in `sched->resmgr` -- EARS-UB-1.2: WHEN the scheduler receives a dataset with `table == Data_data_resvals`, THEN the system shall update `sched->sharedInst` at each value's index position -- EARS-UB-1.3: WHEN either handler encounters a NULL element in the dataset rows, THEN the system shall skip that element and continue processing remaining rows -- EARS-UB-1.4: WHEN either handler encounters a memory allocation failure, THEN the system shall return `-ENOMEM` and free any partially allocated structures - -### Requirement 2: Dead code removal — pushmsg handler branches - -The scheduler SHALL remove the dead `Request_body_pushmsg` branch in `scheduler_req_handler` and the stub functions `sched_handle_jobpush` and `sched_handle_workerpush`. These paths can never execute because the `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` packets into `datamsg` before invoking the request handler. - -**Acceptance criteria:** -- EARS-UB-2.1: WHEN `scheduler_req_handler` is compiled, THEN the `Request_body_pushmsg` case and its two child branches (job/worker push) shall not exist -- EARS-UB-2.2: WHEN the scheduler is linked, THEN the symbols `sched_handle_jobpush` and `sched_handle_workerpush` shall not be present in the binary - -### Requirement 3: NotifyData event handling for job updates - -The scheduler SHALL process `NotifyData_data_schedUpdateJob` notify messages beyond the current trace-log-only behavior. Upon receipt, the system SHALL locate the job in `sched->jobs` by jobid, update the job's mutable fields (state, priority, queue, resource request) from the notify payload, and invoke `plugin_job_update` to notify plugins of the change. - -**Acceptance criteria:** -- EARS-UB-3.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedUpdateJob`, THEN the system shall find the existing job by jobid and update its mutable fields -- EARS-UB-3.2: WHEN the jobid in the update notify does not match any existing job (i.e., the job was never pushed via the initial dataset bulk load), THEN the system shall return `-ENOENT` — jobs are created only via the `Data_data_schedjobs` dataset path, never via notify -- EARS-UB-3.3: WHEN the update payload carries job data that is NULL, THEN the system shall return `-EINVAL` - -### Requirement 4: NotifyData event handling for queue updates - -The scheduler SHALL process `NotifyData_data_schedUpdateQueue` notify messages from the controller. Upon receipt, the system SHALL locate the queue in `sched->queues` by name and update its status, priority, accept_interval, sched_delay, and resource request fields from the notify payload. - -**Acceptance criteria:** -- EARS-UB-4.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedUpdateQueue`, THEN the system shall find the existing queue by name and update its mutable configuration fields -- EARS-UB-4.2: WHEN the queue name in the update notify does not match any existing queue (i.e., the queue was never loaded via the initial config push), THEN the system shall return `-ENOENT` — queues are created only via the `Conf4Sched` config path, never via notify -- EARS-UB-4.3: WHEN the update payload carries queue data that is NULL, THEN the system shall return `-EINVAL` - -### Requirement 5: NotifyData event handling for resource value updates - -The scheduler SHALL process `NotifyData_data_schedUpdateResvals` notify messages from the controller. Upon receipt, the system SHALL iterate the `xce_resval_list_t` payload, and for each resource value, update the per-worker `resvals` array on the named worker, matching by host name. - -**Acceptance criteria:** -- EARS-UB-5.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedUpdateResvals`, THEN the system shall update each named worker's `resvals` array with the new values -- EARS-UB-5.2: WHEN a named host in the updates does not correspond to any known worker, THEN the system shall skip that update entry and continue processing remaining entries -- EARS-UB-5.3: WHEN the update payload carries zero resource values, THEN the system shall return 0 without modifying state - -### Requirement 6: NotifyData event handling for job resource usage - -The scheduler SHALL process `NotifyData_data_schedJobRusages` notify messages from the controller. Upon receipt, the system SHALL locate the job by jobid, store the resource usage data (`xce_sched_job_rusages_t`) on the job object, and invoke `plugin_job_rusage_update` to notify plugins of the usage change. - -**Acceptance criteria:** -- EARS-UB-6.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedJobRusages`, THEN the system shall store the usage data on the matching job -- EARS-UB-6.2: WHEN the jobid in the update does not match any existing job, THEN the system shall return `-ENOENT` without modifying state -- EARS-UB-6.3: WHEN the rusage payload is NULL, THEN the system shall return `-EINVAL` - -### Requirement 7: Error handling consistency - -The scheduler SHALL return meaningful error codes from all message handlers that enable the controller to distinguish between transient failures (ENOMEM) and permanent protocol errors (EINVAL, ENOENT). The scheduler's main loop in `sched_update()` SHALL log a WARN-level message when a handler returns a non-zero error code, including the offending opcode and sub-type for operational debugging. - -**Acceptance criteria:** -- EARS-UB-7.1: WHEN a handler returns a non-zero error code, THEN `sched_update()` shall emit a WARN log including the `xce_sched_op_t` value and the error code -- EARS-UB-7.2: WHEN an unknown notify event type is received, THEN the handler shall return `-EINVAL` (already implemented — verify non-regression) -- EARS-UB-7.3: WHEN a push message arrives with an unknown `body_which` tag, THEN the handler shall return `-EINVAL` (already implemented — verify non-regression) - -### Requirement 8: Human-readable message type names in logs - -The scheduler SHALL log message type identifiers using symbolic names rather than numeric values. The existing helpers in `src/lib/helper.c` — `xce_request_op_name()`, `xce_request_msg_name()`, and `xce_dataset_table()` — already provide name lookups for OPCode, Request body tag, and Data table type. A new `xce_notify_event_name(int)` function SHALL be added to cover the NotifyData event type enum. All TRACE, DEBUG, INFO, and WARN messages in `handler.c` that reference these enums SHALL use these name-lookup functions instead of printing numeric values. - -**Acceptance criteria:** -- EARS-UB-8.1: WHEN `scheduler_req_handler` logs a message that includes an opcode, THEN the log shall call `xce_request_op_name()` to show the symbolic name (e.g., `"push"`) rather than the numeric value -- EARS-UB-8.2: WHEN `sched_handle_dataset` logs a message that includes a data table type, THEN the log shall call `xce_dataset_table()` to show the symbolic name (e.g., `"schedjobs"`) rather than the numeric value (currently `TRACE("%s table = %d", ...)`) -- EARS-UB-8.3: WHEN `sched_handle_notify` logs a message that includes a notify event type, THEN the log shall call `xce_notify_event_name()` to show the symbolic name rather than the numeric value -- EARS-UB-8.4: WHEN `xce_notify_event_name()` receives a numeric value that does not correspond to a known NotifyData enum constant, THEN it shall return `"UNKNOWN(%d)"` with the numeric value substituted (matching the pattern used by the existing helpers) -- EARS-UB-8.5: WHEN `sched_update()` emits a WARN log for a handler error (per Requirement 7), THEN the log shall include symbolic names via the appropriate helper function - -## Non-Requirements - -The following areas are intentionally excluded from this specification: -- **New protocol message types** — only existing, already-defined message types in `controller.capnp` are in scope -- **Controller-side changes** — only the scheduler's inbound message handling is in scope -- **Plugin implementation details** — plugin update functions are called but not modified -- **Bulk (Dataset) job/worker push already works** — the `Data_data_schedjobs` and `Data_data_schedworkers` cases in `sched_handle_dataset` already delegate to `sched_on_jobs`/`sched_on_workers` -- **`pushmsg` handler path** — the `Request_body_pushmsg` branch in `scheduler_req_handler` and the functions `sched_handle_jobpush`/`sched_handle_workerpush` are dead code; the `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg` before the handler is called, so incremental push handling is never needed - -## Next Step - -Run `/kiro:spec-design scheduler-controller-msg-gap` to proceed to design phase. diff --git a/.kiro/specs/scheduler-controller-msg-gap/research.md b/.kiro/specs/scheduler-controller-msg-gap/research.md deleted file mode 100644 index 0d36f521..00000000 --- a/.kiro/specs/scheduler-controller-msg-gap/research.md +++ /dev/null @@ -1,304 +0,0 @@ -# Research Log: scheduler-controller-msg-gap - -> Feature: scheduler-controller-msg-gap -> Created: 2026-05-11 - -## Discovery Type - -Extension (light discovery) — extending existing handler code in an established C codebase. - -## Summary - -The scheduler's `handler.c` dispatcher processes controller messages via `scheduler_req_handler`. Six message cases are stubbed out or empty-bodied. The `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg`, making the `Request_body_pushmsg` branch dead code. Existing plugin hooks (`plugin_job_update`, `plugin_queue_update`, `plugin_worker_update`, `plugin_job_state_change`) exist for notifying plugins of state changes. A `plugin_job_rusage_update` hook does not yet exist and must be added. The `src/lib/helper.c` file already provides `xce_request_op_name()`, `xce_request_msg_name()`, and `xce_dataset_table()` — a new `xce_notify_event_name()` function is needed. - -## Research Log - -### Topic 1: Plugin Hook System - -**Source:** `src/scheduler/include/plugin.h`, `src/scheduler/src/plugin.c` - -**Findings:** -- Hooks follow a `plugin_*_update(void* sched, xce_sched_*_t*)` pattern -- Called after state mutation to notify registered plugins -- Hooks iterate registered plugins and call their function pointers -- `plugin_job_state_change(void*, xce_sched_job_t*, uint32_t)` takes an extra state parameter -- No `plugin_job_rusage_update` hook exists yet — must be added - -**Implication:** New handlers should call the corresponding plugin hook after mutating scheduler state. The rusage handler needs a new hook typedef and global function. - -### Topic 2: Object Conversion Functions - -**Source:** `src/scheduler/src/schedobj.c` - -**Findings:** -- `make_sched_job(void* arg, xce_job_t* j)` — maps jobid, times, state, user/queue/project pointers, resreq, mintasks/maxtasks, allocs, accounts -- `make_sched_queue(void* arg, xce_queue_t* q)` — maps name, attrs, status, priority, accept_interval, sched_delay, resreq -- `make_sched_worker(void* arg, xce_worker_t* w)` — maps name, status, idx, resvals zarray from `w->resvals_to_sched` -- `kill_sched_*` functions free their respective objects - -**Implication:** Job/queue/worker update handlers should use `make_sched_*` for create-or-update semantics. For updates (existing objects), we need update-in-place logic since `make_sched_*` always creates new objects. - -### Topic 3: Resource and Shared Instance Initialization - -**Source:** `src/scheduler/src/scheduler.c` - -**Findings:** -- `sched_init_resources()` — iterates `config->resources[]`, calls `makeResource(sched->resmgr, r)` for each, warns on NULL/duplicate -- `sched_init_sharedInst()` — iterates `config->resvals[]`, uses `MOVE_PTR` to steal val pointers into `sched->sharedInst` zarray -- Both called from `sched_init_config()` during initial config push -- For runtime updates, we need incremental versions that can add new or update existing resources - -**Implication:** Dataset handlers for `Data_data_resources` and `Data_data_resvals` need new update functions that can both create and modify existing entries. - -### Topic 4: Existing Helper Functions - -**Source:** `src/lib/helper.c`, `src/include/helper.h` - -**Findings:** -- `xce_request_op_name(xce_request_t* req)` — returns string for OPCode enum -- `xce_request_msg_name(xce_request_t* req)` — returns string for Request body tag -- `xce_dataset_table(int table)` — returns string for Data table type enum -- All return `"unknown"` for unrecognized values -- `xce_notify_event_name()` does NOT exist yet — must be created -- Signature pattern: `char* func_name(enumerated_type_or_struct_ptr)` returning `char*` - -**Implication:** Create `xce_notify_event_name(int event)` following same pattern. Also consider `char* xce_sched_op_name(xce_sched_op_t op)` for the update request op enum. - -### Topic 5: Cap'n Proto Enum Values (NotifyData) - -**Source:** `src/schema/controller.capnp` - -**Findings:** -All 19 NotifyData event tags: -- `jobPidInfo(0)`, `jobFailInfo(1)`, `jobDispatchFail(2)`, `jobFinishInfo(3)`, `schedStartJob(4)`, `schedExitJob(5)`, `schedFailJob(6)`, `schedUpdateJob(7)`, `schedUpdateResvals(8)`, `stoppedJob(9)`, `resumedJob(10)`, `ckptinfo(11)`, `schedUpdateQueue(12)`, `taskPidInfo(13)`, `taskFailInfo(14)`, `taskFinish(15)`, `postDoneJob(16)`, `schedJobRusages(17)`, `updateJob(18)` - -**Implication:** The `xce_notify_event_name()` function must cover all 19 values. Only the scheduler-relevant ones need handler implementations (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages are the gap items). - -### Topic 6: Dead Code Analysis - -**Source:** `src/lib/xcectx.c` lines 93-106, `src/scheduler/src/handler.c` lines 172-197 - -**Findings:** -- `vtask_entry()` accumulates `pushmsg` rows into `ctx->dataset`, then rewrites `body_which = Request_body_datamsg` and `body.datamsg = ctx->dataset` -- The `Request_body_pushmsg` case in `scheduler_req_handler` (lines 185-198) calls `sched_handle_jobpush()` and `sched_handle_workerpush()`, both of which are stubs returning 0 -- These functions can never be reached via normal message flow - -**Implication:** Safe to remove the pushmsg case branch and both stub functions. - -## Architecture Decisions - -1. **Handler pattern:** New notify handlers follow the same enqueue-to-pend-updates pattern as `sched_on_job_state_change` — create a `xce_sched_update_request_t`, set the op, enqueue via `enq_pend_update()`, process in `sched_update()` main loop. - -2. **Update-in-place for job/queue/worker:** For NotifyData update events, we need update semantics (find existing, mutate fields) rather than create semantics (make_sched_*). New `sched_update_job`, `sched_update_queue`, `sched_update_worker` functions will handle this. - -3. **New XCE_SCHED_OP enum values:** Need `XCE_SCHED_OP_UPDATE_RESVALS` for resource value updates and `XCE_SCHED_OP_JOB_RUSAGE` for rusage updates. - -4. **Plugin hook for rusage:** Add `plugin_job_rusage_hook_t` typedef and `plugin_job_rusage_update()` global function following existing patterns. - -5. **Helper function:** Add `xce_notify_event_name(int)` and `xce_sched_op_name(xce_sched_op_t)` to `src/lib/helper.c` and `src/include/helper.h`. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **scheduler-controller-msg-gap** is **~100% complete**. All 9 task groups, 28 subtasks (1.1–9.9) are `[x]` (9.9 optional). -- **No remaining gaps**: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (`schedUpdateJob`, `schedUpdateQueue`, `schedUpdateResvals`, `schedJobRusages`, `Data_data_resources`, `Data_data_resvals`) are wired with full business logic. Dead code (`pushmsg` branches, `sched_handle_jobpush`, `sched_handle_workerpush`) removed. Symbolic logging via `xce_notify_event_name()` and `xce_sched_op_name()` is in place. -- **Optional gap**: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness. -- **Cross-spec relationship**: `resval-id-decoupling` uses the same pattern (name-based resource lookup via `sched->resmgr`) in `sched_init_sharedInst`. The `XCE_SCHED_OP_UPDATE_RESVALS_FULL` handler added in this spec also resolves by name — consistency is maintained. -- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred. ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -> Method: per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Every requirement re-verified against current source on `persistent_ha` (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c, resobj.c, controller sched.c, testsuite/test.c). This pass **revises the 2026-06-08 audit's "~100% complete" conclusion**: handler wiring is complete, but four acceptance criteria are only partially met, two latent bugs exist in the dequeue paths, payload memory is leaked for the new op types, and the Task 9 test checkboxes substantially overstate actual coverage. - -## 1. Current State Summary - -- All 7 controller-emitted NotifyData events (`src/controller/sched.c`: schedStartJob, schedExitJob, schedFailJob, schedUpdateJob, schedJobRusages, schedUpdateResvals, schedUpdateQueue) have non-stub scheduler handlers in `src/scheduler/src/handler.c` (`sched_handle_notify`, lines 275–323). Unknown events return `-EINVAL`. -- Both previously-empty dataset cases (`Data_data_resources`, `Data_data_resvals`) are wired in `sched_handle_dataset` (handler.c:130–144) with ownership transfer (`dataset->nrows = 0; dataset->rows = NULL`). -- Dead `Request_body_pushmsg` branch and `sched_handle_jobpush`/`sched_handle_workerpush` are fully removed; the only remaining reference is a comment in testsuite/test.c:459. Unknown `body_which` returns `-EINVAL` (handler.c:349–351). -- The enqueue→dequeue two-phase pattern is followed: all new handlers enqueue `xce_sched_update_request_t` via `enq_pend_update()`; `sched_update()` (scheduler.c:727–845) processes the five new ops on the main thread. -- Note: no controller code currently **pushes** `Data_data_resources`/`Data_data_resvals` to the scheduler — `src/controller/sched.c` pushes only schedjobs/schedworkers; `src/controller/opget.c` serves these tables only as get-responses. The Req 1 handlers are forward-looking (harmless, but untriggered in production today). - -## 2. Requirement-to-Asset Map - -| Req / AC | Asset(s) | Status | Notes | -|---|---|---|---| -| 1 / UB-1.1 (resources add **or update**) | handler.c:58–79 `sched_handle_resources`; scheduler.c:770–781 `XCE_SCHED_OP_UPDATE_RESOURCES` | **Partial** | Dequeue calls `makeResource()`, but `makeResource_` (src/lib/resobj.c:108–140) returns NULL with `EEXIST` when the name already exists — **update semantics are missing**; re-pushed catalogs silently fail to modify existing resources. Return value is also ignored (no log), violating the error-handling steering ("never silently swallow errors"). | -| 1 / UB-1.2 (resvals → sharedInst by index) | handler.c:81–102; scheduler.c:782–799 `UPDATE_RESVALS_FULL` | Implemented | Resolves resource by name via `key_find(sched->resmgr,…)`, puts at `r->idx` — consistent with `sched_init_sharedInst` and the resval-id-decoupling spec. Unknown resources WARN-and-skip. | -| 1 / UB-1.3 (skip NULL rows) | scheduler.c:773–775, 786–788 | Implemented | `continue` on NULL entries in both loops. | -| 1 / UB-1.4 (-ENOMEM, free partial) | handler.c calloc guards | Implemented | Handlers return `-ENOMEM`; no partial allocations to free at enqueue level. | -| 2 / UB-2.1, UB-2.2 (dead code gone) | handler.c:325–362 | **Implemented** | `Request_body_pushmsg` case and both stub functions absent from source; `grep` finds no definitions anywhere in src/scheduler/. | -| 3 / UB-3.1 (update mutable fields: state, priority, queue, resreq) | scheduler.c:707–725 `sched_update_existing_job` | **Partial** | Only `job->state` is copied from the update payload. Priority, queue, and resource-request fields named in the requirement are not updated. `plugin_job_update()` is called. | -| 3 / UB-3.2 (-ENOENT unknown jobid) | scheduler.c:716–718 | Implemented | Wait — caveat: `sched_update_existing_job` exists but the `XCE_SCHED_OP_UPDATE_JOB` dequeue case routes to `sched_update_jobs()` (scheduler.c:754–756), which is the bulk create path. Whether a notify-driven single-job update reaches `sched_update_existing_job` needs verification (see Research Needed R-2). | -| 3 / UB-3.3 (-EINVAL NULL payload) | handler.c:180–182 | Implemented | | -| 4 / UB-4.1, 4.2, 4.3 (queue update) | schedobj.c:485–511 `sched_update_existing_queue`; scheduler.c:767–769 | Implemented | status/priority/accept_interval/sched_delay/resreq all updated; resreq freed + reallocated; `-ENOENT` on unknown queue; `-EINVAL` on NULL; `plugin_queue_update()` called. | -| 5 / UB-5.1 (per-worker resvals update) | scheduler.c:801–818 `UPDATE_RESVALS` | **Partial — bug** | `zarray_put(&w->resvals, 0, rv->val)` writes **index 0 unconditionally**. Worker resvals are indexed by resource idx (`zarray_put(&(worker->resvals), r->idx, val)`, schedobj.c:291) and read back via `zarray_get(&w->resvals, i)` in resource.c:161. The handler must resolve `rv->name` → `sched->resmgr` → `r->idx` and put at that index; as written, every update clobbers slot 0 regardless of resource. | -| 5 / UB-5.2 (skip unknown host) | scheduler.c:808–811 | Implemented | `sched_get_worker()` NULL → `continue`. | -| 5 / UB-5.3 (zero values → return 0) | handler.c:235–237 | Implemented | NULL or `count == 0` short-circuits before enqueue. | -| 6 / UB-6.1 (**store** rusage on job + plugin) | scheduler.c:819–831 `JOB_RUSAGE`; plugin.c:361 `plugin_job_rusage_update` | **Partial** | Plugin hook is implemented and called, but the rusage data is **not stored on the job object** as the requirement states — after the plugin call the payload is dropped (and leaked, see §3). | -| 6 / UB-6.2 (-ENOENT unknown jobid) | scheduler.c:822–824 | Implemented | | -| 6 / UB-6.3 (-EINVAL NULL rusage) | handler.c:257–259 | Implemented | | -| 7 / UB-7.1 (WARN with op + errno) | scheduler.c:838–841 | Implemented | `WARN("%s: op %s failed: %s (%d)", …, sched_op_name(req->op), strerror(-rc), rc)` after the switch. | -| 7 / UB-7.2 (unknown notify → -EINVAL) | handler.c:315–317 | Implemented (non-regression confirmed) | | -| 7 / UB-7.3 (unknown body_which → -EINVAL) | handler.c:349–351 | Implemented (non-regression confirmed) | | -| 8 / UB-8.1 (op name in req handler log) | handler.c:330 | Implemented | `TRACE("%s op = %s", __func__, xce_request_op_name(request))`. | -| 8 / UB-8.2 (dataset table name) | handler.c:109 | Implemented | `xce_dataset_table(dataset->table)`. | -| 8 / UB-8.3 (notify event name) | handler.c:279 | Implemented | `xce_notify_event_name(notify->event)`. | -| 8 / UB-8.4 (UNKNOWN(%d) fallback) | src/lib/helper.c:367–415 | Implemented | All 19 NotifyData tags covered + `snprintf(buf, …, "UNKNOWN(%d)", event)`. | -| 8 / UB-8.5 (symbolic op name in WARN) | scheduler.c:685–706 `sched_op_name` | Implemented — **deviation** | Implemented as **static `sched_op_name()` in scheduler.c**, not `xce_sched_op_name()` in `src/lib/helper.c`/`src/include/helper.h` as design Component 8 and tasks 1.1/1.3 (both `[x]`) state. The deviation is arguably correct (the enum is scheduler-internal; libxce cannot see `xce_sched_op_t`), but tasks.md and design.md no longer reflect reality. Uses a non-thread-safe `static char buf[32]` — acceptable since only the main thread calls it; same pattern as the helper. | -| 9 (tests) | src/scheduler/testsuite/test.c | **Partial — checkboxes overstate coverage** | See §4. | - -**Memory-ownership constraint (design risk realized):** `sched_handle_notify` nullifies `notify->data.*` after successful enqueue (handler.c:297–315), transferring ownership from `free_Request()` to the update request — but `sched_pend_update_killer` (scheduler.c:311–321) has empty `break;` for all five new ops and the dequeue cases never free the payloads. Result: every `schedUpdateQueue` leaks an `xce_queue_t`, every `schedUpdateResvals` leaks the `xce_resval_list_t` (entries + name/host strings; only `rv->val` survives into the zarray — and on the index-0 bug path even that gets clobbered without a free), every `schedJobRusages` leaks the rusages struct, and every resources/resvals dataset leaks the row array and row structs (`makeInstance` copies; originals are never freed). The `UPDATE_JOB` path is fine (existing killer frees via `free_Job_ptr`). The design's High-severity risk row predicted exactly this; the mitigation was only half-applied (transfer done, release missing). - -## 3. Identified Gaps (priority order) - -1. **G-1 (bug, correctness): UPDATE_RESVALS writes worker resval index 0 unconditionally** — scheduler.c:813. Breaks scheduling decisions that read per-resource values. (Req 5) -2. **G-2 (bug, correctness): UPDATE_RESOURCES cannot update an existing resource** — `makeResource` → `EEXIST` → NULL, return ignored, no log. "Add or update" is only "add". (Req 1 / UB-1.1) -3. **G-3 (leak): killer/dequeue never free new-op payloads** despite ownership transfer (queue, resval_list, rusages, resource/resval rows). Slow per-message leak in a long-lived daemon. (Design risk #1) -4. **G-4 (requirement shortfall): job update copies only `state`** — priority/queue/resreq not applied. (Req 3 / UB-3.1) -5. **G-5 (requirement shortfall): rusage not stored on job object** — only the plugin hook sees it. Requires a rusage field on `xce_sched_job_t` (none exists today). (Req 6 / UB-6.1) -6. **G-6 (test debt): tasks 9.2, 9.3, 9.4, 9.6, 9.7 are checked `[x]` but absent from test.c.** The only new tests are: 9.1 notify-event-name (real), an enum compile-presence check, 9.5 reduced to a NULL-guard exercised against a **local stub** of `sched_update_existing_queue` defined inside test.c:25–34 (the real schedobj.c implementation is never linked/tested), and 9.8 reduced to `TEST_PASS()` with a comment. `xce_sched_op_name`/`sched_op_name` has no test (9.1 claims it does). None of the enqueue handlers, dequeue cases, or `sched_update_existing_job` are tested. -7. **G-7 (doc drift): design Component 8 / tasks 1.1, 1.3 say `xce_sched_op_name` lives in lib/helper** — it is a static in scheduler.c. Update spec docs or move the function. - -## 4. Implementation Approach Options - -**Option A — Extend in place (minimal correctness patch).** Fix G-1 (resolve `rv->name` → resmgr idx, mirroring schedobj.c:284–291), G-2 (on `EEXIST` find existing resource and mutate fields, or add an `updateResource()` beside `makeResource()` in resobj.c; log NULL returns), G-3 (add `free_*` calls in `sched_pend_update_killer` per op, reusing the deep-free helpers in src/lib/request.c), G-4 (copy priority/queue/resreq in `sched_update_existing_job`), and G-7 (doc edit). Defer G-5/G-6. -- ✅ Smallest diff, all in files already touched by this spec; restores correctness fast. -- ❌ Leaves UB-6.1 unmet and test checkboxes false; test debt persists. - -**Option B — New test program + extracted ops module.** Create `src/scheduler/testsuite/test_msg_handlers.c` (links scheduler objects + generated capnp headers, which test.c cannot include — the stated reason 9.8 became a comment), and optionally extract the five dequeue case bodies into `src/scheduler/src/msgops.c` with non-static functions so they are unit-testable without including scheduler.c. -- ✅ Makes every acceptance criterion genuinely testable; honest Task 9; aligns with testing steering (tiered test_*.c programs already exist in other daemons). -- ❌ Larger diff; new file + meson.build change; refactor risk in scheduler.c. - -**Option C — Hybrid (recommended).** Phase 1: Option A bug/leak fixes (G-1…G-4, G-7) with TDD where linkable. Phase 2: Option B test program covering handler enqueue + dequeue paths, replacing the stub-based 9.5 and the TEST_PASS 9.8 with real assertions; un-check or re-implement tasks 9.2–9.8 accordingly. Phase 3 (optional, needs decision): G-5 — add a rusages field to `xce_sched_job_t` or formally amend Req 6 to "deliver to plugins only". -- ✅ Correctness first, honesty in tracking, requirement closure staged. -- ❌ Two-phase coordination; Phase 3 needs a requirements decision. - -## 5. Effort & Risk - -| Work item | Effort | Risk | Justification | -|---|---|---|---| -| G-1 resvals index fix | S | Low | One-line lookup mirroring existing schedobj.c pattern; resmgr API already used two cases above. | -| G-2 resource update-on-EEXIST | S–M | Medium | Touching shared libxce resobj.c affects controller too; scheduler-side find-and-mutate is the safer S variant. | -| G-3 killer frees | M | Medium | Must trace ownership per type against free_* helpers in request.c; wrong free = double-free under ASAN, wrong no-op = leak persists. | -| G-4 job field updates | S | Low | Copy 3 more fields + string realloc, same shape as queue update. | -| G-5 store rusage on job | M | Medium | New field on `xce_sched_job_t` + lifetime/free in `kill_sched_job`; or amend requirement (zero code). | -| G-6 real test program | M | Medium | New testsuite binary + meson wiring; capnp header dependency is the known friction that derailed Task 9 the first time. | -| G-7 doc sync | S | Low | Edit design.md/tasks.md only. | -| **Total (Option C, Phases 1–2)** | **M (≈4–6 days)** | **Medium** | All within established patterns; risk concentrated in memory-ownership work under ASAN. | - -## 6. Recommendations for Design/Implementation Follow-up - -1. Treat G-1 and G-2 as **defects against the already-approved spec** — fix before marking the spec `implemented`; both silently corrupt or ignore controller state updates. -2. Adopt **Option C**: in-place fixes first (TDD per project rules — failing test in the new test program first where possible), then the dedicated `test_msg_handlers.c` to make Task 9 honest. Un-check tasks 9.2–9.8 until real tests exist, or annotate them with the stub caveat. -3. Resolve G-3 with the design's own mitigation table: request struct owns the payload after nullification ⇒ `sched_pend_update_killer` must deep-free each new op's payload (reuse `free_Queue_ptr`/`free_Resource_ptr`-style helpers from src/lib/request.c if exported; otherwise add small free helpers). -4. Decide G-5 explicitly: either add the job-side rusage storage Req 6 promises, or amend UB-6.1 — do not leave the requirement and code disagreeing. -5. Sync design.md Component 8 and tasks 1.1/1.3 with the actual `sched_op_name` location (G-7). - -## 7. Research Needed - -- **R-1:** Which `free_*` deep-free helpers in src/lib/request.c are exported and safe for `xce_queue_t`, `xce_resval_list_t`, `xce_job_rusages_t`, `xce_resource_t*[]` — confirm against ASAN before wiring into the killer. -- **R-2:** Confirm the notify-driven `schedUpdateJob` actually reaches `sched_update_existing_job()`: the dequeue `XCE_SCHED_OP_UPDATE_JOB` case calls `sched_update_jobs()` (bulk path); verify `sched_update_jobs` performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs. -- **R-3:** Is there a planned controller-side emitter for `Data_data_resources`/`Data_data_resvals` pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec. -- **R-4:** Whether `makeInstance(&resobj_type, r)` deep-copies all of `xce_resource_t` (name string included) — determines whether resource row payloads can be freed immediately after `makeResource` returns. - ---- - -## Gap Re-Audit (2026-06-18) - -> Post-implementation audit. tasks.md = 36/36 `[x]` (9.9 gated), phase=implemented. Method: every EARS AC re-verified against current on-disk source via codegraph (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c) + grep. This pass tracks the 2026-06-09 G-1..G-7 findings: **G-1 and G-2 are now FIXED; G-3, G-4, G-5, G-6, G-7 remain OPEN; one NEW finding (G-8) supersedes the 2026-06-09 R-2 caveat.** - -### 0. Delta vs 2026-06-09 audit - -| Prior gap | 2026-06-18 status | Evidence | -|---|---|---| -| G-1 UPDATE_RESVALS clobbered worker index 0 | **FIXED** | scheduler.c:826-831 now resolves `rv->name`→`key_find(sched->resmgr)`→`r->idx`, `zarray_put(&w->resvals, r->idx, rv->val)`; unknown resource WARN+skip. Matches schedobj.c:283-291. | -| G-2 UPDATE_RESOURCES could not update existing resource | **FIXED** | scheduler.c:775-792: on `makeResource()`==NULL (EEXIST), `key_find` the existing resource and copy `valtype/ordtype/flags/interval`; WARN on update and on lookup-miss. "Add-or-update" now satisfied for mutable scalar fields. | -| G-3 killer/dequeue never free new-op payloads (leak) | **OPEN** | `sched_pend_update_killer` (scheduler.c:311-320) still has empty `break;` for UPDATE_QUEUE / UPDATE_RESOURCES / UPDATE_RESVALS_FULL / UPDATE_RESVALS / JOB_RUSAGE. Dequeue cases (scheduler.c:767-847) free nothing. Ownership IS transferred (handler.c:297-316 nullifies `notify->data.*`; sched_handle_dataset:130-145 nullifies rows) ⇒ confirmed per-message leak. | -| G-4 job update copies only `state` | **OPEN (worse — see G-8)** | `sched_update_existing_job` (scheduler.c:707-725) still copies only `job->state`; priority/queue/resreq untouched. AND it is never reached (G-8). | -| G-5 rusage not stored on job | **OPEN** | `xce_sched_job_t` (sched/types.h:80-96) has NO rusage field. JOB_RUSAGE case (scheduler.c:836-847) calls `plugin_job_rusage_update()` then drops+leaks the payload. UB-6.1 "store the usage data on the job object" unmet. | -| G-6 test debt (9.2/9.3/9.4/9.6/9.7 `[x]` but absent; 9.5 stub; 9.8 TEST_PASS) | **OPEN** | testsuite/test.c still defines a LOCAL stub `sched_update_existing_queue` (test.c:25-34) — real schedobj.c impl never linked. Only real new tests: 9.1 notify-event-name (test.c:399-423) + enum compile check. No tests for any enqueue handler, dequeue case, `sched_update_existing_job`, `sched_op_name`, leaks. | -| G-7 doc drift: `xce_sched_op_name` in lib/helper per design/tasks 1.1,1.3 | **OPEN** | Implemented as static `sched_op_name()` in scheduler.c:685-705, NOT `xce_sched_op_name()` in lib/helper.c. design.md Component 8 + tasks.md 1.1/1.3 (both `[x]`) still describe the wrong location/name. | - -### 1. NEW finding - -- **G-8 (correctness, HIGH — supersedes R-2): `sched_update_existing_job()` is dead code; notify-driven job updates take the bulk CREATE path.** `grep -rn sched_update_existing_job src/` returns ONLY its definition (scheduler.c:707) — zero callers. The `XCE_SCHED_OP_UPDATE_JOB` dequeue case (scheduler.c:755-758) routes to `sched_update_jobs()` (scheduler.c:629-642), which for every job calls `make_sched_job()` (a fresh `calloc`) + `sched_add_job()` (`key_add` into `sched->jobs`). For an existing jobid this either duplicate-keys or replaces the live `xce_sched_job_t` with a new object — discarding `allocs`/`accounts`/`schedobj` runtime state and leaking the old node. There is no `-ENOENT` for an unknown jobid (UB-3.2 unmet) and no in-place mutation (UB-3.1 unmet). Req 3 is effectively NOT closed at the dequeue layer despite the handler-side wiring and the `[x]` on tasks 5.6 / 3.1 / 3.2. - -### 2. Requirement-to-Asset map (tagged) - -| Req / AC | Asset (file:line) | Tag | -|---|---|---| -| 1 / UB-1.1 resources add-or-update | handler.c:59-80; scheduler.c:770-796 | CLOSED | -| 1 / UB-1.2 resvals→sharedInst by idx | handler.c:82-103; scheduler.c:797-815 | CLOSED | -| 1 / UB-1.3 skip NULL rows | scheduler.c:774, 802-804 | CLOSED | -| 1 / UB-1.4 -ENOMEM, free partial | handler.c:68-71,91-94 | CLOSED | -| 2 / UB-2.1, UB-2.2 dead pushmsg gone | handler.c:333-362 (no pushmsg case); grep: no `sched_handle_jobpush`/`workerpush` | CLOSED | -| 3 / UB-3.1 update mutable fields | scheduler.c:707-725 (`state` only) AND unreachable | OPEN (G-4 + G-8) | -| 3 / UB-3.2 -ENOENT unknown jobid | scheduler.c:716-718 — never reached; live path `sched_update_jobs` create-anyway | OPEN (G-8) | -| 3 / UB-3.3 -EINVAL NULL payload | handler.c:181-183 | CLOSED (enqueue layer) | -| 4 / UB-4.1/4.2/4.3 queue update | schedobj.c:485-511; scheduler.c:767-769 | CLOSED | -| 5 / UB-5.1 per-worker resvals | scheduler.c:816-835 (idx fix) | CLOSED | -| 5 / UB-5.2 skip unknown host | scheduler.c:824-825 | CLOSED | -| 5 / UB-5.3 zero values→0 | handler.c:236-238 | CLOSED | -| 6 / UB-6.1 store rusage on job | scheduler.c:836-847 (plugin only; not stored) | OPEN (G-5) | -| 6 / UB-6.2 -ENOENT unknown jobid | scheduler.c:840-841 | CLOSED | -| 6 / UB-6.3 -EINVAL NULL rusage | handler.c:258-260 | CLOSED | -| 7 / UB-7.1 WARN op+errno | scheduler.c:855-858 | CLOSED | -| 7 / UB-7.2 unknown notify→-EINVAL | handler.c:318-320 | CLOSED | -| 7 / UB-7.3 unknown body_which→-EINVAL | handler.c:350-352 | CLOSED | -| 8 / UB-8.1 op name in req log | handler.c:331 | CLOSED | -| 8 / UB-8.2 dataset table name | handler.c:109 | CLOSED | -| 8 / UB-8.3 notify event name | handler.c:279 | CLOSED | -| 8 / UB-8.4 UNKNOWN(%d) fallback | helper.c (xce_notify_event_name) + scheduler.c:703 | CLOSED | -| 8 / UB-8.5 symbolic op name in WARN | scheduler.c:685-705 `sched_op_name` (static, not lib helper) | CLOSED-with-deviation (G-7) | - -**Score: 17 / 23 ACs fully closed.** OPEN: UB-3.1, UB-3.2 (G-8/G-4), UB-6.1 (G-5). Cross-cutting non-AC defects: G-3 (leak), G-6 (test debt), G-7 (doc drift). Of the 8 top-level requirements, **Req 2, 4, 5, 7, 8 are fully CLOSED; Req 1 CLOSED; Req 3 OPEN; Req 6 PARTIAL (delivery yes, storage no).** - -### 3. Residual / deferred gaps (priority order) - -1. **G-8 (HIGH, correctness)** — notify `schedUpdateJob` hits the create path; `sched_update_existing_job` is dead code. Corrupts/duplicates live job state; no -ENOENT. Req 3 unmet. -2. **G-3 (MED, leak)** — five new ops never freed despite ownership transfer. Unbounded slow leak in a long-lived daemon; aggravated by G-8 (old job node also leaked on overwrite). -3. **G-5 (MED, requirement shortfall)** — rusage never stored on job (no field exists). UB-6.1 unmet. -4. **G-4 (LOW, requirement shortfall)** — even once reachable, job update applies only `state`; priority/queue/resreq dropped. -5. **G-6 (MED, test debt / tracking integrity)** — tasks 9.2/9.3/9.4/9.6/9.7 `[x]` but no tests in test.c; 9.5 tests a local stub, not real schedobj.c; 9.8 is TEST_PASS+comment. Checkboxes overstate coverage; none of the dequeue correctness fixes (G-1/G-2) are tested. -6. **G-7 (LOW, doc drift)** — design.md Component 8 + tasks 1.1/1.3 name `xce_sched_op_name` in lib/helper; reality is static `sched_op_name` in scheduler.c. - -### 4. Integration risks - -- Req 1 handlers remain forward-looking: no controller emitter pushes `Data_data_resources`/`Data_data_resvals` to the scheduler today (controller/sched.c pushes only schedjobs/schedworkers; opget.c serves them only as get-responses). G-1/G-2 fixes are correct but untriggered in production — regression-prone because untested AND unexercised. (Prior R-3 still open.) -- G-8 + G-3 interact: overwriting a live job via `sched_add_job` leaks the prior node's `schedobj`/`allocs`/`accounts`. Surfaces under ASAN only when a notify update arrives for an already-loaded job — exactly the persistent_ha post-failover resync scenario. - -### 5. Options for open work (G-8, G-3, G-5, G-4, G-6, G-7) - -**Option A — Minimal correctness patch.** -- G-8: in dequeue `XCE_SCHED_OP_UPDATE_JOB`, branch on `njobs==1 && sched_get_job(jobid)!=NULL` → `sched_update_existing_job()`; else keep `sched_update_jobs()`. OR route the notify-only op to a distinct `XCE_SCHED_OP_*` so the single-job update never enters the bulk path. -- G-4: copy priority/queue/resreq (+resreq realloc) into `sched_update_existing_job`, mirroring `sched_update_existing_queue` (schedobj.c:498-506). -- G-3: add per-op `free_*` in `sched_pend_update_killer` (reuse src/lib/request.c deep-free helpers — confirm exports per prior R-1 against ASAN). -- G-7: edit design.md/tasks.md to record the `sched_op_name` location/name deviation. -- Defer G-5, G-6. -- Effort **S–M**, Risk **Medium** (ownership/free work must pass ASAN; double-free risk). - -**Option B — Close everything incl. requirement + tests.** -- All of A, plus G-5 (add `xce_job_rusages_t* rusages` to `xce_sched_job_t` + free in `kill_sched_job` + store in JOB_RUSAGE case) and G-6 (new `testsuite/test_msg_handlers.c` linking scheduler objects + capnp headers — the friction that derailed Task 9 originally — replacing the local-stub 9.5 and TEST_PASS 9.8 with real assertions; un-check 9.2-9.8 until real). -- Effort **M–L (~5-7 days)**, Risk **Medium-High** (new test binary + meson wiring; struct lifetime changes). - -**Option C — Hybrid (recommended).** -- Phase 1: A's correctness/leak fixes (G-8, G-4, G-3) with a failing test first where linkable; G-7 doc sync. -- Phase 2: G-6 dedicated test program for enqueue+dequeue paths incl. the G-1/G-2 regressions. -- Phase 3 (needs decision): G-5 — add job-side rusage storage, OR formally amend UB-6.1 to "deliver to plugins only" (zero code). Do not leave requirement and code disagreeing. -- Effort **M**, Risk **Medium**; staged so correctness lands first and tracking becomes honest before requirement closure. - -### 6. Recommendation - -Do NOT treat the spec as truly `implemented`. Five top-level requirements (2,4,5,7,8) and Req 1 are genuinely closed; **Req 3 is broken at the dequeue layer (G-8) and Req 6 stores nothing (G-5)**. Adopt Option C. Re-open/annotate tasks 5.6 (job update wiring), 9.2-9.8 (test debt), and 1.1/1.3 (doc drift). G-8 and G-3 are defects against the approved spec and should block a clean `implemented` sign-off. - -### 7. Research still needed - -- **R-1 (carry-over):** Confirm exported `free_*` deep-free helpers in src/lib/request.c for `xce_queue_t` / `xce_resval_list_t` / `xce_job_rusages_t` / `xce_resource_t*[]` and validate under ASAN before wiring the killer (G-3). -- **R-3 (carry-over):** Planned controller emitter for `Data_data_resources`/`Data_data_resvals` to scheduler (persistent_ha resync)? Determines whether Req 1 stays forward-looking. -- **R-4 (carry-over):** Does `makeResource`/`makeInstance` deep-copy `xce_resource_t` (name incl.)? Determines if resource rows can be freed right after enqueue (G-3 scope). diff --git a/.kiro/specs/scheduler-controller-msg-gap/spec.json b/.kiro/specs/scheduler-controller-msg-gap/spec.json deleted file mode 100644 index 368cf47c..00000000 --- a/.kiro/specs/scheduler-controller-msg-gap/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "scheduler-controller-msg-gap", - "description": "Analyze the codebase to find gaps in the scheduler's handling of controller messages \u2014 missing message types, unhandled edge cases, or incomplete message processing paths", - "phase": "implemented", - "created_at": "2026-05-11T01:34:00-07:00", - "updated_at": "2026-08-16T04:30:00-07:00", - "language": "en", - "status": "closed", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "note": "CLOSED 2026-08-16. Both open items resolved: sched_pend_update_killer leak fixed in commit 93124c56 (2026-06-19, bug-765 G-3) -- verified current scheduler.c:394-460 correctly frees all 5 payload types. Test coverage gap substantially closed in commit 276e26c3 (bug-765 G-6) -- added test_handlers.c (6 tests) + test_job_update.c (10 tests); msg-gap-relevant suites now 31/31 (the prior '15/15' predates these files). Bugs 335/336 confirmed fixed 2026-06-10. 3 low-value residuals remain individually documented in tasks.md's Implementation Notes (9.2 -ENOMEM fault-injection, 9.6 dormant worker-resvals apply path, 9.9 log-capture-gated WARN test) -- non-blocking, not worth standalone tracking." -} diff --git a/.kiro/specs/scheduler-controller-msg-gap/tasks.md b/.kiro/specs/scheduler-controller-msg-gap/tasks.md deleted file mode 100644 index f3b2e9c5..00000000 --- a/.kiro/specs/scheduler-controller-msg-gap/tasks.md +++ /dev/null @@ -1,136 +0,0 @@ -# Tasks: scheduler-controller-msg-gap - -> Generated: 2026-05-11T19:30:00-07:00 -> Feature: scheduler-controller-msg-gap -> Status: Tasks Generated - -## Overview - -Implementation tasks for filling gaps in the scheduler's handling of controller messages. The work follows the design's build order: shared helpers first, then header definitions, plugin hooks, business logic, handler wiring, and finally tests. - ---- - -## Task 1: Add name-lookup helpers - -**Requirements:** 8 - -Add the op/event name-lookup helpers, following the `xce_dataset_table()` pattern (`char*` return, `"UNKNOWN(%d)"` fallback). - -> **Layering note (bug-765 G-7 reconciliation, 2026-06-18):** the two helpers live in DIFFERENT layers by necessity. `xce_notify_event_name(int)` is lib-level (it takes an `int` NotifyData event) and lives in `src/lib/helper.c` (declared in `src/include/helper.h`). The scheduler-op name lookup operates on `xce_sched_op_t`, which is a **scheduler-only** type (`src/scheduler/include/scheduler.h`); `src/lib` must not depend on scheduler headers (no layering inversion), so it is implemented as a **static `sched_op_name(xce_sched_op_t)` in `src/scheduler/src/scheduler.c`** — NOT as `xce_sched_op_name` in lib/helper. The original design/tasks text placing it in lib/helper did not account for this layering; the as-built scheduler-layer placement is correct. - -- [x] 1.1 Declare `xce_notify_event_name(int event)` in `src/include/helper.h`. (The scheduler-op name lookup is scheduler-internal — see the layering note — so it is a static `sched_op_name` in `scheduler.c`, not declared in `helper.h`.) -- [x] 1.2 Implement `xce_notify_event_name()` in `src/lib/helper.c` — switch on all `NotifyData_data_*` enum values; return symbolic string; fall back to `"UNKNOWN(%d)"` -- [x] 1.3 Implement the scheduler-op name lookup as static `sched_op_name(xce_sched_op_t op)` in `src/scheduler/src/scheduler.c` — switch on all `XCE_SCHED_OP_*` enum values; return symbolic string; fall back to `"UNKNOWN(%d)"`. (Kept in the scheduler layer per the layering note above, not in lib/helper.) - ---- - -## Task 2: Extend scheduler.h with new op types and request fields - -**Requirements:** 1, 3, 4, 5, 6 - -Add five new `xce_sched_op_t` enum values and five corresponding union members in `xce_sched_update_request_t` to carry the payloads for queue update, resource update, full resval update, incremental resval update, and job rusage. - -- [x] 2.1 Add `XCE_SCHED_OP_UPDATE_QUEUE`, `XCE_SCHED_OP_UPDATE_RESOURCES`, `XCE_SCHED_OP_UPDATE_RESVALS_FULL`, `XCE_SCHED_OP_UPDATE_RESVALS`, and `XCE_SCHED_OP_JOB_RUSAGE` to the `xce_sched_op_t` enum in `src/scheduler/include/scheduler.h` -- [x] 2.2 Add the corresponding named union members (`queueupdate`, `resourceupdate`, `resvalfullupdate`, `resvalupdate`, `job_rusage`) to `xce_sched_update_request_t` - ---- - -## Task 3: Add plugin rusage hook - -**Requirements:** 6 - -Add the `plugin_job_rusage_hook_t` typedef, the `rusage_hook` field in `plugin_hooks_t`, and the `plugin_job_rusage_update()` global function. This follows the identical pattern used by `plugin_job_state_change()`. - -- [x] 3.1 Add `plugin_job_rusage_hook_t` typedef and `rusage_hook` field to `plugin_hooks_t` in `src/scheduler/include/plugin.h` -- [x] 3.2 Implement `plugin_job_rusage_update()` in `src/scheduler/src/plugin.c` — iterate registered plugins and call their `rusage_hook` when non-NULL - ---- - -## Task 4: Add queue update function in schedobj.c - -**Requirements:** 4 - -Implement `sched_update_existing_queue()` in `src/scheduler/src/schedobj.c`. The function locates a queue by name via `sched_get_queue()`, updates its mutable fields (status, priority, accept_interval, sched_delay, resreq string), calls `plugin_queue_update()`, and returns `-ENOENT` if the queue is unknown. - -- [x] 4.1 Implement `sched_update_existing_queue(xce_scheduler_t*, xce_queue_t*)` in `src/scheduler/src/schedobj.c` with correct lookup, field update, string reallocation for resreq, plugin call, and error returns - ---- - -## Task 5: Extend scheduler.c dequeue processing and error logging - -**Requirements:** 1, 3, 4, 5, 6, 7 - -Wire the five new op types into the `switch (req->op)` in `sched_update()`, implement `sched_update_existing_job()`, and add WARN-level error logging after the switch when a handler returns non-zero. - -- [x] 5.1 Add `XCE_SCHED_OP_UPDATE_QUEUE` case — delegate to `sched_update_existing_queue()`, return `-ENOENT` when queue not found -- [x] 5.2 Add `XCE_SCHED_OP_UPDATE_RESOURCES` case — iterate `resdata->resources[]`, call `makeResource(sched->resmgr, r)` for each non-NULL entry -- [x] 5.3 Add `XCE_SCHED_OP_UPDATE_RESVALS_FULL` case — iterate `resvaldata->vals[]`, find resource in `sched->resmgr` by name, update value; log WARN and skip unknown resources -- [x] 5.4 Add `XCE_SCHED_OP_UPDATE_RESVALS` case — iterate `resvals->vals[]`, find worker by host name via `sched_get_worker()`, update worker's `resvals` zarray; silently skip unknown hosts -- [x] 5.5 Add `XCE_SCHED_OP_JOB_RUSAGE` case — find job by jobid, **store rusage data ON THE JOB** (`xce_job_rusages_t`/`xce_sched_job_rusages_t` field added to `xce_sched_job_t`, replacing any prior, freed in `kill_sched_job`), call `plugin_job_rusage_update()`; return `-ENOENT` if job not found. _Remediation (bug-765, gap G-5): rusage was never stored — `xce_sched_job_t` has no rusage field; Req 6 UB-6.1 unmet. See research.md §Gap Re-Audit._ -- [x] 5.6 Implement AND WIRE `sched_update_existing_job()` — find job by jobid via `sched_get_job()`, update ALL mutable fields per Req 3 (state, priority, queue, resource request — resreq string realloc like `sched_update_existing_queue`), call `plugin_job_update()`; return `-ENOENT` when jobid unknown, `-EINVAL` on NULL. **CRITICAL routing fix:** the notify `schedUpdateJob` handler (`sched_handle_sched_update_job`, handler.c:200) and the dataset bulk-load (`sched_on_jobs`, handler.c:31) BOTH enqueue `XCE_SCHED_OP_UPDATE_JOB`, and the dequeue (scheduler.c:755) routes BOTH to `sched_update_jobs()` (create/replace). Distinguish the notify path (introduce a distinct op, e.g. `XCE_SCHED_OP_UPDATE_JOB_FIELDS`, enqueued only by the notify handler, dequeued to `sched_update_existing_job`) so a notify update mutates-in-place / returns `-ENOENT` and NEVER creates a job (Req 3: jobs created only via the dataset path). _Remediation (bug-765, gap G-8+G-4): `sched_update_existing_job` was dead code (0 callers) and copied only `state`; UB-3.1/UB-3.2 unmet. See research.md §Gap Re-Audit._ -- [x] 5.7 Add WARN log in `sched_update()` after switch block for non-zero `rc` — include symbolic op name via the static `sched_op_name()` (scheduler.c, see Task 1 layering note) and `strerror(-rc)` -- [x] 5.8 Free new-op payloads in `sched_pend_update_killer` (and any dequeue early-exit) for all new op types (`UPDATE_QUEUE`/`UPDATE_RESOURCES`/`UPDATE_RESVALS_FULL`/`UPDATE_RESVALS`/`JOB_RUSAGE`/the new `UPDATE_JOB_FIELDS`) — ownership IS transferred to the request (handler nullifies the notify/dataset payload pointers), so the empty `break;` cases leak. Use the lib deep-free helpers; validate ASAN-clean (no double-free). _Remediation (bug-765, gap G-3): killer had no-op `break` cases → confirmed per-message leak. See research.md §Gap Re-Audit._ - ---- - -## Task 6: Wire notify handlers in handler.c - -**Requirements:** 3, 4, 5, 6, 8 - -Add four static handler functions for the currently-stubbed NotifyData events, wire them into `sched_handle_notify()`, and update the TRACE log calls in `sched_handle_notify()` and `sched_handle_dataset()` to use symbolic name helpers. - -- [x] 6.1 Implement `sched_handle_sched_update_job()` — validate non-NULL, enqueue `XCE_SCHED_OP_UPDATE_JOB`; return `-EINVAL` on NULL, `-ENOMEM` on alloc failure -- [x] 6.2 Implement `sched_handle_sched_update_queue()` — validate non-NULL, enqueue `XCE_SCHED_OP_UPDATE_QUEUE`; return `-EINVAL` on NULL, `-ENOMEM` on alloc failure -- [x] 6.3 Implement `sched_handle_sched_update_resvals()` — skip if NULL or count==0, enqueue `XCE_SCHED_OP_UPDATE_RESVALS`; return `-ENOMEM` on alloc failure -- [x] 6.4 Implement `sched_handle_sched_job_rusage()` — validate non-NULL, enqueue `XCE_SCHED_OP_JOB_RUSAGE`; return `-EINVAL` on NULL, `-ENOMEM` on alloc failure -- [x] 6.5 Wire all four handlers into `sched_handle_notify()` switch — replace stub `break` / TRACE-only cases -- [x] 6.6 Update TRACE log in `sched_handle_dataset()` from `%d` table integer to `%s` via `xce_dataset_table()` (8.2), and update notify event logs to use `xce_notify_event_name()` (8.3) - ---- - -## Task 7: Add dataset handlers in handler.c - -**Requirements:** 1, 8 - -Add `sched_handle_resources()` and `sched_handle_resvals()` static functions and wire them into the `sched_handle_dataset()` switch for the two currently-empty Data table cases. Update the existing opcode log in `scheduler_req_handler` to use `xce_request_op_name()`. - -- [x] 7.1 Implement `sched_handle_resources()` — skip if NULL or count==0, enqueue `XCE_SCHED_OP_UPDATE_RESOURCES`; return `-ENOMEM` on alloc failure -- [x] 7.2 Implement `sched_handle_resvals()` — skip if NULL or count==0, enqueue `XCE_SCHED_OP_UPDATE_RESVALS_FULL`; return `-ENOMEM` on alloc failure -- [x] 7.3 Wire `sched_handle_resources()` and `sched_handle_resvals()` into `sched_handle_dataset()` switch replacing the empty `break` cases -- [x] 7.4 Update opcode log in `scheduler_req_handler` to call `xce_request_op_name()` instead of printing the numeric value (8.1) - ---- - -## Task 8: Remove dead code pushmsg branches - -**Requirements:** 2 - -Delete the `Request_body_pushmsg` case block from `scheduler_req_handler()` and the static functions `sched_handle_jobpush()` and `sched_handle_workerpush()`. These paths are unreachable because `vtask_entry()` rewrites all `pushmsg` into `datamsg` before invoking the handler. - -- [x] 8.1 Remove the `Request_body_pushmsg` case and its two nested branches (`job`/`worker`) from `scheduler_req_handler()` in `src/scheduler/src/handler.c` -- [x] 8.2 Remove the static function definitions `sched_handle_jobpush()` and `sched_handle_workerpush()` from `src/scheduler/src/handler.c` - ---- - -## Task 9: Write unit tests for all new message handlers - -**Requirements:** 1, 2, 3, 4, 5, 6, 7, 8 - -Extend `src/scheduler/testsuite/test.c` with unit tests covering every acceptance criterion. Tests must run without a live server — use stub schedulers, table-driven cases, and the existing testsuite harness. - -- [x] 9.1 Test `xce_notify_event_name()` (lib helper) — verify known values return correct strings, unknown values return `"UNKNOWN(%d)"` format. (The scheduler-op name lookup is a static `sched_op_name` in scheduler.c — scheduler-internal, not directly unit-testable from `test.c`; it is exercised indirectly via the Req 7 WARN path.) -- [x] 9.2 Test `sched_handle_resources()` and `sched_handle_resvals()` handler functions — verify enqueue on valid input, skip on NULL/empty. **REAL** (bug-765 G-6): `testsuite/test_handlers.c` (white-box `#include "../src/handler.c"`) asserts each enqueues the correct op / skips on NULL/empty. (`-ENOMEM` path requires calloc fault injection — not covered; low value.) -- [x] 9.3 Test `sched_handle_sched_update_job()`, `sched_handle_sched_update_queue()`, `sched_handle_sched_update_resvals()`, `sched_handle_sched_job_rusage()` — verify `-EINVAL` on NULL, enqueue on valid input. **REAL** (G-6): `test_handlers.c` — incl. the 5.6 assertion that update_job enqueues `UPDATE_JOB_FIELDS` (not the bulk `UPDATE_JOB`). -- [x] 9.4 Test `sched_update_existing_job()` — verify `-ENOENT` for unknown jobid, correct field update for known jobid. **REAL** (G-6): `testsuite/test_job_update.c` (in-place mutation + identity + `-ENOENT` no-create + `-EINVAL`). -- [x] 9.5 Test `sched_update_existing_queue()` — verify `-ENOENT` for unknown queue, correct field update for known queue; `-EINVAL` on NULL. **REAL** (G-6): `test_job_update.c` exercises the LINKED `schedobj.c` impl (replaces the former local stub in `test.c`). -- [~] 9.6 Test dequeue path for `XCE_SCHED_OP_UPDATE_RESVALS` — verify unknown host skipped, known host updated. **PARTIAL** (G-6): the ENQUEUE side (`sched_handle_sched_update_resvals` → `UPDATE_RESVALS`) is tested in `test_handlers.c`. The dequeue worker-resvals APPLY is NOT unit-tested — it is a dormant production path (research R-3: no controller emits `Data_data_resvals` to the scheduler today) and follows the same find→mutate pattern as the tested appliers (job/queue/rusage). Follow-up: extract a `sched_apply_resvals` helper + worker/resmgr fixture if the path is activated. -- [x] 9.7 Test dequeue path for `XCE_SCHED_OP_JOB_RUSAGE` — verify `-ENOENT` for unknown jobid, rusage stored for known jobid; `-EINVAL` on NULL. **REAL** (G-6): `test_job_update.c` tests `sched_apply_job_rusage` (store-on-job, replace-frees-prior, `-ENOENT`, `-EINVAL`). -- [x] 9.8 Verify dead code symbols are absent — `sched_handle_jobpush` / `sched_handle_workerpush` not defined (2.1, 2.2). Documented compile-time guarantee in `test.c` (`test_dead_pushmsg_symbols_absent`): the symbols were removed in Task 8; referencing them would fail the build. (Symbol-absence is inherently a build/link property, not a runtime assertion.) -- [~] 9.9 *(gated: log capture infrastructure)* Verify WARN log is emitted in `sched_update()` when any handler returns a non-zero error code (optional — may require log capture infrastructure) - -## Implementation Notes -- 5.6 (bug-765 remediation, gaps G-8+G-4, done 2026-06-18): Introduced a distinct op `XCE_SCHED_OP_UPDATE_JOB_FIELDS` (+ `jobfieldsupdate.job` union member) so the notify `schedUpdateJob` handler routes to `sched_update_existing_job` (now in schedobj.c) which mutates state/priority/queue/resreq IN PLACE and returns `-ENOENT` for unknown jobid — never creating a job. The dataset bulk-create path (`sched_on_jobs` → `XCE_SCHED_OP_UPDATE_JOB` → `sched_update_jobs`) is unchanged. Killer frees the new op's payload via `free_Job_ptr`. New white-box test `testsuite/test_job_update.c` (separate meson executable `sched_job_update_test`, white-box `#include "../src/schedobj.c"` + stubs for scheduler-internal externs): 3/3 PASS, regression `sched_test` 15/15, ASAN clean (leaks-off per suite convention). VERIFY NOTES: (1) the handler→dequeue ROUTING is verified by code review, NOT an automated integration test (an end-to-end test through `sched_handle_notify` needs NotifyData capnp plumbing — follow-up); (2) `free(job->resreq)` on replace is shallow but consistent with the codebase — `kill_sched_job` is a no-op stub, so sched jobs are not deep-freed anywhere; the inner-resreq leak is pre-existing, not introduced here. (3) test_job_update.c originally had an accidental `*/` inside a comment (`make_*/kill_*`) + non-ASCII chars + a `make_sched_obj` stub conflicting with the static def in schedobj.c — rewritten ASCII-clean with correct stub set. Task 9.4 (test sched_update_existing_job) is now genuinely satisfied by this real test. -- 5.5 (bug-765 remediation, gap G-5, done 2026-06-18): Added `xce_job_rusages_t* rusages` to `xce_sched_job_t` (sched/types.h). New `sched_apply_job_rusage()` (schedobj.c) stores the report ON the job (ownership transferred), replacing any prior via the generated `free_JobRusages_ptr`; returns `-ENOENT` unknown / `-EINVAL` NULL. The dequeue `JOB_RUSAGE` case calls it and NULLs `req->u.job_rusage.rusages` on success so the killer doesn't double-free; `kill_sched_job` (was a no-op stub) now releases `job->rusages`. NOTE: used the generated capnp deep-freer `free_JobRusages_ptr` (NOT a hand-rolled freer — it correctly frees pids/pgids/usages incl. resname per the capnp allocation model; an earlier hand-rolled `free_job_rusages` was removed). 4 new white-box tests (store / replace-frees-prior / -ENOENT / -EINVAL): sched_job_update_test 7/7. -- 5.8 (bug-765 remediation, gap G-3, done 2026-06-18): `sched_pend_update_killer` now frees all five previously-leaked op payloads with the exact generated deep-freers — `free_QueueConf_ptr` (xce_queue_t), `free_Resource_ptr[]`+free(arr) (xce_resource_t), `free_ResourceVal_ptr[]`+free(arr) (xce_resval_t), `free_ResValList_ptr` (xce_resval_list_t), `free_JobRusages_ptr` (xce_job_rusages_t). Safe because: ownership is transferred (notify `sched_handle_notify` and dataset `sched_handle_dataset` nullify their pointers on enqueue), `makeResource` constructs a fresh instance via `makeInstance` (no alias), and the dequeue copies values (no alias). VERIFY NOTE: the JOB_RUSAGE killer path interacts with 5.5's ownership; the other four killer frees are review-verified (type-exact generated freers + confirmed transfer/non-alias) but NOT runtime-ASAN-exercised — those notify/dataset ops are not emitted to the scheduler in production today (research R-3 dormant paths) and constructing capnp payloads for a dequeue test is a follow-up. regression sched_test 15/15; no bad-free/double-free under ASAN. -- G-7 (bug-765 doc drift, resolved 2026-06-18, DOC-ONLY): the scheduler-op name lookup is `static sched_op_name(xce_sched_op_t)` in `scheduler.c`, NOT `xce_sched_op_name` in `lib/helper`. RESOLUTION = fix the docs (code is correct): `xce_sched_op_t` is a scheduler-only type (`scheduler.h`) and `src/lib` has zero scheduler-header dependencies, so placing the helper in lib/helper would be a layering inversion. Only `xce_notify_event_name(int)` (lib-level) belongs in lib/helper. Updated Task 1 (intro + 1.1/1.3 + 5.7 + 9.1) and design.md Component 8/9 + file table + risks to reflect the scheduler-layer placement. No code change (kept `sched_op_name` in origin per user direction). -- G-6 (bug-765 test debt, addressed 2026-06-18): converted the falsely-`[x]` Task 9 tests into REAL tests via two new white-box executables (registered as meson tests under `--suite scheduler`, leaks-off per convention): `testsuite/test_handlers.c` (`#include "../src/handler.c"`; `enq_pend_update` + handlers are static → self-contained, only `sched_init_config` stubbed) covers 9.2 + 9.3 (6 tests); `testsuite/test_job_update.c` (`#include "../src/schedobj.c"`) covers 9.4 (job in-place), 9.5 (queue in-place — replaces the old local-stub test), 9.7 (rusage) — 10 tests. Suite: sched_test 15/15 + sched_job_update_test 10/10 + sched_handlers_test 6/6, all GREEN, ASAN-clean. REMAINING (honest): 9.6 dequeue worker-resvals APPLY is PARTIAL — enqueue tested, apply is a dormant path (R-3); 9.8 is a documented compile-time symbol-absence guarantee; 9.9 stays gated (log-capture infra); 9.2 `-ENOMEM` calloc-fault path not injected. These are low-value/dormant; the correctness-critical handler + applier paths are now genuinely covered. diff --git a/.kiro/specs/scheduler-core/.solomd/session.94ffc737-bfba-49ef-8dcb-f6e6a1bf0915.json b/.kiro/specs/scheduler-core/.solomd/session.94ffc737-bfba-49ef-8dcb-f6e6a1bf0915.json deleted file mode 100644 index 675a9871..00000000 --- a/.kiro/specs/scheduler-core/.solomd/session.94ffc737-bfba-49ef-8dcb-f6e6a1bf0915.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "device_id": "94ffc737-bfba-49ef-8dcb-f6e6a1bf0915", - "device_label": "Linux x86_64", - "saved_at": 1786169069, - "active_index": 3, - "tabs": [ - { - "file_path": "/home/rshen/projects/xce/worktrees/scheduler-framework/.kiro/specs/scheduler-core/tasks.md", - "file_name": "tasks.md", - "cursor_line": null, - "cursor_col": null, - "rel_path": "tasks.md" - }, - { - "file_path": "/home/rshen/projects/xce/worktrees/scheduler-framework/.kiro/specs/scheduler-core/spec.json", - "file_name": "spec.json", - "cursor_line": null, - "cursor_col": null, - "rel_path": "spec.json" - }, - { - "file_path": "/home/rshen/projects/xce/worktrees/scheduler-framework/.kiro/specs/scheduler-core/research.md", - "file_name": "research.md", - "cursor_line": null, - "cursor_col": null, - "rel_path": "research.md" - }, - { - "file_path": "/home/rshen/projects/xce/worktrees/scheduler-framework/.kiro/specs/scheduler-core/requirements.md", - "file_name": "requirements.md", - "cursor_line": null, - "cursor_col": null, - "rel_path": "requirements.md" - } - ] -} \ No newline at end of file diff --git a/.kiro/specs/scheduler-core/brief.md b/.kiro/specs/scheduler-core/brief.md deleted file mode 100644 index c2504736..00000000 --- a/.kiro/specs/scheduler-core/brief.md +++ /dev/null @@ -1,168 +0,0 @@ -# Brief: scheduler-core - -## Problem - -The XCE scheduler has a plugin architecture but critical foundational gaps: - -1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation. - -2. **Monolithic scheduler tied to controller I/O** — `xce_scheduler_t` holds both scheduling state and controller API context (`apictx`, `cluster_cfg`). No way to replay recorded events for debugging or test scheduling policies without a live cluster. - -3. **No picker plugin interface** — the existing plugin framework has `job_elect_fn` and `job_alloc_fn`, but no defined contract for how the picker receives state-change events, manages sessions, or communicates decisions back. The FCFS picker is tightly coupled to the scheduler internals. - -4. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. - -Operators need configurable scheduling policies, and developers need a way to debug/tune scheduling behavior without a live cluster. - -## Current State - -### What exists -- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn` -- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs by (priority desc, queue_time asc). Communicates via `jbuf` cbuf. -- **Scheduling loop** (`scheduler.c`): pick job → allocate → pend or dispatch → next -- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`. -- **Select expression memoization** (`select_eval_ctx_t`): per-session AST caching - -### What's missing -- Pluggable job picker with defined interface contract -- Configurable picker name (no recompilation) -- Core/driver separation: scheduling logic decoupled from controller I/O -- Simulation mode: replay recorded events, virtual clock, deterministic execution -- Runtime recording for debugging/tuning -- State snapshot/restore for replay efficiency - -## Desired Outcome - -### 1. Pluggable Job Picker - -The core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The picker retains today's dedicated thread, communicating with the allocator thread via the existing `jbuf` cbuf (superseded by the full three-thread producer/consumer contract in requirements.md Req 1 AC 3-4 — this brief predates that decision). - -Picker plugin interface (`xce_job_picker_iface_t`): - -| Category | Hooks | Purpose | -|----------|-------|---------| -| Lifecycle | `init`, `fini` | Setup and teardown | -| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called when events arrive | -| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle | -| Post-allocation | `on_dispatch`, `on_pend` | After allocation attempt | -| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) | - -### 2. Core/Driver Separation - -Scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it: - -| Mode | Driver | Input | Output | -|------|--------|-------|--------| -| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller | -| Simulation | Replay file | `.replay` capnp binary stream → core events | Core decisions → log | - -- **Mode-agnostic core**: no mode flag, no mode-specific code paths -- **Virtualized clock**: `xce_core_now(core)` with two-base + speed formula -- **Discrete event-driven virtual clock**: jumps to next event timestamp -- **Snapshot + delta replay**: load snapshot, replay delta events -- **Deterministic guarantee**: same initial state + same events = same decisions -- **Runtime recording control**: xceadmin `scheduler record start/stop` - -### 3. Single Builtin Picker - -`builtin_fcfs` internally implements queue-priority + FCFS job ordering — replicates today's behavior. -Per-queue picker overrides supported. - -### Queue vs Project — Roles - -| | Queue | Project | -|---|---|---| -| **Role** | Scheduling mechanism | Resource ownership | -| **Question** | HOW are jobs ordered? | WHO owns resources? | -| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy (future) | -| **Cardinality** | 1 job → 1 queue | 1 job → 1 project | - -## Approach - -**Event-Driven Core with Pluggable Picker + Dual-Mode Driver**: - -### Core scheduling loop - -```c -picker->begin_session(picker); -while ((job = picker->next_job(picker)) != NULL) { - if (try_allocate(job) > 0) { - picker->on_dispatch(picker, job->jobid); - on_decision(driver_ctx, DISPATCH, job); - } else if (can_preempt(job)) { - on_decision(driver_ctx, PREEMPT, job, victims); - picker->on_pend(picker, job->jobid); - } else { - on_decision(driver_ctx, PEND, job); - picker->on_pend(picker, job->jobid); - } -} -picker->end_session(picker); -``` - -Core knows nothing about project/queue/job ordering. `next_job()` hides all selection. - -### Clock virtualization - -```c -uint64_t xce_core_now(core) { - wall_ns = clock_gettime(CLOCK_MONOTONIC); - elapsed = wall_ns - core->real_clock_base; - return core->virtual_clock_base + (elapsed * speed_num) / speed_den; -} -``` - -Normal: bases=0, speed=1/1 → returns wall clock. Simulation discrete: speed=0/1 → frozen at event ts. - -### What stays unchanged -- Plugin framework hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved -- Allocation logic — unchanged (in scheduler-allocation sibling spec) -- Controller communication in driver — extended, not replaced - -### What's new -- `xce_job_picker_iface_t` — pluggable picker with full hook specification -- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t` -- `xce_core_event_t` — typed event interface -- `xce_core_decision_cb_t` — decision callback -- `.replay` capnp binary stream format + snapshot/restore -- `xce_core_now()` — virtualized clock accessor -- `builtin_fcfs` — single builtin picker -- `job_pick_plugin` on scheduler + per-queue override -- xceadmin recording commands - -## Scope - -### In -- Pluggable job picker interface with full hook specification -- Single builtin picker (`builtin_fcfs`) -- Configurable picker plugin with per-queue override -- Core/driver separation: passive core, normal driver, simulation driver -- Virtualized clock (`xce_core_now()` with two-base + speed formula) -- Discrete event-driven virtual clock -- `.replay` capnp binary stream format (HEADER + SNAPSHOT + events + DECISIONS) -- State snapshot/restore -- Deterministic replay guarantee -- Runtime recording control via xceadmin commands -- Data structures: `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t` -- Unit tests: picker hook behavior, simulation replay, snapshot/restore, deterministic verification -- HA soak regression for picker changes - -### Out -- Allocation iterators and parjob allocator (→ scheduler-allocation) -- Preemption policy and SIGSTP pipeline (→ scheduler-preemption) -- New picker policies beyond builtin_fcfs -- Hierarchical project resource quotas (→ project-resource-ownership) -- Dead field on `xce_sched_job_t` - -## Sibling Specs -- **scheduler-allocation**: Iterator-driven allocation, parjob allocator, resreq cache -- **scheduler-preemption**: Preemption policy, victim selection, SIGSTP pipeline - -## Constraints -- **C (gnu17)** — no C++, no external libraries beyond existing -- **Cancel-safe** — `next_job()` has pthread_cancel cancellation points -- **Session isolation** — picker state reset on begin_session -- **Backward compatible** — existing FCFS behavior unchanged -- **Deterministic replay** — same state + events → same decisions -- **Mode-agnostic core** — no mode flag, no mode-specific code paths -- **No wall-clock time in core** — all time through xce_core_now() diff --git a/.kiro/specs/scheduler-core/design.md b/.kiro/specs/scheduler-core/design.md deleted file mode 100644 index b0f12e1b..00000000 --- a/.kiro/specs/scheduler-core/design.md +++ /dev/null @@ -1,794 +0,0 @@ -# Technical Design: scheduler-core - -## Overview - -**Purpose**: This feature replaces the scheduler's hardcoded FCFS job-selection logic and its hard-wired coupling to the live controller with a pluggable picker interface and a passive, mode-agnostic scheduling core. The core is driven by either a normal-mode driver (live controller) or a simulation-mode driver (recorded `.replay` file), and both modes are required to produce byte-identical scheduling decisions given identical input — this is the spec's central, testable invariant. - -**Users**: Cluster operators configure job-picking policy without a rebuild (Req 2) and gain a `xceadmin scheduler record`/`dump` workflow. Scheduler developers gain a `--sim`/`--interactive` simulation driver to debug and tune scheduling policy against recorded production traces without a live cluster (Req 5). - -**Impact**: `xce_scheduler_t` (today: scheduling state + `apictx` + `cluster_cfg` in one struct, `scheduler.c`) splits into `xce_sched_core_t` (pure scheduling state, zero I/O) plus `xce_sched_driver_t` (normal, controller-attached) and `xce_sched_sim_driver_t` (simulation, replay-attached). The hardcoded `"xce_sched_fcfs"` string at `scheduler.c:43` is replaced by a configurable, plugin-registered picker. A new controller opcode (`OPCode_schedAdmin`) and a new xceadmin command group (`scheduler record`/`scheduler dump`) are added as pass-through admin plumbing. - -### Goals -- Job-picking policy configurable per-scheduler and per-queue without recompilation, with `builtin_fcfs` shipping as the zero-behavior-change default (Req 1-4). -- Scheduling core with zero controller I/O, wall-clock, or filesystem dependencies, driven identically by a normal or simulation driver (Req 5 AC 1-8). -- Runtime recording of live scheduling sessions to a self-contained `.replay` file, and a simulation driver + interactive REPL that replays it deterministically (Req 5 AC 9-22). -- Normal↔sim equivalence as a verifiable regression test: record a live session, replay it, diff the decisions — zero divergence (Req 5 AC 7.1). - -### Non-Goals -- New job-picking policies beyond `builtin_fcfs` (interface only; sibling work). -- Allocation iterators / `try_allocate()` internals (`scheduler-allocation`). -- Preemption policy / victim selection (`scheduler-preemption`) — this design defines `PreemptDecision`'s **wire shape** only (needed for `Req 5 AC 5`'s decision table to compile), not preemption logic. -- Hierarchical project resource quotas (`project-resource-ownership`). -- REPL state mutation, reverse execution, or scripting (explicitly deferred in Req 5 AC 22). - -## Boundary Commitments - -### This Spec Owns -- `xce_job_picker_iface_t` and the `builtin_fcfs` implementation. As-built: a new `src/scheduler/src/plugins/builtin_fcfs.c` implements the interface; `plugins/fcfs.c` was deliberately left unchanged (still `job_elect_fn`-only) as the live exercise target for `plugin.c`'s legacy compat wrapper (see Components — BuiltinFcfsPicker). -- `xce_sched_core_t`: all scheduling state, the picker/allocator thread pair, the driver↔core protocol (`core_push_events`, `core_go`, `on_decision`), and `xce_core_now()` clock virtualization. -- `xce_sched_driver_t` (normal-mode): controller event translation, decision publishing, and recording control (start/stop/status, periodic snapshot). -- `xce_sched_sim_driver_t` (simulation-mode): `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`, and the interactive REPL. -- The `.replay` capnp wire format (`ReplayHeader`, `SnapshotEntry`, `ReplayEvent`, `ReplaySession`, `ReplayDecision`, and their payload sub-structs) in `src/schema/controller.capnp`. -- The `OPCode_schedAdmin` request/reply pair and its **pass-through** dispatch case in the controller's `myreq_handler` (`src/controller/cluster.c`), plus its accompanying `src/controller/testsuite/` coverage — this spec's only footprint outside `src/scheduler/`. -- `xceadmin scheduler record start|stop|status` and `xceadmin scheduler dump <FILE>` (new `src/xceadmin/src/xceadmin/scheduler.py`). -- `job_pick_plugin` (global + per-queue) and `project_name` (per-queue) fields; `xce_sched_project_t` scaffolding (name + priority only — no quota logic). - -### Out of Boundary -- `try_allocate()` and the allocation iterator (`scheduler-allocation` owns the implementation; this spec only calls it from the allocator thread, unchanged signature). -- `can_preempt()` and victim-selection semantics (`scheduler-preemption` owns the algorithm; this spec owns only `PreemptDecision`'s wire shape, see Data Models). -- Hierarchical project quota, `resource_quota`/`resource_used`/`parent_project` on `xce_sched_project_t` (`project-resource-ownership`). -- Any controller state or persistence for recording (`OPCode_schedAdmin` is explicitly a stateless pass-through — the controller commits nothing, owns no WAL entry for it). -- REPL write-side mutation, semantic snapshot diff in xceadmin (both explicitly deferred, Req 5 AC 22). - -### Allowed Dependencies -- Existing plugin framework (`plugin.h`/`plugin.c`) — `xce_job_picker_iface_t` is a new plugin category registered through the same `plugin_register`/`key_find` mechanism, not a parallel system. -- Existing select-expression evaluator (`resreq.h`, `select_eval_ctx_t`) — used as-is by the allocator thread; the core does not re-implement or wrap it. -- libxce data-structure subset only inside the core: `ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti` (Req 5 AC 3). The core must **not** link `netio`, `delay`, `xceapi`, or `idgen`. -- `xcecli_ctx_t` API (`xcecli_jobdispatch`, `xcecli_set_pend_reason`, `xcecli_preempt`) — normal-mode driver only, never referenced from `xce_sched_core_t` or `xce_sched_sim_driver_t`. -- `xcecommon.wire`/`xcecommon.schema`/`xceadmin.discovery` (Python) — the new `xceadmin/scheduler.py` module reuses these exactly as `push.py` does; it does not open its own socket or reload the schema. - -### Revalidation Triggers -- Any change to `xce_job_picker_iface_t`'s hook signatures — every picker plugin (present and future) must be re-verified. -- Any change to `jbuf` cbuf capacity away from 1 — invalidates the determinism argument in Req 5 AC 7; `scheduler-allocation` and `scheduler-preemption` must both re-verify ordering assumptions. -- Any change to the `.replay` capnp schema's message set or `ReplayDecision.body` union — `scheduler-preemption`'s `PreemptDecision` consumer and any external tooling reading `.replay` files must re-check compatibility. -- Any change to `OPCode_schedAdmin`'s request/reply shape — `xceadmin.scheduler` module must be updated in lockstep (same repo, same PR). - -## Architecture - -### Existing Architecture Analysis - -Today, `xce_scheduler_t` (`src/scheduler/include/scheduler.h`) owns scheduling state (`jobs`, `queues`, `workers`, `plugins` kvmgrs), controller I/O (`apictx: xcecli_ctx_t`, `cluster_cfg: xce_cluster_config_t`), and three threads (`jobpicker`, `allocator`, main) wired through three `cbuf`s (`wbuf` cap 1, `jbuf` cap `njobs`≡1 today, `decisions` cap `ndecisions`≡5). `main.c`'s loop calls `sched_update` → `sched_begin_session` → `sched_go` → drains `decisions` cbuf with a hardcoded `switch (decision->op)` → `sched_end_session`. The picker plugin (`fcfs.c`) is selected by a literal string (`scheduler.c:43`) and implements the old `xce_sched_plugin_t->job_elect_fn` hook, tightly coupled to `sched->queues`/`sched->jobs` kvmgrs directly. - -**Preserved without change**: the three-thread/three-cbuf pattern (`wbuf`→picker, `jbuf`→allocator, `decisions`→driver); `plugin_register`/`key_find` plugin lookup; `select_eval_ctx_t` per-session memoization; `sched_queue_is_ready()`/`sched_job_is_ready()` (currently stub `return 1;` in `scheduler.c`, called as-is by both today's `fcfs_elect_job` and the new `builtin_fcfs`). - -**What changes**: `xce_scheduler_t` splits along the controller-I/O seam into `xce_sched_core_t` (no `apictx`/`cluster_cfg`) and `xce_sched_driver_t` (holds them). The picker plugin category becomes `xce_job_picker_iface_t`, replacing `job_elect_fn` (deprecated with a compatibility wrapper, Req 5 AC "Migration from old job_elect_fn"). - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph DriverLayer[Driver Layer] - NormalDriver[Normal Driver] - SimDriver[Simulation Driver] - Controller[Live Controller] - ReplayFile[Replay File] - Controller --> NormalDriver - ReplayFile --> SimDriver - end - - subgraph CoreLayer[Scheduler Core] - Core[Scheduler Core State] - Picker[Job Picker Plugin] - Allocator[Allocator Thread] - Core --> Picker - Picker --> Allocator - end - - NormalDriver -->|core_push_events core_go| Core - SimDriver -->|core_push_events core_go| Core - Core -->|on_decision| NormalDriver - Core -->|on_decision| SimDriver - - NormalDriver -->|xcecli| Controller - SimDriver -->|write| RecordFile[Sim Output Replay File] - - subgraph AdminPath[Admin Control Path] - XceAdmin[xceadmin scheduler CLI] - SchedAdminOp[OPCode_schedAdmin dispatch] - XceAdmin --> SchedAdminOp - SchedAdminOp --> Controller - end - Controller -->|forward request-reply| NormalDriver -``` - -**Architecture Integration**: -- **Selected pattern**: Event-sourced core with a swappable driver (ports-and-adapters, applied narrowly — the core is the hexagon, the two drivers are the only adapters). Selected in requirements.md as the only pattern that makes normal↔sim equivalence a structural property rather than a discipline (see `research.md`, Architecture Pattern Evaluation). -- **Domain boundaries**: Core owns scheduling *decisions*; drivers own I/O translation *only* — no scheduling logic in either driver. `OPCode_schedAdmin` dispatch is a third, thin adapter (controller-side) that never touches core or driver state directly, only forwards bytes. -- **Existing patterns preserved**: three-thread/cbuf pipeline, plugin registration, select-expression memoization. -- **New components rationale**: `SimDriver` and the `.replay` format are net-new (no existing analog); `SchedAdminOpcode` is new but structurally mirrors the existing `OPCode_upgradeConfig` case (see `research.md`). -- **Steering compliance**: no new wall-clock/network dependency inside `src/scheduler/`'s core files (`ha-async-io.md`'s async-I/O discipline extended to a synchronous core by design, not violated — the core has no I/O at all); `.capnp` used for the new wire format per `structure.md`'s "Cap'n Proto for wire formats" convention. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Scheduler core / drivers | C (gnu17) | `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t` | No new language/runtime; existing `src/scheduler/` toolchain | -| Wire format | Cap'n Proto (existing `controller.capnp`) | `.replay` stream, `OPCode_schedAdmin` request/reply | Extends the existing schema file, not a new one | -| Admin CLI | Python / `click` (existing xceadmin stack) | `scheduler record`/`scheduler dump` commands | New module (`xceadmin/scheduler.py`) in the existing package, same `pycapnp` schema loader | -| Build | Meson ≥ 1.5 (existing) | New `.c` files added to `src/scheduler/meson.build`; testsuite entries | No new build system | - -### Dependency Direction - -`Picker Plugin Interface → Scheduler Core → {Normal Driver, Simulation Driver} → {Controller (xcecli), Replay File I/O}`. Controller-side `SchedAdminOpcode` dispatch depends only on the capnp schema (shared, not owned by the core) and forwards to the driver without depending on `xce_sched_core_t` internals. xceadmin's `scheduler.py` depends on `xcecommon`/`xceadmin.discovery` only — never on any C header. Each layer imports only from layers to its left; the core never imports driver headers (enforced by the "Allowed Dependencies" list above — no `xcecli_ctx_t`/`netio` symbol may appear in `core.c`). - -## File Structure Plan - -### Directory Structure - -**As-built** (updated 2026-08-07 after implementation; see tasks.md's `## Implementation Notes` for the full rationale behind each split — all were 800-line-file-cap splits made during implementation, not scope changes): - -``` -src/scheduler/ -├── include/ -│ ├── scheduler.h # unchanged Phase 1; Phase 2 shrinks to xce_scheduler_t compat shim -│ ├── plugin.h # + xce_job_picker_iface_t (Req 1 AC 2), deprecation of job_elect_fn -│ ├── core.h # xce_sched_core_t, xce_core_event_t, decision cb type -│ ├── driver.h # xce_sched_driver_t (normal-mode) -│ ├── sim.h # xce_sched_sim_driver_t -- built as sim.h, not sim_driver.h (task 11.1, documented deviation) -│ ├── recording.h # record start/stop/status types, ReplayEvent/ReplaySession writer contract -│ ├── repl.h # REPL hook wrappers consumed by sim_loop.c -- no-op unless sim->interactive -│ ├── decision.h # pre-existing; extended with XCE_DECISION_PREEMPT (task 4.4) -│ └── sched/types.h # + job_pick_plugin, project_name on xce_sched_queue_t (Req 2) -├── src/ -│ ├── scheduler.c # sched_job_picker(): job_elect_fn != NULL delegates to the legacy loop, NULL runs the generic xce_job_picker_iface_t loop -│ ├── core.c # xce_sched_core_init/fini, jbuf(cap=1)/decisions cbuf plumbing -│ ├── core_events.c # xce_core_push_events() -- split out of core.c (800-line cap; task 4.3) -│ ├── core_session.c # xce_core_begin_session/xce_core_end_session -- split out of core.c (task 4.4) -│ ├── core_clock.c # xce_core_now(), tlog format integration -- the ONE file allowed to call clock_gettime() (task 4.5) -│ ├── driver.c # normal-mode driver: capnp<->core event translation, decision publishing, deferred admin-request handling -│ ├── recording.c # record start/stop/status, header/snapshot writes -│ ├── recording_events.c # ReplayEvent writes -- split out of recording.c (task 9.2/9.3) -│ ├── recording_session.c # ReplaySession begin/end writes -- split out of recording.c (task 9.2/9.3) -│ ├── repl.c # REPL command loop core (execution control, Req 5 AC 22) -│ ├── repl_break.c # REPL breakpoints + event/decision hook wrappers -- split out of repl.c (task 12.2) -│ ├── repl_info.c # REPL state-inspection commands -- split out of repl.c (task 12.2) -│ ├── repl_params.c # REPL snapshot/simulation-parameter commands -- split out of repl.c (task 12.3) -│ ├── repl_internal.h # shared repl_*.c internal declarations (task 12.2) -│ ├── snapshot.c # save_snapshot()/load_snapshot() orchestration, picker rebuild -│ ├── snapshot_wire.c # SnapshotData capnp encode/decode -- split out of snapshot.c (task 8.1) -│ ├── snapshot_load.c # snapshot restore application -- split out of snapshot.c (task 8.1) -│ ├── sim.c # simulation startup (Req 5 AC 13): replay header/snapshot load, virtual clock seed -│ ├── sim_loop.c # simulation event loop + REPL hook call sites (Req 5 AC 14/18) -│ ├── sim_record.c # --sim-output ReplayEvent/ReplaySession writes -- split out of sim_loop.c (task 11.5) -│ ├── sim_verify.c # --sim-verify decision comparator -- split out of sim_loop.c (task 11.4) -│ ├── handler.c # unchanged — capnp push/notify enqueue into pend_updates; scheduler_req_handler dispatches SchedAdminRequest -│ ├── plugin.c # + picker plugin registration category, + job_elect_fn compat - # wrapper (adapts old-style job_elect_fn-only plugins to - # xce_job_picker_iface_t, logs a deprecation warning at register time) -│ └── plugins/ -│ ├── builtin_fcfs.c # the native xce_job_picker_iface_t default picker (Req 3) -- NEW file, see note below -│ └── fcfs.c # UNCHANGED -- still registers job_elect_fn only; kept as the live exercise target for plugin.c's compat wrapper (task 1.2). NOT rewritten as originally planned here; see BuiltinFcfsPicker component notes. -├── testsuite/ -│ ├── test_picker_iface.c, test_picker_fcfs.c, test_picker_fcfs_session.c, test_picker_wiring.c, -│ │ test_picker_config_fields.c, test_picker_core_loop_boundary.c, test_picker_pendreason_factory_audit.c, -│ │ test_plugin_picker_registration.c # picker interface + builtin_fcfs + registration/compat-wrapper coverage -│ ├── test_core_events.c, test_core_session.c, test_core_full_cycle.c, test_core_isolation.c, -│ │ test_core_jbuf_capacity_pin.c # core contract + determinism-pin coverage -│ ├── test_clock.c # xce_core_now() two-base+speed formula -│ ├── test_driver_translate.c, test_driver_decision_cb.c, test_driver_session_ordering.c -│ ├── test_recording_path_validation.c, test_record_lifecycle.c, test_admin_request.c -│ ├── test_snapshot_save_load.c -│ ├── test_sim_startup.c, test_sim_event_loop.c, test_sim_isolation.c, test_sim_alt_events.c, -│ │ test_sim_verify.c, test_sim_record_output.c, test_sim_equivalence.c -│ └── test_repl_execution_control.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c, -│ test_repl_equivalence.c -src/controller/ -├── cluster.c # + case OPCode_schedAdmin (Req 5 AC 10) -└── testsuite/ - └── test_sched_admin_dispatch.c # standby rejection, disconnected-scheduler, timeout, success round-trip - # (name differs from this plan's original test_sched_admin.c; same coverage, - # registered in controller_test.c same as existing opcode tests) -src/schema/ -└── controller.capnp # + SchedAdminRequest/Reply, ReplayHeader/.../ReplayDecision + payload structs (see Data Models); - # + WalCatchupEntry.payload/batchPayloads switched Data->List(UInt8) (bug-1505, follow-up - # outside this spec's boundary, same codec-crash class as SnapshotData.internalState/bug-1504) -src/xceadmin/src/xceadmin/ -├── cli.py # + @cli.group() scheduler; nested @scheduler.group() record with - # start/stop/status commands, plus @scheduler.command() dump (thin layer only) -└── scheduler.py # record_start/stop/status(), dump() — mirrors push.py's pattern -src/xceadmin/tests/ -├── test_scheduler.py # mock-TLS-server tests for record start/stop/status + dump parsing -└── test_cli_scheduler_mock.py # additional CLI-level mock-server coverage (task 10.6, beyond this plan's original file list) -``` - -### Modified Files -- `src/scheduler/src/scheduler.c` — Phase 1: hardcoded picker string removed; Phase 2: `sched_job_picker()` branches `job_elect_fn != NULL` (legacy plugins, delegates to their own self-contained loop) vs `NULL` (native `xce_job_picker_iface_t` pickers, generic per-session loop) — `scheduler.c` was not retired; it remains the picker-dispatch and allocator-loop home. -- `src/scheduler/src/plugins/fcfs.c` — **as-built: left unchanged**, not rewritten. `plugins/builtin_fcfs.c` (new file) implements `xce_job_picker_iface_t` and is the registered default (`schedobj.c`); `fcfs.c` keeps registering `job_elect_fn` only, deliberately, as the live exercise target for `plugin.c`'s legacy compat wrapper (task 1.2) — see BuiltinFcfsPicker component notes for the full rationale. -- `src/scheduler/src/main.c` — decision-drain `switch` replaced by `on_decision` callback registration (Req 5 AC 6); also gained the `--sim`/`--interactive`/`--record-dir`/`--sim-*` CLI surface (tasks 9.1, 11.1). -- `src/controller/cluster.c` — one new `case OPCode_schedAdmin:` block, structurally mirroring `OPCode_upgradeConfig` (primary-gate, delegate, explicit reply every path). -- `src/schema/controller.capnp` — additive only: new `OPCode_schedAdmin` enum value, new structs, two new union members on existing `Request.body`/`RequestReply.data` unions (ordinals `@31`/`@19` respectively — next available, verified against current schema in `research.md`). -- `src/xceadmin/src/xceadmin/cli.py` — one new `@cli.group() scheduler`, containing a nested `@scheduler.group() record` (with `start`/`stop`/`status` commands) plus a sibling `@scheduler.command() dump` — matching the literal CLI syntax `xceadmin scheduler record start`/`xceadmin scheduler dump`, not 4 flat commands. - -## System Flows - -### Normal-Mode Scheduling Round (driver ↔ core) - -```mermaid -sequenceDiagram - participant Ctrl as Controller - participant Drv as Normal Driver - participant Core as Scheduler Core - participant Pick as Picker Thread - participant Alloc as Allocator Thread - - Ctrl->>Drv: capnp push or notify - Drv->>Drv: buffer into pend_updates - Drv->>Core: core_push_events - Core->>Pick: state change hooks, synchronous, picker idle - Drv->>Core: core_go, non blocking - Core->>Pick: signal via wbuf - Pick->>Pick: begin_session then next_job loop - Pick->>Alloc: send jobid via jbuf per job - Alloc->>Alloc: try_allocate then can_preempt - Alloc->>Drv: send decision via decisions cbuf - Drv->>Ctrl: on_decision publishes dispatch, pend reason, or preempt - Alloc->>Drv: end of session marker via decisions cbuf - Drv->>Core: core_end_session -``` - -Key decision not obvious from the diagram: decisions publish to the controller **immediately per-decision**, not batched at session end (Req 5, "Decision publishing") — the drain loop in the driver thread interleaves `cbuf_get`/`on_decision` one at a time. - -### `OPCode_schedAdmin` Recording Control (admin path) - -```mermaid -sequenceDiagram - participant CLI as XceAdmin SchedulerModule - participant Ctrl as Controller ClusterDispatch - participant Sched as Scheduler NormalDriver - - CLI->>Ctrl: schedAdmin request, action plus recordFile - Ctrl->>Ctrl: check ha_is_primary, reject standby - Ctrl->>Sched: forward via scheduler endpoint, five second timeout - Sched->>Sched: validate recordFile, reject or resolve under record_dir - Sched->>Sched: save_snapshot then write header and snapshot entry - Sched-->>Ctrl: schedAdmin reply, ok plus recording state and counters - Ctrl-->>CLI: relay reply to caller -``` - -Key decision not obvious from the diagram: the controller is a pure pass-through — it does not interpret `SchedAdminRequest` fields beyond the opcode dispatch, and owns no recording state (Boundary Commitments, "This Spec Owns" — recording logic lives entirely in the scheduler driver). - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|---|---|---|---|---| -| 1.1 | Core loop calls `next_job()` only, no ordering logic | SchedulerCore, JobPicker | `xce_job_picker_iface_t.next_job` | Normal-Mode Round | -| 1.2 | Picker hook categories (lifecycle/state/session/post-alloc/optional) | JobPicker | `xce_job_picker_iface_t` | — | -| 1.3 | Three-thread cbuf model | SchedulerCore | `wbuf`/`jbuf`/`decisions` cbufs | Normal-Mode Round | -| 1.4 | Picker/allocator/main-loop pseudocode contract | SchedulerCore, JobPicker | `core_push_events`/`core_go` | Normal-Mode Round | -| 1.5 | Cancel-safe, session-aware picker | JobPicker | `begin_session`/`end_session`/`next_job` | — | -| 1.6 | Replace hardcoded picker string | SchedulerCore | `plugin_register`/`key_find` | — | -| 1.7 | Preserve thread architecture, decision publish via callback | SchedulerCore | `on_decision` cb | Normal-Mode Round | -| 2.1 | `job_pick_plugin` on `xce_scheduler_t`, default `builtin_fcfs` | SchedulerCore | — | — | -| 2.2 | `job_pick_plugin` per-queue override | JobPicker | `xce_sched_queue_t.job_pick_plugin` | — | -| 2.3 | `project_name` on queue | SchedulerCore | `xce_sched_queue_t.project_name` | — | -| 2.4 | Config-driven picker name, no rebuild | SchedulerCore | scheduler config | — | -| 2.5 | Unregistered plugin fails loud | SchedulerCore | `plugin_register`/`key_find` | — | -| 2.6 | Picker contract documented (cancel-safe, session-aware) | JobPicker | `xce_job_picker_iface_t` | — | -| 3.1 | `builtin_fcfs` auto-registered as default | BuiltinFcfsPicker | plugin registration | — | -| 3.2 | Hook-by-hook FCFS implementation | BuiltinFcfsPicker | `xce_job_picker_iface_t` | — | -| 3.3 | Private `ordq_t` state, no core queries in `next_job` | BuiltinFcfsPicker | — | — | -| 3.4 | Ordering equivalence with today's `fcfs_elect_job` | BuiltinFcfsPicker | — | — | -| 3.5 | Dedicated picker thread retained, cancel points | BuiltinFcfsPicker, SchedulerCore | `next_job`/`cbuf_get(wbuf)` | Normal-Mode Round | -| 3.6 | `end_session` no-op, `pendJobs` carries over | BuiltinFcfsPicker | — | — | -| 4.1 | Job-picking order regression check | BuiltinFcfsPicker | — | test_picker_fcfs.c | -| 4.2 | Existing scheduler unit tests pass unmodified | SchedulerCore | — | testsuite/ | -| 4.3 | HA soak — no job loss/divergence | SchedulerCore | — | `/xce-test-ha` | -| 5.1 | Mode-agnostic core, no mode flag | SchedulerCore | `xce_sched_core_init()` | — | -| 5.2 | Passive event-driven interface, zero I/O deps | SchedulerCore | `core_push_events`/`on_decision` | — | -| 5.3 | Virtualized clock, two-base+speed formula | SchedulerCore | `xce_core_now()` | — | -| 5.4 | Core event types (typed payloads) | SchedulerCore | `xce_core_event_t` | — | -| 5.5 | Decision output types (DISPATCH/PEND/PREEMPT) | SchedulerCore | `xce_decision_t` | — | -| 5.6 | Decision output callback | SchedulerCore, NormalDriver | `xce_core_decision_cb_t` | Normal-Mode Round | -| 5.7 | Deterministic guarantee + normal↔sim equivalence | SchedulerCore | `jbuf` cap=1, hash-iteration audit, PRNG seed | test_sim_equivalence.c | -| 5.8 | Driver pending-event queue | NormalDriver | `pend_updates` dataq | Normal-Mode Round | -| 5.9 | Controller integration (capnp translate, publish decisions) | NormalDriver | `xcecli_ctx_t` | Normal-Mode Round | -| 5.10 | Runtime recording control (`OPCode_schedAdmin`) | NormalDriver, SchedAdminOpcode, XceAdminSchedulerCLI | `SchedAdminRequest`/`Reply` | Admin Path | -| 5.11 | Periodic snapshot during recording | NormalDriver | `record_snapshot_interval` | — | -| 5.12 | Session response time metrics | SchedulerCore | `SessionSummary` | — | -| 5.13 | Simulation mode startup (load header+snapshot) | SimDriver | `.replay` load sequence | — | -| 5.14 | Discrete event-driven virtual clock | SimDriver | `--sim-speed` | — | -| 5.15 | `--sim-verify` deterministic replay verification | SimDriver | `decision_eq` | — | -| 5.16 | `--sim-alt-events` what-if branching | SimDriver | alt-events file | — | -| 5.17 | Simulation isolation (no controller deps) | SimDriver | `xce_sched_sim_driver_t` | — | -| 5.18 | `ReplaySession` as session boundary | SimDriver | event loop | — | -| 5.19 | `--sim-output` simulation record file | SimDriver | `.replay` writer | — | -| 5.20 | `.replay` capnp stream format | ReplayFormat | schema structs | — | -| 5.21 | Snapshot serialization / picker rebuild | SchedulerCore, ReplayFormat | `save_snapshot`/`load_snapshot` | — | -| 5.22 | Interactive REPL | SimDriver | command loop | — | -| 6.1 | Resreq immutability | SchedulerCore | — | — | -| 6.2 | Pend reason factory (`make_pendreason()`) | SchedulerCore | `lib/preason.c` | — | -| 6.3 | Pointer-identity fast path | SchedulerCore | — | — | -| 6.4 | Select expression memoization preserved | SchedulerCore | `select_eval_ctx_t` | — | -| 6.5 | Snapshot forward-compatible with object sharing | ReplayFormat | `SnapshotData` | — | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|--------------|--------|---------------|---------------------------|-----------| -| JobPicker (interface) | Scheduler Core | Defines the pluggable job-selection contract | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 | plugin.h (P0) | Service | -| BuiltinFcfsPicker | Scheduler Core | Default picker replicating today's FCFS order | 3.1-3.6, 4.1 | JobPicker (P0) | Service | -| SchedulerCore | Scheduler Core | Passive, mode-agnostic scheduling engine | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 | JobPicker (P0), allocation/preemption (P1, sibling specs) | Service, State | -| NormalDriver | Scheduler Driver | Controller-attached driver, recording control | 5.6, 5.8-5.11 | SchedulerCore (P0), `xcecli_ctx_t` (P0) | Service, Event | -| SimDriver | Scheduler Driver | Replay-attached driver, virtual clock, REPL | 5.13-5.19, 5.22 | SchedulerCore (P0), ReplayFormat (P0) | Service, State | -| ReplayFormat | Wire Protocol | `.replay` capnp schema, `SnapshotData` | 5.20, 5.21, 6.5 | `controller.capnp` (P0) | Batch | -| SchedAdminOpcode | Controller | Pass-through dispatch for `OPCode_schedAdmin` | 5.10 | `cluster.c` `myreq_handler` (P0) | API | -| XceAdminSchedulerCLI | Tooling | `scheduler record`/`scheduler dump` commands | 5.10 | `xcecommon.wire`/`discovery` (P0) | API | - -### Scheduler Core - -#### JobPicker - -| Field | Detail | -|-------|--------| -| Intent | Contract every job-selection strategy implements; hides all ordering logic from the core loop | -| Requirements | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 | - -**Responsibilities & Constraints** -- Owns iteration order over pending jobs; the core never inspects queue/job internals for ordering. -- Cancel-safe (`pthread_cancel` points in `next_job()`) and session-scoped (`begin_session` resets iteration state). -- State-change hooks run synchronously in the driver thread during `core_push_events()`; session hooks run in the picker thread; post-allocation hooks run in the allocator thread (Req 1 AC 3). - -**Dependencies** -- Inbound: SchedulerCore — invokes every hook (P0) -- Outbound: none — picker implementations must not call back into core internals outside the hook parameters they're given - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -typedef struct { - int (*init)(void* picker, xce_sched_core_t* core); - void (*fini)(void* picker); - int (*begin_session)(void* picker); - xce_sched_job_t* (*next_job)(void* picker); - int (*end_session)(void* picker); - int (*on_dispatch)(void* picker, uint64_t jobid); - int (*on_pend)(void* picker, uint64_t jobid); - int (*on_queue_update)(void* picker, xce_sched_queue_t* queue); - int (*on_job_push)(void* picker, xce_sched_job_t* job); - int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state); - int (*on_job_update)(void* picker, xce_sched_job_t* job); - int (*on_config)(void* picker, xce_conf4sched_t* config); /* MAY be NULL */ - int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); /* MAY be NULL */ -} xce_job_picker_iface_t; -``` -- Preconditions: `init()` called exactly once before any other hook; `begin_session()` precedes every `next_job()` sequence. -- Postconditions: `next_job()` returns NULL exactly when no ready job remains for the current session. -- Invariants: hook implementations never block indefinitely (picker thread holds no lock the allocator needs); `next_job()` never allocates a job the same session has already returned. - -**Implementation Notes** -- Integration: registered via `plugin_register`/`key_find` under the picker plugin category (new `plugin.c` registration table entry). -- Validation: `test_picker_fcfs.c` feeds hooks directly and asserts `next_job()` order — no live core needed (Req 3 AC 3's testability rationale). -- Risks: a picker implementation that queries core state directly during `next_job()` (bypassing the hook-fed private state pattern) would still compile but silently reintroduce lock contention — not enforceable at the interface level, called out for code review. - -#### BuiltinFcfsPicker - -| Field | Detail | -|-------|--------| -| Intent | Default `xce_job_picker_iface_t` implementation; replicates today's `fcfs_elect_job` ordering exactly through the new interface | -| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1 | - -**Responsibilities & Constraints** -- Maintains private state only — `ordq_t queues` (ordered by `queue->priority desc`) and, per queue, an `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). Never queries `sched->queues`/`sched->jobs` during `next_job()` (Req 3 AC 3). -- Every hook's behavior is fully specified in requirements.md Req 3 AC 2's hook-by-hook table — not repeated here; this block covers integration and testability, not the algorithm. -- Auto-registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` at scheduler startup (Req 3 AC 1). -- `end_session()` is a no-op; undispatched jobs carry over in `pendJobs` to the next round unchanged (Req 3 AC 6). - -**Dependencies** -- Inbound: SchedulerCore — invokes every `xce_job_picker_iface_t` hook, same as any picker (P0) -- Outbound: none — implements the interface, depends on nothing beyond it - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -Implements `xce_job_picker_iface_t` (struct shown under JobPicker, above) with the `fcfs_plugin_data_t` private state described in Responsibilities. No additional public surface beyond the interface itself. -- Preconditions/Postconditions/Invariants: inherits JobPicker's contract in full; adds no relaxations or extensions. - -**Implementation Notes** -- Integration, as-built: implemented in a **new** file, `plugins/builtin_fcfs.c`, registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` (`schedobj.c`) — NOT a rewrite of `plugins/fcfs.c` in place, as originally planned above. `fcfs.c` was deliberately left untouched (still `job_elect_fn`-only, unregistered as default) because task 1.2's `job_elect_fn` compat wrapper needs a real legacy-style plugin to exercise; `fcfs.c` fills that role, and `scheduler.c`'s `sched_job_picker()` dispatch (`job_elect_fn != NULL` vs `NULL`) depends on both variants existing side by side. Functionally equivalent to the original plan (one default picker implementing the new interface, ordering-equivalent to legacy `fcfs_elect_job`); only the file-layout mechanics differ. -- Validation: `test_picker_fcfs.c` is this component's primary test — feeds `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly and asserts `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order, matching today's `fcfs_elect_job` (Req 4.1's regression check). -- Risks: ordering-equivalence regression is the main risk (any drift from today's `fcfs_elect_job` output is a behavior change operators didn't opt into) — mitigated by `test_picker_fcfs.c` asserting against the documented ordering rule directly, plus the HA soak in Req 4.3. - -#### SchedulerCore - -| Field | Detail | -|-------|--------| -| Intent | Owns all scheduling state; drives the picker/allocator threads; exposes the driver↔core protocol | -| Requirements | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 | - -**Responsibilities & Constraints** -- Zero direct dependency on controller communication, capnp, network I/O, filesystem, or wall-clock time (Req 5 AC 2) — enforced by the "Allowed Dependencies" link list, not just convention. -- `jbuf` cbuf capacity is hardcoded to 1 inside `xce_sched_core_init()` — never a parameter (Req 5 AC 7, pinned against the existing `njobs` FIXME in `scheduler.c:384`). -- All internal time via `xce_core_now()`; no direct `time()`/`clock_gettime()`/`gettimeofday()` calls anywhere in `core.c`. - -**Dependencies** -- Inbound: NormalDriver, SimDriver — both call `core_push_events`/`core_go`, register `on_decision` (P0) -- Outbound: JobPicker (P0); `try_allocate()`/`can_preempt()` from sibling specs (P1 — signature-stable dependency, no behavior assumption) -- External: none - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -int xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed); -int xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents); -int xce_core_go(xce_sched_core_t*); -int xce_core_end_session(xce_sched_core_t*); -uint64_t xce_core_now(xce_sched_core_t*); -int xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out); -int xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in); -void xce_sched_core_fini(xce_sched_core_t*); -``` -- Preconditions: `core_push_events()` only valid when no session is active (`session_active == 0`); `core_go()` returns `-EINVAL` if a session is already active. `xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()` are likewise only valid when `session_active == 0` — both return `-EBUSY` if called mid-session, rather than silently capturing or restoring torn state. Every caller in this design (record start, periodic checkpoint, simulation startup) already invokes them between sessions by construction; the `-EBUSY` return is a defensive contract check, not a path any correct caller should hit. -- Postconditions: after `on_decision` delivers the end-of-session `NULL` marker, `session_active` is guaranteed to transition to 0 on the subsequent `core_end_session()` call. -- Invariants: given identical `(config, event sequence)`, `core_push_events`+`core_go` produce an identical decision sequence (Req 5 AC 7) — the whole point of the design. - -##### State Management -- State model: scheduling state (jobs/queues/workers/resources/resvals/users/projects) held in `kvmgr`s inside `xce_sched_core_t`, mirroring today's `xce_scheduler_t` layout minus `apictx`/`cluster_cfg`. -- Persistence & consistency: no persistence inside the core itself — `save_snapshot()`/`load_snapshot()` are the only serialization boundary, and only the driver decides when to invoke them. -- Concurrency strategy: three-thread cbuf pipeline unchanged from today; clock atoms (`virtual_clock_base`, `real_clock_base`, `speed_num`, `speed_den`) are lock-free `_Atomic uint64_t`, updated by the driver as a group before each event dispatch. - -**Implementation Notes** -- Integration: `core_push_events`/`core_go`/`on_decision` are the only surface either driver touches — no driver reaches into `xce_sched_core_t` internals. -- Validation: `test_core_events.c` (contract-level), `test_sim_equivalence.c` (record→replay→diff, Req 5 AC 7.1 — the spec's central regression test). -- Risks: the `kvmgr` iteration-order audit (Req 5 AC 7, "Stabilized hash table iteration") is a Phase 2 deliverable with real but bounded scope — jobs/queues/users/projects only (per `research.md` and requirements.md's explicit Phase 2 task). - -### Scheduler Driver - -#### NormalDriver - -| Field | Detail | -|-------|--------| -| Intent | Connects the core to a live controller; owns recording control | -| Requirements | 5.6, 5.8, 5.9, 5.10, 5.11 | - -**Responsibilities & Constraints** -- Translates capnp push/notify into `xce_core_event_t`, buffers between sessions in `pend_updates` (existing dataq swap mechanism). -- Publishes each decision immediately via `xcecli_jobdispatch`/`xcecli_set_pend_reason`/`xcecli_preempt` as it drains from `decisions` cbuf — not batched at session end. -- Recording is boolean-gated: zero file I/O, zero extra allocation when `recording == false` (Req 5 AC 10, Performance). -- Validates `recordFile` before ever calling `fopen()`: reject `/` or `..`, resolve under `record_dir`, confirm containment via `realpath()` (Req 5 AC 10, "Record file path validation" — this design inherits, not re-derives, that requirement). -- `record_dir` is provisioned exactly one way: a `--record-dir DIR` scheduler startup CLI flag (default `/var/log/xce/`), parsed alongside the existing normal-mode arguments in `main.c` — never a capnp field on any request path (`SchedAdminRequest` included), so nothing reachable over the wire can redirect where recordings are written. Matches the existing `--sim-*` flag family's provisioning style (CLI-only, not config-pushed). - -**Dependencies** -- Inbound: `main.c` (process entry point) (P0) -- Outbound: SchedulerCore (P0), `xcecli_ctx_t` (P0) -- External: `SchedAdminOpcode` dispatch (P1 — receives forwarded requests, not a caller relationship) - -**Contracts**: Service [x] / Event [x] - -##### Service Interface -```c -typedef struct { - xce_sched_core_t core; - xcecli_ctx_t apictx; - xce_cluster_config_t cluster_cfg; - dataq_t* pend_updates[2]; - char* record_dir; /* from --record-dir CLI flag at startup; default "/var/log/xce/"; - never settable via any request-path field */ - char* record_file_path; - FILE* record_file; - bool recording; - uint64_t record_event_count; - uint64_t record_snapshot_interval; - uint64_t record_start_ts; -} xce_sched_driver_t; - -int sched_driver_run(xce_sched_driver_t*, xce_conf4sched_t*, xce_cluster_config_t*, void* handler); -int sched_driver_record_start(xce_sched_driver_t*, const char* record_file /* bare filename or NULL */); -int sched_driver_record_stop(xce_sched_driver_t*); -int sched_driver_record_status(xce_sched_driver_t*, SchedAdminReply* out); -``` -- Preconditions: `record_start` fails with `-EALREADY`-equivalent if already recording; `record_stop` fails if not recording. -- Postconditions: on `record_start` success, `ReplayHeader` + initial `SnapshotEntry` are durably on disk before the reply is sent. -- Invariants: `recording == true` iff `record_file != NULL`. - -##### Event Contract -- Subscribed events: capnp push/notify from the controller (existing handler.c path, unchanged). -- Published events: decisions to the controller via `xcecli_ctx_t`; `SchedAdminReply` back through the forwarding path on recording-control requests. -- Ordering / delivery guarantees: decisions publish in allocator output order; recording writes events/decisions in the same order they cross the driver thread. - -**Implementation Notes** -- Integration: `main.c`'s mode-selection branch calls `sched_driver_run()` when `--sim` is absent. -- Validation: recording lifecycle tested against a real filesystem (temp dir) — start/stop/status round trip, path-validation rejection cases, disk-full simulated via a read-only mount point. -- Risks: `OPCode_schedAdmin` ships with the same primary/standby gate as `OPCode_upgradeConfig` and no additional RBAC/auth check — a deliberate scope decision (project owner, 2026-08-02), not an oversight. Strengthening authentication for this opcode (and `config push`, if it also lacks one) is deferred to a follow-up spec. See Open Questions/Risks. - -#### SimDriver - -| Field | Detail | -|-------|--------| -| Intent | Connects the core to a `.replay` file; owns virtual clock, verification, and the REPL | -| Requirements | 5.13-5.19, 5.22 | - -**Responsibilities & Constraints** -- Contains no `xcecli_ctx_t`, `xce_cluster_config_t`, capnp socket, or controller RPC of any kind (Req 5 AC 17) — enforced by the struct definition itself having no such fields. -- Discrete virtual clock: jumps directly to the next event's timestamp, never a fixed tick (Req 5 AC 14). -- REPL is a single `fgets` loop in the same thread as the simulation loop — no REPL thread (Req 5 AC 22, Implementation note). - -**Dependencies** -- Inbound: `main.c` (P0, when `--sim` is present) -- Outbound: SchedulerCore (P0), ReplayFormat (P0) -- External: none - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -typedef struct { - xce_sched_core_t core; - FILE *replay_file, *alt_events_file, *record_file; - uint64_t sim_speed_num, sim_speed_den; - bool verify; - bool interactive; -} xce_sched_sim_driver_t; - -int sched_sim_driver_run(xce_sched_sim_driver_t*, /* CLI arguments struct */ const void* args); -``` -- Preconditions: replay file's second message must be a `SnapshotEntry` (rejected otherwise, Req 5 AC 20). -- Postconditions: with `--sim-output`, every session cycle appends exactly one `ReplaySession` to the output file before the next `ReplayEvent` batch begins. -- Invariants: REPL `step`/`continue` to completion produces byte-identical output to batch `--sim` (Req 5 AC 22, "Deterministic replay is preserved"). - -##### State Management -- State model: same `xce_sched_core_t` as normal mode, loaded via `load_snapshot()` instead of live events. -- Persistence & consistency: `--sim-output` writes a new independent `.replay` file; never mutates the input file. -- Concurrency strategy: single-threaded driver loop around the same core thread pair; REPL blocks the driver loop intentionally (no wall-clock deadline in simulation). - -**Implementation Notes** -- Integration: `main.c`'s mode-selection branch calls `sched_sim_driver_run()` when `--sim` is present. -- Validation: `test_sim_equivalence.c` drives this component directly (record via NormalDriver's test harness, replay via this one, diff). -- Risks: none beyond what's already captured under SchedulerCore (the determinism guarantee is a core property; SimDriver only needs to not violate it, e.g. by never bypassing `core_push_events`/`core_go`). - -### Wire Protocol - -#### ReplayFormat - -| Field | Detail | -|-------|--------| -| Intent | `.replay` capnp stream schema and `SnapshotData` | -| Requirements | 5.20, 5.21, 6.5 | - -**Responsibilities & Constraints** -- Additive-only changes to `controller.capnp` — reuses existing structs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) wherever they already exist (verified in `research.md`); only defines new structs where none exist. -- `SnapshotData` field shapes track the *actual current* C struct field sets (e.g. `xce_sched_user_t` is `{name}` only today, `xce_sched_project_t` is `{name}` only today) — the schema does not invent fields the C structs don't have yet. - -**Dependencies** -- Inbound: SchedulerCore (`save_snapshot`/`load_snapshot`), NormalDriver (recording writer), SimDriver (reader/writer) (all P0) -- Outbound: none -- External: none - -**Contracts**: Batch [x] - -##### Batch / Job Contract -- Trigger: `record start` (normal mode) or `--sim-output` (simulation mode) write a `.replay` stream incrementally; `--sim`/`xceadmin scheduler dump` read one. -- Input / validation: first message must be `ReplayHeader`; second must be `SnapshotEntry` (Req 5 AC 20) — both enforced at read time, not just documented. -- Output / destination: append-only file; `record stop` / clean simulation completion writes `ReplayEvent{endSession}`. -- Idempotency & recovery: a write failure mid-message truncates cleanly to the last complete message (Req 5 AC 10, error handling) — no partial-message corruption possible because each message is a full capnp frame written atomically at the `fwrite` level. - -**New capnp structs** (additive to `controller.capnp`; `ReplayHeader`/`ReplayEvent`/`ReplaySession`/`ReplayDecision`/`SessionSummary` wrapper structs already fully specified in requirements.md Req 5 AC 20 — not repeated here): - -```capnp -# New event payloads (JOBS_PUSH/WORKERS_PUSH/etc reuse existing Conf4Sched, -# SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages — -# verified present in controller.capnp, research.md). - -struct JobStateChangeEvent $C.mapname("job_state_change_event_t") { - jobid @0: UInt64; - oldState @1: UInt32; - newState @2: UInt32; - exitCode @3: Int32; -} - -struct JobUpdateEvent $C.mapname("job_update_event_t") { - jobid @0: UInt64; - changedFields @1: UInt32; # bitmask: JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ - job @2: Job; # full updated record; reuses existing Job struct -} - -# New decision payloads for ReplayDecision.body (Req 5 AC 5). -# DISPATCH/PEND reuse existing JobAlloc/JobPendReason; PREEMPT is new -# (wire shape owned here per Boundary Commitments; victim-selection -# semantics owned by scheduler-preemption — see research.md). -# -# AC 5's field list for DISPATCH is "jobid, workerid, allocations[]" -# (singular workerid). JobAlloc already generalizes this correctly for -# multi-host jobs: JobAlloc.jobid covers jobid, JobAlloc.allocs is -# List(HostAlloc) with each HostAlloc.host as that allocation's worker -# id and HostAlloc.allocs: List(ResAlloc) as its resource/amount pairs -# -- verified against controller.capnp, no new fields needed. jobid -# lives ONLY in JobAlloc.jobid (not duplicated at DispatchDecision -# level) to avoid two unsynchronized sources of truth for the same job. - -struct DispatchDecision $C.mapname("sched_dispatch_decision_t") { - alloc @0: JobAlloc; -} - -struct PendDecision $C.mapname("sched_pend_decision_t") { - jobid @0: UInt64; - reason @1: JobPendReason; -} - -struct PreemptVictim $C.mapname("preempt_victim_t") { - jobid @0: UInt64; - workerid @1: Text; -} - -struct PreemptDecision $C.mapname("sched_preempt_decision_t") { - jobid @0: UInt64; - victims @1: List(PreemptVictim); - reason @2: JobPendReason; -} - -# SnapshotData — full core-state checklist from requirements.md Req 5 AC 21, -# given a concrete wire shape. Reuses existing List(...) payload structs -# for domain objects already carried elsewhere on the wire; the two -# scheduler-internal-only caches (shared-resource tracking, unsatisfiable- -# resreq cache) and session/PRNG state have no existing external contract, -# so they are versioned opaque blobs the core alone reads/writes -- no -# consumer outside save_snapshot()/load_snapshot() ever inspects their -# internal shape (WHAT not HOW: the boundary that matters is "round-trips -# through the core, opaque to everyone else"). - -struct SnapshotData $C.mapname("sched_snapshot_data_t") { - config @0: Conf4Sched; - jobs @1: SchedJobData; - queues @2: List(QueueConf); - workers @3: WorkerData; - resources @4: ResourceData; - resvals @5: ResValList; - users @6: List(SchedUser); - projects @7: List(SchedProject); - prngSeed @8: UInt64; - internalState @9: Data; # shared-resource tracking + unsatisfiable-resreq - # cache + session/iterator state, core-opaque - internalVersion @10: UInt32; # format tag for internalState, so a future - # incompatible internal layout fails loud - # instead of misinterpreting old bytes -} - -struct SchedUser $C.mapname("xce_sched_user_t") { - name @0: Text; # matches xce_sched_user_t today (name only) -- - # extend in lockstep if xce_sched_user_t gains fields -} - -struct SchedProject $C.mapname("xce_sched_project_t") { - name @0: Text; - priority @1: Int32; -} -``` - -**Implementation Notes** -- Integration: `save_snapshot()`/`load_snapshot()` are the only functions that construct/consume `SnapshotData`; no other component touches `internalState`'s bytes. -- Validation: as-built, round-trip coverage for every new struct lives in `src/controller/testsuite/test_e2e_capnp.c` (task 7.5), not a dedicated `test_replay_format.c` — this repo's existing convention is one shared capnp round-trip test file per schema, and this spec's new structs were added to it rather than starting a second one; `internalVersion` mismatch is a dedicated negative test within it. -- Risks: `internalState`'s opaque-blob design trades wire-level inspectability for implementation flexibility — acceptable because AC 21 only requires the *core* to round-trip correctly, never that `xceadmin scheduler dump` render its internals meaningfully (dump can print `internalState`'s byte length and `internalVersion` only). - -### Controller - -#### SchedAdminOpcode - -| Field | Detail | -|-------|--------| -| Intent | Pass-through dispatch for `OPCode_schedAdmin` in the controller's request handler | -| Requirements | 5.10 | - -**Responsibilities & Constraints** -- Structurally mirrors the existing `OPCode_upgradeConfig` case (`cluster.c:670-705`) for the primary-gate/delegate/explicit-reply shape: primary-only gate via `ha_is_primary()`, explicit reply on every path, 5s timeout. -- Owns zero recording state — every field of `SchedAdminRequest`/`Reply` passes through untouched. -- **Blocking-wait admission (hardened 2026-08-19, design review follow-up)**: like `OPCode_upgradeConfig` (in-process commit-barrier wait, `COMMIT_WAIT_RESERVED`-admitted), this case now also admission-gates its blocking wait. Originally (accepted, project owner, 2026-08-02) this case blocked its `vtask` handler thread for up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock`, with no reservation — a boundary case reviewed against `ha-async-io.md`'s vtaskmgr admission rule and accepted on the grounds that `xceadmin scheduler` commands are cluster-administrator-triggered, low-frequency, human-paced traffic, not expected to burst concurrently the way heartbeat/replication traffic does. A retrospective design review flagged this as a real (if low-probability) starvation risk under a burst of concurrent/scripted requests, since each blocked vtask thread occupies a worker for up to 5s and `c->sched.lock` already serializes them one at a time — N concurrent requests could occupy N threads for up to N×5s. Fix: `vtaskmgr_blocking_acquire(&c->vtaskmgr)`/`vtaskmgr_blocking_release(&c->vtaskmgr)` now wrap the ENTIRE forward+wait region (acquired before `mylock_lock(&c->sched.lock)`, released on every exit path) — the same shared budget `commit.c`'s `wait_admit_locked()`/`wait_release_locked()` already gate on (wired via `commit_set_pool(&c->commit, &c->vtaskmgr)`). Admission-exhausted requests fast-reject with `{ok=false, error="server busy, try again"}` before ever touching `c->sched.lock` — never queued. See requirements.md Req 5 AC 10's own Amendment for the full rationale. - -**Dependencies** -- Inbound: XceAdminSchedulerCLI, via TLS+capnp (P0) -- Outbound: NormalDriver (P0, forwards the request) -- External: none - -**Contracts**: API [x] - -##### API Contract -| Method | Endpoint (opcode) | Request | Response | Errors | -|--------|--------------------|---------|----------|--------| -| Request/Reply | `OPCode_schedAdmin` | `SchedAdminRequest` | `SchedAdminReply` | `-EAGAIN` (not primary), `"scheduler not connected"`, `"request timeout"`, `"invalid record file name"`, `"already recording"`, `"not recording"`, `"server busy, try again"` (admission exhausted) (full table: requirements.md Req 5 AC 10) | - -**Implementation Notes** -- Integration: one new `case` in `myreq_handler`'s switch (`cluster.c`); no new file. -- Validation: `test_sched_admin_dispatch.c` (`src/controller/testsuite/`; named differently from this plan's original `test_sched_admin.c`, same coverage) asserting the reply-on-every-path property (standby rejection, disconnected-scheduler rejection, timeout) — mirrors existing `OPCode_upgradeConfig` test coverage style. -- Risks: no RBAC/auth check is visible for the comparable `OPCode_upgradeConfig` case in `cluster.c` (see `research.md`), and `OPCode_schedAdmin` inherits that same gap by deliberate decision — auth hardening for this opcode is deferred to a follow-up spec, not silently accepted as permanent. - -### Tooling - -#### XceAdminSchedulerCLI - -| Field | Detail | -|-------|--------| -| Intent | `xceadmin scheduler record start/stop/status` and `xceadmin scheduler dump <FILE>` | -| Requirements | 5.10 | - -**Responsibilities & Constraints** -- `cli.py` stays a thin `click` layer; all logic lives in the new `scheduler.py` module, mirroring `push.py`'s split. -- `record_start/stop/status()` go through `xceadmin.discovery.follow_redirects` + `decode_reply` (request/reply opcode, needs redirect-following, unlike `probe_config()`). -- `dump()` has no network dependency — decodes a local `.replay` file only. - -**Dependencies** -- Inbound: operator CLI invocation (P0) -- Outbound: `xcecommon.wire`/`schema`, `xceadmin.discovery` (P0); SchedAdminOpcode via the wire (P0, for record/status only) -- External: none - -**Contracts**: API [x] - -##### API Contract -| Method | Endpoint | Request | Response | Errors | -|--------|----------|---------|----------|--------| -| CLI | `scheduler record start [FILE]` | `SchedAdminRequest{recordStart}` | `SchedAdminReply` | `XceAdminError` subclasses (connection, redirect, decode) | -| CLI | `scheduler record stop` | `SchedAdminRequest{recordStop}` | `SchedAdminReply` | same | -| CLI | `scheduler record status` | `SchedAdminRequest{recordStatus}` | `SchedAdminReply` | same | -| CLI | `scheduler dump <FILE>` | (local file) | formatted text to stdout | `SchemaNotFoundError`, file-not-found | - -**Implementation Notes** -- Integration: `cli.py` imports `record_start`/`record_stop`/`record_status`/`dump` from `scheduler.py`, same import shape as today's `build_cluster_config`/`push_config`. -- Validation: `test_scheduler.py` — mock-TLS-server tests for the three record commands (mirrors `test_config_push_mock.py`), plus golden-file tests for `dump()` against a small fixture `.replay` file. -- Risks: none beyond ReplayFormat's (this component only formats/transmits, never interprets scheduling semantics). - -## Data Models - -Covered in full under **Components and Interfaces → ReplayFormat** above (new capnp structs) — not duplicated here per the dedup guidance in design-principles.md. `SnapshotData`'s reuse of existing `Conf4Sched`/`SchedJobData`/`WorkerData`/`ResourceData`/`ResValList`/`QueueConf` is the only cross-cutting data-model decision; see `research.md`, "Reusable capnp payload types," for the verification trail. - -## Error Handling - -### Error Strategy -Fail-loud, no silent fallback — consistent with the controller's existing "fail loud, stay alive" design invariant (`src/controller/CLAUDE.md`), applied to the scheduler process at the scope this feature touches. - -### Error Categories and Responses -- **Configuration errors** (unregistered picker plugin, Req 2 AC 5): scheduler startup fails with a clear error — no silent fallback to `builtin_fcfs`. -- **Recording errors** (disk full, permission error, invalid `recordFile`): logged, recording state cleanly torn down (`recording=false`, file closed, `record_file_path` freed), scheduling continues unaffected (Req 5 AC 10, error handling) — recording failure is never fatal to the scheduler process. -- **Simulation errors** (missing/corrupt replay file, missing initial `SnapshotEntry`): simulation driver fails at startup with a clear message — no partial/best-effort simulation. -- **Admin-path errors** (`OPCode_schedAdmin`): every failure mode gets an explicit `{ok=false, error=...}` reply — never a dropped request (mirrors the "no silent drop" precedent already established for replication-class messages, `src/controller/CLAUDE.md`'s protocol checklist). - -### Monitoring -No new metrics infrastructure — `SessionSummary` (Req 5 AC 12) provides session-duration diagnostics through the existing `.replay` recording path; no separate telemetry system is introduced. - -## Testing Strategy - -- **Unit — Picker contract** (`test_picker_fcfs.c`): feed `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly, assert `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order — verifies Req 3.2-3.4 without a live core. -- **Unit — Core event contract** (`test_core_events.c`): `core_push_events()` followed by `core_go()` with a stub picker asserts state-change hooks fire exactly once per event, synchronously, before `core_go()` returns — verifies Req 5.1-5.8. -- **Unit — Clock virtualization** (`test_clock.c`): `xce_core_now()` against the four mode rows in Req 5 AC 3's table (normal, discrete, real-time, fast/slow) — asserts the two-base+speed formula directly, no wall-clock flakiness (fixed `real_clock_base`/mocked `clock_gettime` delta). -- **Unit — Replay format round-trip** (as-built: `src/controller/testsuite/test_e2e_capnp.c`, not a dedicated `test_replay_format.c` — added to this repo's existing shared capnp round-trip test file instead): encode/decode every new capnp struct (`JobStateChangeEvent`, `JobUpdateEvent`, `DispatchDecision`, `PendDecision`, `PreemptDecision`, `SnapshotData`) — field-for-field equality after round-trip. -- **Integration — Deterministic replay** (`test_sim_equivalence.c`, Req 5.7/5.7.1): record a synthetic scheduling session through `NormalDriver`'s test harness → replay through `SimDriver` → assert `ReplayDecision` sequences match exactly. This is the design's highest-value test — it directly exercises the central invariant. -- **Integration — Recording lifecycle** (driver-level, Req 5.10-5.11): `record start` → submit synthetic events → `record stop` → verify file structure (`ReplayHeader`, `SnapshotEntry`, alternating `ReplayEvent`/`ReplaySession`, periodic checkpoints at the configured interval, final `endSession`). -- **Integration — `OPCode_schedAdmin` dispatch** (`test_sched_admin_dispatch.c`, controller-level; named differently from this plan's original `test_sched_admin.c`): standby rejection, disconnected-scheduler rejection, timeout, and successful forward/reply round trip — mirrors existing `OPCode_upgradeConfig` test coverage. -- **Integration — xceadmin scheduler CLI** (`test_scheduler.py`): mock-TLS-server tests for `record start/stop/status` (redirect-following, reply decoding) plus golden-file tests for `dump()` against a fixture `.replay` file. -- **E2E — HA soak regression** (Req 4.3, Req 5 per-phase verification): `/xce-test-ha` sequential and chaotic rounds after each phase, asserting zero new job loss/divergence attributable to this feature — required before each phase merges, not a one-time gate at the end. -- **E2E — REPL equivalence** (Req 5.22): `--interactive` with `step`/`continue` to completion produces the same decision sequence as batch `--sim` on the same replay file. - -## Security Considerations - -_Feature-specific decisions only; baseline RBAC/auth practices are in `.kiro/steering/security.md`._ - -- **`recordFile` path validation** is already fully specified at the requirements level (Req 5 AC 10, "Record file path validation" — reject `/`/`..`, resolve under configured `record_dir`, `realpath()` containment check before `fopen()`). This design's `NormalDriver` and `SchedAdminOpcode` components inherit that requirement directly. `record_dir` itself is provisioned only via a `--record-dir` scheduler startup CLI flag — never from `SchedAdminRequest` or any other request-path field — so the validation boundary can't be redirected by a caller. -- **`OPCode_schedAdmin` ships with no RBAC/auth check beyond the primary/standby gate — accepted scope decision, deferred hardening.** The comparable `OPCode_upgradeConfig` case has no visible RBAC/auth check in `cluster.c` either (see `research.md`). Project owner decision (2026-08-02): authentication for this opcode is enhanced in a follow-up spec, not blocking this one. Phase 3 ships `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current (unauthenticated-beyond-primary-gate) posture — this is a known, accepted gap, not a silent one. -- **`.replay` files may contain sensitive cluster state** (full job/user/resource snapshots) — file permissions on `record_dir` are an operational concern (steering-level, not new to this feature) but worth calling out since this is the first feature to write full scheduler snapshots to disk in a human-decodable format (`xceadmin scheduler dump`). - -## Performance & Scalability - -_Feature-specific targets only; general practices are in steering docs._ - -- **Recording overhead**: zero file I/O, zero extra allocation when `recording == false` — a single boolean branch per session (Req 5 AC 10, explicit requirement, inherited here as a design constraint on `NormalDriver`). -- **Determinism vs throughput**: `jbuf` cbuf capacity is permanently 1 (rendezvous, not a buffer) — this bounds picker→allocator throughput to strictly sequential handoff. This is an intentional trade-off (determinism over pipelining) already made at the requirements level (Req 5 AC 7) and is not revisited here; any future throughput work must find a different mechanism that preserves the ordering guarantee (e.g. deterministic batching with an explicit order tag), not simply raise the cbuf capacity. -- **Simulation fast mode**: `speed_num=0` means the virtual clock never advances between events at wall-clock cost — `--sim` on a large `.replay` file runs at CPU-bound speed, not wall-clock speed. No specific throughput target is set here (none given in requirements); implementation should avoid introducing any per-event syscall beyond what decoding the capnp stream requires. -- **`OPCode_schedAdmin` vtask occupancy (hardened 2026-08-19)**: each request blocks its vtask handler thread up to 5s (cross-process wait on the scheduler's reply) while holding `c->sched.lock` — now gated by the same `vtaskmgr_blocking_acquire`/`_release` admission reservation `commit.c`'s commit-wait barrier shares (see `SchedAdminOpcode` component above). Admission-exhausted bursts fast-reject instead of occupying vtask threads; not separately load-tested against a real concurrent burst (unit-tested only, `test_schedadmin_fast_rejects_when_admission_exhausted`), but the mechanism itself is the same one already proven under `test_vtask_admission.c` and production commit-wait traffic. - -## Open Questions / Risks - -1. **`OPCode_schedAdmin` authorization mechanism** (P2 — accepted gap, deferred). Ships at parity with `OPCode_upgradeConfig`'s current posture (primary/standby gate only, no RBAC/auth check). Project owner decision (2026-08-02): auth hardening for this opcode is a follow-up spec, not a Phase 3 blocker. Tracked in `.kiro/steering/roadmap.md`'s existing `peer-authentication-hardening` backlog item (updated 2026-08-02 to cover `OPCode_schedAdmin` alongside `OP_upgradeConfig` — same `validate_host()` gate closes both). See Security Considerations and `research.md`. -2. **`PreemptDecision`'s wire shape is pre-committed ahead of `scheduler-preemption`'s own design** (P1). Flagged as a revalidation trigger in Boundary Commitments; `scheduler-preemption`'s design should explicitly confirm or extend this shape early, not discover it mid-implementation. -3. **`SnapshotData.internalState` opaque-blob scope** (P2). If a future spec needs `xceadmin scheduler dump` to render shared-resource-tracking or unsatisfiable-resreq-cache contents meaningfully (not just byte length), `internalState` will need to become a structured capnp field instead of opaque `Data` — deferred here because no current requirement asks for it (AC 20's dump description only requires human-readable output for what dump already needs to show, and internals were never asked to be inspectable). -4. **`xce_sched_core_t`'s `config`/`plugins`/`jobs`/`queues`/`workers`/`users`/`projects` field block is left as `var ...` in requirements.md's data-structure summary** (not fully typed) — Phase 2 implementation must finalize these as concrete `kvmgr`/pointer types when `core.c` is written; this design does not re-type them since requirements.md already commits to "moved from `xce_scheduler_t`" as the authoritative source of truth for their shapes. -5. ~~**`OPCode_schedAdmin` blocks a vtask handler thread up to 5s per request with no admission reservation**~~ **FIXED 2026-08-19** (design-review follow-up). Now gated by `vtaskmgr_blocking_acquire`/`_release` against the same `c->vtaskmgr` reservation `commit.c`'s commit-wait barrier shares; admission-exhausted requests fast-reject with `{ok=false, error="server busy, try again"}` instead of blocking. See `SchedAdminOpcode` component above and `research.md`'s "SUPERSEDES" note. -6. ~~**Req 5 AC 3's tlog virtual-clock logging integration is built but never activated**~~ **FIXED 2026-08-19** (design-review follow-up). `sched_sim_driver_start()` (`sim.c`) now calls `xce_core_register_tlog_format()`/`xce_core_set_active_for_logging(&sim->core)` right after `sim_set_virtual_clock()`, exactly at the site this entry identified; `sched_sim_driver_run()` (`sim_loop.c`) deactivates (`xce_core_set_active_for_logging(NULL)`) on all 7 of its return paths to avoid the ASAN stack-use-after-scope hazard this entry flagged — confirmed by fixing a real instance of exactly that hazard, caught live when the new test was added (bug-1598, `test_sim_event_loop.c`'s fixtures switched to `static` storage). No sim-driver fini function was added; the per-return-path deactivation is the shipped mitigation instead. -7. **`plugins/fcfs.c` vs `plugins/builtin_fcfs.c` split was undocumented in this design until 2026-08-07** (P3 — documentation gap only, now fixed here). Found by `/kiro-validate-impl`: the file-structure and BuiltinFcfsPicker sections above originally said `fcfs.c` would be rewritten; the actual implementation added a new `builtin_fcfs.c` and deliberately left `fcfs.c` untouched (task 1.2's compat-wrapper exercise target). Functionally correct and confirmed live (`builtin_fcfs` is the registered default), just wasn't reflected here until this update. Tasks 2.1-2.6 in tasks.md still carry no `_Verified:` annotations (unlike every other task group) — worth backfilling for traceability. diff --git a/.kiro/specs/scheduler-core/requirements.md b/.kiro/specs/scheduler-core/requirements.md deleted file mode 100644 index ad605905..00000000 --- a/.kiro/specs/scheduler-core/requirements.md +++ /dev/null @@ -1,1348 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE scheduler has a plugin architecture but critical foundational gaps. Job selection policy is hardcoded (`sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal). The scheduler is tightly coupled to controller I/O (`apictx`, `cluster_cfg` embedded in `xce_scheduler_t`). There is no way to swap picking policies without recompilation, and no way to debug or tune scheduling behavior without a live cluster. - -This spec establishes one central design invariant and three architectural foundations to achieve it. - -**Central invariant — Normal↔Sim equivalence**: Given identical initial configuration and identical event sequences, the scheduler shall produce identical scheduling decisions regardless of whether it runs in production (normal mode, controller-connected, wall-clock time) or in simulation (replay file, virtual clock). Record a live run → replay it in simulation → the decisions match byte-for-byte. This is NOT a debugging convenience — it is the **validation contract** for the entire architecture. If normal and simulation diverge, the core/driver separation is broken. This invariant enables every simulation-mode use case: debugging is faithful, tuning is predictive, regression testing is trustworthy, and what-if analysis reflects real behavior. - -Three foundations deliver this invariant: - -**Pluggable job picker**: A `xce_job_picker_iface_t` — the picker thread calls `picker->next_job()` to get the next schedulable job and hands it to the allocator thread via cbuf, with zero knowledge of how jobs are selected. The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior. Per-queue picker overrides are supported. The picker runs in a dedicated thread communicating with the allocator via cbuf (existing producer/consumer pattern preserved). - -**Core/driver separation**: The scheduler core is a passive, event-driven engine with zero controller I/O dependencies. A normal-mode driver connects it to a live controller; a simulation-mode driver replays recorded cluster events through a discrete event-driven virtual clock, enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. - -**Core is mode-agnostic**: The core has no concept of which mode it runs in. It processes events identically whether the driver is in normal mode or simulation mode. There is no mode flag, no mode-specific code path, and the same init function is used for both modes. - -> Full discovery context is in `.kiro/specs/scheduler-core/brief.md`. Sibling specs: `scheduler-allocation`, `scheduler-preemption`. - -## Queue vs Project — Roles - -These are fundamentally different concerns: - -| | Queue | Project | -|---|---|---| -| **Role** | Scheduling mechanism | Resource ownership | -| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? | -| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster | -| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy, members (future) | -| **Cardinality to job** | 1 job → 1 queue | 1 job → 1 project | -| **Cardinality to each other** | Many-to-many | Many-to-many | - -- **Queue** = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, priority (for inter-queue ordering within the picker), per-queue picker override, and preemption config. -- **Project** = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2). - -## Boundary Context - - - **In scope**: Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic behind a single plugin; single builtin picker (`builtin_fcfs`) replicating today's behavior through the new interface; configurable picker plugin with per-queue override (`queue->job_pick_plugin`); **core/driver separation** with passive event-driven core interface, mode-agnostic core (no mode flag, no mode-specific code paths), normal-mode driver (controller-connected), simulation-mode driver as a **standalone investigation tool** (replay recorded events with discrete event-driven virtual clock, zero controller dependencies, self-contained `.replay` file — single capnp binary stream with header + snapshot + events + decisions + periodic checkpoints in one artifact, what-if editing via event file substitution, diff-based comparison of simulated vs recorded decisions via `--sim-output`), virtualized clock access (`xce_core_now()` with two-base + speed formula), runtime recording control via xceadmin commands, including the **controller-side changes this requires** (new `OPCode_schedAdmin` + `SchedAdminRequest`/`SchedAdminReply` capnp types in `controller.capnp`, and pass-through dispatch logic added to the controller's `myreq_handler` per AC 10 — this spec's boundary extends into `src/controller/` for this one opcode, it is not scheduler-only), `.replay` capnp stream format, state snapshot/restore for replay efficiency, **interactive REPL** with single-step, breakpoints, state inspection, and snapshot capture via `--interactive`, and deterministic replay guarantee for debugging/tuning; **normal↔sim equivalence guarantee** (same config + same events → identical decisions regardless of mode, AC 7.1); `xceadmin scheduler dump <FILE>` for human-readable replay file inspection; documentation of picker plugin interface contract (hooks, cancel-safe, session-aware); unit tests for picker hook behavior, **event recording/replay, snapshot/restore, and deterministic replay**; HA soak regression for picker changes. - -- **Out of scope**: Allocation iterators and parjob allocator (→ `scheduler-allocation` spec); preemption policy and SIGSTP pipeline (→ `scheduler-preemption` spec); new job selection policies beyond the builtin FCFS picker — the pluggable picker interface enables them, but none ship in this spec beyond `builtin_fcfs`; hierarchical project resource quotas (deferred to `project-resource-ownership` spec — only `project_name` on queue and `xce_sched_project_t` scaffolding ship here); decision payload capnp schema for `DispatchDecision`/`PendDecision`/`PreemptDecision` (deferred to design phase — replay file format schema is in AC 20); distributed/cluster-mode simulation (v1 is single-scheduler replay only); visual inspection / UI tool for replay files (deferred — `xceadmin scheduler dump` provides human-readable output, `diff` of dumps is the primary comparison workflow for v1). - -- **Adjacent expectations**: Depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking — none of these change. Sibling spec `scheduler-allocation` provides the `try_allocate()` function this spec's core loop calls. Sibling spec `scheduler-preemption` provides the `can_preempt()` check and preemption decision type. - -## Implementation Phases - -The spec is implemented in three sequential phases. Each phase ships independently, passes HA soak regression, and preserves existing scheduling behavior. Phases are ordered by dependency — earlier phases produce primitives that later phases consume. - -### Phase 1: Pluggable Picker (Requirements 1–4) - -**Scope**: Replace the hardcoded `"xce_sched_fcfs"` literal with a configurable `xce_job_picker_iface_t`. Implement `builtin_fcfs` as the default plugin, replicating today's FCFS behavior exactly through the new interface hooks. - -**Changes to existing code**: -- Add `xce_job_picker_iface_t` with lifecycle, state-change, session, and post-allocation hooks (Req 1) -- Implement `builtin_fcfs` plugin with private `ordq_t` state, all MUST hooks (Req 3) -- Replace `sched_job_picker()` hardcoded string with `plugin_register`/`key_find` lookup (Req 1 AC 7) -- Add `job_pick_plugin` to `xce_scheduler_t` and `xce_sched_queue_t` (Req 2) -- Add `xce_sched_project_t` scaffolding + `project_name` on queue (Req 2 AC 3) - -**What does NOT change**: -- Today's main loop (`sched_update` → `sched_begin_session` → `sched_go` → decision drain → `sched_end_session`) is untouched -- Today's threads (picker, allocator) and cbuf communication are untouched -- Today's controller integration (`scheduler_req_handler`, capnp) is untouched -- Today's build system, test suite structure is untouched - -**Verification**: HA soak — same job-picking order as today, same decisions, zero divergence. - -**Why first**: Phase 2's `core_push_events()` must call picker hooks (`on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update`). Implementing the picker first means Phase 2 only moves existing hook call sites — same calls, different location. No dead wiring, no incomplete interfaces. - -### Phase 2: Core/Driver Separation (Requirement 5, ACs 1–8) - -**Scope**: Extract the scheduling core into a passive, mode-agnostic engine (`xce_sched_core_t`). Introduce the three-call driver↔core protocol (`core_push_events`, `core_go`, `on_decision`). Build the normal-mode driver that recreates today's controller-connected behavior. - -**Changes to existing code**: -- Extract `xce_sched_core_t` from `xce_scheduler_t` — all scheduling state, zero controller deps (AC 1) -- Implement `core_push_events()` — batch-push events, call picker hooks synchronously (AC 2, AC 4) -- Implement `core_go()` — non-blocking signal to picker thread via `cbuf_put(&wbuf)` (AC 6) -- Implement `on_decision()` callback — per-decision emission to driver (AC 6) -- Build normal-mode driver: capnp → core events → decisions → xcecli (AC 8) -- Replace today's `main.c` decision switch with driver callback -- Audit `kvmgr`-backed iteration sites (jobs, queues, users, projects) for order-stability and fix any found unstable (AC 7); hardcode `jbuf` capacity to 1 inside `xce_sched_core_init()`, do not expose it as a parameter (AC 7) - -**What does NOT change**: -- Picker interface and `builtin_fcfs` plugin (Phase 1 already done) -- Threading: picker → jbuf → allocator → decisions cbuf pattern preserved -- Controller integration: same capnp messages, same xcecli API, same behavior -- Simulation: not yet wired — core is mode-agnostic but no sim driver exists - -**Verification**: HA soak — same decisions as Phase 1, same controller interaction, zero regression. - -**Why second**: Core extraction needs the picker hooks to exist (Phase 1 done). Core extraction is a prerequisite for Phase 3 — simulation needs a mode-agnostic core that is already tested and passing HA soak in production. - -### Phase 3: Simulation Mode + Recording + Interactive REPL (Requirement 5, ACs 9–22) - -**Scope**: Add virtual clock (`xce_core_now()`), snapshot/restore (`save_snapshot`/`load_snapshot`), simulation driver (replay file → events → decisions → record file), interactive REPL (step, breakpoints, state inspection, snapshot capture), and runtime recording hooks in the normal-mode driver. - -**Changes to existing code**: -- Add clock virtualization atoms to core: `virtual_clock_base`, `real_clock_base`, `speed_num`, `speed_den` (AC 3) -- Implement `save_snapshot()` / `load_snapshot()` with picker rebuild from core state (AC 21) -- Build simulation driver: load HEADER+SNAPSHOT, replay events, emit decisions to `--sim-output` (AC 13, AC 19) -- Build interactive REPL: stdin/stdout command loop, breakpoint matching, state inspection queries (AC 22) -- Add recording state to normal driver: `scheduler record start/stop`, event writing, periodic snapshots (AC 10–11, AC 12) -- Add simulation CLI flags: `--sim`, `--sim-speed`, `--sim-verify`, `--sim-output`, `--interactive` / `-i` - -**What does NOT change**: -- Core is already mode-agnostic (Phase 2 done) — no `if (simulation)` branches added -- Picker is already clean (Phase 1 done) — no changes to picker interface -- Normal driver is already split (Phase 2 done) — recording hooks are boolean-gated, zero overhead when off -- Production scheduling behavior is identical — simulation is an opt-in mode - -**Verification**: HA soak (normal mode — no regression). **Normal↔sim equivalence** (AC 7.1): record a live run, replay in simulation, `diff <(xceadmin scheduler dump recorded.replay) <(xceadmin scheduler dump simulated.replay)` → decision content identical. **Deterministic replay** (AC 7): same `.replay` file twice → same output. **Recording**: file written correctly, simulation replays recorded file correctly. **Interactive REPL** (AC 22): `--interactive` mode steps, breaks, and completes to identical decisions as batch `--sim` mode. State inspection commands report accurate live state (verified against known-good snapshot dumps). - -**Why last**: Simulation needs Phase 1 (picker hooks exist) and Phase 2 (core is mode-agnostic, driver boundary is clean). It is the only phase that adds net-new capability rather than restructuring existing code — all risk is in Phase 1 and 2. By Phase 3, the new architecture has already been validated in production via HA soak. - -### Phase Dependency Graph - -``` -Phase 1: Pluggable Picker ──► Phase 2: Core/Driver Split ──► Phase 3: Simulation - (Req 1-4) (Req 5 AC 1-8) (Req 5 AC 9-22) - │ │ │ - │ No deps │ Needs picker hooks │ Needs mode-agnostic core - │ │ from Phase 1 │ + clean driver boundary - │ │ │ from Phase 2 - ▼ ▼ ▼ - HA soak pass HA soak pass HA soak pass -``` - -Each phase: TDD (red → green → refactor) → HA soak regression → merge. No phase gates on unfinished work from a later phase. Each phase ships value independently. - -## Requirements - -### Requirement 1: Pluggable job picker interface - -**Objective:** As a maintainer of the scheduler, I want the core scheduling loop to call a single `next_job()` plugin interface with no knowledge of how jobs are selected, so that picking strategies (FCFS, priority, fair-share, deadline, etc.) are truly pluggable and the core loop never changes regardless of the picking algorithm. - -#### Acceptance Criteria -1. The Scheduler core loop shall call `picker->next_job(picker)` to get the next schedulable job (or NULL if none ready), and shall not contain any logic about project iteration, queue iteration, or job ordering — those concerns are entirely encapsulated inside the picker plugin. - -2. The job picker plugin interface (`xce_job_picker_iface_t`) shall define these operations, organized by category: - - **Lifecycle hooks** (MUST): - - | Operation | Signature | Semantics | - |-----------|-----------|-----------| - | `init` | `int (*init)(void* picker, xce_sched_core_t* core)` | One-time setup, receives core reference for state queries | - | `fini` | `void (*fini)(void* picker)` | Teardown and cleanup of all picker-owned resources | - - **State-change hooks** (MUST — called by core when events arrive from driver): - - | Operation | Signature | Semantics | Triggered by | - |-----------|-----------|-----------|-------------| - | `on_queue_update` | `int (*on_queue_update)(void* picker, xce_sched_queue_t* queue)` | Add/update queue in picker's internal ordering | `QUEUE_UPDATE` event | - | `on_job_push` | `int (*on_job_push)(void* picker, xce_sched_job_t* job)` | Add new pending job to picker's candidate set | `JOBS_PUSH` event | - | `on_job_state_change` | `int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t new_state)` | Remove job from candidates if no longer pending; re-add if returning to pending | `JOB_STATE_CHANGE` event | - | `on_job_update` | `int (*on_job_update)(void* picker, xce_sched_job_t* job)` | Reorder job in candidates if priority/queue changed | `JOB_UPDATE` event | - - **Session hooks** (MUST — called at round boundaries): - - | Operation | Signature | Semantics | - |-----------|-----------|-----------| - | `begin_session` | `int (*begin_session)(void* picker)` | Reset iteration cursor; called at start of each scheduling round | - | `next_job` | `xce_sched_job_t* (*next_job)(void* picker)` | Return next schedulable job per picker's ordering, or NULL when exhausted | - | `end_session` | `int (*end_session)(void* picker)` | End-of-round notification; called before core sleeps between rounds | - - **Post-allocation hooks** (MUST — called after allocation attempt): - - | Operation | Signature | Semantics | - |-----------|-----------|-----------| - | `on_dispatch` | `int (*on_dispatch)(void* picker, uint64_t jobid)` | Job was allocated; remove from candidates permanently | - | `on_pend` | `int (*on_pend)(void* picker, uint64_t jobid)` | Job was pended; keep for next round but may deprioritize | - - **Optional hooks** (MAY — NULL if not needed): - - | Operation | Signature | Semantics | When needed | - |-----------|-----------|-----------|------------| - | `on_config` | `int (*on_config)(void* picker, xce_conf4sched_t* config)` | Rebuild picker strategy from config | Config-driven pickers (deadline, fair-share weights) | - | `on_job_rusage` | `int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages)` | Track per-job resource usage | Resource-aware pickers (fair-share, backfill) | - - Hooks not defined on the picker interface (worker updates, resource updates, resval updates) are not forwarded to the picker — picking is about job ordering, not resource availability. - -3. **Threading model**: The core uses three threads communicating via cbuf (bounded FIFO with mutex + condition variables, `src/include/mylock.h`): - - | Thread | Role | cbuf In | cbuf Out | Blocked On | - |--------|------|---------|----------|------------| - | Picker | Iterates queues/jobs via `next_job()`, pushes jobids | `wbuf` (from core main) | `jbuf` (to allocator) | `cbuf_get(&wbuf)` between sessions | - | Allocator | Calls `try_allocate()`/`can_preempt()`, builds decisions | `jbuf` (from picker) | `decisions` (to driver) | `cbuf_get(&jbuf)` between jobs | - | Core Main (driver context) | Calls `core_push_events()`, signals `core_go()`, drains decisions | `decisions` (from allocator) | `wbuf` (to picker) | `cbuf_get(&decisions)` during session | - - **Driver → Core event delivery**: State-change hooks (`on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update`) are called synchronously during `core_push_events()` in the driver thread, before the call returns. This is safe because `core_push_events()` is only called between sessions when the picker thread is idle (blocked on `cbuf_get(&wbuf)`) and the allocator thread is idle (blocked on `cbuf_get(&jbuf)`). No mid-session event injection. - - **Session hooks** (`begin_session`, `next_job`, `end_session`) run in the picker thread. **Post-allocation hooks** (`on_dispatch`, `on_pend`) run in the allocator thread. - -4. The core scheduling loop shall use a **producer/consumer pattern across three threads** communicating via cbuf: - - **Picker thread** (`fcfs_elect_job` equivalent): - ```c - while (1) { - int go = 0; - cbuf_get(&core->wbuf, &go); // block until core_go() signals - picker->begin_session(picker); - xce_sched_job_t* job; - while ((job = picker->next_job(picker)) != NULL) { - cbuf_put(&core->jbuf, &job->jobid); // hand off to allocator - } - uint64_t end_marker = 0; - cbuf_put(&core->jbuf, &end_marker); // end-of-session signal - picker->end_session(picker); - } - ``` - - **Allocator thread**: - ```c - while (1) { - uint64_t jobid = 0; - cbuf_get(&core->jbuf, &jobid); // block until picker sends job - if (jobid == 0) { - // end-of-session marker - cbuf_put(&core->decisions, NULL); - continue; - } - xce_sched_job_t* job = sched_get_job(core, jobid); - int rc = try_allocate(job); - if (rc > 0) { - picker->on_dispatch(picker, job->jobid); - decision = make_dispatch_decision(job->allocs); - } else if (can_preempt(job)) { - decision = make_preempt_decision(job, victims); - picker->on_pend(picker, job->jobid); - } else { - decision = make_pend_decision(job->reason); - picker->on_pend(picker, job->jobid); - } - cbuf_put(&core->decisions, &decision); - } - ``` - - **Main loop** (driver thread context — `core.go()`): - ```c - atomic_store(&core->session_active, 1, memory_order_release); - core_begin_session_internal(core); // select eval ctx, reset workers - cbuf_put(&core->wbuf, &go); // NON-BLOCKING: signal picker, return - // Driver then drains decisions cbuf: - while (1) { - cbuf_get(&core->decisions, &decision); - if (decision == NULL) break; // session exhausted - core->on_decision(core->driver_ctx, decision); // publish immediately - } - core_end_session_internal(core); // GC, reset workers - atomic_store(&core->session_active, 0, memory_order_release); - ``` - - The picker and allocator contain zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`. The allocator contains zero knowledge of how the driver publishes decisions — it only calls `cbuf_put(&decisions, &decision)`. - -5. The picker plugin shall be cancel-safe: `next_job()` shall have defined `pthread_cancel` cancellation points. The picker shall be session-aware: `begin_session()` resets iteration state, `end_session()` is called before sleep-between-rounds. - -6. The hardcoded `"xce_sched_fcfs"` literal at `scheduler.c:43` (in `sched_job_picker()`) shall be replaced by loading the configured `sched->job_pick_plugin` via the existing `plugin_register`/`key_find` mechanism. If the configured plugin name is not registered, the Scheduler shall fail with a clear configuration error. - -7. The existing multi-threaded producer/consumer architecture (picker thread → `jbuf` cbuf → allocator thread → `decisions` cbuf → main loop) shall be preserved. The driver thread signals the picker thread via `wbuf` cbuf (non-blocking `cbuf_put`). The only thread eliminated is the conceptual one: today's FCFS picker is a dedicated thread, and it stays a dedicated thread — but the core scheduling loop in the driver is simplified because decision publishing moves from a hardcoded `main.c` switch to a driver-registered `on_decision` callback. - -### Requirement 2: Configurable picker plugin - -**Objective:** As a cluster operator, I want to configure which job-picking policy the scheduler uses without recompiling, and override it per-queue, so that FCFS, priority, or custom pickers can be swapped operationally. - -#### Acceptance Criteria -1. The Scheduler shall add a `job_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_fcfs"` when unset — this is the global picker plugin name. - -2. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_queue_t`, defaulting to NULL (inherit from global). When non-NULL, the configured global picker plugin shall use this per-queue override for jobs within that queue. How the picker plugin uses per-queue overrides is an implementation detail of the plugin, not the core loop. - -3. The Scheduler shall add a `project_name` field to `xce_sched_queue_t`, set from queue configuration, establishing the queue→project relationship. The picker plugin may use this field internally for project-scoped iteration. - -4. The Scheduler shall accept the picker plugin name from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild. - -5. When the configured plugin name is not registered via the existing `plugin_register`/`key_find` mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default. - -6. The picker plugin interface contract (`xce_job_picker_iface_t`) shall be documented: each picker must be cancel-safe (responds to `pthread_cancel` at defined points in `next_job()`) and session-aware (resets its iteration state on `begin_session()`). - -### Requirement 3: Builtin default picker - -**Objective:** As a maintainer, I want the new architecture to ship with a builtin picker plugin that replicates today's exact scheduling behavior, so that operators who don't configure a custom picker see no behavioral change. - -#### Acceptance Criteria - -1. **Registration**: `builtin_fcfs` shall be registered automatically at scheduler startup as `"builtin_fcfs"` and set as the default `sched->job_pick_plugin`. - -2. **Hook implementation** — `builtin_fcfs` shall implement every MUST and SHOULD hook from `xce_job_picker_iface_t` (Req 1 AC 2) as follows: - - | Hook | FCFS Implementation | - |------|-------------------| - | `init` | Allocate `fcfs_plugin_data_t`. Initialize `ordq_t queues` (ordered by `queue->priority desc`). Set iteration cursor to NULL. | - | `fini` | Free all per-queue `ordq_t pendJobs`, the `ordq_t queues`, and plugin data. | - | `on_queue_update` | Upsert queue into `ordq_t queues`. If queue exists, update its priority and reorder. If new, allocate `fcfs_queue_t` with empty `ordq_t pendJobs`. | - | `on_job_push` | If `job->state & JOB_STAT_PEND`: find the job's queue in `ordq_t queues`, add `fcfs_job_t{job}` to that queue's `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). | - | `on_job_state_change` | If `new_state` is not pending (EXIT/FAIL/DONE): remove job from its queue's `pendJobs` by jobid. If `new_state` returns to PEND: re-add to `pendJobs`. | - | `on_job_update` | If job's `priority` or `queue` field changed: remove from old position in `pendJobs`, re-insert at correct position per ordering. | - | `begin_session` | Reset iteration cursor to head of `ordq_t queues`. | - | `next_job` | Walk `ordq_t queues` from cursor. For each queue, skip if `sched_queue_is_ready()` returns false. Within queue, walk `ordq_t pendJobs`. Return first job where `sched_job_is_ready()` returns true. Advance cursor. Return NULL when all queues and jobs exhausted. | - | `end_session` | No-op for FCFS (iteration state reset in `begin_session`). | - | `on_dispatch` | Remove job from its queue's `ordq_t pendJobs` by jobid. | - | `on_pend` | No-op for FCFS — job remains in `pendJobs`, will be retried next round in same position. | - | `on_config` | NULL — FCFS is config-agnostic. | - | `on_job_rusage` | NULL — FCFS is not resource-aware. | - -3. **Internal data structures** — `builtin_fcfs` shall maintain its own private state, not query the core's `sched->queues`/`sched->jobs` during `next_job()`. All candidate management happens in the state-change hooks. This ensures: - - `next_job()` is fast — no core lock acquisition, no hash table scans - - The picker is independently testable — feed hooks, verify `next_job()` order - - Simulation snapshots capture picker state naturally via the hooks already recorded in the event stream - -4. **Ordering equivalence** — `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`: inter-queue by descending `queue->priority`, intra-queue by `priority desc, queue_time asc`. Behavior with per-queue picker overrides is as specified in Req 2 AC 2. - -5. **Dedicated picker thread retained** — `builtin_fcfs` shall implement `next_job()` as a synchronous call within the picker's dedicated thread. The picker thread communicates with the allocator thread via `jbuf` cbuf (existing pattern). `pthread_cancel` cancellation points shall exist in the `next_job()` iteration loop and in `cbuf_get(&wbuf)` where the picker blocks between sessions. - -6. **Session state** — `end_session()` is a no-op. Iteration state is reset in `begin_session()`. Undispatched jobs remain in `pendJobs` across sessions (they carry over to the next round). - -### Requirement 4: No regression to existing picker behavior or HA soak - -**Objective:** As a maintainer, I want the picker rework and core/driver separation scoped so that existing FCFS behavior is unaffected, making the framework change safe to ship independently of any new policy. - -#### Acceptance Criteria -1. `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`, given the same queue and job state — same inter-queue priority ordering and same intra-queue (priority desc, queue_time asc) ordering. -2. Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string and the refactored `sched_go()` / decision publishing path this feature replaces. The picker thread, allocator thread, and cbuf communication pattern are preserved. -3. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. - -### Requirement 5: Core/Driver Separation with Dual-Mode Operation - -**Objective:** As a scheduler developer, I want the scheduling core separated from all external I/O so that the same core logic runs deterministically in production (driven by a live controller) and in simulation (driven by recorded cluster events with a discrete event-driven virtual clock), enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. - -#### Acceptance Criteria - -**Driver ↔ Core Protocol (overview)** - -The driver and core communicate through a three-call synchronous interface with cbuf-based internal threading. The driver buffers controller events externally between sessions, pushes them as a batch before each scheduling round, and receives decisions one-by-one as they are produced. - -| Direction | Call | Semantics | Thread | -|-----------|------|-----------|--------| -| Driver → Core | `core_push_events(core, events, n)` | Batch-push events before session. State-change hooks called synchronously. Only valid when picker is idle (between sessions). | Driver thread | -| Driver → Core | `core_go(core)` | Start scheduling session. **NON-BLOCKING** — signals picker thread via cbuf, returns immediately. Returns `-EINVAL` if session already active. | Driver thread | -| Core → Driver | `on_decision(ctx, decision)` | Each scheduling decision emitted immediately as produced. Called from allocator thread context. | Allocator thread | -| (implicit) | picker sends `jobid=0` through `jbuf` | End-of-session marker. Allocator forwards as `NULL` through `decisions` cbuf. | Picker → Allocator → Driver | - -**Event buffering discipline**: The driver accumulates controller events (capnp push/notify messages from the handler thread) into `pend_updates` (existing dataq swap mechanism). Between sessions, the driver drains `pend_updates` into `xce_core_event_t` array and calls `core_push_events()`. No events are injected mid-session — the picker thread is blocked on `cbuf_get(&wbuf)` and the allocator thread on `cbuf_get(&jbuf)`, so `core_push_events()` has exclusive access to picker state. - -**Decision publishing**: Each decision is published immediately to the controller (not batched at session end). The driver's `on_decision` callback calls `xcecli_jobdispatch()` / `xcecli_set_pend_reason()` / `xcecli_preempt()` synchronously. This matches today's `main.c:181-196` behavior. - -**Mode selection**: Mode is chosen at startup via CLI flags. The core has zero knowledge of which mode is active: - -```c -// main.c — mode selection (single branch) -if (arguments.sim_file) { - sched_sim_driver_run(&sim_driver, arguments); -} else { - sched_driver_run(&driver, &config, cluster_cfg, handler); -} -``` - -| Flag | Mode | Purpose | -|------|------|---------| -| (none) | Normal | Controller-connected (current behavior) | -| `--sim FILE` | Simulation | Replay `.replay` file | -| `--sim-speed N` | Simulation | 0=fast, 1=real-time, N=N× | -| `--sim-start-ts NS` | Simulation | Start replay from logical time | -| `--sim-verify` | Simulation | Compare decisions against recorded | -| `--interactive` / `-i` | Simulation | Open REPL for step-through debugging | -| `--sim-alt-events FILE` | Simulation | What-if branching | -| `--sim-output FILE` | Simulation | Write simulated decisions to `.replay` file | - -**Migration from old `job_elect_fn`**: The existing `xce_sched_plugin_t->job_elect_fn` (which `fcfs_elect_job` implements) is superseded by `xce_job_picker_iface_t->next_job()`. The old `job_elect_fn` shall be deprecated with a warning at plugin register time. The new `builtin_fcfs` picker implements the new interface exclusively. Existing plugins that only implement `job_elect_fn` continue to work via a compatibility wrapper during the transition period, but new pickers shall implement `xce_job_picker_iface_t`. - ---- - -**Core Interface Contract** - -1. **Mode-agnostic core**: The Scheduler core shall have no concept of which mode it runs in. It processes events identically whether the driver is in normal mode (controller-connected, wall-clock time) or simulation mode (replay file, virtual clock). Specifically: - - There shall be no mode flag (`bool simulation_mode` or equivalent) anywhere in the core - - There shall be no `if (simulation) { ... } else { ... }` code path in the core - - `xce_sched_core_init()` shall take identical parameters regardless of mode - - The core's decision callback shall be the same function pointer type in both modes - - The core's `save_snapshot()` and `load_snapshot()` shall be identical in both modes - -2. **Passive event-driven interface**: The core shall expose a pure event-driven interface — it receives typed events from the driver, updates internal scheduling state, and emits decisions through a driver-registered callback. The core shall have zero direct dependency on: controller communication, capnp serialization, network I/O, filesystem access, or wall-clock time sources. - -3. **Virtualized clock access**: The core shall access time exclusively through `xce_core_now()`. The core shall never call `time(NULL)`, `clock_gettime()`, `gettimeofday()`, or any other wall-clock time function directly. - - **Clock virtualization mechanism — two-base + speed formula:** - - The core stores four fields (atomically updated by the driver as a group): - - | Field | Type | Description | - |-------|------|-------------| - | `virtual_clock_base` | `_Atomic uint64_t` | Virtual-time anchor (nanoseconds) | - | `real_clock_base` | `_Atomic uint64_t` | Wall-clock anchor captured when `virtual_clock_base` was set (nanoseconds, `CLOCK_MONOTONIC`) | - | `speed_num` / `speed_den` | `_Atomic uint64_t` | Speed multiplier as rational fraction | - - `xce_core_now(core)` computes: - ```c - uint64_t wall_ns = clock_gettime(CLOCK_MONOTONIC); // only wall-clock call in the core - uint64_t elapsed = wall_ns - core->real_clock_base; - return core->virtual_clock_base + (elapsed * core->speed_num) / core->speed_den; - ``` - - | Mode | `virtual_clock_base` | `real_clock_base` | `speed` | Behavior | - |------|----------------------|-------------------|---------|----------| - | Normal | 0 | 0 | 1/1 | `xce_core_now()` = wall clock (degraded: `0 + (wall - 0) × 1`) | - | Simulation discrete | event `ts` | wall at load | 0/1 | `xce_core_now()` = frozen at event `ts` (time does not advance between events) | - | Simulation real-time | snapshot `ts` | wall at load | 1/1 | `xce_core_now()` tracks wall clock from virtual anchor | - | Simulation fast/slow | snapshot `ts` | wall at load | N/1 or 1/N | `xce_core_now()` runs N× faster or slower than wall clock | - - The driver atomically updates all four fields before each event dispatch. **Multi-thread safety**: `xce_core_now()` reads atomic fields lock-free. Both picker and allocator threads independently compute the same virtual time from the shared bases — no mutex needed. - - All internal time uses go through `xce_core_now()`: scheduling logic, logging (via tlog format function using `userptr`), error messages, debug traces. - Job `sub_time` and `queue_time` are set by the driver before pushing `JOBS_PUSH`. - - **Third-party library constraint**: any library linked into the core that needs time must obtain it through `xce_core_now()` or a core-provided wrapper. Specifically: - - **Logging (tlog)**: The core shall register its own format function via `tlog_reg_format_func(fn, core_ptr)` where `core_ptr` is the `xce_sched_core_t*`. This format function shall ignore `info->time` and instead read `xce_core_now(core_ptr)`, converting nanoseconds to `struct tm` fields via `gmtime_r()` (deterministic — pure computation). - - **libxce subset**: The core shall link only against the minimal data-structure subset of libxce (`ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti`) — not against `netio`, `delay`, `xceapi`, or `idgen` which call wall-clock time internally. - - **pthread timed-blocking**: The core is multi-threaded. In simulation mode, thread interleaving is made deterministic — `pthread_cond_timedwait()` and `pthread_mutex_timedlock()` are not needed in the deterministic execution path. - - Libraries that cannot avoid calling wall-clock time directly must not be linked into the core; such concerns stay in the driver layer. - -4. **Core event types — detailed payload specification**: The core interface shall accept at minimum these input events. Each event carries a timestamp (`ts`, nanoseconds since epoch) and a type-specific payload. The field set below is normative for implementation; the exact capnp message schema is deferred to the design phase. - - **`CONFIG_PUSH`** — Full configuration push (scheduler startup or config reload): - | Field | Type | Description | - |-------|------|-------------| - | `config` | `xce_conf4sched_t*` | All queues, resource thresholds, preemption config, time windows, global policy settings | - | → Picker hook: `on_config(picker, config)` | | | - - **`JOBS_PUSH`** — Newly submitted jobs: - | Field | Type | Description | - |-------|------|-------------| - | `njobs` | `int` | Number of jobs in this batch | - | `jobs` | `xce_job_t*[njobs]` | Job records: jobid, user, project, queue, priority, state, queue_time, resreq | - | → Picker hook: `on_job_push(picker, job)` for each job | | | - - **`WORKERS_PUSH`** — Worker list update (nodes added/removed): - | Field | Type | Description | - |-------|------|-------------| - | `nworkers` | `int` | Number of workers in this update | - | `workers` | `xce_worker_t*[nworkers]` | Worker records: workerid, host, resources, state, assigned_jobs | - - **`RESOURCES_PUSH`** — Resource definitions (new resource types registered): - | Field | Type | Description | - |-------|------|-------------| - | `n` | `int` | Number of resource definitions | - | `resources` | `xce_resource_t*[n]` | Resource records: name, type, total, flags | - - **`RESVALS_FULL`** — Full resource value snapshot (periodic or after large change): - | Field | Type | Description | - |-------|------|-------------| - | `n` | `int` | Number of resource value entries | - | `resvals` | `xce_resval_t*[n]` | Each: workerid, resource_name, available, used, reserved | - - **`RESVALS_DELTA`** — Incremental resource value update (fine-grained changes): - | Field | Type | Description | - |-------|------|-------------| - | `deltas` | `xce_resval_list_t*` | Linked list of individual resval changes (add/subtract) | - - **`JOB_STATE_CHANGE`** — Job started, exited, or failed: - | Field | Type | Description | - |-------|------|-------------| - | `jobid` | `uint64_t` | Job identifier | - | `old_state` | `uint32_t` | Previous job state (for context) | - | `new_state` | `uint32_t` | New job state: RUNNING, EXIT, FAIL, DONE, PEND | - | `exit_code` | `int32_t` | Exit code (meaningful only for EXIT/FAIL) | - | → Picker hook: `on_job_state_change(picker, job, new_state)` | | | - - **`JOB_UPDATE`** — In-place job field modification (priority change, queue reassignment): - | Field | Type | Description | - |-------|------|-------------| - | `jobid` | `uint64_t` | Job identifier | - | `changed_fields` | `uint32_t` | Bitmask of fields that changed (JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ) | - | `job` | `xce_job_t*` | Full job record with updated fields | - | → Picker hook: `on_job_update(picker, job)` | | | - - **`QUEUE_UPDATE`** — Queue definition added or modified: - | Field | Type | Description | - |-------|------|-------------| - | `queue` | `xce_queue_t*` | Queue record: name, priority, RUNLIMIT, MAX_RUNNING, state, time_window, job_pick_plugin, project_name | - | → Picker hook: `on_queue_update(picker, queue)` | | | - - **`JOB_RUSAGE`** — Job resource usage update (for resource-aware pickers): - | Field | Type | Description | - |-------|------|-------------| - | `jobid` | `uint64_t` | Job identifier | - | `rusages` | `xce_job_rusages_t*` | Resource usage metrics: cpu_time, mem_bytes, io_bytes | - | → Picker hook: `on_job_rusage(picker, job, rusages)` (optional — NULL if not registered) | | | - - **`BEGIN_ROUND`** — Trigger a scheduling round (session boundary, see AC 18): - | Field | Type | Description | - |-------|------|-------------| - | `timestamp_ns` | `uint64_t` | Logical time at which the round should execute | - - **`END_SESSION`** — Explicit end-of-recording marker (written to replay file, never sent to core): - | Field | Type | Description | - |-------|------|-------------| - | `timestamp_ns` | `uint64_t` | Logical time when recording stopped | - -5. **Decision output types**: The core shall emit decisions through the `on_decision` callback. Three decision types exist: - - | Decision Type | Fields | Driver Action | - |---------------|--------|---------------| - | `DISPATCH` | `jobid`, `workerid`, `allocations[]` (resource, amount per allocation) | `xcecli_jobdispatch()` | - | `PEND` | `jobid`, `reason` (code + message) | `xcecli_set_pend_reason()` | - | `PREEMPT` | `jobid`, `victims[]` (jobid, workerid per victim), `reason` | `xcecli_preempt()` | - - These are recorded as `ReplayDecision` messages within `ReplaySession.decisions`. The exact decision payload schema is deferred to the design phase. - -6. **Decision output callback**: The core shall emit all scheduling decisions (`DISPATCH`, `PEND`, `PREEMPT`) through a `xce_core_decision_cb_t` callback registered by the driver at init time. Signature: `int (*on_decision)(void* driver_ctx, xce_decision_t* decision)`. This replaces the current internal `decisions` cbuf poll loop in `main.c`. - -7. **Deterministic guarantee**: Given identical initial state and identical event sequences with identical timestamps, the core shall produce identical decision sequences. The core is multi-threaded (picker + allocator threads communicating via cbuf). Deterministic replay is achieved through: - - **cbuf capacity=1 serialization**: The `jbuf` cbuf between picker and allocator has capacity 1 (`cbuf_init(&jbuf, 1, sizeof(uint64_t))`). This acts as a rendezvous — the picker blocks on `cbuf_put` until the allocator takes the current jobid. Jobs flow through the pipeline one at a time in fully determined order. **This capacity is a determinism invariant, not a tuning knob**: today's code sets it via `int njobs = 1;` with a `/* FIXME: make this configurable */` comment (`scheduler.c:384`). That FIXME shall NOT be resolved to any value other than 1 without re-deriving the determinism argument in this AC — raising `jbuf` capacity turns the rendezvous into a buffer, and the picker may enqueue jobs faster than the allocator drains them, making allocator-observed job order depend on scheduling/timing rather than on the event sequence alone. Phase 2 (Req 5 AC 1-8, core extraction) shall either hardcode this capacity to 1 inside `xce_sched_core_init()` or leave the `njobs` FIXME untouched; it shall not expose `jbuf` capacity as a driver- or config-supplied parameter. - - **Stabilized hash table iteration**: All hash table / key-value store iteration that affects scheduling output order shall be stabilized (sorted by key before iterating). **Phase 2 ownership**: as part of extracting `xce_sched_core_t` (Req 5 AC 1), the implementer shall audit every `kvmgr`-backed iteration site reachable from the scheduling loop (jobs, queues, users, projects — `scheduler.c:404-412`) and confirm each is already order-stable as a function of (keys, insert/delete sequence) with no process-random seeding; any site found unstable shall be fixed in Phase 2, before Phase 3 simulation work depends on it. This audit is a Phase 2 deliverable, not an implicit byproduct. - - **PRNG for tie-breaking**: Any required randomness (e.g., tie-breaking) shall use a PRNG seeded by the driver at init time (`core->prng_seed`). - - **Same events → same picker state**: Since `core_push_events()` is called between sessions (picker idle), the picker's internal data structures (`ordq_t queues`, `ordq_t pendJobs`) are populated identically given identical event sequences. The picker iteration then produces jobids in identical order. - - **7.1 Normal-mode ↔ simulation equivalence**: Given identical initial configuration (`xce_conf4sched_t`) and identical event sequences (same job submissions, state changes, resource updates at the same logical positions), the core shall produce identical decision sequences in both normal mode and simulation mode. Specifically: - - - Record a normal-mode scheduling session (AC 10) → produce `.replay` file R - - Run simulation on R (AC 13) → produce `.replay` file S - - `diff <(xceadmin scheduler dump R) <(xceadmin scheduler dump S)` → only non-decision metadata may differ (timestamps in `ReplayHeader`, wall-clock annotations in `SessionSummary`). Every `ReplayDecision` shall match identically: same jobid, same workerid, same allocation amounts, same pend reasons. - - This constraint is the **validation contract** for the entire core/driver architecture. If normal mode and simulation mode diverge, one of three things is broken: - - The core has a hidden dependency on wall-clock time or controller I/O (violates AC 1) - - The event recording is lossy — dropping events that matter to scheduling decisions (violates AC 10) - - The simulation driver's input translation from capnp or `.replay` format is not faithfully reproducing the normal-mode event stream - - This constraint turns recording + replay from a debugging convenience into a **regression test**: after any scheduler change, record a live run, replay it in simulation, and verify the decisions are byte-for-byte identical. Any divergence is a bug. - -8. **Driver event queue**: The driver shall retain an internal pending-event queue to buffer events received from the controller between scheduling rounds. Events accumulate in the driver between `BEGIN_ROUND` invocations. - -**Normal-Mode Driver** - -9. **Controller integration**: The normal-mode driver shall receive capnp push/notify messages from the controller, translate each to the corresponding core event type, and invoke the core's event push interface. The driver shall publish core-emitted decisions back to the controller via the existing `xcecli_ctx_t` API. - -10. **Runtime recording control — detailed mechanism**: The scheduler shall support runtime recording of all scheduling events and decisions to a replay file for later simulation and analysis. - - **Admin command flow**: xceadmin CLI connects directly to the controller via TLS + capnp (same as `xceadmin config push` — no apiserver, no REST). The controller receives `OPCode_schedAdmin` and forwards it to the scheduler via the existing `scheduler_t.ep` endpoint. The scheduler processes the request and replies; the controller relays the reply back to xceadmin. This is a request-reply opcode, NOT a one-way notify — recording `status` returns live scheduler state that the controller does not own. - - **Why request-reply (not notify)**: The existing `OPCode_notify` is fire-and-forget — no reply path. Recording `start`/`stop` could theoretically use notify, but `status` must return live state (`recording`, `file`, `event_count`, `started_at`). Using two different mechanisms for three commands on the same subsystem is worse than one consistent request-reply path. Additionally, `start`/`stop` benefit from a synchronous reply: `start` returns the auto-generated filename, and `stop` returns the final event count. - - **Why a dedicated opcode (not reusing OP_upgradeConfig)**: `OP_upgradeConfig` carries a config blob through the controller's own config-replication pipeline. Recording control is a scheduler-scoped admin action — it touches no controller state, needs no WAL persistence, and should be forwarded directly to the scheduler without controller-side interpretation. A dedicated `OPCode_schedAdmin` keeps these concerns separate. - - **Cap'n Proto schema additions** (in `controller.capnp`): - - New OPCode: - ```capnp - enum OPCode { - ... - schedAdmin @42; - } - ``` - - New types: - ```capnp - enum SchedAdminAction { - recordStart @0; - recordStop @1; - recordStatus @2; - } - - struct SchedAdminRequest $C.mapname("sched_admin_req_t") { - action @0: SchedAdminAction; - recordFile @1: Text; # filename (not a path) for recordStart; empty = auto-generate. - # See AC 10 "Record file path validation" — the scheduler - # resolves this beneath a configured recording directory; - # it is never used as a caller-controlled absolute path. - } - - struct SchedAdminReply $C.mapname("sched_admin_reply_t") { - ok @0: Bool; - error @1: Text; # human-readable error if !ok - - # Fields populated for recordStatus (also returned by recordStart/recordStop - # for confirmation — caller can verify state after each action): - recording @2: Bool; # true between start and stop - file @3: Text; # record file path (auto-generated name returned by start) - eventCount @4: UInt64; # events written since recording started - startedAt @5: UInt64; # logical time recording started (nanoseconds since epoch) - } - ``` - - New Request.body union entry: - ```capnp - schedadminmsg @31: SchedAdminRequest; - ``` - - New RequestReply.data union entry: - ```capnp - schedadminreplymsg @19: SchedAdminReply; - ``` - - **Controller forwarding** (in `myreq_handler`): - ``` - case OPCode_schedAdmin: - → validate xceadmin auth (admin RBAC, same as config push) - → lock(&c->sched.lock) - → if sched.ep == NULL: - unlock; reply {ok=false, error="scheduler not connected"} - → forward to sched.ep via existing capnp push/request - → await scheduler reply (timeout: 5s) - → unlock - → relay SchedAdminReply back to xceadmin via RequestReply - ``` - - The controller is a **pass-through** — it owns no recording state, interprets no fields beyond the opcode dispatch. All recording logic lives in the scheduler's normal-mode driver. In the new architecture, recording is a driver-layer concern — the core has zero knowledge of whether recording is active. - - **Record file path validation**: `recordFile` is admin-authenticated but caller-supplied, and the scheduler process opens it for writing — it shall NOT be treated as a trusted absolute path. The scheduler shall: - - Read a configured recording directory from scheduler config (e.g. `record_dir`, default `/var/log/xce/`). - - Reject any `recordFile` value containing a path separator (`/`) or a `..` path segment — only a bare filename is accepted, never a path. - - Resolve the final write target as `record_dir/recordFile` (or `record_dir/<auto-generated-name>` when `recordFile` is empty), and additionally verify via `realpath()` (or equivalent) that the resolved path stays within `record_dir` before `fopen()`. - - Reply `{ok=false, error="invalid record file name"}` if validation fails — no file is opened, no directory is created. - - This mirrors the "never trust external data" input-validation boundary applied elsewhere in this codebase (e.g. RBAC/JWT-gated config endpoints) — `recordFile` is user input crossing a process boundary (xceadmin → controller → scheduler) and is validated at that boundary, not assumed safe because the caller is already admin-authenticated. - - **Edge cases**: - - **Scheduler not connected** (`sched.ep == NULL`): Controller replies `{ok=false, error="scheduler not connected"}` immediately. No capnp round-trip needed. - - **Scheduler disconnects mid-request**: The 5s timeout triggers; controller replies `{ok=false, error="request timeout"}`. - - **Controller is not primary**: xceadmin's existing redirect-following logic handles this — if the contacted controller is not the primary, it redirects to the primary (same as `config push`). The scheduler is a singleton attached to the primary controller, so recording commands always reach the primary's scheduler. - - **Invalid `recordFile`** (path separator, `..` segment, or resolves outside `record_dir`): scheduler replies `{ok=false, error="invalid record file name"}` before touching the filesystem (see "Record file path validation" above). - - **xceadmin CLI surface**: - ``` - xceadmin scheduler record start [FILE] - → SchedAdminRequest{action=recordStart, recordFile=FILE or ""} - → Success: {ok=true, file="<auto or specified>", recording=true, - eventCount=0, startedAt=<logical_time>} - → Error: {ok=false, error="already recording", recording=true, file="<current>"} - - xceadmin scheduler record stop - → SchedAdminRequest{action=recordStop} - → Success: {ok=true, recording=false, eventCount=14273, file="<was>", - startedAt=<original_start>} - → Error: {ok=false, error="not recording", recording=false} - - xceadmin scheduler record status - → SchedAdminRequest{action=recordStatus} - → Success: {ok=true, recording=true|false, file="<current or empty>", - eventCount=N, startedAt=<original_start or 0>} - ``` - - **Reply field semantics by action**: - | Field | recordStart (ok) | recordStart (err) | recordStop (ok) | recordStop (err) | recordStatus | - |-------|-------------------|--------------------|------------------|--------------------|--------------| - | `ok` | true | false | true | false | true | - | `error` | "" | reason (e.g. "already recording") | "" | reason (e.g. "not recording") | "" | - | `recording` | true | true (unchanged) | false | false (unchanged) | true/false | - | `file` | current path | current path | last path (informational) | "" | current or "" | - | `eventCount` | 0 | current count | final count | 0 | current | - | `startedAt` | start time | original start | original start (historical) | 0 | original or 0 | - - **Error codes reference**: - | Condition | `ok` | `error` | - |-----------|------|---------| - | `record start` while already recording | false | `"already recording"` | - | `record stop` while not recording | false | `"not recording"` | - | Scheduler not connected to this controller | false | `"scheduler not connected"` | - | Request timeout (5s) | false | `"request timeout"` | - | Disk full / write error (start) | false | `"cannot write to <file>: <errno>"` | - | `recordFile` contains `/` or `..`, or resolves outside `record_dir` | false | `"invalid record file name"` | - | Admission reservation exhausted (see Amendment below) | false | `"server busy, try again"` | - - **Amendment (2026-08-19, design review follow-up)**: the controller's forward-and-block-for-reply wait (this AC's own "Admin command flow" — held for the full duration under `c->sched.lock`, up to the 5s deadline) is bounded blocking on a vtask thread, which `ha-async-io.md`'s async-I/O discipline permits only under an admission reservation. A burst of concurrent `schedAdmin` requests could otherwise occupy N vtask threads for up to N×5s each, starving the pool the reservation exists to protect. The dispatch case now acquires the SAME shared admission budget `commit.c`'s commit-wait barrier already gates on (`vtaskmgr_blocking_acquire`/`vtaskmgr_blocking_release` against `c->vtaskmgr`, wired via `commit_set_pool`) before ever locking `c->sched.lock`, and fast-rejects with `{ok=false, error="server busy, try again"}` — never queuing — when the reservation is exhausted. Sharing the budget with commit-wait traffic is deliberate: the reservation protects a fixed number of vcpus from any class of admitted bounded-blocking work, not a per-caller quota. - - Status output (xceadmin formats the capnp reply): - ``` - recording: yes - file: /var/log/xce/scheduler-replay-prod-cluster-1-20260729-020900.replay - events: 14273 - started at: 2026-07-29T02:09:00 - ``` - - **Permissions**: All three commands require admin-level authentication (same RBAC role as `config push`). The controller validates the xceadmin auth before forwarding to the scheduler — the scheduler trusts the controller's forwarding (same trust model as existing notify events from the controller). - - **Recording state** (in `xce_sched_driver_t`): - ```c - bool recording; // true between start and stop - char* record_file_path; // strdup'd path for status queries - FILE* record_file; // replay file handle (NULL when not recording) - uint64_t record_event_count; // counter for periodic snapshot - uint64_t record_snapshot_interval; // N events between snapshots (0 = disabled) - uint64_t record_start_ts; // logical time when recording started (0 = not started) - ``` - - `record_file_path` is a `strdup`'d copy of the auto-generated or user-specified path, owned by the record file lifecycle: `strdup` on start, `free`+NULL on stop or error. This field exists so `SchedAdminReply.file` can be populated even after `record_file` is closed — it persists for informational queries (e.g., `status` after a disk-full auto-stop). - - **Recording lifecycle — what gets written at each step:** - - 1. **On `record start`**: - - Take full core snapshot via `save_snapshot(core, &snapshot_data)` — fills `SnapshotData` struct in memory - - Write `ReplayHeader` as first message of the file (recorder version, config hash, cluster identity, timestamps, sched policy) - - Serialize `SnapshotEntry{ts, data=snapshot_data}` and write as second message - - Set `recording = true`, `record_event_count = 0` - - 2. **On each `core_push_events()` call** (driver thread, between sessions): - - For each event in the batch, serialize and write the corresponding `ReplayEvent` message to the `.replay` file - - Increment `record_event_count` for each event written - - 3. **On `core_go()`** (driver thread, start of session): - - Begin building a `ReplaySession` message with `beginRound = xce_core_now(core)` - - 4. **In the decision drain loop** (driver thread, during session): - - For each decision from `cbuf_get(&decisions)`: append a `ReplayDecision` to the current `ReplaySession.decisions` list - - Decisions are in allocator output order, before the NULL end-of-session marker - - 5. **On `core_end_session()`** (driver thread): - - Finalize the `ReplaySession` with session summary, write the complete message to the `.replay` file - - 6. **On periodic checkpoint** (driver thread): - - If `record_event_count > 0 && record_event_count % record_snapshot_interval == 0`: - - Serialize and write `SnapshotEntry` message from `save_snapshot(core, &snapshot_data)` - - 7. **On `record stop`**: - - Write `ReplayEvent{endSession}` message - - `fflush()` + `fclose()` the record file - - Set `recording = false`, `record_file = NULL` - - **Recording insert points in the normal-mode driver main loop** (annotated): - ```c - while (!sched.stop) { - sched_update(&sched); // process capnp push/notify - // [RECORDING] write each buffered event as ReplayEvent message - // [RECORDING] periodic checkpoint check - - sched_begin_session(&sched); - // [RECORDING] build ReplaySession{beginRound = now} - sched_go(&sched); - - while (cbuf_get(&sched.decisions, &d), d) { - // [RECORDING] append ReplayDecision to session - switch (d->op) { - case DISPATCH: sched_publish_dispatch(...); break; - case PEND: sched_publish_pend_reason(...); break; - case PREEMPT: sched_publish_preempt(...); break; - } - } - - sched_end_session(&sched); - // [RECORDING] finalize + write ReplaySession message - // [RECORDING] periodic checkpoint after session - sleep(1); - } - ``` - - **Performance**: When recording is inactive (`recording == false`), the scheduler shall incur zero overhead — no file I/O, no extra allocations, no event buffering beyond what the non-recording path already does. The recording check is a single boolean branch per session. - - **Error handling**: If writing to the record file fails (disk full, permission error), the driver shall log the error, set `recording = false`, `fflush()` + `fclose()` the record file, `free(record_file_path)` + set `record_file_path = NULL`, and continue normal scheduling operation. Recording failure shall not cause the scheduler to stop or crash. - -11. **Periodic snapshot during recording**: While recording is active, the driver shall periodically append fresh snapshots to the replay file (default: every `N` events, configurable via `--record-snapshot-interval N`, N=0 disables periodic snapshots). - -12. **Session response time metrics**: The core shall track session duration for diagnostics. At `core_go()`, the core stores `session_start = xce_core_now()`. At `core_end_session()`, the core computes `session_duration_ns = xce_core_now() - session_start`. The duration is written to the `ReplaySession.summary` field as a `SessionSummary` struct. The `SessionSummary` is optional for replay correctness (the simulation driver ignores it) but provides valuable diagnostics: - - **Normal mode**: reveals real scheduling latency — is the picker slow? is allocation the bottleneck? - - **Simulation mode**: in real-time mode, compare virtual vs wall-clock session duration; in fast mode, the virtual clock freezes during `core_go()` so `duration_ns` is typically 0 (all decisions at same `ts`) - - **Tuning**: diff session durations before and after a policy change to quantify performance impact - - The core stores `session_start` internally — no new API functions, no change to the `on_decision` callback, no overhead beyond a single `xce_core_now()` call per session. - -**Simulation-Mode Driver** - -13. **Simulation mode startup**: The scheduler shall support simulation mode activated by `--sim FILE` and `--sim-start-ts TIMESTAMP`. The startup sequence is: - - **Step 1 — Open and validate replay file:** - ``` - sim->replay_file = fopen(arguments.sim_file, "r"); - read first capnp message → validate it is ReplayHeader - ``` - If the file is missing, unreadable, or the first message is not a valid `ReplayHeader`, the simulation driver shall fail with a clear error message. - - **Step 2 — Locate starting snapshot:** - ``` - scan forward from ReplayHeader, decode each SnapshotEntry - find the most recent SnapshotEntry with ts <= --sim-start-ts - ``` - If no `SnapshotEntry` exists after `ReplayHeader`, the simulation driver shall reject the file. Without initial state, simulation has no starting point. - - **Step 3 — Load snapshot and rebuild picker:** - ``` - xce_sched_core_load_snapshot(&sim->core, snapshot_entry.data) - // Walk core state to rebuild picker private state: - for each queue in core.queues: - picker->on_queue_update(picker, queue) - for each job in core.jobs where job.state & JOB_STAT_PEND: - picker->on_job_push(picker, job) - ``` - After this step, core state and picker state are fully restored. The core is ready to run. - - **Step 4 — Set virtual clock:** - ``` - sim->core.virtual_clock_base = snapshot_ts - sim->core.real_clock_base = clock_gettime(CLOCK_MONOTONIC) // wall anchor - sim->core.speed_num = arguments.sim_speed // 0=fast, 1=real-time, N>1=N× - sim->core.speed_den = 1 - ``` - From this point, `xce_core_now()` returns virtual time computed from the snapshot's logical timestamp. - - **Step 5 — Enter event loop (AC 18):** accumulate events → BEGIN_ROUND → `core_push_events` → `core_go` → drain decisions → `core_end_session` → repeat until EOF. - - The `ReplayHeader` + `SnapshotEntry` together form the complete self-contained starting point. Nothing outside the replay file is needed to begin simulation. - -14. **Discrete event-driven virtual clock**: The simulation driver shall implement a virtual clock that advances by jumping directly to the timestamp of the next queued event — no fixed tick interval, no wall-clock waiting. Strictly monotonically increasing. Same-timestamp events delivered in file order. Supports: - - `--sim-speed N` — N=0 as-fast-as-possible, N=1 real-time, N>1 N× real-time - - `--interactive` / `-i` — open REPL for step-through debugging, breakpoints, state inspection, and snapshot capture (AC 22) - -15. **Deterministic replay verification**: In simulation mode with `--sim-verify`, the driver shall compare each core-emitted decision against the recorded decisions in the replay file. Any mismatch shall be reported as a regression with exact logical-time and decision-detail context. - -16. **What-if branching**: The simulation driver shall accept `--sim-alt-events FILE`. When both a base `.replay` file (for snapshot loading) and an alt-events file are provided, the driver loads the snapshot from the base file, then replays events from the alt-events file (same `.replay` capnp stream format) — enabling "what-if" analysis. - -17. **Simulation isolation**: In simulation mode, the scheduler process shall not establish any connection to the controller. The `xce_sched_sim_driver_t` shall contain no `xcecli_ctx_t`, no `xce_cluster_config_t`, no capnp socket, and no RPC to any controller. All scheduling input comes exclusively from the replay file. All scheduling decisions go to the simulation record file (see AC 19) and/or stdout. Attempting to use a controller-facing API from simulation mode is a programming error that shall be caught at compile time (via struct field absence) or fail with a clear error at startup. - -18. **BEGIN_ROUND as session boundary**: The simulation driver shall treat each `ReplaySession` message as the trigger for one complete scheduling session. The driver accumulates `ReplayEvent` messages into a batch, and upon encountering `ReplaySession`, executes the full session cycle: - - ```c - // Simulation driver event loop - xce_core_event_t buf_events[MAX_EVENTS_PER_ROUND]; - int buf_n = 0; - - while (decode_next_message(replay_file, &msg) == 0) { - switch (msg.which) { - case ReplayEvent: - buf_events[buf_n++] = replay_event_to_core_event(&msg.event); - break; - - case ReplaySession: // ← SESSION BOUNDARY - // 1. Push accumulated events (picker hooks called synchronously) - core_push_events(&sim->core, buf_events, buf_n); - - // 2. Kick scheduling round (non-blocking signal to picker thread) - core_go(&sim->core); - - // 3. Drain decisions until end-of-session marker (NULL) - xce_decision_t* d; - int di = 0; - while (cbuf_get(&sim->core.decisions, &d) == 0 && d != NULL) { - // 4. Compare against recorded decisions (if verify enabled) - if (sim->verify && di < msg.session.decisions.len) { - xce_decision_t expected = from_replay_decision(msg.session.decisions[di]); - if (!decision_eq(&expected, d)) - log_mismatch(sim, &expected, d); // --sim-verify - di++; - } - if (sim->record_file) - write_replay_decision(sim->record_file, d); // record output - free_decision(d); - } - - // 5. End session (GC, reset workers) - core_end_session(&sim->core); - buf_n = 0; // reset for next round - break; - - case SnapshotEntry: - // Mid-replay snapshots (periodic checkpoints): skipped during replay. - // Only the initial SnapshotEntry (loaded at startup) is consumed. - break; - } - } - ``` - - The driver thread blocks during the decision drain loop (step 3-4) because `cbuf_get(&decisions)` blocks until the allocator produces a decision or the end-of-session marker. The `core_go()` call itself is non-blocking (it only signals the picker thread via `cbuf_put(&wbuf)` and returns immediately). After all decisions are drained and the NULL marker arrives, `core_end_session()` runs cleanup, and the driver resumes reading the next message from the `.replay` file. - - **Decision ordering constraint**: `ReplayDecision` messages appear within the `ReplaySession.decisions` list. Their order matches the allocator thread's output order through the `decisions` cbuf. A `ReplayDecision` appearing outside a `ReplaySession` is a format error. - -19. **Simulation record file**: In simulation mode with `--sim-output FILE`, the driver shall write a `.replay` file in the identical capnp stream format as the input replay file (`ReplayHeader` → `SnapshotEntry` → `ReplayEvent` → `ReplaySession` cycles). This record file captures the simulation's actual decisions, enabling: - - - **Diff-based debugging**: `diff <(xceadmin scheduler dump original.replay) <(xceadmin scheduler dump simulated.replay)` reveals exactly which decisions changed - - **Tuning feedback loop**: Modify picker parameters, re-run simulation, diff the record files to quantify the impact - - **What-if comparison**: Compare `--sim-alt-events` run output against the base run output - - The record file starts with a `ReplayHeader` (carrying scheduler version + config hash from the sim run, NOT copied from the replay file), followed by the same `SnapshotEntry` that was loaded, then events + `ReplaySession` cycles exactly as produced by the core. If `--sim-output` is not specified, decisions are printed to stdout only and no record file is written. - - **Symmetry with normal-mode recording**: Both the normal-mode driver (when recording is active, AC 10) and the simulation-mode driver (with `--sim-output`) produce `.replay` files in the identical capnp stream format. The same file layout, same message types, same decision encoding. This symmetry is intentional: - - | Aspect | Normal-mode recording | Simulation `--sim-output` | - |--------|----------------------|---------------------------| - | Trigger | `scheduler record start` (runtime) | `--sim-output FILE` (CLI at startup) | - | Input source | Controller capnp events | Replay file events | - | Writes events? | Yes — records all input events as `ReplayEvent` messages | Yes — passes through events from replay file as `ReplayEvent` messages | - | Writes decisions? | Yes — records scheduler decisions in `ReplaySession` messages | Yes — records simulated decisions in `ReplaySession` messages | - | Output file | `.replay` file (for later simulation) | `.replay` file (for diff analysis) | - | `ReplayHeader` origin | Live cluster metadata | Sim run metadata | - | `SnapshotEntry` origin | `save_snapshot(core)` at record start | `SnapshotEntry` from replay file | - | Purpose | Capture reality | Capture simulation result | - - This means `diff <(xceadmin scheduler dump original.replay) <(xceadmin scheduler dump simulated.replay)` works out of the box — both files have the same structure, same timestamps, same event types. Only the decisions may differ, which is exactly what you want to compare. - -**Replay File Format** - -20. **Single-file capnp stream format (`.replay`)**: The replay file shall be a single file containing consecutive Cap'n Proto messages in the standard framing (segment table + data per message). Each message is one of the types below. The file is appendable — recording writes each message as it occurs; no need to know final size upfront. - - **File suffix**: `.replay` — self-documenting, no collision with schema files (`.capnp`), configuration blobs (`.bin`), or text formats (`.json`, `.jsonl`). - - **Schema** (in `controller.capnp`): - - ```capnp - struct ReplayHeader { - recorderVersion @0: Text; # scheduler binary version that wrote this file - configHash @1: Data; # sha256 of xce_conf4sched_t - cluster @2: ReplayCluster; - recordingStartTs @3: UInt64; # logical time recording started (ns) - recordingStartWall @4: Text; # wall-clock time recording started (ISO 8601) - schedPolicy @5: Text; # active picker plugin name (e.g. "builtin_fcfs") - } - - struct ReplayCluster { - name @0: Text; - controllers @1: List(Text); # ["host1:9991", "host2:9992", "host3:9993"] - recordingHost @2: Text; # controller hostname that made the recording - } - - struct SnapshotEntry { - ts @0: UInt64; # logical time of snapshot - data @1: SnapshotData; # full core state - } - - struct ReplayEvent { - ts @0: UInt64; # logical time - body @1: union { - configPush @2: Conf4Sched; - jobsPush @3: SchedJobData; - workersPush @4: WorkerData; - resourcesPush @5: ResourceData; - resvalsFull @6: ResValList; - resvalsDelta @7: ResValList; - jobStateChange @8: JobStateChangeEvent; - jobUpdate @9: JobUpdateEvent; - queueUpdate @10: QueueConf; - jobRusage @11: SchedJobRusages; - endSession @12: Void; # recording stop marker - } - } - - struct ReplaySession { - beginRound @0: UInt64; # logical time (ts value) - decisions @1: List(ReplayDecision); - summary @2: SessionSummary; # optional diagnostic metadata - } - - struct ReplayDecision { - ts @0: UInt64; - body @1: union { - dispatch @2: DispatchDecision; - pend @3: PendDecision; - preempt @4: PreemptDecision; - } - } - - struct SessionSummary { - durationNs @0: UInt64; - nDecisions @1: UInt32; - nDispatched @2: UInt32; - nPended @3: UInt32; - } - ``` - - **File structure**: The `.replay` file is a single stream of consecutive capnp messages: - - ``` - [msg 1] ReplayHeader - [msg 2] SnapshotEntry ← initial state (MUST be present, MUST be first after header) - [msg 3] ReplayEvent ← input event - [msg 4] ReplayEvent ← input event - ... - [msg N] ReplaySession ← BEGIN_ROUND → decisions → SESSION_SUMMARY - [msg N+1] ReplayEvent - ... - [msg M] SnapshotEntry ← periodic checkpoint (every N events, AC 11) - ... - [msg K] ReplayEvent{endSession} ← recording stop marker - ``` - - - **`ReplayHeader`**: first message. Identifies the file type (`.replay` files start with a known capnp struct, not an ambiguous byte sequence). Carries metadata about the recording origin — recorder version, config hash, cluster identity, timestamps, active scheduling policy. - - **The `.replay` file is fully self-contained**: the `ReplayHeader` identifies which config was active (`configHash`), and the first `SnapshotEntry` carries the full config payload plus all scheduling state. No external config file, controller, or database is needed to replay — one file has everything. - - **`SnapshotEntry`**: the first data message after `ReplayHeader` MUST be a `SnapshotEntry`. The simulation driver shall reject any `.replay` file whose second message is not a `SnapshotEntry` — without initial state, simulation has no starting point. Mid-stream `SnapshotEntry` messages (from periodic recording, AC 11) provide seek points for `--sim-start-ts`. - - **`ReplayEvent`**: input event carrying payload in a discriminated union. The `body` union tag selects the event type (`configPush`, `jobsPush`, etc.). No separate `"type"` field needed — the capnp union tag is the type discriminator. - - **`ReplaySession`**: groups a `BEGIN_ROUND` trigger with its resulting decisions and session summary. A `ReplaySession` between `ReplayEvent` messages marks a session boundary; a `ReplayDecision` outside a `ReplaySession` is a format error. Decision order within `decisions` matches allocator output order. - - **`ReplayEvent{endSession}`**: final message in a completed recording. Signals clean shutdown. The simulation driver ignores it (replay runs until EOF). - - No text parsing, no JSON schema, no two-format code paths — the same capnp decode path reads every message. The simulation driver decodes each message sequentially from the stream. - - **Human inspection**: `xceadmin scheduler dump <FILE>` decodes a `.replay` file to human-readable text. xceadmin already bundles `controller.capnp` schemas for `config push`/`config build` — `scheduler dump` adds one new subcommand reusing the same schema infrastructure. - - **Recording (append)**: The `.replay` format is append-friendly — `save_snapshot()` and event writes serialize one capnp message at a time. On `record start`: write `ReplayHeader` + initial `SnapshotEntry`. Each event: write one `ReplayEvent`. Each session: write one `ReplaySession`. Periodic checkpoint: write one `SnapshotEntry`. On `record stop`: write `ReplayEvent{endSession}` + fflush + fclose. Recording failure mid-write: the file is truncated to the last complete message — no partial-message corruption. - - **Simulation (read)**: `--sim-start-ts NS` seeks forward through `SnapshotEntry` messages (deserialize each, check `ts <= NS`, keep latest match), loads the matching snapshot, then replays all messages with `ts >= NS`. - -**State Snapshot** - -21. **Snapshot serialization**: The core shall support `save_snapshot()` producing a `SnapshotData` capnp struct containing all scheduler configuration (`xce_conf4sched_t*` — queues, priorities, resource thresholds, preemption config, time windows), jobs, workers, resources, resvals, users, projects, shared-resource tracking state, unsatisfiable-resreq cache, PRNG state, and session state. The core shall support `load_snapshot()` restoring from `SnapshotData` to that exact state. - - **The snapshot is fully self-contained for simulation.** After `load_snapshot()` and picker rebuild, the core is ready to process `core_push_events()` + `core_go()` and produce decisions — no other state, external config, or preceding events are needed. Given a `.replay` file, the simulation driver can start from any `SnapshotEntry` (via `--sim-start-ts`): load the snapshot, rebuild picker, set virtual clock to snapshot `ts`, then replay all messages with `ts >= snapshot.ts`. The result is identical to replaying from the beginning — the snapshot is a complete checkpoint. - - **Picker state is NOT stored in snapshot.** The builtin picker's private state (`ordq_t queues`, per-queue `ordq_t pendJobs`) is derived from core state: every queue → `on_queue_update`, every pending job → `on_job_push`. After `load_snapshot()` restores core state, the driver shall rebuild the picker's private state by walking the core's queues and jobs and calling the picker's state-change hooks. This ensures: - - Snapshots capture only canonical core state (no duplicate of picker internals) - - Picker rebuild is deterministic: same core state + same hook sequence = same picker state - - Custom pickers with additional state can rebuild from the same core-state walk - - **Snapshot contents checklist** — everything required to run a scheduling session: - - | Category | Contents | - |----------|----------| - | Config | `xce_conf4sched_t*` — all queues, priorities, resource thresholds, preemption config, time windows | - | Jobs | All job records with current state, priority, queue assignment, queue_time | - | Queues | All queue definitions with RUNLIMIT, MAX_RUNNING, state, per-queue picker override | - | Workers | All worker records with resources, state, assigned jobs | - | Resources | Resource definitions (resreq templates) | - | Resvals | Current resource value snapshots for all workers | - | Users | User records with resource limits, group membership | - | Projects | Project scaffolding with priority (v1: flat, v2: hierarchical quota) | - | Shared-resource | Shared resource tracking state across workers | - | Unsatisfiable cache | Resreqs that could not be satisfied in previous rounds (for early skip) | - | Session state | Current scheduling round, iterator positions, GC state | - | PRNG state | Current PRNG value (for deterministic tie-breaking resume mid-file) | - -22. **Interactive simulation REPL**: In simulation mode with `--interactive` (or `-i`), the scheduler shall open a read-eval-print loop on stdin/stdout after loading the replay file but before the first event is processed. The REPL gives the operator full control over simulation execution — single-step, breakpoints, state inspection, and snapshot capture — turning the simulation driver into a real debugger for scheduling policy. - - **Interaction model**: The simulation process blocks on stdin at startup (pre-replay pause) and at each breakpoint hit. No controller, no network, no authentication — this is a local investigation tool. The REPL prompt is `(sim) `. - - **Minimum command set**: - - **Execution control**: - | Command | Description | - |---------|-------------| - | `step [N]` | Advance N sessions (default 1), pause after. Processes all events up to and including the Nth `BEGIN_ROUND` from the current position, then returns to REPL. | - | `continue` | Run until next breakpoint hit or EOF. If no breakpoints are set, runs to completion. | - | `quit` | Exit the scheduler immediately. No cleanup guarantees beyond `fclose` on open files. | - - **Breakpoints**: - | Command | Description | - |---------|-------------| - | `break <event_type> [key=value ...]` | Set breakpoint on event type (e.g., `JOB_STATE_CHANGE`, `JOB_PUSH`, `BEGIN_ROUND`, `DECISION`). Optional key=value filters narrow the match: `break JOB_STATE_CHANGE jobid=12345`, `break JOB_STATE_CHANGE state=RUN`, `break DECISION type=DISPATCH queue=normal`. Multiple filters are AND'd. Key names match the field names in the capnp schema for each message type. | - | `info break` | List all breakpoints with their IDs, event types, and filters. | - | `delete <id>` | Delete breakpoint by ID. `delete all` removes all breakpoints. | - - **State inspection**: - | Command | Description | - |---------|-------------| - | `info session` | Show current simulation position: logical time (`xce_core_now()`), event index (Nth message processed), session count, last `SessionSummary` if available. | - | `info job <jobid>` | Dump one job: state, queue, priority, queue_time, resreq, allocations. Jobid uses the same format as xcecli (e.g., `12345`). | - | `info queue <name>` | Dump one queue: priority, state (open/closed), RUNLIMIT, MAX_RUNNING, job_pick_plugin, list of queued job IDs with priorities. `info queue all` lists all queues. | - | `info worker <name>` | Dump one worker: state, resources (total + available), assigned jobs. `info worker all` lists all workers. | - | `info resource <name>` | Dump one resource: type, total capacity, available, allocation count. `info resource all` lists all resources. | - | `info summary` | Quick overview: `N_jobs (M pending, K running), N_workers (M online), N_queues, N_resources`. | - - **Snapshots**: - | Command | Description | - |---------|-------------| - | `snapshot [FILE]` | Write a full `save_snapshot()` to FILE as a standalone `.replay` file (contains `ReplayHeader` + `SnapshotEntry`). If FILE is omitted, auto-generate: `<replay_basename>-snap-<event_index>.replay`. The snapshot is a complete state dump (same format as periodic recording checkpoints). The operator can restart simulation from this snapshot via `--sim <snapshot_file> --sim-start-ts <time_at_snapshot>`. | - - **Help**: - | Command | Description | - |---------|-------------| - | `help` | Print available commands with one-line descriptions. Covers all REPL commands: execution control (`step`, `continue`, `quit`), breakpoints (`break`, `info break`, `delete`), state inspection (`info session/job/queue/worker/resource/summary`), snapshots (`snapshot`), sim parameters (`set`, `info config`). | - - **Simulation parameters** (interactive `set` / `info config`): - | Command | Description | - |---------|-------------| - | `set speed <N>` | Set simulation speed from this point forward. `0` = as-fast-as-possible (no wall-clock delay), `1` = real-time, `N` = N× real-time. Overrides the `--sim-speed` CLI value. | - | `set output <FILE>` | Start writing simulation decisions to FILE in the same `.replay` capnp stream format as `--sim-output`. Opens `ReplayHeader`+`SnapshotEntry` at the current position if the file is new. File is written incrementally — each subsequent session's decisions land in the file. | - | `set output off` | Close the current output file. Subsequent decisions are not recorded. `info config` confirms the file path and event count at close time. | - | `set verify on\|off` | Enable or disable decision verification against the recorded replay file. When enabled, each decision is compared against the corresponding recorded decision at the same logical time. Mismatches are printed immediately. Default: matches `--sim-verify` CLI flag. | - | `info config` | Show current sim parameters: `speed=N, output=<path or "off">, verify=on|off, events_processed=N, current_ts=<logical_time>, sessions=N`. | - - **Why interactive parameters matter**: Without these, tuning simulation requires restart: change `--sim-speed`, re-run from snapshot, re-check results. With `set speed` and `set output`, the operator can fast-forward through known-good regions (`set speed 0`), slow down for suspicious sections (`set speed 0.1`), and only start recording output when the interesting behavior starts — all in one session. This turns the REPL from a passive inspector into an active investigation tool where parameters co-evolve with the operator's understanding of the trace. - - **REPL integration with the simulation loop**: - ``` - load replay file + snapshot - if --interactive: - print "[sim] loaded: N events, snapshot @ ts=..." - print "[sim] ready. type 'help' for commands." - repl_pause() // pre-replay pause — operator can set breakpoints - - for each event in replay_file: - // ── sim parameter: speed ── - if sim->sim_speed_num > 0: - wall_clock_delay(delta_ns * sim->sim_speed_den / sim->sim_speed_num) - - if check_breakpoints(event): - print "[sim] hit breakpoint @ event #I: <event_type> <key>..." - repl_pause() - - core_push_events(core, event) - - if msg.which == ReplaySession: - core_go(core) - for each decision: - // ── sim parameter: verify ── - if sim->verify: - compare_and_report(decision, expected) // AC 15; REPL 'set verify on/off' - - // ── sim parameter: output ── - if sim->output_file: - write_replay_decision(sim->output_file, decision) // AC 19; REPL 'set output' - - if check_breakpoints(decision): - print "[sim] hit breakpoint @ decision: <type> job=<jobid>..." - repl_pause() - - if --interactive: - print "[sim] replay complete. N_sessions processed." - if sim->output_file: - print "[sim] output written to: <path>" - ``` - - **What is NOT in v1**: - - **Write-side mutations to scheduler state**: No `set queue priority`, `set job state`, `add job`, `remove job`. The `set` command in v1 only tunes simulation parameters (`speed`, `output`, `verify`) — it never mutates core scheduling state. Operator inspects state but cannot alter it. What-if analysis uses `--sim-alt-events` (AC 16) instead. State mutation from REPL is a follow-up. - - **Reverse execution**: No `step back` or time-travel undo. Operator can restart from snapshots captured at earlier points. - - **Expression evaluation**: No `print core->sessions_count` or arbitrary C evaluation. Commands are fixed. - - **Scripting/batch mode**: No `source FILE` or `-c "step 5; info summary"`. The REPL is interactive-only. For automated simulation runs, use the batch flags (`--sim`, `--sim-output`, `--sim-verify`) without `--interactive`. - - **No semantic `snapshot diff` in xceadmin**: `xceadmin scheduler dump` produces human-readable output. Comparing two dumps (`diff <(xceadmin scheduler dump snap1) <(xceadmin scheduler dump snap2)`) works but is generic text diff — no scheduler-aware semantic diff ("queue priority changed from 10 to 20", "3 new jobs added"). Semantic diff is a v2 feature. - - **Deterministic replay is preserved**: The REPL never alters core state or event stream. It only controls *when* events are processed (by pausing), not *what* events are processed. The same replay file with the same breakpoints ignored (treated as skip) produces identical decisions as batch mode. Interactive use with `step` and `continue` that runs to completion produces the same output as `--sim` without `--interactive`. - - **Implementation note**: The REPL is a single `fgets` loop in the simulation driver. It is not a separate thread — when the REPL blocks on stdin, the simulation loop is parked. This is correct and intentional: the operator is in control, and the simulation has no wall-clock deadlines. - - **Amendment (2026-08-19, live-operator testing — task 12.5)**: driving the REPL against a real replay file surfaced five gaps in the v1 design above. All five are now part of this acceptance criterion, not a follow-up: - - 1. **`continue` with no breakpoints must not end the session.** The original pseudocode's `for each event in replay_file: ...` loop, once it reaches EOF with no breakpoint ever hit, simply falls out the bottom and the process exits — the operator never gets to inspect the final state. Corrected behavior: reaching EOF while `--interactive` prints the existing "[sim] replay complete. N_sessions processed." banner and then **re-enters the REPL command loop** instead of returning. State-inspection commands (`info job/queue/worker/resource/summary/session`) remain fully usable against the final simulated state. - - 2. **The simulator shall never terminate except via an explicit `quit` (or stdin EOF, which this REPL has always treated as an implicit `quit`).** This is the direct corollary of (1): once replay is complete and the REPL has re-entered its prompt, `step`/`continue` at that point are no-ops that print "[sim] replay already complete -- no more events to process. Use 'quit' to exit." and re-prompt, rather than being treated as "hand control back to a driver loop that no longer has anything to drive." - - 3. **`step`/`continue` shall echo the input events and resulting decisions for the current session, in the order they actually occur.** V1 only ever printed decisions (`sim_print_decision()`); the events that drove them (`JOB_PUSH`, `JOB_STATE_CHANGE`, `WORKERS_PUSH`, etc.) were completely silent unless a breakpoint happened to match one. An operator single-stepping through a session had no visibility into what changed before the resulting dispatch/pend decisions appeared. Corrected behavior: every event is echoed one line at a time as it is decoded (the same point breakpoint-matching already inspects it, `xce_repl_on_event()`), so the printed sequence is event(s) → BEGIN_ROUND → decision(s), matching real execution order. - - 4. **`info resource <name>|all` shall report resource *metadata*, restricted to globally-shared resources — never a per-host aggregation.** V1's implementation summed `total`/`available` across every worker's per-host resource value for every resource, host-based ones included (e.g. `cpus`, `mem`) — a fabricated, meaningless number for a per-host consumable already correctly inspected via `info worker <name>`, and it never surfaced the resource *definition* (description, value type, which of `XCE_RESF_BUILTIN`/`DYNAMIC`/`GLOBAL`/`SHARED`/`EXTERNAL`/`RELEASE`/`DEFAULT_HOST`/`CONSUMABLE`/`LIMIT` apply, reporting interval). Corrected behavior: `info resource <name>` prints the resource's definition metadata; `info resource all` lists only resources flagged `XCE_RESF_GLOBAL` or `XCE_RESF_SHARED` (host-based/per-worker resources are out of scope for this command — they are `info worker`'s job). A global/shared resource's current value is read from the core's single shared-instance pool (`core->sharedInst`, indexed by the resource's `idx` — the same pool `RESVALS_FULL`/`RESVALS_DELTA` events and snapshot save/load already treat as the defining store for these resources), never aggregated across workers. - - 5. **REPL command arguments accept an optional pair of surrounding quotes.** An operator copying a jobid straight out of `info jobs`' own output (e.g. `0-15-1787129769-0[0]`) and habitually quoting it (`info job "0-15-1787129769-0[0]"`) got a silent wrong-answer failure (`parse_jobid()`'s regex requires a whole-string anchor match, so the literal quote characters broke the match, and the `strtoull()` fallback parsed the leading `"` as a nonnumeric prefix, silently producing jobid 0) instead of an intelligible error. Corrected behavior: `repl_split_tokens()` strips one matching pair of leading/trailing `"` or `'` characters from each token before it reaches any command handler. - -### Requirement 6: Object sharing forward-compatibility - -**Objective:** As a maintainer, I want the core's data-ownership patterns to leave room for future resreq/pend-reason object sharing, so that a later optimization spec can introduce sharing without retrofitting these invariants under load-bearing code. - -Full object sharing (resreq template registry, pend reason dedup pool) is deferred to a follow-up spec. However, the architecture must not block sharing from being added later: - -#### Acceptance Criteria - -1. **Resreq immutability**: Resreq objects shall be treated as immutable once created. No code path shall mutate a resreq in place. This ensures sharing via `const` pointer is safe to introduce later. - -2. **Pend reason factory**: All pend reason allocation shall go through `make_pendreason()` (the single factory in `lib/preason.c`). Callers never `calloc(sizeof(xce_pendreason_t))` directly. - -3. **Pointer-identity fast path**: Code comparing two resreqs or pend reasons shall use pointer equality (`a == b`) as a fast path before deep comparison. - -4. **Select expression memoization**: The existing `select_eval_ctx_t` per-session memoization shall be preserved and extended. Parsed/evaluated form attached to resreq on first evaluation, reused for subsequent evaluations. - -5. **Snapshot forward-compatibility**: The snapshot format shall not preclude object sharing. Shared objects shall be serializable by identity (object table with IDs). - -These invariants impose zero overhead today but rule out design choices that would make sharing expensive or impossible to retrofit. - -## Data Structure Changes Summary - -### `xce_scheduler_t` additions -```c -char* job_pick_plugin; // default: "builtin_fcfs" (Req 2) -``` - -### `xce_sched_queue_t` additions -```c -char* project_name; // owning project (picker uses internally) -char* job_pick_plugin; // per-queue picker override (NULL = use global) -``` - -### `xce_sched_project_t` additions -```c -int32_t priority; // inter-project ordering (future spec) -// future: resource_quota, resource_used, parent_project -``` - -### `xce_job_picker_iface_t` (new) -```c -typedef struct { - int (*init)(void* picker, xce_sched_core_t* core); - void (*fini)(void* picker); - int (*begin_session)(void* picker); - xce_sched_job_t* (*next_job)(void* picker); - int (*end_session)(void* picker); - int (*on_dispatch)(void* picker, uint64_t jobid); - int (*on_pend)(void* picker, uint64_t jobid); - int (*on_queue_update)(void* picker, xce_sched_queue_t* queue); - int (*on_job_push)(void* picker, xce_sched_job_t* job); - int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state); - int (*on_job_update)(void* picker, xce_sched_job_t* job); - int (*on_config)(void* picker, xce_conf4sched_t* config); - int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); -} xce_job_picker_iface_t; -``` - -### Core/Driver split -```c -// Core event (driver → core) -typedef enum { XCE_CORE_EVENT_CONFIG_PUSH = 0, XCE_CORE_EVENT_JOBS_PUSH, - XCE_CORE_EVENT_WORKERS_PUSH, XCE_CORE_EVENT_RESOURCES_PUSH, - XCE_CORE_EVENT_RESVALS_FULL, XCE_CORE_EVENT_RESVALS_DELTA, - XCE_CORE_EVENT_JOB_STATE_CHANGE, XCE_CORE_EVENT_JOB_UPDATE, - XCE_CORE_EVENT_QUEUE_UPDATE, XCE_CORE_EVENT_JOB_RUSAGE, - XCE_CORE_EVENT_BEGIN_ROUND } xce_core_event_type_t; - -typedef struct { - xce_core_event_type_t type; - uint64_t ts; - union { /* event-type-specific payload */ } payload; -} xce_core_event_t; - -typedef int (*xce_core_decision_cb_t)(void* driver_ctx, xce_decision_t* decision); - -typedef struct { - // Scheduling state (moved from xce_scheduler_t) - xce_conf4sched_t* config; var plugins; var jobs, queues, workers, users, projects; - // ... (full state) - - // Core/driver boundary - void* driver_ctx; - xce_core_decision_cb_t on_decision; - - // Internal threading — cbuf producer/consumer - pthread_t picker_thread, allocator_thread; - cbuf_t wbuf; // driver→picker trigger (int, cap=1) - cbuf_t jbuf; // picker→allocator job handoff (uint64_t, cap=1) - cbuf_t decisions; // allocator→driver decision output (xce_decision_t*, cap=5) - _Atomic int session_active; // 1 during session, 0 between - - // Clock virtualization — atomically updated by driver - _Atomic uint64_t virtual_clock_base, real_clock_base, speed_num, speed_den; - uint64_t prng_seed; - - // Picker plugin slot - char* job_pick_plugin; -} xce_sched_core_t; - -typedef struct { - xce_sched_core_t core; - xcecli_ctx_t apictx; - xce_cluster_config_t cluster_cfg; - dataq_t* pend_updates[2]; - // Recording state (driver-layer only — core is unaware) - char* record_dir; // configured recording directory (AC 10 path validation); - // loaded from scheduler config at startup, default "/var/log/xce/" - char* record_file_path; // strdup'd path for status queries (AC 10) - FILE* record_file; // replay file handle (NULL when not recording) - bool recording; - uint64_t record_event_count; - uint64_t record_snapshot_interval; - uint64_t record_start_ts; -} xce_sched_driver_t; - -typedef struct { - xce_sched_core_t core; - FILE *replay_file, *alt_events_file, *record_file; - // sim parameters (mutable via REPL 'set' command — AC 22) - uint64_t sim_speed_num, sim_speed_den; // 0/1=fast, 1/1=real-time, N/1=Nx - bool verify; // decision comparison on/off - bool interactive; // REPL active -} xce_sched_sim_driver_t; - -// Core interface -int xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed); -int xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents); // batch push before session -int xce_core_go(xce_sched_core_t*); // NON-BLOCKING: signal picker thread -int xce_core_end_session(xce_sched_core_t*); // GC, reset workers -uint64_t xce_core_now(xce_sched_core_t*); // virtualized clock accessor -int xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out); // fills out with core state -int xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in); // restores from snapshot -void xce_sched_core_fini(xce_sched_core_t*); -``` - -## Architecture Diagrams - -### Internal Core Thread Architecture (Multi-Threaded, cbuf-Based) - -``` -┌──────────────────────────────────────────────────────────────────┐ -│ CORE (process) │ -│ │ -│ [PICKER THREAD] [ALLOCATOR THREAD] │ -│ │ -│ cbuf_get(&wbuf) ──┐ cbuf_get(&jbuf) ──┐ │ -│ begin_session() │ try_allocate() │ │ -│ while next_job(): │ can_preempt() │ │ -│ cbuf_put(&jbuf,─┼──► cbuf_put( │ │ -│ &jobid) │ &decisions,──► on_decision callback │ -│ cbuf_put(&jbuf, 0) │ &decision) │ │ -│ end_session() │ (NULL on EOS) │ │ -│ ┌─────────────────┘ ┌─────────────────┘ │ -│ │ │ │ -│ │ wbuf jbuf │ decisions │ -│ │ (int) (uint64_t) │ (xce_decision_t*) │ -│ │ cap=1 cap=1 │ cap=5 │ -│ └─────────────────────────┴──────────────────────────────────────│ -│ │ -│ [Clock atoms — lock-free] [Core state] │ -│ virtual_clock_base queues, jobs, workers │ -│ real_clock_base plugins, resources, resvals │ -│ speed_num, speed_den picker plugin instance │ -│ prng_seed users, projects │ -└──────────────────────────────────────────────────────────────────┘ -``` - -### Core Scheduling Loop (per-round) - -``` - ┌─────────────────────┐ - │ DRIVER THREAD │ - │ │ - │ core_push_events() │ ← batch: CONFIG_PUSH, JOBS_PUSH, - │ picker hooks │ JOB_STATE_CHANGE, QUEUE_UPDATE, - │ called sync │ WORKERS_PUSH, RESVALS_*, etc. - │ │ - │ core_go() ──────────┼── cbuf_put(&wbuf, &go) - │ NON-BLOCKING │ - │ │ - │ while (1): │ - │ cbuf_get(&decisions)── block per decision - │ if NULL: break │ ← end-of-session marker - │ on_decision(d) ───┼── publish to controller NOW - │ │ (xcecli_jobdispatch / - │ core_end_session() │ xcecli_set_pend_reason / - │ │ xcecli_preempt) - │ sleep(1) or next │ - │ event trigger │ - └─────────────────────┘ -``` - -### Core/Driver Architecture - -``` -┌──────────────────────────────────────────────────────────────┐ -│ DRIVER LAYER │ -│ Normal Mode Simulation Mode │ -│ controller capnp ──┐ replay.replay ──┐ │ -│ handler → events │ virtual clock │ │ -│ → publish decisions│ → record/log │ │ -│ │ ▲ │ │ ▲ │ │ -└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘ - │ │ decisions │ │ decisions - ▼ │ ▼ │ -┌──────────────────────────────────────────────────────────────┐ -│ SCHEDULER CORE │ -│ core_push_event() → picker hooks → picker->next_job() │ -│ → try_allocate() → on_decision() → DISPATCH / PREEMPT / PEND │ -│ ZERO dependencies on: controller, capnp, network, wall-clock, filesystem │ -└──────────────────────────────────────────────────────────────┘ -``` - -## Future: Project Resource Ownership - -Deferred to a separate `project-resource-ownership` spec. When implemented: - -- Projects form a tree with resource quota at each node -- A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue -- The core scheduling loop requires zero code changes — only a new picker plugin replaces `builtin_fcfs` -- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes -- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends - -The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. The only difference is which picker plugin is configured. diff --git a/.kiro/specs/scheduler-core/research.md b/.kiro/specs/scheduler-core/research.md deleted file mode 100644 index 7dda3e21..00000000 --- a/.kiro/specs/scheduler-core/research.md +++ /dev/null @@ -1,98 +0,0 @@ -# Research & Design Decisions: scheduler-core - -## Summary -- **Feature**: `scheduler-core` -- **Discovery Scope**: Complex Integration (extension of an existing C daemon, new wire protocol, new cross-component opcode) -- **Key Findings**: - - `requirements.md` is unusually design-complete for a requirements doc — most C-level contracts (struct layouts, thread model, capnp wire framing) are already normative, not left to design. Design's real job here is: (1) the still-deferred `ReplayEvent`/`ReplayDecision` payload capnp schemas, (2) the concrete File Structure Plan, (3) closing two boundary tensions the requirements doc surfaces but doesn't resolve. - - Almost every payload type `ReplayEvent.body`'s union needs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) **already exists verbatim** in `src/schema/controller.capnp` — the requirements author already picked names that match the live schema. Only `JobStateChangeEvent` and `JobUpdateEvent` are genuinely new. This is a Build-vs-Adopt win: reuse, don't reinvent. - - The scheduler component's real file layout (`src/scheduler/src/{decision.c,handler.c,main.c,plugin.c,scheduler.c,plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c},resource.c,schedobj.c}`) is substantially larger than `.wolf/anatomy.md`'s cached listing (which only knew `handler.c`, `main.c`, `scheduler.c`). File Structure Plan below is built from the live `find`, not the stale cache. - -## Research Log - -### Existing scheduler component layout -- **Context**: Need concrete file paths for the File Structure Plan; `structure.md` gives the pattern (`include/`, `src/`, `testsuite/`, plugin subdir), but not the current file set. -- **Sources Consulted**: `find src/scheduler -name '*.c' -o -name '*.h'`, `src/scheduler/testsuite/meson.build`. -- **Findings**: `include/`: `decision.h`, `plugin.h`, `sched/types.h`, `scheduler.h`. `src/`: `decision.c`, `handler.c`, `main.c`, `plugin.c`, `scheduler.c`, `resource.c`, `schedobj.c`, `plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c}`. `testsuite/`: `test.c` (runner harness), `test_handlers.c`, `test_job_array_expand.c`, `test_job_array_wire_e2e.c`, `test_job_update.c` — standalone TUs, several `#include ../src/*.c` directly (white-box pattern) rather than linking. -- **Implications**: New core/driver files (`xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`) are new `src/` files, not modifications of `scheduler.c` wholesale — `scheduler.c` keeps today's `xce_scheduler_t`-owning code during Phase 1, then Phase 2 extracts state into a new `core.c`. Test files for the new pieces follow the existing standalone-TU-with-`#include`, one-binary-per-unit convention already established in `testsuite/meson.build`. - -### Controller-side opcode dispatch location and pattern -- **Context**: AC 10 requires a new `OPCode_schedAdmin` case; need to confirm which file owns `myreq_handler` and the exact style of an existing opcode case for consistency. -- **Sources Consulted**: `grep -rln myreq_handler src/controller/*.c`, `cluster.c` around `OPCode_fetchfile` and `OPCode_upgradeConfig`. -- **Findings**: `myreq_handler` lives in `src/controller/cluster.c` (also referenced from `ha.c`/`ha_convergence.c`, but the dispatch switch itself is in `cluster.c`). The closest structural precedent is `OPCode_upgradeConfig` (`cluster.c:670-705`): gated by `ha_is_primary(&c->ha)` (reject standby with `-EAGAIN`, `RequestReply_data_nodata`), then delegates to a `handle_*()` function. **No RBAC/auth check is visible in that case block** — the comment trail (`cluster.c:672-693`, bug-fix 2026-07-25) discusses primary/standby gating exhaustively but never mentions an authentication check at this opcode. If `OPCode_upgradeConfig` has one, it happens earlier (TLS handshake / connection-accept time) or not at all at this layer. -- **Implications**: AC 10's claim "same RBAC role as config push" cannot be verified against a concrete gate in the code as it stands — carried into design.md as an **Open Question**, not asserted as a resolved fact. `OPCode_schedAdmin`'s case block should structurally mirror `OPCode_upgradeConfig` (primary-only gate, `handle_*()` delegation, explicit reply on every path) since that is the only comparable admin-opcode precedent in the file. - -### xceadmin CLI extension pattern -- **Context**: AC 10's `xceadmin scheduler record start/stop/status` and AC 20's `xceadmin scheduler dump` need a concrete module plan consistent with the existing `config push`/`config build` implementation. -- **Sources Consulted**: `src/xceadmin/src/xceadmin/{cli.py,push.py,build.py,discovery.py}`, `src/xceadmin/src/xcecommon/{wire.py,schema.py}`. -- **Findings**: `cli.py` is a thin `click` layer only (`@cli.group()` per resource, `@group.command()` per verb) — all business logic lives in per-resource modules (`push.py`, `build.py`) that a) load the schema once via `xcecommon.schema.load_schema()`, b) build/decode capnp messages, c) go through `xcecommon.wire.connect`/`send_request` for the socket, and d) for request/reply opcodes, route through `xceadmin.discovery.follow_redirects` + `xceadmin.discovery.decode_reply` (which unwraps the `Request{opcode: OPCode_reply, body: replymsg}` envelope every reply is wrapped in — a real bug fixed 2026-07-30, documented in `discovery.py`'s own docstring). Every raising path uses one exception hierarchy (`XceAdminError` subclasses), caught once in `cli.py`. -- **Implications**: A new `xceadmin/scheduler.py` module is the natural home for `record_start()`, `record_stop()`, `record_status()`, and `dump()` — same shape as `push_config()`/`probe_config()`, reusing `discovery.follow_redirects`/`decode_reply` (this is a request/reply opcode per AC 10, so unlike `probe_config()` it DOES need redirect-following). `dump()` differs from the other three: it operates on a local `.replay` file only, no network call — it decodes a capnp stream with `xcecommon.schema.load_schema()`'s module object and formats each message type to text; it belongs in the same module for CLI cohesion but has no wire dependency. - -### Reusable capnp payload types for ReplayEvent / SnapshotData -- **Context**: Requirements AC 4 explicitly defers the exact capnp schema for event payloads to design; AC 20's `ReplayEvent.body` union names `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `JobStateChangeEvent`, `JobUpdateEvent`, `QueueConf`, `SchedJobRusages` without defining the two `*Event` types. -- **Sources Consulted**: `grep -n '^struct ' src/schema/controller.capnp`. -- **Findings**: `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages` **all already exist** in `controller.capnp` with those exact names (the requirements author evidently cross-checked the schema before naming the union members). `JobStateChangeEvent` and `JobUpdateEvent` do not exist. The existing `Job` struct (`controller.capnp:288`) already carries `jobid` + `state`, and `JobAlloc`/`JobPendReason` exist for allocation/pend-reason payloads that the new `DispatchDecision`/`PendDecision` need. -- **Implications**: Design only needs to define 2 new event payload structs and 3 new decision payload structs (reusing `JobAlloc` and `JobPendReason` internally), not 9. See Data Models section of `design.md`. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Event-sourced core with pluggable driver (selected) | Core holds all scheduling state, driven purely by typed events + a pull-based clock; two driver implementations (normal, simulation) translate their respective input sources to the same event stream | Normal↔sim equivalence becomes a structural property, not a discipline; picker/allocator threading is unchanged from today, minimizing regression surface | Requires strict discipline (no wall-clock calls, no controller calls) inside the core — a single violation breaks the central invariant silently until someone runs `--sim-verify` | Already fully specified in requirements.md; design.md's job is to make the boundary and remaining schema concrete, not to re-litigate the pattern | -| Mode-flagged single scheduler (rejected in requirements) | Keep `xce_scheduler_t` as-is, add `if (sim_mode)` branches for clock/IO | No new types, smallest diff | Directly violates the central invariant requirements establishes (normal/sim would share code paths only by discipline, not by construction) — rejected by requirements.md Req 5 AC 1, not re-opened here | — | - -## Design Decisions - -### Decision: New capnp structs for JOB_STATE_CHANGE / JOB_UPDATE events -- **Context**: AC 4 specifies the field set (jobid, old_state, new_state, exit_code for state-change; jobid, changed_fields, full job record for update) but no capnp shape exists. -- **Alternatives Considered**: - 1. Reuse `Job` alone and let the driver infer old/new state from two full records — rejected, doubles payload size for a two-field delta and loses `exit_code`. - 2. Define thin new structs that carry only the delta plus embed `Job` where a full record is genuinely needed — selected. -- **Selected Approach**: `JobStateChangeEvent{jobid, oldState, newState, exitCode}`; `JobUpdateEvent{jobid, changedFields, job: Job}` (embeds the existing `Job` struct, not a re-derived one). -- **Rationale**: Matches AC 4's field table exactly; embedding `Job` for the update case (where the driver already has to build a full record) avoids a second job-shape definition; the state-change case stays flat since AC 4 requires no other job fields. -- **Trade-offs**: `JobUpdateEvent` duplicates the full `Job` payload even for a single-field priority change — acceptable, matches AC 4's explicit field table (`job: xce_job_t*` full record), not a design deviation. -- **Follow-up**: None — field set is normative from requirements, only the capnp wrapper is new. - -### Decision: Decision payload structs (DispatchDecision / PendDecision / PreemptDecision) — and a boundary note on PreemptDecision -- **Context**: AC 5 fully specifies fields for all three decision types. Separately, the Boundary Context assigns "the preemption decision type" to the `scheduler-preemption` sibling spec. -- **Alternatives Considered**: - 1. Omit `PreemptDecision` entirely from this design, leave a placeholder in the `ReplayDecision.body` union for `scheduler-preemption` to add later — rejected, AC 5's decision output table is a Requirement 5 AC in *this* spec, and the `on_decision` callback signature / `ReplayDecision` union must compile today (Phase 2) before `scheduler-preemption` exists. - 2. Define `PreemptDecision`'s wire shape now per AC 5's field table, treat victim-selection *semantics* (not the wire shape) as what `scheduler-preemption` owns — selected. -- **Selected Approach**: `PreemptDecision{jobid, victims: List(PreemptVictim{jobid, workerid}), reason: JobPendReason}` defined in this spec's schema additions. -- **Rationale**: The wire struct is a passive data shape with no scheduling logic in it; nothing about defining it here binds `scheduler-preemption` to any victim-selection algorithm. Leaving the union member undefined would make `xce_decision_t`/`ReplayDecision` incomplete types this spec's own Phase 2 code needs to reference (`can_preempt()`'s caller, per requirements.md Req 1 AC 4's allocator pseudocode, already branches on a preempt outcome). -- **Trade-offs**: If `scheduler-preemption` later needs additional fields on `PreemptDecision`, that spec must extend this schema (a revalidation trigger, listed in design.md's Boundary Commitments) rather than owning it outright from scratch. -- **Follow-up**: Flag explicitly to whoever picks up `scheduler-preemption` that the wire shape is pre-committed here; the spec should confirm the field set is sufficient before implementation. - -### Decision: `OPCode_schedAdmin` authorization mechanism — deferred to a follow-up spec (project owner call, 2026-08-02) -- **Context**: AC 10 states "same RBAC role as config push," but the only comparable opcode (`OPCode_upgradeConfig`) has no visible authorization check in `cluster.c` — only a primary/standby gate. -- **Alternatives Considered**: - 1. Assert a specific mechanism (e.g. "add a `cluster:control` RBAC check") without verifying one exists for `config push` today — rejected, would document a fictitious parity claim. - 2. Block Phase 3 until the implementer traces `config push`'s real auth boundary and applies it here too — rejected by project owner: not worth gating this spec's delivery on. - 3. Ship `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current posture (primary-gate only, no RBAC check), document the gap explicitly, and track auth hardening as separate future work — **selected**. -- **Selected Approach**: Option 3. `design.md`'s Open Questions/Risks and Security Considerations sections record this as an accepted, deliberate gap rather than an unresolved blocker. -- **Rationale**: `config push`'s current authorization boundary could not be located in the controller-side handler during discovery. Rather than block this spec on retrofitting auth for two opcodes at once, the project owner chose to scope that work separately. -- **Trade-offs**: `OPCode_schedAdmin` (and, unchanged, `OPCode_upgradeConfig`) remain gated only by primary/standby routing until the follow-up lands — acceptable short-term since both are already TLS-connected, admin-tooling-only surfaces (xceadmin), not general network-reachable endpoints. -- **Follow-up**: Add an "auth hardening for admin opcodes (`OPCode_upgradeConfig`, `OPCode_schedAdmin`)" item to the project roadmap (see `design.md` Open Questions/Risks #1). Track as a future spec, not a task inside `scheduler-core`. - -### Decision: `OPCode_schedAdmin`'s blocking cross-process wait — accepted as a boundary case, no admission reservation (project owner call, 2026-08-02) -- **Context**: Design review (2026-08-02) flagged that `myreq_handler` runs as a `vtask_t` handler (bounded 8-thread pool, `cluster.c:238`), and `OPCode_schedAdmin` blocks that thread up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock` — unlike `OPCode_upgradeConfig`'s in-process, `COMMIT_WAIT_RESERVED`-admitted commit-barrier wait. `ha-async-io.md`'s vtaskmgr rule requires admission reservation for bounded blocking and forbids unbounded blocking on this pool; the `raft-drive-async` family fixed prior real violations of this exact rule (`config-push`'s busy-poll+join was one of them). -- **Alternatives Considered**: - 1. Add a `COMMIT_WAIT_RESERVED`-style admission reservation for `OPCode_schedAdmin` before Phase 3 ships — the design-review recommendation. - 2. Make the scheduler forward asynchronous (vtask completion callback instead of blocking wait) — larger structural change, deferred as unnecessary for current usage. - 3. Accept the blocking wait as a boundary case given expected request volume — **selected**. -- **Selected Approach**: Option 3. `xceadmin scheduler record`/`dump` commands are cluster-administrator-triggered, human-paced, low-frequency — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism is added. -- **Rationale**: The `COMMIT_WAIT_RESERVED` precedent exists because commit-barrier waits are on the hot path of every job submission (high, uncontrolled frequency). Admin recording control is operator-initiated and rare by nature; the risk this pattern protects against in the commit-barrier case doesn't apply at comparable severity here. -- **Trade-offs**: If usage patterns change (e.g. automated/scripted recording control invoked at scale, or a monitoring tool polling `record status` frequently), this becomes a live risk without warning — no test or admission gate would catch it. -- **Follow-up**: If `scheduler record`/`dump` usage ever becomes automated or high-frequency, revisit with a `COMMIT_WAIT_RESERVED`-style admission reservation. -- **SUPERSEDES (2026-08-19, design-review follow-up)**: Option 3 above is no longer the shipped behavior. A retrospective design review flagged that `c->sched.lock` already serializes concurrent `schedAdmin` requests one at a time — a burst of N concurrent requests could occupy N vtask threads for up to N×5s each even under Option 3's "accepted, human-paced" framing, which this Decision's own Trade-offs bullet didn't fully account for. `vtaskmgr_blocking_acquire`/`vtaskmgr_blocking_release` (Option 1's mechanism, reusing `commit.c`'s existing `c->vtaskmgr` reservation rather than a new one) now wrap the whole forward+wait region; admission-exhausted requests fast-reject with `{ok=false, error="server busy, try again"}` instead of queuing. See `design.md`'s `SchedAdminOpcode` component ("Blocking-wait admission (hardened 2026-08-19)") and `requirements.md` Req 5 AC 10's own Amendment for the current, authoritative description. - -## Risks & Mitigations -- **`OPCode_schedAdmin` ships without an RBAC/auth check beyond the primary/standby gate — accepted, not blocking.** Mitigation: documented explicitly in `design.md`; tracked as a future roadmap item (see Decision above), not a Phase 3 dependency. -- **`OPCode_schedAdmin` blocks a vtask thread for up to 5s per request — MITIGATED 2026-08-19.** Now gated by the same `vtaskmgr_blocking_acquire`/`_release` admission reservation `commit.c`'s commit-wait barrier shares (see the Decision's "SUPERSEDES" note above); admission-exhausted bursts fast-reject rather than occupying vtask threads. -- **`PreemptDecision` wire shape is pre-committed before `scheduler-preemption`'s own design exists** — mitigation: documented as a revalidation trigger; `scheduler-preemption` must review this shape early rather than discovering it mid-implementation. -- **`jbuf` capacity=1 determinism invariant depends on a `FIXME: make this configurable` comment never being resolved carelessly** — mitigation: already hard-pinned in requirements.md Req 5 AC 7 with an explicit Phase 2 task; design.md's File Structure Plan marks `xce_sched_core_init()` as the single place this is enforced. - -## References -- `src/scheduler/CLAUDE.md`, `.kiro/steering/{structure.md,tech.md,ha-async-io.md}` — component layout and controller HA invariants. -- `src/schema/controller.capnp` — canonical wire schema; source of truth for reused struct names. -- `src/xceadmin/src/xceadmin/push.py`, `discovery.py` — reference pattern for the new `scheduler.py` xceadmin module. diff --git a/.kiro/specs/scheduler-core/spec.json b/.kiro/specs/scheduler-core/spec.json deleted file mode 100644 index 97192ecc..00000000 --- a/.kiro/specs/scheduler-core/spec.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "feature_name": "scheduler-core", - "created_at": "2026-07-27T00:00:00Z", - "updated_at": "2026-08-19T00:00:00Z", - "language": "en", - "phase": "implementation-complete", - "parent_spec": "scheduler-framework", - "description": "Core/driver separation, pluggable job picker interface, and simulation mode for the XCE scheduler", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "blocked_on": [], - "sibling_specs": ["scheduler-allocation", "scheduler-preemption"] -} diff --git a/.kiro/specs/scheduler-core/tasks.md b/.kiro/specs/scheduler-core/tasks.md deleted file mode 100644 index 0e265118..00000000 --- a/.kiro/specs/scheduler-core/tasks.md +++ /dev/null @@ -1,431 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: Picker Interface & Plugin Registration - -- [x] 1.1 Define the job picker plugin contract - - Add the lifecycle hooks (init, fini), state-change hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update), session hooks (begin_session, next_job, end_session), post-allocation hooks (on_dispatch, on_pend), and optional hooks (on_config, on_job_rusage MAY be NULL) to the picker plugin header - - Document the contract: every hook signature, which thread each hook category runs in (driver/picker/allocator), cancel-safety requirement for `next_job()`, and session-reset requirement for `begin_session()` - - Observable: the header compiles standalone and defines a complete function-pointer struct matching every hook category - - _Requirements: 1.2, 1.5, 2.6_ - -- [x] 1.2 Register the picker plugin category and deprecate the old election hook - - Add a picker plugin registration category to the plugin registration table, parallel to existing plugin categories - - Add a compatibility wrapper that adapts an old-style `job_elect_fn`-only plugin to the new interface, logging a deprecation warning at registration time - - Observable: registering a plugin that implements only `job_elect_fn` succeeds via the wrapper and logs a deprecation warning; registering one that implements the new interface directly succeeds without a warning - - _Requirements: 1.6, 2.5_ - - _Depends: 1.1_ - -- [x] 1.3 (P) Add picker configuration fields - - Add a global picker plugin name field to the scheduler, defaulting to `builtin_fcfs` when unset - - Add a per-queue picker plugin override field, defaulting to NULL (inherit global) - - Add a project-name field to the queue struct, and extend the project struct with a priority field (scaffolding only — no quota fields) - - Observable: a queue config with no override inherits the global picker name; a queue config with an override reports that override, not the global default - - _Requirements: 2.1, 2.2, 2.3, 2.4_ - - _Boundary: SchedulerCore config fields_ - -- [x] 2. Core: Builtin FCFS Picker - -- [x] 2.1 Implement builtin_fcfs lifecycle and private state - - Implement `init`/`fini`: allocate/free the picker's private ordered-queue state (queue ordering by priority desc, and one job-ordering structure per queue) - - Never query the scheduler's own queue/job tables from inside the picker after init — all candidate state lives in the picker's private structures - - Observable: after `init()`, the picker holds an empty private candidate set; after `fini()`, all picker-owned memory is released (verified under a leak-checked test run) - - _Requirements: 3.1, 3.3_ - - _Depends: 1.1, 1.2_ - - _Verified: 2026-08-03 (commit `f7921b96`) — backfilled 2026-08-07 by `/kiro-validate-impl` (this task and 2.2-2.6 were implemented and reviewed at the time but never got a `_Verified:` line). New `src/scheduler/src/plugins/builtin_fcfs.c` (not a `fcfs.c` rewrite — see design.md's BuiltinFcfsPicker notes and Implementation Notes below for why): `fcfs_plugin_data_t` holds `ordq_t queues` plus per-queue `ordq_t pendJobs`, allocated in `init()`/freed in `fini()`; never queries `sched->queues`/`sched->jobs`._ - -- [x] 2.2 Implement builtin_fcfs state-change hooks - - `on_queue_update`: upsert a queue into the picker's ordered queue list, keyed by descending priority - - `on_job_push`: add a newly-pending job into its queue's candidate set, ordered by priority desc then submission/queue time asc - - `on_job_state_change`: remove a job from candidates when it leaves pending state, re-add it when it returns to pending - - `on_job_update`: reposition a job in its candidate set when its priority or queue assignment changes - - Observable: feeding a sequence of these hooks directly (no live core) produces a candidate set whose order matches the documented ordering rule, verifiable by a direct hook-driven test - - _Requirements: 3.2_ - - _Depends: 2.1_ - - _Verified: 2026-08-03 (commit `b7214ebe`) — backfilled 2026-08-07. `on_queue_update`/`on_job_push`/`on_job_state_change`/`on_job_update` implemented on `fcfs_plugin_data_t`'s `ordq_t` structures, ordering `(queue priority desc, job priority desc, queue_time asc)`; `test_picker_fcfs.c` feeds the hooks directly (no live core) and asserts the resulting order._ - -- [x] 2.3 Implement builtin_fcfs session and post-allocation hooks - - `begin_session`: reset the iteration cursor to the head of the ordered queue list - - `next_job`: walk queues then jobs in ordering, skipping not-ready queues/jobs, returning the first ready job and advancing the cursor; returns NULL when exhausted - - `end_session`: no-op (iteration state already reset in `begin_session`) - - `on_dispatch`: remove the job from its queue's candidate set permanently - - `on_pend`: no-op — job remains in its candidate set for the next round in the same position - - Observable: a full begin_session → repeated next_job → end_session cycle against a fixed candidate set returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order and returns NULL exactly once all candidates are exhausted - - _Requirements: 3.2, 3.4, 3.6_ - - _Depends: 2.2_ - - _Verified: 2026-08-03 (commit `fcb32a8b`) — backfilled 2026-08-07. `begin_session`/`next_job`/`end_session`/`on_dispatch`/`on_pend` implemented per plan; this task's own review found the real cross-thread race later closed by 2.4 (picker thread's `next_job()` vs allocator thread's `on_dispatch`/`on_pend` racing unsynchronized `fcfs_plugin_data_t` state) — see `## Implementation Notes` below, "Task 2.3's review found a real concurrency gap for task 2.4 to close."_ - -- [x] 2.4 Wire builtin_fcfs as the registered default and remove the hardcoded picker string - - Auto-register `builtin_fcfs` at scheduler startup under the name `"builtin_fcfs"` and set it as the default picker plugin - - Replace the hardcoded `"xce_sched_fcfs"` literal with a lookup of the configured picker plugin name through the existing plugin registration/lookup mechanism; fail scheduler startup with a clear error if the configured name isn't registered - - Confirm the existing three-thread/cbuf pipeline (picker thread, allocator thread, driver-signaled `wbuf`/`jbuf`/`decisions` cbufs) is unchanged by this wiring — no new threads, no cbuf capacity changes - - Add cancel points in `next_job()`'s iteration loop and confirm the picker thread's existing block-on-signal point remains cancel-safe - - Observable: starting the scheduler with no picker config uses `builtin_fcfs` automatically; starting it with an unregistered picker name fails startup with a clear error instead of silently falling back - - _Requirements: 1.3, 1.4, 1.6, 2.5, 3.1, 3.5_ - - _Depends: 2.3, 1.3_ - - _Verified: 2026-08-03 (commit `7ee9a16f`) — backfilled 2026-08-07. `builtin_fcfs` registered and defaulted (`schedobj.c:364`); registered "alongside the untouched legacy `xce_sched_fcfs`" (`scheduler.c:466`'s own comment) rather than replacing `plugins/fcfs.c` in place — `fcfs.c` stays as task 1.2's `job_elect_fn` compat-wrapper exercise target. `sched_job_picker()` (`scheduler.c`) branches `job_elect_fn != NULL` (legacy, self-contained loop) vs `NULL` (native picker, generic per-session loop). Closed this task's own review-found race (see 2.3's note) with `fcfs_plugin_data_t.lock` (`mylock_t`), held by exactly 7 hooks (`on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update`, `begin_session`, `next_job`, `on_dispatch`) — `on_pend`/`end_session`/`init`/`fini` correctly excluded (never touch the locked state)._ - -- [x] 2.5 Remove ordering logic from the core scheduling loop - - The core loop calls only `picker->next_job()` to obtain the next schedulable job — no project/queue/job iteration or ordering logic remains in the loop itself - - Observable: a code-level check confirms the scheduling loop function contains no direct queue/job traversal — all selection reads happen through the picker interface - - _Requirements: 1.1, 1.7_ - - _Depends: 2.4_ - - _Verified: 2026-08-03 (commit `9d91d470`) — backfilled 2026-08-07. Confirmed via source read: `sched_job_picker()`'s native-picker branch calls only `picker_iface->next_job()` to obtain the next job — no queue/job traversal or ordering logic in the loop itself; a code-level guard test asserts this._ - -- [x] 2.6 Picker regression tests and data-sharing invariant checks - - Assert `builtin_fcfs`'s `next_job()` ordering matches today's pre-change FCFS ordering exactly, given identical queue/job state - - Confirm the existing scheduler unit test suite passes unmodified except at the specific assertions that exercised the removed hardcoded string and the decision-publishing path this feature replaces - - Verify resreq objects are never mutated in place by any touched code path, all pend-reason allocation goes through the existing factory function (no direct `calloc` of the reason struct), comparisons use pointer equality as a fast path before deep comparison, and the existing per-session select-expression memoization is untouched - - Observable: the regression suite passes with the documented ordering rule as its oracle, and a grep-based or code-review check confirms no new `calloc(sizeof(...pendreason...))` call site was introduced - - _Requirements: 4.1, 4.2, 6.1, 6.2, 6.3, 6.4_ - - _Depends: 2.5_ - - _Verified: 2026-08-03 (commit `604a5649`) — backfilled 2026-08-07. `test_picker_pendreason_factory_audit.c` and the picker regression suite confirm ordering-equivalence against legacy `fcfs_elect_job` and the pend-reason/resreq data-sharing invariants (Req 6.1-6.4); part of the 41/41 green suite reconfirmed by this validation pass._ - -- [x] 3. Run Phase 1 HA soak regression - - Run sequential and chaotic HA failover soak rounds against a cluster running the new picker architecture - - Observable: zero new job loss, decision divergence, or liveness stall attributable to this phase's changes across the soak run - - _Requirements: 4.3_ - - _Depends: 2.6_ - - _Verified: 15-round sequential soak, 2026-08-03 — PASS: 15 rounds, 48 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)_ - -- [x] 4. Foundation: Scheduler Core Extraction - -- [x] 4.1 Extract the scheduling core from the monolithic scheduler struct - - Split scheduling state (jobs, queues, workers, resources, resvals, users, projects, plugin instances) into its own core structure holding zero controller I/O state (no API client context, no cluster config) - - Confirm the core links only against the approved minimal data-structure subset and never links network/timer/API libraries — no wall-clock time call appears anywhere in the extracted core's source - - Observable: the core structure compiles and links with zero symbols from the networking/API libraries; a build-time or link-time check confirms this - - _Requirements: 5.1, 5.2_ - - _Depends: 3_ - - _Verified: 2026-08-03 — new xce_sched_core_t (core.h/core.c) holds all scheduling state, xce_scheduler_t shrank to {core; apictx; cluster_cfg}; nm-based isolation test (test_core_isolation.c) proves zero netio/xcecli/idgen/delaymgr symbols; 14/14 scheduler suite passing; independent review APPROVED (own fresh build/test/nm run + reproduced RED)_ - -- [x] 4.2 Pin the determinism invariants - - Hardcode the picker-to-allocator handoff channel's capacity to 1 inside core initialization — never expose it as a parameter - - Audit every hash-table/key-value-store iteration reachable from the scheduling loop (jobs, queues, users, projects) and confirm each produces order deterministic in (keys, insert/delete sequence) with no process-random seeding; fix any site found unstable - - Observable: core initialization rejects (or simply has no code path to accept) a caller-supplied handoff-channel capacity; the iteration-order audit's findings are recorded and any fix is covered by a targeted test - - _Requirements: 5.7_ - - _Verified: 2026-08-03 — jbuf capacity hardcoded to literal 1 (core.c), regression test test_core_jbuf_capacity_pin.c; kvmgr audit found jobs/queues/users/projects already fully order-stable (xzarray ascending-slot walk, no process-random seeding), no fix needed, findings recorded durably as a comment in core.c above xce_sched_core_init(); 15/15 scheduler suite passing; two-round review (round 1 REJECTED on missing durable record only, round 2 APPROVED after fix); core->workers flagged as an open follow-up (see Implementation Notes), not this task's scope_ - - _Depends: 4.1_ - -- [x] 4.3 Implement the core event interface and batch event push - - Define the typed core event representation (timestamp plus type-specific payload) covering configuration push, job push, worker push, resource push, resource-value full/delta push, job state change, job update, queue update, and job resource-usage events - - Implement batch event push: state-change picker hooks fire synchronously for each event in the batch, valid only when no scheduling session is active - - Observable: pushing a batch of events with the picker idle fires the corresponding state-change hook exactly once per event, synchronously, before the push call returns - - _Requirements: 5.4, 5.8_ - - _Depends: 4.1_ - - _Verified: 2026-08-03 — xce_core_event_type_t (9 kinds) + xce_core_event_t in core.h; xce_core_push_events() in new core_events.c (split out of core.c to stay under the 800-line cap — design.md's file-structure sketch not updated yet); rejects -EBUSY mid-session, fires each hook exactly once synchronously (JOBS_PUSH once per job); additive only, not wired into the live path (task 5.1/5.2's job). Round 1 REJECTED: CONFIG_PUSH didn't decompose queues/resources/resvals like sched_init_config() does — fixed round 2 (core_apply_config_resources/resvals/queues + shared core_upsert_queue(), also used by QUEUE_UPDATE), round-2 review APPROVED with live RED→GREEN reproduction. 16/16 scheduler suite passing._ - -- [x] 4.4 Implement the session-start signal and decision callback protocol - - Implement the non-blocking session-start call that signals the picker thread and returns immediately, rejecting a call while a session is already active - - Define the three decision output shapes (dispatch, pend, preempt) and the decision callback invoked once per decision as the allocator produces it, plus the explicit end-of-session marker - - Observable: starting a session while one is already active is rejected with a distinct error; each decision reaches the registered callback in allocator-output order, followed by the end-of-session marker - - _Requirements: 5.5, 5.6_ - - _Depends: 4.3_ - - _Verified: 2026-08-03 — xce_core_go()/xce_core_end_session() (new core_session.c) implement the non-blocking session-start signal (-EINVAL on reentrant call, proven a true no-op via cbuf_tryget) and end-session cleanup; xce_core_set_decision_cb() registers the driver's on_decision callback (separate setter, not an xce_sched_core_init() param, to avoid touching its live sched_init() call site); XCE_DECISION_PREEMPT added to decision.h matching the DISPATCH/PEND/PREEMPT AC 6 table exactly. Decision-drain loop confirmed driver-owned (not a core function) via 3 independent requirements.md/design.md citations, re-derived independently by review. Additive only, sched_go/sched_begin_session/sched_end_session/sched_allocator/main.c untouched. REJECTED round 1 on a clang-format alignment regression in core.h (mid-struct insertion de-aligned 24 pre-existing lines) — fixed with clang-format -i, whitespace-only, re-verified 17/17 green._ - -- [x] 4.5 Implement clock virtualization and session timing - - Implement the virtualized clock accessor using the two-anchor-plus-speed formula, backed by atomically-updated fields; no direct wall-clock call anywhere else in the core - - Register the core's own log-timestamp formatter, bound to the virtualized clock accessor instead of the logging library's default wall-clock read — this is the one third-party dependency inside the core that would otherwise call wall-clock time on every log line, silently reintroducing the exact dependency task 4.1 confirmed absent at link time - - Track session start/end timestamps internally and compute session duration for diagnostics, exposed through the session summary - - Observable: the clock accessor's output matches the documented formula for all four operating modes (normal, discrete simulation, real-time simulation, fast/slow simulation) given fixed anchor values; a log line emitted while the virtual clock is frozen at a fixed simulated timestamp shows that timestamp, not the real wall-clock time at the moment of the call - - _Requirements: 5.3, 5.12_ - - _Verified: 2026-08-03 — xce_core_now() (new core_clock.c, kept out of core.c so test_core_isolation.c's wall-clock scan still passes) implements the exact two-anchor+speed formula; all 4 modes proven with fixed anchors, no sleep-based flakiness. requirements.md's literal tlog_reg_format_func(fn, core_ptr) call shape doesn't exist in the real vendored tlog API (verified against tlog.h: registration takes only the callback, userptr is per-call-site) — resolved via a module-level active-core pointer read by a new core-scoped format function, not wired into mylog_init()/main.c yet. session_start/session_duration_ns stamped correctly in xce_core_go()/xce_core_end_session() (task 4.4's behavior otherwise untouched). 18/18 scheduler suite passing, clang-format clean, review APPROVED first pass. Follow-up noted: core_clock.c is the 3rd core-adjacent file (after core_events.c, core_session.c) outside test_core_isolation.c's nm-based audit — manually verified clean, not mechanically enforced._ - - _Depends: 4.1_ - -- [x] 4.6 Core contract tests - - Test the full push-events → start-session → drain-decisions → end-session cycle against a stub picker (no driver needed), asserting hook firing order and timing - - Observable: the contract test suite passes, exercising both the batch-push precondition (rejects mid-session) and the double-session-start rejection - - _Requirements: 5.1, 5.2_ - - _Depends: 4.4_ - - _Verified: 2026-08-03 — new test_core_full_cycle.c drives push_events→go→drain→end_session against a real picker pthread (running the real xce_job_picker_iface_t hooks on a test-local stub picker) plus a main-thread stand-in allocator (sched_allocator() is xce_scheduler_t*-coupled, out of pure-core scope) calling real on_dispatch/on_pend/make_dispatch_decision/make_pend_decision. Both preconditions (-EINVAL reentrant core_go, -EBUSY mid-session push_events) proven against a genuinely active session, not a hand-set flag, and proven to un-latch after the session ends. Two live RED→GREEN reproductions (neutered each guard in turn), both independently reproduced by review. 19/19 scheduler suite passing, clang-format clean, 30/30 stress runs with zero flakiness, review APPROVED first pass. Zero production files touched — test-only, as scoped._ - -- [x] 5. Core: Normal-Mode Driver - -- [x] 5.1 Build the normal-mode driver's event translation path - - Translate controller capnp push/notify messages into core events, buffering them in a pending-event queue between scheduling rounds - - Observable: a sequence of simulated controller push/notify messages, once drained, produces a core event batch whose contents match the input messages one-to-one - - _Requirements: 5.8, 5.9_ - - _Verified: 2026-08-03 — new driver.h/driver.c: xce_sched_driver_translate_pending_events(xce_sched_core_t*) drains pend_updates via the existing dataq-swap pattern, translates all 10 op types to xce_core_event_t (field-verified against core_events.c's actual consumption, not assumed), stamps via xce_core_now(), pushes through the real xce_core_push_events(). Deliberately scoped to xce_sched_core_t* alone (not the full design.md xce_sched_driver_t) since pend_updates already lives on core since task 4.1 and this task is translation-only — task 5.2 can wrap trivially with &driver->core. handler.c untouched. Two flagged risk items (RESVALS zarray dangling-pointer shape, zero-filled changed_fields/exit_code for fields the legacy payload can't supply) both independently verified as pre-existing/honestly-inert, not new bugs. 20/20 scheduler suite passing, real-ASan clean, clang-format clean, review APPROVED first pass._ - - _Depends: 4.4_ - -- [x] 5.2 Wire decision publishing, the mode-selection entry point, and retire the hardcoded decision switch - - Publish each decision to the controller immediately as it's produced (dispatch, pend-reason, preempt) via the existing client API — not batched at session end - - Replace the scheduler process's hardcoded decision-type switch statement with registration of the driver's decision callback - - Add the process entry point's mode-selection branch (normal-mode driver vs. simulation driver based on whether a replay file was given) and normal-mode's own CLI flags - - Observable: submitting a synthetic session that produces a mix of dispatch and pend decisions results in each being published to the (simulated) controller in the order produced, with no batching delay; starting the process with no replay file argument runs the normal-mode driver - - _Requirements: 1.7, 5.6, 5.9_ - - _Depends: 5.1_ - - _Verified: 2026-08-04 — main.c's live loop rewired: xce_sched_driver_translate_pending_events() -> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs() -> xce_core_signal_go() -> decision-drain calling the registered xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(), all in the new exported xce_sched_driver_run_normal_mode_round() (driver.c). xce_core_go() (task 4.4) split into xce_core_begin_session()/xce_core_signal_go() as a pure decomposition (test_core_session.c unmodified, still passing) specifically so plugin_begin_session()/sched_account_jobs() can run in the exact safety-critical window (after resval reset, strictly before the picker is signaled) that task 4.4 deliberately excluded from pure-core scope. Empirically proven, not just documented, by test_driver_session_ordering.c (real cross-thread happens-before assertions) with a live RED->GREEN reproduction, independently reproduced by review. xcecli_preempt() confirmed not to exist anywhere in client.h; PREEMPT decisions fail loudly (-ENOSYS, logged) rather than fabricating a client API. --sim CLI flag added (parse-only, loud not-yet-implemented failure, since SimDriver is Phase 3). 22/22 scheduler suite passing, --help smoke check clean, review APPROVED after one clang-format fix. Found (not introduced, not fixed, flagged for follow-up) a genuine pre-existing plugin_end_session() guard/call mismatch bug in plugin.c predating this spec — affects the legacy xce_sched_fcfs picker's end-of-session cursor reset, not builtin_fcfs._ - -- [x] 6. Run Phase 2 HA soak regression - - Run sequential and chaotic HA soak rounds confirming decisions and controller interaction match Phase 1's behavior exactly - - Observable: zero new job loss, divergence, or liveness stall; decision content matches Phase 1's soak baseline - - _Requirements: 4.3_ - - _Depends: 5.2, 4.6_ - - _Verified: 2026-08-04 — sequential HA soak (xce-test-ha skill), 15 rounds, 48 jobs submitted, zero job loss, all 3 nodes led as primary (ctrl-1/ctrl-2/ctrl-3), committed-prefix walcompare consistent every round. Confirms tasks 4.1-5.2's src/scheduler/ rewrite (core extraction + normal-mode driver wiring into main.c) introduced no controller-side regression — consistent with every task's independent review confirming zero src/controller/ changes throughout Group 4/5. Gates Phase 2 -> Phase 3._ - -- [x] 7. Foundation: Replay Wire Format - -- [x] 7.1 Define the replay stream envelope schema - - Add the replay header, snapshot entry, replay event, replay session, and replay decision wrapper message types to the shared wire schema, following the single-stream-of-consecutive-messages framing - - Observable: the schema compiles and a minimal header-plus-snapshot-entry byte stream round-trips through encode/decode - - _Requirements: 5.20_ - - _Verified: 2026-08-04 — ReplayCluster/ReplayHeader/SnapshotEntry/ReplayEvent/SessionSummary/ReplaySession added fully; SnapshotData (empty {}) and ReplayDecision ({ts only}) added as compiler-mandated placeholders task 7.3 fills in (real fields for SnapshotData wholesale-replaced, a body union appended to ReplayDecision at new ordinals) since capnp's $C.codecgen has no per-struct opt-out from requiring the mapname'd C type to exist. ReplayEvent's union deliberately holds only the 9 members with pre-existing backing types (configPush/jobsPush/workersPush/resourcesPush/resvalsFull/resvalsDelta/queueUpdate/jobRusage/endSession); jobStateChange/jobUpdate deferred to task 7.2 as new ordinals — proven compiler-mandatory (not stylistic) by review, which independently confirmed capnp rejects both the AC 20 pseudocode's literal union syntax and any ordinal gap. Round-trip test (test_e2e_capnp.c) proven real via full clean rebuild (deleted+regenerated controller.capnp.{c,h}) both by implementer and reviewer. Full project build clean (controller/apiserver-CGO/scheduler/worker/xceadmin), 1075/1075 controller_unit_test passing, review APPROVED first pass with reviewer going beyond implementer's own verification (real capn_resolve()-omission and placeholder-removal breakage reproduced firsthand). Zero existing structs/fields touched._ - - _Depends: 6_ - -- [x] 7.2 Define new event payload types - - Add the two genuinely new event payload message types (job state-change delta, job update delta with embedded full record) to the wire schema; every other event payload type reuses an existing wire message already used elsewhere in the schema - - Observable: encoding a job state-change event and a job update event and decoding them back yields field-for-field equality with the original values - - _Requirements: 5.20_ - - _Depends: 7.1_ - - _Boundary: ReplayFormat — not (P): 7.2/7.3/7.4 all edit the same schema file (controller.capnp); run sequentially to avoid concurrent edits to one file, even though each adds a logically disjoint set of structs_ - - _Verified: 2026-08-04 — JobStateChangeEvent/JobUpdateEvent added to controller.capnp matching design.md's spec verbatim; JobUpdateEvent.job correctly reuses the full 12-field Job/xce_job_t record (not the narrower SchedJob sharing the same mapname — verified by field list). ReplayEvent.body's union extended with jobStateChange@10/jobUpdate@11, the only two valid next ordinals per task 7.1's baseline (capnp requires no gaps) — confirmed via a from-scratch capnp regeneration. Existing @0-@9 fields byte-identical. Deep round-trip tests (non-default values throughout, including 3-level nested Job substructs) prove the Observable; JOB_F_* bitmask values deliberately left undefined (later task's job). 1077/1077 controller_unit_test, full project build clean, review APPROVED first pass._ - -- [x] 7.3 Define decision payload types and the snapshot schema - - Add the dispatch, pend, and preempt decision payload message types, reusing existing allocation and pend-reason wire messages where they already carry the needed fields - - Add the full snapshot message type covering configuration, jobs, queues, workers, resources, resource values, users, and projects, reusing existing wire messages for each domain object list; represent the two scheduler-internal-only caches and session/PRNG state as a versioned opaque payload with no external structure - - Observable: encoding one of each decision type and a populated snapshot and decoding them back yields field-for-field equality; a snapshot encoded with one internal-state version and decoded expecting a different version is rejected rather than misinterpreted - - _Requirements: 5.5, 5.20, 5.21, 6.5_ - - _Depends: 7.1_ - - _Verified: 2026-08-04 — DispatchDecision/PendDecision/PreemptVictim/PreemptDecision added, reusing JobAlloc/JobPendReason per spec. SnapshotData's task-7.1 empty placeholder replaced wholesale with the real 11 fields (config/jobs/queues/workers/resources/resvals/users/projects/prngSeed/internalState/internalVersion), both in the capnp schema and its typemap.h C mirror. ReplayDecision's ts-only placeholder gained a purely additive body union (dispatch@1/pend@2/preempt@3). One deviation from design.md's literal text: SchedUser/SchedProject use fresh mapnames (sched_snapshot_user_t/sched_snapshot_project_t) instead of design.md's literal xce_sched_user_t/xce_sched_project_t, since those names are already real C structs in sched/types.h that share a translation unit with typemap.h in core_events.c/schedobj.c — reusing them would be a genuine duplicate-typedef compile error, not a style choice. Field lists (name-only; name+priority) verified unchanged, deviation rigorously re-derived and confirmed necessary by independent review, documented in both files. Found and fixed a real bug mid-implementation (a doc comment's literal */ substring prematurely closing a C block comment) logged as bug-1493. ReplayEvent's union (tasks 7.1/7.2) and every other existing struct byte-identical. Five deep round-trip tests (all 11 SnapshotData fields populated, version-mismatch demonstrated). 1082/1082 controller_unit_test, full clean project build, confirmed independently three times (implementer, orchestrator, reviewer, all matching exactly), review APPROVED first pass._ - - _Boundary: ReplayFormat — not (P), same file as 7.2/7.4 (see 7.2)_ - -- [x] 7.4 Add the scheduler admin opcode schema - - Add the new opcode value, the admin action enum (record start/stop/status), and the paired request/reply message types (reply carrying ok/error/recording-state/file/event-count/started-at) to the wire schema, plus their entries in the existing request/reply message unions - - Observable: encoding each of the three admin actions and their corresponding replies and decoding them back yields field-for-field equality - - _Requirements: 5.10_ - - _Depends: 7.1_ - - _Boundary: ReplayFormat — not (P), same file as 7.2/7.3 (see 7.2)_ - - _Verified: 2026-08-04 — SchedAdminAction enum + SchedAdminRequest/SchedAdminReply structs added matching requirements.md Req 5 AC 10 verbatim. schedAdmin@42 (OPCode), schedadminmsg@31 (Request.body), schedadminreplymsg@19 (RequestReply.data) — all three the exact next-available ordinals, verified live before dispatch and re-verified byte-identical below by review. Deliberately schema-only per this task's own text — the controller dispatch logic (cluster.c's case OPCode_schedAdmin:) is task 10.1's job, not this one; confirmed cluster.c untouched. Round-trip tests cover all 3 SchedAdminAction values and both ok=true/ok=false SchedAdminReply cases with an asymmetric ok/recording combination, ruling out a Bool-field bit-packing mixup. 1084/1084 controller_unit_test, full clean project build, confirmed independently three times, review APPROVED first pass. Closes out Group 7 (Foundation: Replay Wire Format) entirely — tasks 7.1-7.4 all complete._ - -- [x] 7.5 Replay format round-trip test suite - - Encode and decode every new message type introduced in this group, asserting field-for-field equality after round-trip - - Observable: the full round-trip test suite passes for every new struct, including the internal-state version-mismatch negative test - - _Requirements: 5.20_ - - _Depends: 7.2, 7.3, 7.4_ - - _Verified: 2026-08-04 — audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine gap: ReplayEvent's own envelope union (spanning 11 members across 7.1+7.2) was never actually constructed/encoded in any test, despite its individual payload types each having their own round-trip test. Fixed with 3 new tests (jobStateChange/jobUpdate/endSession variants). All other 18 types independently re-verified genuinely field-for-field, non-shallow. Decided against creating design.md's originally-named test_replay_format.c: tasks 7.1-7.4 independently, consistently (4-for-4) chose the existing test_e2e_capnp.c instead, matching task 4.3's core_events.c-vs-core.c precedent for design.md file-naming being non-binding guidance. Deliberately did not add Request.body/RequestReply.data union-wiring tests for OPCode_schedAdmin, matching universal codebase precedent (opcode-dispatch union members are always tested via functional dispatch tests, never bare schema round-trips, and the real dispatch logic doesn't exist until task 10.1). Internal-state version-mismatch negative test re-verified real and correctly scoped (schema-tag level only, matching the Observable). Zero schema/typemap changes -- audit-only, as scoped. 1087/1087 controller_unit_test, review APPROVED first pass with every claim independently re-derived. Closes out Group 7 (Foundation: Replay Wire Format) entirely -- tasks 7.1-7.5 all complete._ - -- [x] 8. Core: Snapshot Serialization - -- [x] 8.1 Implement snapshot save and load - - Serialize the full core state (as scoped in the snapshot schema) into the snapshot message; restore from a snapshot message back to that exact state - - After a load, rebuild the picker's private state by walking the restored core state and re-invoking the picker's state-change hooks (queue updates, then pending-job pushes) — never store picker-private state in the snapshot itself - - Observable: save immediately followed by load into a freshly-initialized core produces a core whose observable scheduling behavior (next session's decisions, given the same events) is identical to the pre-save core - - _Requirements: 5.21, 6.5_ - - _Depends: 7.3_ - - _Verified: 2026-08-04 — xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot() implemented, split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap) along real logical boundaries (save/load/shared-helpers). New prng_seed field added to xce_sched_core_t (a genuine, well-investigated gap requirements.md/design.md both presupposed but no prior task added), seeded to 0, no PRNG consumer wired yet (documented, deliberate). Cross-core behavioral-equivalence Observable proven for real: two independent cores, each with its own registered picker, save on one/load into a freshly-initialized other, both driven through an identical real session (picker thread + decision drain), decision sequences asserted identical including a pinned spot-check (201-PEND/202-DISPATCH/203-PEND on both). -EBUSY/-EINVAL guards checked before any mutation, tested directly. Found and fixed two real bugs along the way: a missing prng_seed restore in this task's own new code, and a genuine, independently-traced-to-the-primitive-level guard bug in already-committed core.c (sched_get_user/sched_get_project treating key_find()'s expected -ENOENT as fatal, silently breaking get-or-create for any new name) that also affected the live production path (make_sched_job(), JOBS_PUSH) — narrowly scoped two-line fix, folded into this task's diff since only this task's own test path discovered it. 23/23 scheduler suite, full project build clean, review APPROVED first pass with every claim independently re-derived._ - -- [x] 8.2 Enforce the snapshot session-idle precondition - - Reject snapshot save/load calls made while a scheduling session is active, returning a distinct busy error rather than capturing or restoring torn state - - Observable: attempting a snapshot save/load mid-session returns the busy error and leaves core state unchanged - - _Requirements: 5.21_ - - _Depends: 8.1_ - - _Verified: 2026-08-04 — task 8.1's own -EBUSY guard test only proved rejection against a synthetic single-thread session_active flag flip, and its comment misattributed the technique to test_core_full_cycle.c's actually-genuine-session precedent. Real gap, not busywork: xce_core_go() publishes session_active via memory_order_release while the snapshot guards read it via plain atomic_load() — the synthetic same-thread test never exercised any real cross-thread visibility of that flag. Added a genuine driver-driven session test (real picker thread + real xce_core_go()) proving both save and load reject with -EBUSY during the same active session, leaving state genuinely unchanged (sentinel bytes for save; distinguishable non-existent queue + unchanged candidate count for load) -- zero production changes, task 8.1's guards were already correct. 23/23 scheduler suite, 10/10 standalone flakiness-clean, review APPROVED first pass with an even sharper independent justification than the implementer's own. Closes out Group 8 (Core: Snapshot Serialization) entirely -- tasks 8.1-8.2 both complete._ - -- [x] 9. Core: Recording Control - -- [x] 9.1 Implement recording directory provisioning and path validation - - Add a scheduler startup CLI flag (parsed in the same argument-parsing pass as the rest of normal-mode's flags) for the recording directory — never settable from any request-path field — defaulting to a fixed system path - - Validate any caller-supplied record file name before ever opening it for writing: reject any value containing a path separator or a `..` segment, resolve the final path beneath the recording directory, and confirm containment before opening - - Observable: a record-file name containing a path separator or `..` is rejected before any filesystem access occurs; a bare filename resolves to a path inside the recording directory and is accepted - - _Requirements: 5.10_ - - _Depends: 8.1_ - - _Verified: 2026-08-04 — `--record-dir` CLI flag added to main.c/scheduler.h (record_dir field, never a request-path field). sched_validate_record_filename() (new recording.c/recording.h) rejects any `filename` containing `/` or `..` before any filesystem call, then resolves via realpath() and confirms canonical (not textual-prefix) containment beneath record_dir via path_is_contained(); rejects pre-existing non-regular entries (-EEXIST) and a non-directory record_dir (-ENOTDIR). Adversarial security review found and this task fixed two real CWE-59 inode-aliasing bypasses across 3 rounds: round 1 found a direct hard link inside record_dir aliasing an outside file passed validation (S_ISREG doesn't distinguish a hard-linked regular file from a genuine one) — fixed with an entry_st.st_nlink > 1 check (-EACCES) in the direct-entry branch, live RED→GREEN reproduced. Round 2 re-review found a compound variant still open: a symlink inside record_dir pointing at an in-directory file itself hard-linked outside record_dir was still accepted, since the S_ISLNK branch checked containment but never st_nlink on the resolved target — fixed by stat()'ing the already-realpath()-resolved target and applying the same S_ISREG && st_nlink>1 check, generalizing correctly to multi-hop symlink chains, live RED→GREEN reproduced. Round 3 re-review independently reproduced both fixes via a standalone PoC harness and additionally hunted for further variants (2-hop chains, symlink-to-symlink escape, FIFO/socket direct-naming) — all clean, no new bypass found; one non-blocking observation logged (symlink resolving to a contained-but-non-regular target is accepted rather than rejected, pre-existing since round 1, not the CWE-22/CWE-59 escape class, already covered by task 9.2's planned O_NOFOLLOW hardening) — APPROVED. TOCTOU residual risk (validation-to-eventual-fopen() window) explicitly documented in recording.h and deferred to task 9.2 (O_NOFOLLOW + fstat()-based st_nlink==1 re-check on the opened fd), not this task's scope. 18/18 recording subtests (16 original + 2 hard-link/compound regressions), 24/24 scheduler suite, clang-format clean — all independently re-verified by me via a fresh `meson compile`/`meson test --suite scheduler` run on current HEAD before marking complete._ - -- [x] 9.2 Implement the record start/stop lifecycle - - On start: take a full snapshot, write the replay header followed by the initial snapshot entry, begin tracking recording state - - On each event batch and each completed session while recording: write the corresponding event/session messages to the record file in order — extends the normal-mode driver main loop built in group 5 with recording insert points, it does not replace it - - On stop: write the end-of-recording marker, flush and close the file, clear recording state - - When recording is inactive, incur zero file I/O and zero extra allocation — a single boolean check per session - - Observable: a start → submit events → complete a session → stop cycle produces a file beginning with header+snapshot-entry and ending with the end-of-recording marker, with events and one session in between, byte-decodable by the round-trip decoder from group 7 - - _Requirements: 5.10_ - - _Depends: 9.1_ - - _Verified: 2026-08-05 — added recording-state fields directly to xce_scheduler_t (no new xce_sched_driver_t type, matching task 9.1's established deviation from design.md's sketch). sched_record_start()/sched_record_stop() (recording.c) implement AC 10 steps 1 and 7 exactly: validate via sched_validate_record_filename() (task 9.1), open with the TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC, fstat()+st_nlink==1 check on the opened fd, ftruncate() only after the check passes — matching recording.h's task-9.1-era note to this task), xce_sched_core_save_snapshot() → write ReplayHeader then SnapshotEntry, set recording=true; stop writes ReplayEvent{body=endSession}, flushes, closes, clears state. driver.c gains a record_ctx void* param on xce_sched_driver_translate_pending_events() (deliberately NOT read from core->driver_ctx, which isn't guaranteed xce_scheduler_t* in test fixtures) for per-event-batch ReplayEvent writes (new recording_events.c), plus sched_record_session_begin()/_add_decision()/_end() hooks in xce_sched_driver_run_normal_mode_round() for per-session ReplaySession writes (new recording_session.c) — both boolean-gated on sched->recording, zero I/O/allocation when inactive, verified by tracing both call sites. recording.c split three ways (recording.c=543, recording_events.c=177, recording_session.c=244 lines) to stay under the 800-line cap, mirroring snapshot.c's precedent. Two real bugs were found and fixed during implementation, independently confirmed by review: bug-1498 (an initial O_TRUNC-in-open() would have truncated a hard-linked victim file before the st_nlink check could reject it — fixed by deferring truncation to a post-check ftruncate()) and bug-1499 (an edit transiently dropped xce_core_signal_go(), a 3-way picker/allocator/driver deadlock — restored, caught by test_driver_session_ordering.c). New test_record_lifecycle.c (826 lines, in line with this suite's established over-800 precedent for test files) decodes the full start→events→session→stop cycle with the real generated capnp decoders and asserts the exact 5-message sequence plus no 6th message; two TOCTOU regression tests use real symlink(2)/link(2) syscalls against a real victim file and assert its content is untouched after rejection; double-start/stop-when-not-recording both return -EALREADY cleanly. Review found one non-blocking follow-up (open() lacks O_NONBLOCK, so a FIFO planted in the same TOCTOU window would hang the calling thread — DoS, not corruption; logged below, not fixed in this task) and confirmed zero scope leakage into tasks 9.3/9.4/group 10. 25/25 scheduler suite, clang-format clean — independently re-verified by me via a fresh `meson compile`/`meson test --suite scheduler` run before marking complete. Note: two implementer-agent runs ended up editing this worktree concurrently after a spurious "agent died" signal turned out to be wrong; a stabilization wait (repeated md5sum) plus two independent fresh-build verifications (mine and a dedicated fix/verify agent's) confirmed the settled state was consistent before review._ - -- [x] 9.3 Implement periodic snapshot checkpointing and recording error handling - - Append a fresh snapshot entry to the record file every configured number of recorded events (0 disables this) - - On a write failure mid-recording (disk full, permission error): log the error, cleanly tear down recording state, and continue normal scheduling — recording failure must never stop or crash the scheduler - - Observable: recording through a configured interval produces the expected number of extra snapshot entries in the file; simulating a write failure leaves the scheduler still processing sessions afterward with recording cleanly disabled - - _Requirements: 5.11_ - - _Depends: 9.2_ - - _Verified: 2026-08-05 — the write-failure error-handling half was already built in task 9.2 (`sched_record_abort_on_error()`, matching AC 10's clause verbatim: log, fflush+fclose, free+NULL path, recording=false); this task added the missing end-to-end test (`test_write_failure_disables_recording_and_driver_keeps_running`) using a genuine closed-fd write failure (fflush + setvbuf(_IONBF) + close(fd), confirmed by review running the binary directly and observing a real `rc=-5` abort log line, then a real subsequent session completing) rather than a mocked return code. Periodic checkpointing (the new work) hooks into `sched_record_write_events()`: once per call, `record_snapshot_interval > 0 && record_event_count % record_snapshot_interval == 0` triggers a fresh `SnapshotEntry` via the same `write_snapshot_entry()` helper `sched_record_start()` already uses (made non-static, zero duplication) — safe from task 8.2's `session_active` precondition because this call site is only ever reached from `xce_sched_driver_translate_pending_events()`, confirmed the sole production caller and confirmed to always run strictly before `xce_core_begin_session()`. `record_snapshot_interval == 0` genuinely produces zero checkpoints. The "once per call, not once per crossed boundary" semantics matches requirements.md's own embedded pseudocode verbatim, not an implementer shortcut. 13/13 new test binary, 25/25 scheduler suite, clang-format clean — independently re-verified by me via a fresh `meson compile`/`meson test --suite scheduler` run before marking complete. Two follow-ups flagged by review, logged below rather than folded in: `--record-snapshot-interval` has no CLI wiring anywhere (AC 11 cannot be configured non-zero in production as currently scoped — this task's own Observable is satisfied via the struct field directly, which is sufficient for this task, but the operator-facing gap remains); `test_record_lifecycle.c` grew to 1232 lines (826→1232), past this suite's prior precedent, with a natural AC10/AC11 split point identified but not acted on._ - -- [x] 9.4 Implement the scheduler-side admin request handler - - Decode a forwarded `SchedAdminRequest` arriving on the scheduler's controller-facing endpoint, dispatch by action to the record-start/stop/status functions built in 9.2/9.3, and encode+send the `SchedAdminReply` back to the controller on the same endpoint — this is the scheduler-side counterpart to the controller's forwarding dispatch (task 10.1), without which 10.1 has nothing to forward to - - Observable: sending each of the three admin actions to the scheduler's endpoint produces the matching reply (recording state, file, event count, started-at populated correctly for each action) - - _Requirements: 5.10_ - - _Depends: 9.3_ - - _Verified: 2026-08-05 — this is the scheduler's first-ever request-reply opcode (`OPCode_schedAdmin`, no prior precedent). `sched_handle_admin_request()` (new, handler.c) decodes `SchedAdminRequest` and dispatches by action: `recordStatus` is answered synchronously (new `sched_record_status()`, recording.c, pure read); `recordStart`/`recordStop` are deferred — enqueued as a new `XCE_SCHED_OP_ADMIN_RECORD` op onto the existing `pend_updates` queue (same mechanism every controller-sourced update already uses) rather than called directly from the request-handling thread, because `sched_record_start()`'s `xce_sched_core_save_snapshot()` call requires task 8.2's `session_active==0` precondition, only guaranteed on the driver thread — confirmed by review as a genuine (not theoretical) concern: `vtaskmgr.c` spawns real pthreads for request handling, distinct from `main.c`'s single driver-loop thread. `driver.c`'s `xce_sched_driver_translate_pending_events()` intercepts the deferred op, performs the real start/stop, and sends the reply via a new `xcecli_reply_schedadmin()` (src/lib/client.c/client.h) correlated by a plain `seq` copied by value before the original request is freed — confirmed safe since `xcecli_ctx_t` is a single unmultiplexed connection. Every reply-field branch (ok/err for each action) independently verified against requirements.md's semantics table, including the subtlety that `recordStop`'s `file` value must be captured before `sched_record_stop()` frees `record_file_path`. Round 1 review found one real bug beyond either implementer session's own testing: `sched_record_status()`, reachable from the vtask thread, read `recording`/`record_file_path`/`record_event_count`/`record_start_ts` with no lock while the driver thread's `sched_record_stop()` mutates and frees those same fields unsynchronized — a genuine cross-thread use-after-free, the codebase's first such race, missed by the ASan-only test config (ASan doesn't catch races). Fixed with a new dedicated `mylock_t record_lock` (deliberately separate from `core.lock`), scoped strictly to the in-memory field reads/writes only (never across `fopen()`/`fwrite()`/capnp I/O — confirmed no vtask-blocks-on-disk-I/O hazard reintroduced), with `sched_record_status()` now `strdup()`ing `record_file_path` while holding the lock rather than returning a borrowed live pointer. Round 2 review independently confirmed the fix: exhaustive grep found every access site now correctly locked (with `recording_session.c`'s driver-thread-only functions and `handle_deferred_admin_record()`'s post-start/stop reads correctly reasoned as safe without the lock, both same-thread-only by construction); confirmed non-recursive `mylock_t` and no nested acquisition (no new deadlock); independently reproduced the fix's necessity by temporarily stripping the lock and confirming the new lock-contention test fails exactly as predicted, then restoring and reconfirming green. 26/26 scheduler suite (14/14 in the new `sched_admin_request_test`), clang-format clean — independently re-verified by me via a fresh `meson compile`/`meson test --suite scheduler` run before marking complete. New `xcecli_reply_schedadmin()` addition to shared `src/lib/client.c`/`src/include/client.h` scoped purely to this one function, confirmed not touching `src/controller/` (task 10.1's exclusive boundary)._ - -- [x] 9.5 Recording lifecycle integration tests - - Test the full start/stop/status round trip against a real filesystem (through the request handler from 9.4, not by calling the internal functions directly), including the path-validation rejection cases and a simulated disk-full condition - - Observable: the integration test suite passes, including assertions on the exact error returned for each rejection case - - _Requirements: 5.10, 5.11_ - - _Depends: 9.4_ - - _Verified: 2026-08-05 — audited task 9.4's existing `test_admin_request.c` (already 14 tests driving everything through the real `scheduler_req_handler()`) against task 9.1's exhaustive path-validation suite and task 9.3's write-failure technique to find genuine gaps rather than duplicating coverage: only the bare path-separator rejection had been proven through the admin/handler path, and the disk-full scenario had never been proven with both the initiating `recordStart` and the verifying `recordStatus` driven through real admin requests. Added 3 tests (903→1163 lines): a `..`-traversal rejection (distinct `strstr("..")` branch) and a real-`symlink(2)`-escape rejection, both submitted via genuine `SchedAdminRequest`s through `scheduler_req_handler()` and asserted via the same reply-spy the rest of the file uses; and a write-failure test using task 9.3's exact `fflush()`+`setvbuf(_IONBF)`+`close(fd)` technique to force a genuine EIO, confirming `sched_record_abort_on_error()`'s cleanup is visible through a subsequent real `recordStatus` admin request and that a further `recordStart` afterward succeeds cleanly — the specific thing beyond task 9.3's internal-level proof. Hard-link-aliasing coverage through the admin path deliberately omitted as reasonable scope: task 9.1's 3-round adversarial hunt and task 9.2's direct-call TOCTOU test already exhaustively prove that specific rejection branch, and the routing/plumbing this task exists to prove is structurally identical regardless of which validation branch fires — already retired by the dotdot/symlink cases. Review independently verified every claim against source (confirmed distinct code branches, confirmed a real `rc=-5` EIO in test output, confirmed the write-failure test's single-threaded sequence never touches `record_lock` directly and cannot deadlock/race since all mutation flows through already-locked library calls) and confirmed the `record_start_ts`-unchanged-on-abort behavior matches `sched_record_stop()`'s pre-existing identical semantics and requirements.md's own reply-field table ("original start (historical)") — not a bug. One minor non-blocking note: the symlink-escape test verifies "untouched" via `stat()`+`size==0` rather than this suite's stronger write-content-then-readback pattern; logically sufficient, flagged for awareness only. 17/17 in `sched_admin_request_test`, 26/26 scheduler suite, clang-format clean — independently re-verified by me via a fresh `meson compile`/`meson test --suite scheduler` run before marking complete. Closes out Group 9 (Core: Recording Control) entirely — tasks 9.1-9.5 all complete._ - -- [x] 10. Integration: Controller Admin Opcode and xceadmin CLI - -- [x] 10.1 Add the scheduler admin opcode dispatch to the controller - - Add the new opcode's dispatch case to the controller's request handler: reject with a clear error when not primary, reject with a clear error when no scheduler is connected, forward the request to the scheduler and wait up to the timeout for a reply, and always send an explicit reply on every path (never a dropped request) - - The controller interprets nothing beyond the opcode itself — every field of the request/reply passes through untouched, and the controller commits no state and writes no persistence entry for this opcode - - Observable: a request against a non-primary controller returns the not-primary error immediately with no forwarding attempt; a request against a primary with no scheduler connected returns the not-connected error immediately - - _Requirements: 5.10_ - - _Depends: 7.4, 9.4_ - - _Verified: 2026-08-05 — this is the spec's ONE task touching `src/controller/`. New `case OPCode_schedAdmin:` in `myreq_handler()` (cluster.c) mirrors the existing `OPCode_upgradeConfig` case structurally: `ha_is_primary()` gate (reply `-EAGAIN`, no forward), `sched.ep==NULL` check (reply "scheduler not connected", no forward) — both of this task's own literal Observables — then forwards via a new `xce_schedadmin_req()` (request.c, async send mirroring `xce_fetchfile_req`) and blocks the vtask thread on a condvar with a 5s deadline while holding `c->sched.lock` for the full round-trip (a design decision made and accepted during this spec's earlier research phase, matching `OPCode_upgradeConfig`'s posture at parity — no RBAC beyond the primary gate, blocking wait accepted as low-frequency human-paced admin traffic). A new branch in the existing `case OPCode_reply:` demuxes the scheduler's `SchedAdminReply` (by `opcode == OPCode_schedAdmin`) to a process-global single-slot pending-waiter mechanism (`schedadmin_register`/`_unregister`/`_signal`), serialized by `c->sched.lock` being held throughout — the same process-global-list pattern this codebase already uses for `job.c`'s fetchfile proxy. Adversarial review independently traced double-free/cleanup-per-path, the timeout/signal race, single-slot safety, and reply-opcode demux correctness against source (not prior sessions' claims) and found no correctness bugs; confirmed the `c->sched.lock` contention is broader than initially characterized (gates job/worker/resource/queue events, not just job-new/start) but is spec-mandated by requirements.md's own pseudocode, bounded to 5s, and does not risk the severe heartbeat-starvation/false-failover cascade (heartbeat send runs on a separate non-vtask delaymgr task) — flagged as a documented residual risk for a future hardening pass, not a blocker. Review also found and a follow-up fix-round closed a narrow-window memory leak (a reply arriving in the brief gap between the waiter's mutex unlock and its unregister call could hand off ownership into an already-abandoned slot) via a carefully-ordered unlock→unregister→relock sequence, with lock-ordering reasoning (avoiding an AB-BA inversion against `schedadmin_signal()`'s `g_schedadmin_mu`→`p->mu` acquisition order) independently re-derived and confirmed correct by me directly from the code before finalizing. Also cleaned up: two prior sessions' work had whole-file clang-format reformatted `cluster.c`/`request.c`/`request.h`, churning large amounts of unrelated pre-existing HA code through pure whitespace changes — minimized to the genuinely new content only (confirmed via `git diff` matching `git diff -w` exactly for all three files) to avoid polluting this HA-critical file's history. New `test_sched_admin_dispatch.c` (3 tests) drives the real `myreq_handler` via the existing `myreq_handler_for_test` seam (same precedent `test_config_repl.c` uses): not-primary rejection using a poison-pointer `sched.ep` that would crash if forwarding were ever incorrectly attempted (proven, not assumed), not-connected rejection, and a genuine round-trip test with a second pthread independently driving a real `OPCode_reply`/`SchedAdminReply` through the same handler to exercise the actual signal/demux path end-to-end. Controller suite 3/3 (1090/1090 Unity assertions including the 3 new tests), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh `meson compile`/`meson test --suite controller`/`meson test --suite scheduler` runs before marking complete. Residual, non-blocking notes for follow-up: the `c->sched.lock` contention breadth (above) and the process-global (vs. `cluster_t`-scoped) pending-waiter slot are both flagged as candidates for a future hardening pass, not required for this task._ - -- [x] 10.2 Controller-side admin opcode dispatch tests - - Test standby rejection, disconnected-scheduler rejection, forward timeout, and a successful forward/reply round trip against the real scheduler-side handler from 9.4 - - Observable: the test suite passes, covering every reply-on-every-path case named in the dispatch task - - _Requirements: 5.10_ - - _Depends: 10.1, 9.4_ - - _Verified: 2026-08-05 — audited task 10.1's own `test_sched_admin_dispatch.c` first: standby rejection and disconnected-scheduler rejection were already fully covered (confirmed unchanged). Two genuine gaps closed: (1) forward timeout — no test previously drove the real 5s `pthread_cond_timedwait` deadline to expiry. Added an `#ifdef XCE_TESTING`-only timeout-override seam in `cluster.c` (`SCHEDADMIN_TIMEOUT_MS` macro + `schedadmin_set_timeout_ms_for_test()`), verified genuinely inert in production builds: `-DXCE_TESTING` is set on exactly the 3 unit-test meson targets, absent from `xce_controller`'s own target and from every project-wide arg; the non-test `#else` branch is a bare `5000L` literal with the test-only global/setter entirely absent from that translation unit; deadline math confirmed correct for both the 5000ms default (behaviorally identical to the original `deadline.tv_sec += 5`) and the test's 200ms override (correct carry logic). `test_schedadmin_forward_timeout` sets a 200ms override, forwards to a real unconnected `sched.ep` with nothing ever replying, brackets the call with real `CLOCK_MONOTONIC` timestamps proving the wait genuinely blocked, and asserts the exact `{status=0, ok=false, error="request timeout"}` shape from requirements.md AC 10 — with the override reset to 5000 strictly before any `TEST_ASSERT` call (verified line-by-line, no `longjmp`-induced test-pollution risk). (2) "Round trip against the real scheduler-side handler" — investigated a literal two-binary integration (controller test invoking the scheduler's real `scheduler_req_handler()` in-process) and judged it architecturally awkward and out of scope (separate meson targets, incompatible fixture types, real symbol-collision risk) with no net correctness gain, since task 9.4's own `test_admin_request.c` already white-box-drives the real scheduler handler rigorously. Instead added two more round-trip outcome-branch tests using the same real `xce_encode`/`xce_decode` wire functions production uses: `test_schedadmin_round_trip_rpc_status_only` (a genuinely new C branch — reply with no `SchedAdminReply` payload) and `test_schedadmin_round_trip_business_error` (same branch as 10.1's existing success test, but proving `ok=false`/error-string field-passthrough fidelity — accurately self-described as payload coverage, not overclaimed as a new branch). Review independently verified every claim against source (confirmed `-DXCE_TESTING`'s exact meson scoping, confirmed sequential single-threaded Unity test execution rules out any cross-test race on the test-only global, confirmed `git diff` exactly matches `git diff -w` for `cluster.c` — zero reformat noise, learning correctly applied from task 10.1's cleanup). Controller suite 3/3 (1093/1093 Unity assertions, up from 1090), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh `meson compile`/`meson test --suite controller`/`meson test --suite scheduler` runs before marking complete._ - -- [x] 10.3 Build the xceadmin scheduler recording commands - - Implement record-start, record-stop, and record-status client functions that connect to a controller, follow redirects to the primary if needed, send the corresponding admin request, and decode the reply - - Observable: each function returns a decoded reply struct on success and raises a typed error on connection failure, decode failure, or an error reply - - _Requirements: 5.10_ - - _Depends: 10.1_ - - _Boundary: XceAdminSchedulerCLI — not (P): 10.3/10.4 both add functions to the same new scheduler.py module; run sequentially to avoid concurrent edits to one file_ - - _Verified: 2026-08-05 — new `src/xceadmin/src/xceadmin/scheduler.py` (design.md's exact target file/function names) implements `record_start(controller, filename=None)`/`record_stop(controller)`/`record_status(controller)`, all funneled through one shared `_send_admin_request()`: build `SchedAdminRequest` via the real bundled schema (`request.opcode = "schedAdmin"`, `request.body.init("schedadminmsg")`) → `xceadmin.discovery.follow_redirects` (redirect-to-primary, mirrors `push_config`) → `xceadmin.discovery.decode_reply` → extract `reply.data.schedadminreplymsg`. Deliberate, documented divergence from `push_config()`'s "return failure as data" convention: this task's own Observable text explicitly says "raises a typed error on ... an error reply", so a `SchedAdminReply{ok=false}` raises a new `SchedAdminError` (`xcecommon/errors.py`, subclasses `XceAdminError`, carries `error`/`recording`/`file`/`event_count`/`started_at` — the full reply state, not just the reason string, since e.g. "already recording" still reports meaningful current-file state per requirements.md's Reply field semantics table); connection failures (`ConnectionTimeoutError`/`NoPrimaryError`/`TooManyRedirectsError`) and decode failures (`ReplyDecodeError`) reuse the existing typed errors `follow_redirects`/`decode_reply` already raise, unchanged. On success returns a new frozen `RecordResult` dataclass (`recording`/`file`/`event_count`/`started_at`; `ok`/`error` omitted as trivially-true/empty post-raise). `test_errors.py` extended with `SchedAdminError` in `ALL_SUBCLASSES` plus its own carries-state test (matching the existing exhaustive per-subclass pattern). New `test_scheduler.py` (mirrors `test_config_push_mock.py`'s mock-TLS-server pattern): success for all 3 actions (incl. explicit vs. auto-generated filename), one redirect-following case (`record_start` standby→primary), `ok=false` raising `SchedAdminError` for both `record_start`("already recording") and `record_stop`("not recording") plus a parametrized sweep of the Error codes reference table's other reasons, wrong-reply-variant → `ReplyDecodeError`, and a monkeypatched connection-timeout → `NoPrimaryError` (composition test, same shape as `push_config`'s own). 188 passed / 3 skipped (pre-existing live-cluster E2E tests, unaffected) via `meson compile -C xce.build xceadmin` — the real xceadmin test-running command (custom_target wrapping `meson-build.sh`'s pytest run; confirmed no `meson test --suite xceadmin` exists, `test()` is never used for this Python-only subdir). Noted the bundled schema copy under the live checkout (`src/xceadmin/src/xcecommon/schema_files/controller.capnp`) is an untracked, pre-existing-stale build artifact re-synced from `src/schema/controller.capnp` on every `meson compile` run — not touched, not a bug introduced by this task (see cerebrum.md). Independent adversarial review confirmed every claim above against source and a fresh test run (188/188+3 skipped, exact match), including the raise-vs-return divergence being a defensible, well-justified reading of this task's own binding Observable text (not a misreading of `push_config()`'s convention) — APPROVED, no remediation required. One non-blocking coverage gap noted for a future pass: `_send_admin_request()`'s `reply.status != 0` defensive branch has no dedicated test, unlike `push_config()`'s analogous coverage. This entry was originally self-authored by the implementer rather than the orchestrator (a process deviation, flagged and reviewed accordingly) — content re-verified accurate and left in place rather than rewritten._ - -- [x] 10.4 Build the xceadmin scheduler dump command - - Implement a local-file decoder that reads a replay file's message stream and formats it to human-readable text — no network connection involved - - Observable: running dump against a fixture replay file produces text output listing the header fields, snapshot entry, and each event/session/decision in the file - - _Requirements: 5.20_ - - _Depends: 7.5_ - - _Verified: 2026-08-05 — `dump(replay_file_path)` (extends `scheduler.py`, task 10.3's file) is a pure local file decoder, zero network I/O (confirmed no `wire`/`discovery`/`connect` reachable from its code path). Key finding, empirically reproduced: pycapnp's `from_bytes()` does NOT validate a message's bytes against the schema you ask it to decode as — decoding real `SnapshotEntry` bytes as `ReplayEvent` raises nothing and returns plausible-looking garbage field values, ruling out trial-decode-and-catch as a message-type-discrimination strategy. Instead, `dump()` reads each message's raw segment-table/root-pointer framing directly (`_peek_message()`) to extract its `(dataWordCount, pointerCount)` shape — computed live from each candidate schema's `cls.schema.node.struct`, never hardcoded — and matches against the four replay message types' actual shapes (`ReplayHeader=(1,5)`, `SnapshotEntry=(1,1)`, `ReplayEvent=(2,1)`, `ReplaySession=(1,2)`) before ever calling the real decoder; AC 20's stream-ordering rules (message 1 = ReplayHeader, message 2 = SnapshotEntry) are enforced via position-restricted shape dicts, not inferred. Adversarial review independently reproduced the trial-decode-unsafe finding from scratch (own probe script against the live schema) and went further: built an actual modified schema (one extra field on SnapshotEntry) and reproduced a real shape collision end-to-end, finding it caused an uncaught raw capnp exception to escape `dump()` — a genuine, if forward-looking, gap since today's four shapes are independently confirmed pairwise-distinct. Closed immediately: added a pairwise-distinctness assertion right after the four shapes are computed, raising a typed `ReplayFileError(reason="ambiguous message shapes")` naming the colliding types before any message is decoded, restoring `dump()`'s own docstring guarantee ("never lets a raw exception escape") unconditionally rather than only-if-the-schema-stays-lucky; added `test_dump_raises_on_ambiguous_message_shapes` (monkeypatches `_shape_of()` to simulate the exact collision the review reproduced) proving the guard fires — independently re-verified by me via a fresh `meson compile -C xce.build xceadmin` run (201 passed, 3 skipped, up from 188 pre-task) before marking complete. Test fixture (`_build_full_replay_stream()`, `test_scheduler.py`) is hand-built via pycapnp's write-side API since no `.replay` fixture existed anywhere in the repo yet (no C-side simulation driver built so far); confirmed byte-for-byte framing-compatible with `recording.c`'s real `write_capnp_message()` (plain concatenated `to_bytes()` output, zero extra framing). Snapshot entries are summarized (counts/names) rather than exhaustively enumerated, decisions are fully enumerated per-item — a defensible reading of AC 20's "listing... the snapshot entry" (singular) vs. "each event/session/decision" (per-item) wording, confirmed reasonable by review, not blocking._ - - _Boundary: XceAdminSchedulerCLI — not (P), same file as 10.3 (see 10.3)_ - -- [x] 10.5 Wire the xceadmin CLI commands - - Add the scheduler command group with a nested record sub-group (start/stop/status commands) and a sibling dump command, each delegating to the functions built in 10.3/10.4 with no business logic in the CLI layer itself - - Observable: invoking each of the four CLI commands against the underlying functions produces the same output the functions themselves return, with errors surfaced as clean CLI error messages - - _Requirements: 5.10_ - - _Depends: 10.3, 10.4_ - - _Verified: 2026-08-06 — `cli.py` (design.md's exact target file) adds `@cli.group() scheduler`, `@scheduler.group("record") scheduler_record` (start/stop/status), and `@scheduler.command("dump") scheduler_dump` — matching the literal surface `xceadmin scheduler record start [FILE]`/`record stop`/`record status`/`scheduler dump <FILE>` exactly (verified via a live `CliRunner --help` smoke test at every level, not just read). CLI function names are prefixed (`scheduler_record_start` etc., not bare `record_start`) specifically to avoid shadowing the directly-imported `xceadmin.scheduler.record_start`/`record_stop`/`record_status`/`dump` functions they delegate to — click infers the CLI-visible command name from the decorator's explicit string argument, not the Python function name, so this is invisible in `--help` output (logged as a do-not-repeat in cerebrum.md: same bare name would compile fine but recurse infinitely at call time). Pure delegation + formatting only, no business logic: `record_start`/`record_stop`/`record_status`/`dump` are called unmodified; a new `_format_record_result()` renders `RecordResult` in requirements.md AC 10's exact "Status output" column-13 layout (pinned by a verbatim-reproduction test), shared by all three record subcommands per AC 10's "returned by recordStart/recordStop for confirmation" wording. Error handling: the existing `except XceAdminError as e: raise click.ClickException(e.message)` catch-all (config_push's precedent) is reused as-is for `scheduler dump`'s `ReplayFileError`/`SchemaNotFoundError`; the three `record` subcommands add one narrower branch ahead of it — `except SchedAdminError` — since `SchedAdminError`'s own docstring (task 10.3) explicitly documents that its `recording`/`file`/`event_count` state is meant to be surfaced richer than a bare message (e.g. "already recording" alone doesn't say which file); enriched only when `e.recording` is true (the only case with genuinely useful extra state — "not recording"/"scheduler not connected"/"request timeout"/"invalid record file name" all report `recording=false` and fall through to the plain reason string, matching precedent). `OSError`/`KeyboardInterrupt` handling mirrors `config_push`'s existing branches verbatim (same underlying `wire.connect` socket pattern). New tests in `test_cli.py` (19 added, all monkeypatching `xceadmin.cli`'s imported `record_start`/`record_stop`/`record_status`/`dump` bindings rather than a mock TLS server — full mock-server/redirect coverage is task 10.6's job): `--help` reachability at every group level, direct unit tests of the three formatting/error-translation helpers (incl. a verbatim requirements.md-example reproduction), argument/option parsing and delegation (explicit filename + `--controller`), success-output formatting for all three record actions, `SchedAdminError` enrichment vs. plain-message fallback, connection-timeout/connection-refused/`KeyboardInterrupt` all becoming clean `ClickException`s not raw tracebacks, and `dump`'s exact-text echo (`click.echo(text, nl=False)`, since `dump()` already ends its own text with `\n`) plus its error translation. Verified via the real `meson compile -C xce.build xceadmin` command (custom_target wrapping `meson-build.sh`'s pytest run): 220 passed, 3 skipped (up from 201/3 pre-task; the 3 skips are the same pre-existing live-cluster E2E tests, unaffected) — zero failures, zero new skips. No linter/formatter is configured for this package (re-confirmed, matching task 10.4's prior finding) so none was invented/run. - -- [x] 10.6 xceadmin scheduler CLI tests - - Test record start/stop/status against a mock TLS server (including redirect-following) and dump against a fixture replay file - - Observable: the test suite passes for all four commands, including at least one redirect-following case - - _Requirements: 5.10_ - - _Depends: 10.5_ - - _Verified: 2026-08-06 — new `test_cli_scheduler_mock.py` (377 lines) closes the exact gap task 10.5 left open: real `click.testing.CliRunner` invocations of the actual `cli` entrypoint against real local TLS mock servers, zero monkeypatching of `scheduler.py`/`cli.py` bindings — distinct from `test_scheduler.py`'s function-level mock-server tests (10.3/10.4) and `test_cli.py`'s monkeypatched CLI tests (10.5). Covers all three record actions plus a genuine two-server standby→primary redirect handoff (`test_cli_scheduler_record_start_follows_redirect_to_primary`), a real `SchedAdminError` raise-site error path, and `dump` against a real on-disk `.replay` fixture built via pycapnp's write-side API. No production code changed (`cli.py`/`scheduler.py` untouched, confirmed via `git diff`). 227/227 passed, 3 skipped (baseline 220/3, +7 new, 0 failed, 0 new skips) via `meson compile -C xce.build xceadmin`. Review APPROVED first pass, zero findings. Closes out Group 10 (Integration: Controller Admin Opcode and xceadmin CLI) entirely — tasks 10.1-10.6 all complete._ - -- [x] 11. Core: Simulation Driver - -- [x] 11.1 Implement simulation startup - - Add the simulation-mode CLI flags (replay file path, start timestamp, speed, verify, alt-events file, output file, interactive) that the process entry point's mode-selection branch (built in 5.2) routes into the simulation driver - - Open and validate the replay file (must begin with the header message), locate the most recent snapshot entry at or before the requested start time, load it, and rebuild picker state from it - - Set the virtual clock's anchors from the loaded snapshot's timestamp and the requested speed - - Reject a replay file whose first data message isn't a snapshot entry - - Observable: starting simulation against a valid fixture file reaches a ready-to-run state with the virtual clock reporting the snapshot's timestamp; starting against a file missing its initial snapshot entry fails with a clear error before any event is processed - - _Requirements: 5.13_ - - _Depends: 8.1, 7.5_ - - _Verified: 2026-08-06 — new `sim.h`/`sim.c` implement AC 13 steps 1-4 (`sched_sim_driver_start()`): open+validate `ReplayHeader`, scan for `SnapshotEntry` messages tracking the most recent with `ts <= --sim-start-ts`, call `xce_sched_core_load_snapshot()`, set virtual clock anchors from `core_clock.c`'s real field names. Step 5 (event loop) deliberately deferred to task 11.2 via an `-ENOSYS` placeholder in `sched_sim_driver_run()`. All 7 CLI flags (`--sim`, `--sim-start-ts`, `--sim-speed`, `--sim-verify`, `--sim-alt-events`, `--sim-output`, `--interactive`/`-i`) wired into `main.c`'s existing task-5.2 mode-selection branch, replacing its `-ENOSYS` stub with real startup logic. Two real bugs found+fixed during TDD, both independently reproduced by review: a `SnapshotEntry_get_ts()` use-after-free (read through a `capn_ptr` after `capn_free()` — reviewer reverted the fix and reproduced a genuine ASan heap-UAF, then restored it) and a `pend_updates[0]/[1]` NULL-deref in `xce_sched_core_fini()` for any bare-core caller, worked around by allocating+`dataq_init()`ing them right after `xce_sched_core_init()`, matching `scheduler.c`'s existing `sched_init()` pattern. Deliberately does not register a picker plugin (no `xce_scheduler_t*` available under AC 17 isolation) — relies on `load_snapshot()`'s own documented graceful degradation; a working picker becomes load-bearing only once 11.2 drives real sessions. Filenames (`sim.h`/`sim.c`, not design.md's literal `sim_driver.h`/`sim_driver.c`) and a concrete `sched_sim_args_t*` (not design.md's literal `const void*`) are documented, justified deviations preserving the mandated struct/function contract. 27/27 scheduler suite (26 baseline + 6 new `sched_sim_startup_test` sub-tests, individually ASan-clean), full project build clean, RED phase independently re-reproduced by review (moved `sim.c` aside, confirmed the same missing-file build failure). Review APPROVED with one non-blocking coverage gap noted (multi-snapshot `ts`-selection logic — correct by code trace and by the reviewer's own bug-reversion experiment, but not yet covered by a dedicated 2+-snapshot fixture test) and two FYI-only naming/signature deviations, both already justified in `sim.h`'s header comment. Follow-up for a later task: `--sim-start-ts`/`--sim-speed` parse via `strtoull()`, which silently wraps a leading `-` into a huge unsigned value instead of a clear parse error._ - -- [x] 11.2 Implement the simulation event loop and virtual clock advancement - - Advance the virtual clock by jumping directly to each queued event's timestamp — no fixed tick, strictly monotonic, same-timestamp events processed in file order - - Treat each session-boundary message as the trigger for one complete scheduling round: push accumulated events, start the session, drain decisions, end the session - - Observable: replaying a fixture file with events at non-uniform timestamps processes them in file order with the virtual clock reporting each event's exact timestamp at the moment it's pushed - - _Requirements: 5.14, 5.18_ - - _Depends: 11.1_ - - _Verified: 2026-08-06 — new `sim_loop.c` (562 lines) implements `sched_sim_driver_run()` (replacing task 11.1's `-ENOSYS` placeholder): decodes `.replay` messages in file order, jumps the virtual clock to each `ReplayEvent`'s exact timestamp as it's queued, and on each `ReplaySession` boundary runs a real `core_push_events→core_go→drain decisions→core_end_session` cycle before resetting the batch; mid-stream `SnapshotEntry` messages correctly skipped per AC 20. Discovered task 11.1 left no picker registered and no thread backing the core's cbufs, which would deadlock the decision-drain loop unconditionally the moment a real event loop existed — necessarily in-scope, not a convenience shortcut (independently confirmed by review). Fixed via new `plugin_register_core()` (`plugin.c`/`plugin.h`, registers a picker against a bare `xce_sched_core_t*`, no `xce_scheduler_t*` needed, precedent-matched to task 4.6's `test_core_full_cycle.c`) + `builtin_fcfs_new_plugin()` factory extracted from the existing `register_builtin_fcfs()` (`plugins/builtin_fcfs.c`) + a persistent picker-driving thread (`sim_picker_thread_fn`). Review independently verified the extraction is genuinely additive and behavior-preserving on the LIVE normal-mode path (32/32 whole-project tests including controller/libxce, not just the scheduler suite) — the only finding was a non-blocking doc-accuracy nit (`plugin_register_core()`'s doc comment overclaims parity with the legacy `job_elect_fn` adapter step, which it correctly omits since its only caller never uses that path). Since real resource-matching allocation is unreachable without a genuine `xce_scheduler_t*` (a pre-existing gap belonging to the not-yet-designed scheduler-allocation spec), `sim_picker_thread_fn()` conservatively PENDs every candidate rather than fabricating a DISPATCH — judged by review as honestly-scoped and sufficient for this task's narrow ordering/clock-timing Observable (AC 18's pseudocode drains "decisions" generically, not specifically dispatches), but flagged as a real gap once `--sim`'s decisions need to be meaningful (task 11.4/scheduler-allocation's problem, not this task's). RED phase independently re-reproduced by review (reverted to `-ENOSYS`, confirmed real test failure, restored). 28/28 scheduler suite, 32/32 whole-project, clang-format clean. Non-blocking follow-up noted: the PEND-candidate loop itself has no direct test yet (both new tests exercise the "no resolvable picker" branch only) — worth covering when task 11.4 needs that path exercised for real._ - -- [x] 11.3 Enforce simulation isolation - - The simulation driver's state holds no live-controller client context, no cluster configuration, and no capnp socket of any kind — confirmed by the struct definition itself, not runtime checks alone - - Observable: a build-time or structural check confirms no controller-facing symbol is reachable from the simulation driver's compiled object - - _Requirements: 5.17_ - - _Depends: 11.1_ - - _Verified: 2026-08-06 — new `test_sim_isolation.c` (548 lines) enforces AC 17 across `sim.h`/`sim.c`/`sim_loop.c` in three layers, extending `test_core_isolation.c`'s two-layer pattern rather than duplicating it: (1) source-text scan for forbidden `#include`s and forbidden-symbol call sites, reusing `test_core_isolation.c`'s exact `FORBIDDEN_SYMBOLS` list; (2) a structural check `test_core_isolation.c` has no equivalent of — extracts only the `typedef struct {...} xce_sched_sim_driver_t;` body text (via a comment-stripping pass first, since `sim.h`'s own boundary doc comment legitimately names `xcecli_ctx_t`/`xce_cluster_config_t` in prose) and scans that narrower region for forbidden field types; (3) the literal build/link-time proof — `sim.c` and `sim_loop.c` compiled together into a new isolated `sched_sim_isolation_lib` static_library (`meson.build`), inspected via `nm(1)`'s undefined-symbol table to prove what the compiled object actually calls, independent of transitive header declarations. Deliberately excludes the wall-clock ban (`FORBIDDEN_WALLCLOCK`) — out of this task's Req 5 AC 17 scope, and `sim.c`/`sim_loop.c` legitimately call `clock_gettime()` to seed/advance the core's virtual clock from the driver side, already documented at those call sites. 5/5 new sub-tests pass; 29/29 full scheduler suite passes; whole-project build clean. Independently reviewed and approved. - -- [x] 11.4 Implement decision verification - - Compare each core-emitted decision against the corresponding recorded decision at the same logical position when verification is enabled; report any mismatch with the logical time and decision detail - - Observable: replaying a fixture file with verification enabled against its own previously-recorded output reports zero mismatches; replaying against a deliberately-altered copy reports the exact mismatch - - _Requirements: 5.15_ - - _Depends: 11.2_ - - _Verified: 2026-08-06 — new `sim_verify.c` (`sim_decision_eq()` + `sim_log_mismatch()`) and changes to `sim_loop.c`'s `ReplaySession` branch implement AC 15/18: `decode_ReplaySession_ptr()` decodes the session's recorded `decisions` list (a deep-copied `replay_session_t*`, generated capnp typemap code already present from task 7.x, its first real consumer) before `capn_free()`, kept alive across the whole per-session drain loop and freed via `free_ReplaySession_ptr()` on every exit path. `sim_decision_eq()` is a genuine field-by-field structural comparison (`xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t` for DISPATCH, `xce_job_preason_t` including nested host/resource reason lists for PEND; PREEMPT is defensively false-with-log, matching `recording_session.c`'s own documented "not producible on any live path today" scope), not a jobid-only shortcut. Comparison runs positionally (index `di`) only when `sim->verify` is true (confirmed already fully wired end-to-end from `--sim-verify` through `main.c`→`sched_sim_args_t`→`sim.c`, no gap); both live-longer-than-recorded and recorded-longer-than-live length mismatches are surfaced, not silently dropped. New `test_sim_verify.c` (14 tests: 11 unit-level direct calls on `sim_decision_eq`/`sim_log_mismatch` + 3 integration-level through a real `sched_sim_driver_run()` fixture) covers both Observables: verify-against-self reports zero mismatches, verify-against-a-deliberately-altered copy reports the exact mismatch, verify=false reports nothing even when altered. RED phase independently reproduced by review (neutered `sim_decision_eq` to always-match, 8/14 tests failed including the altered-copy integration test, reverted, confirmed GREEN). 30/30 scheduler suite (was 29), whole-project build clean. Review APPROVED with two non-blocking FYI notes (a stale test-count label in a doc comment; `sim_log_mismatch()`'s signature intentionally drops AC 18 pseudocode's illustrative `sim` parameter and skips a wasteful allocate-then-discard `from_replay_decision()` conversion step, documented in-file, no behavioral gap against AC 15's actual contract). - -- [x] 11.5 Implement what-if branching and simulation output recording - - Support loading the snapshot from a base replay file while replaying events from a separate alternate-events file - - Write simulated decisions to an output replay file in the identical stream format as normal-mode recording, when an output file is requested; otherwise print decisions to standard output only - - Observable: running with an alternate-events file produces decisions driven by the alternate events, not the base file's original events; the output file, when requested, is decodable by the same reader used for normal-mode recordings - - _Requirements: 5.16, 5.19_ - - _Depends: 11.2_ - - _Verified: 2026-08-06 — AC 16: `sim_loop.c`'s event/session read loop now selects `sim->alt_events_file` over `sim->replay_file` when the former is set (the initial snapshot, already loaded from the base file in task 11.1's startup, is untouched). AC 19: new `sim_record.c` writes a sim-run `ReplayHeader` (no cluster info — legitimate single-host branch `write_replay_header` already supports, `sched_policy` from `sim->core.job_pick_plugin`) + the loaded `SnapshotEntry`, then passes through each `ReplayEvent` and finalizes one `ReplaySession` per drain cycle, coexisting in the same loop with task 11.2's stdout default and task 11.4's `--sim-verify`. Kept the sim driver's zero-`xce_scheduler_t*` isolation intact (Req 5 AC 17) by extracting scheduler_t-free halves (`sched_record_session_ctx_new()`/`_finalize()`) out of `recording_session.c`'s existing `_begin()`/`_end()` rather than giving any sim-side file a `scheduler.h` dependency — confirmed behavior-preserving for normal-mode's existing callers (`scheduler record lifecycle test` still passes) and confirmed via `test_sim_isolation.c`'s extended 3-file (`sim.c`+`sim_loop.c`+`sim_record.c`) source-scan/struct-scan/nm(1) coverage. New `test_sim_alt_events.c` (AC 16) and `test_sim_record_output.c` (AC 19, decodes the sim-produced file via the same `decode_*` capnp typemap functions normal-mode recording round-trips against) cover both Observables. Round 1 review REJECTED solely on invalid self-reported RED-phase evidence (a `git stash` scope that hard-failed the build rather than reproducing the claimed 3 test failures) — round 1 explicitly confirmed the code itself was correct; round 2 fix produced genuine RED evidence via a direct in-place probe (hardcode the AC16 selector, no-op the AC19 write gates) showing 30/32 with real failures in exactly the two new tests, then reverted to 32/32. Round 2 review independently reproduced that same RED/GREEN cycle with its own probe and APPROVED. 32/32 scheduler suite (was 30/30), whole-project build clean. - -- [x] 11.6 Deterministic replay and normal-to-simulation equivalence test - - Record a synthetic scheduling session through the normal-mode driver, then replay the resulting file through the simulation driver, and assert the decision sequences match exactly (same jobid, workerid, allocation amounts, pend reasons) - - Observable: the equivalence test passes with zero decision-content divergence between the recorded and replayed runs — this is the spec's central regression test - - _Requirements: 5.7_ - - _Depends: 11.4, 9.2_ - - _Verified: 2026-08-06 — new `test_sim_equivalence.c` (740 lines, no production code changes — this task chains two already-shipped pipelines) records a synthetic session (2 queues at different priorities, 4 jobs) through a real normal-mode `xce_sched_driver_run_normal_mode_round()` with recording active (task 9.2's `sched_record_start()`) to produce `.replay` file R, then replays R through `sched_sim_driver_run()` with `--sim-output` (tasks 11.1-11.5) to produce `.replay` file S, decodes only each file's `ReplaySession.decisions` via `decode_ReplaySession_ptr()` (deliberately never decoding `ReplayHeader`/`SnapshotEntry`/events, so the test structurally cannot assert on the timestamp/wall-clock metadata AC 7.1 explicitly excludes), and asserts R and S's decisions match in count, type, and full structural content (reusing `sim_verify.c`'s `xce_job_alloc_t`/`xce_job_preason_t` field-by-field comparators, not a jobid-only shortcut). RED phase independently reproduced by review (corrupted a decoded jobid post-decode, confirmed FAIL, reverted, confirmed PASS). Honestly scoped to PEND-only decisions: independently confirmed (by review, reading `sim_loop.c`'s `sim_picker_thread_fn()` directly, committed under task 11.2) that the simulation picker thread has no code path capable of emitting a live DISPATCH decision at all — a genuine, pre-existing structural gap predating this task (real resource-matching allocation belongs to the not-yet-designed `scheduler-allocation` spec), not a shortcut taken here; documented prominently rather than silently avoided. 33/33 scheduler suite (was 32/32), whole-project build clean. Review APPROVED, zero blocking findings. - -- [x] 12. Core: Interactive REPL - -- [x] 12.1 Implement REPL execution control - - Implement a single input-loop REPL (not a separate thread) that pauses before the first event and blocks the simulation loop while active, supporting step-N-sessions, continue-to-next-breakpoint-or-EOF, and quit - - Observable: starting simulation with the REPL enabled pauses before any event is processed; issuing a step command advances exactly the requested number of sessions and returns control to the prompt - - _Requirements: 5.22_ - - _Note: use readline() for the input loop's line editing (history, arrow keys) — xce_test's REPL already sets this precedent (src/controller/testsuite/test.c:3827) and readline is already a linked project dependency, zero new cost. Confirm readline's GPL licensing is acceptable for this build target before committing to it (vs. libedit/linenoise) if that hasn't already been settled elsewhere in the codebase._ **Resolved: switched project-wide to libedit (BSD) instead — see Implementation Notes.** - - _Depends: 11.2_ - - _Verified: RED-phase reproduction performed during review round 2 — temporarily changed `repl_should_pause()` (repl.c) from `return (rs->step_budget <= 0);` to `return false;`, rebuilt, ran `sched_repl_execution_control_test`: 2/4 tests failed exactly as expected (`test_should_pause_fires_on_exactly_the_nth_session`, `test_step_1_processes_exactly_one_session_then_quit_exits`), the 2 unrelated tests (`test_should_pause_never_fires_in_continue_mode`, `test_pre_replay_pause_blocks_before_any_event_is_processed`) stayed green. Reverted; rebuilt; full scheduler suite 34/34 GREEN._ - -- [x] 12.2 Implement REPL breakpoints and state inspection - - Support setting/listing/deleting breakpoints on event type with optional field filters, and inspection commands reporting current simulation position, and job/queue/worker/resource/summary state - - Observable: setting a breakpoint on a specific event type and field value pauses the REPL exactly when a matching event is about to be processed; each inspection command reports state matching what a snapshot dump of the same moment would show - - _Requirements: 5.22_ - - _Depends: 12.1_ - - _Verified: 2026-08-07 — repl.c (task 12.1, 370 lines) split three ways under this project's 800-line cap (task 12.2's own feature surface did not fit in one file): repl.c (416 lines, execution control + dispatch, unchanged in scope), repl_break.c (breakpoint kind/filter matching, breakpoint CRUD, and the three new driver-facing hooks xce_repl_on_event()/xce_repl_on_begin_round()/xce_repl_on_decision()), repl_info.c (`info job`/`queue`/`worker`/`resource`/`summary`/`session`/`break` dumps) -- all sharing private state via a new repl_internal.h (the full xce_sched_repl_state_s definition, moved out of repl.c). sim_loop.c gained exactly 3 new bare call expressions (xce_repl_check_event()/xce_repl_check_begin_round()/xce_repl_check_decision(), repl.h inline wrappers gated on sim->interactive, same pattern as task 12.1's two existing hooks) at the three points design.md's pseudocode requires: per decoded ReplayEvent before it joins the accumulated batch, per ReplaySession boundary before the batch is pushed, and per drained decision after sim_print_decision() -- landed sim_loop.c at exactly 800/800 lines (trimmed its own header comment to make room). repl_should_pause()'s stale doc comment (which wrongly predicted this task would extend it) was corrected: breakpoints are a separate mechanism, never folded into the `step N` pause path. test_sim_isolation.c (task 11.3, already extended by 12.1) extended again to scan repl_break.c/repl_info.c/repl_internal.h (8-file precedent). New test_repl_breakpoints_inspection.c (17 tests, 4 groups: command parsing/listing, pure matcher predicates, state-inspection dumps against a real xce_sched_core_t, full-integration scripted-stdin pause proof for both an event breakpoint and a decision breakpoint). RED-phase: two REAL bugs surfaced and fixed during development (bug-1506: test fixtures calling repl_dispatch()'s `break` command without first calling xce_sched_core_init() left makeString()'s backing xstr_table uninitialized, silently returning NULL breakpoint fields instead of failing loud; bug-1507: repl_breakpoint_free() called a bare free() on a makeString()-produced xstring -- RTTI reference-counted, not a plain malloc() chunk -- causing a heap-corruption abort, fixed by never freeing those fields at all, matching kill_sched_job()/kill_sched_queue()'s existing "makeString()'d fields live for the process's lifetime" precedent). PLUS two targeted synthetic RED-phase probes reproduced independently after both real bugs were fixed: (1) repl_find_event_breakpoint() hardcoded to always return NULL -- rebuilt, ran the suite, exactly 3 of 17 tests failed (test_event_breakpoint_matches_jobid_filter, test_event_breakpoint_matches_state_filter, test_event_breakpoint_pauses_repl_at_matching_event), all others (including decision/begin_round matching and every info dump) stayed green; (2) repl_job_state_name() hardcoded to return a fixed bogus string -- exactly 1 of 17 failed (test_info_job_reports_state_queue_priority). Both probes reverted (grep-confirmed zero residue), rebuilt, reconfirmed green. Final: 35/35 scheduler suite (was 34/34), whole-project build clean. Remediation (2026-08-07): review found `repl_info_job()`'s resreq= field reported only a `set`/`(none)` presence flag instead of real content, despite `format_resreq()` (resreq.h) already existing and already used one file away by `build_wire_job()` (snapshot_wire.c) for this exact purpose; fixed by calling `format_resreq()` in `repl_info_job()` (mirroring `build_wire_job()`'s WARN-and-fallback error handling, freed after printing since this dump is transient), corrected the function's stale doc-comment claim that no print-form existed, and extended `test_info_job_reports_state_queue_priority` to assert the real formatted text (`"Primary:"`/`"Tasks: min = 0, max = 0"`/`"Rusages:"`) appears instead of the old boolean flag. Still 35/35 scheduler suite. - -- [x] 12.3 Implement REPL snapshot capture and simulation parameter commands - - Support writing a full snapshot to a file at the current simulation position, and adjusting speed/output-file/verify-mode from the prompt without restarting - - Observable: a snapshot command produces a file that the simulation driver can later load via a fresh `--sim` invocation starting from that exact position - - _Requirements: 5.22_ - - _Depends: 12.1_ - - _Verified: 2026-08-07 — new repl_params.c (fourth sibling in the repl.c/repl_break.c/repl_info.c family, repl_internal.h's own "Why a split at all" section extended again): `snapshot [FILE]` and `set speed|output|verify ...` (repl_cmd_snapshot()/repl_cmd_set()), reusing sim_record.c's sim_record_write_header_and_snapshot() (task 11.5) unchanged for both the one-shot standalone snapshot and `set output`'s live-stream preamble -- neither hand-rolls capnp encoding. `set speed <N>` mutates BOTH sim->sim_speed_num/sim_speed_den (the driver-level shadow copy) AND sim->core.speed_num/speed_den via the SAME atomic_store_explicit() pair sim_set_virtual_clock() (sim.c) uses at startup -- confirmed by reading core_clock.c's xce_core_now() that it reads the atomic pair fresh, uncached, on every call, so sim_loop.c needed zero new call sites (verified: `sim->sim_speed_num`/`sim->sim_speed_den` alone are never read by sim_loop.c or core_clock.c -- only the atomic `core.speed_num`/`core.speed_den` pair is; the shadow copy exists purely for the REPL/`info config` to read without reaching into `core`, per sim.h's own doc comment -- both fields are kept in sync by repl_set_speed() so this distinction never surfaces as a bug). `info config` (repl_info.c, alongside every other `info` dump) added as the read-only counterpart. `snapshot`'s auto-generated filename (`<basename>-snap-<event_index>.replay`) uses a documented fixed basename fallback ("sim") since xce_sched_sim_driver_t retains no string form of the replay path (sim.h has only a bare FILE*) -- a deliberate, documented deviation matching repl_count_jobs()'s own AC22-pseudocode-deviation precedent, judged not worth adding a retained-path field to sim.h/sim.c for. sim_loop.c untouched, still exactly 800/800 lines. New test_repl_snapshot_params.c (7 tests): the central Observable proved end to end via a forked child (`step 1`/`snapshot FILE`/`quit` against a real fixture) followed by a SECOND, real sched_sim_driver_start() call in the parent against the file the child wrote, asserting success, an exact reloaded snapshot_ts (2000, frozen clock), the pre-session job present and the post-session job absent (proves the snapshot captured the position at command time, not the whole file); plus auto-filename generation, `set speed`/`set verify` state+malformed-input unit tests, `set output` open/preamble-readback/off round trip, `info config` field-format assertions, and a `help` listing regression check. Every other sim-family white-box test file (`test_sim_equivalence.c`/`test_sim_event_loop.c`/`test_repl_breakpoints_inspection.c`/`test_sim_record_output.c`/`test_repl_execution_control.c`/`test_sim_alt_events.c`/`test_sim_verify.c`) needed a one-line `#include "../src/repl_params.c"` addition alongside their existing `repl.c` include -- repl.c's repl_dispatch() now unconditionally references repl_cmd_snapshot()/repl_cmd_set(), the same "unconditional extern forces every white-box TU to add the new source file" pattern this file family already established. RED phase: initial build failed exactly as expected (`meson setup` error "File src/repl_params.c does not exist") after wiring the test file, repl_internal.h extern declarations, and both meson.build files but before writing the implementation; after implementation, a real bug surfaced and was self-fixed before commit (an accidental `*/` inside a prose doc comment -- `xce_scheduler_t*/record_dir` -- prematurely closed a block comment, a variant of this codebase's known recurring `-Wcomment` pitfall). Synthetic RED-phase probe on the finished implementation: removed both `atomic_store_explicit()` calls from `repl_set_speed()` (leaving only the shadow-copy assignment), rebuilt, ran the suite: exactly 1 of 7 tests failed (`test_set_speed_updates_shadow_and_core_atomics`, "Expected 7 Was 0"), all 6 others stayed green; reverted, rebuilt, reconfirmed clean. test_sim_isolation.c extended to scan repl_params.c too (9-file precedent, seventh sched_sim_isolation_lib archive member). Final: 36/36 scheduler suite (was 35/35), whole-project build clean._ - -- [x] 12.4 REPL equivalence end-to-end test - - Run a fixture file through the REPL using step/continue to completion and through batch simulation mode, and assert both produce the same decision sequence - - Observable: both runs produce byte-identical decision sequences - - _Requirements: 5.22_ - - _Depends: 12.2, 12.3_ - - _Verified: 2026-08-07 — TEST-ONLY task, no new production code. New test_repl_equivalence.c (683 lines) runs the SAME sched_sim_driver_run() (sim_loop.c) against ONE hand-built `.replay` fixture (two queues at different priorities "high"=5/"normal"=1, four jobs split across THREE ReplaySession boundaries) twice: once in batch mode (interactive=false, straight through), once REPL-driven (interactive=true, stdin freopen()'d from a scripted "step 1\ncontinue\n" -- exactly 2 lines, sufficient and safe regardless of session count since CONTINUE mode never re-reads stdin once entered, avoiding the readline()-EOF-triggers-exit(0) trap). This is a DIFFERENT comparison than task 11.6's test_sim_equivalence.c (which compares normal-mode driver vs simulation driver) -- here both sides are the identical simulation driver, differing only in `sim->interactive`. Both runs set `sched_sim_args_t.output_path` directly (confirmed unconditional on `interactive` by reading sim.c's `args->output_path != NULL` gate), so both write a full `.replay`-format decision stream to their own file with no `set output` REPL command needed. Since the fixture has THREE session boundaries (not task 11.6's one), decoding uses a new decode_all_sessions() that scans the whole output file structurally via sim.c's shared `msg_root_has_shape()` (ReplaySession's distinct (1,2) word/pointer-count shape) rather than a fixed message-skip count. Comparator reuses sim_verify.c's leaf-level structural comparators (`sim_verify_job_alloc_eq()`/`sim_verify_job_preason_eq()`, both `static` but visible via white-box #include) through a duplicated `replay_decision_content_eq()`, same technique test_sim_equivalence.c's own function of that name establishes. Non-tautology guards: session 1 asserted to have EXACTLY 2 decisions (an unambiguous floor -- only 2 jobs could possibly be pending that early), job 501 independently pinned as present in session 1's batch-side content, total batch decisions asserted >= 4, and every decision's type independently pinned as PEND (matching sim_picker_thread_fn()'s documented DISPATCH gap) BEFORE the cross-run equality check -- so a buggy always-`return 1` comparator could not pass silently. Observed (not hand-predicted) that a PEND'd job persists as a picker candidate into later sessions (session 1: jobs 501/502 -> 2 decisions; session 2: 501/503/502 -> 3; session 3: 501/503/502/504 -> 4; total 9 on both sides), confirming the fixture is genuinely non-trivial without needing a fragile hardcoded per-session prediction. Synthetic RED-phase probe: analysis first showed that defeating `repl_should_pause()` (task 12.1's own already-verified probe target) would NOT turn this test red, because that only changes *when* stdin is read/control returns, never *what* gets processed -- every session still eventually completes regardless of pause timing, and this file's own CONTINUE-is-sticky script design tolerates any pause schedule. The correct probe for THIS invariant ("never alters ... what events are processed") had to touch content, and only under `sim->interactive`: temporarily added one line to `xce_repl_on_event()` (repl_break.c) -- which takes a non-const `xce_core_event_t*` and fires (via repl.h's `sim->interactive`-gated wrapper) strictly before an event joins the batch `xce_core_push_events()` later applies -- corrupting a JOBS_PUSH job's jobid (`+= 1000000`) whenever interactive. Rebuilt, ran the full scheduler suite: exactly 4/37 failed (this new test plus three pre-existing REPL tests that also exercise JOBS_PUSH under interactive mode -- sched_repl_execution_control_test, sched_repl_snapshot_params_test, sched_repl_breakpoints_inspection_test -- all correctly detecting the same real divergence), all 33 others stayed green. This test's own failure output: "REPL step/continue-to-completion must produce a byte-identical decision to batch mode at this session and position" (jobid 501 on the batch side vs 1000501 on the REPL side, directly visible in the two runs' printed PEND lines). Reverted (diff-confirmed byte-identical to the pre-probe file), rebuilt, reconfirmed 37/37 scheduler suite (was 36/36), whole-project build clean. - -- [x] 12.5 REPL live-operator-testing fixes (Requirement 5.22 amendment) - - Fix the five gaps a live operator session surfaced (see requirements.md's AC 22 "Amendment (2026-08-19)" for the full rationale on each): (1) `continue` reaching EOF with no breakpoint hit re-enters the REPL instead of exiting the process; (2) `step`/`continue` issued after replay is already complete re-prompt with a message instead of being treated as "hand control to a driver loop"; (3) `step`/`continue` echo each input event (not just decisions) as it is processed, in real event/decision order; (4) `info resource <name>|all` reports resource definition metadata restricted to `XCE_RESF_GLOBAL`/`XCE_RESF_SHARED` resources, reading the shared value from `core->sharedInst` instead of aggregating host-based resources across every worker; (5) `repl_split_tokens()` strips one matching pair of surrounding quote characters from each token. - - Observable: `--sim --interactive` against a replay file with no breakpoints set, driven by `continue`, stays at the `(sim) ` prompt after "[sim] replay complete" instead of the process exiting -- `info summary`/`info job`/etc. still work at that point, and only `quit` (or stdin EOF) ends the process; a `step` while events are pending prints each event's own one-line summary before that session's decisions; `info resource all` lists only global/shared resources with metadata fields, never a host-based resource's fabricated total/available; `info job "<jobid>"` (quoted) parses identically to the unquoted form. - - _Requirements: 5.22_ - - _Depends: 12.4_ - - _Bugs: bug-1593 (info resource host-vs-shared), bug-1594 (continue/no-quit exit), bug-1595 (events not echoed in step/continue), bug-1596 (quoted REPL argument parsing)_ - - _Verified: 2026-08-19 — Independently re-verified via `/kiro-verify-completion` (fresh evidence, not a prior self-report) across three layers. **(1) Fresh build + full suite**: `xce-build.sh scheduler asan --incremental` compiled clean; scheduler meson suite 41/42 (the one failure, `test_recordstart_symlink_escape_replies_error`, is the pre-existing unrelated bug-1560). Every task-12.5 unit test confirmed PASS by exact name in the fresh testlog: `test_continue_with_no_breakpoints_stays_alive_after_replay_complete`, `test_continue_after_replay_complete_reprompts_instead_of_exiting`, `test_step_echoes_events_before_session_decisions`, `test_info_resource_all_lists_only_global_shared_resources`, `test_info_resource_named_lookup_reports_host_based_note`, `test_info_resource_reports_string_value`, `test_info_job_accepts_quoted_jobid_argument`. **(2) Live subprocess smoke test of the real `xce_scheduler` binary** (fork/exec, real stdin/stdout, not the compiled-in test harness) for all five gaps: gaps 1/2 confirmed via a no-breakpoint replay driven by `continue`/`step` past completion -- process stays at the prompt, `info summary` still answers, `quit` exits 0; gap 3 confirmed via a fixture with `QUEUE_UPDATE`/`JOB_PUSH`/two `JOB_STATE_CHANGE` events -- all four event lines printed before `[sim] replay complete`, in file order; gap 5 confirmed via `info job <id>` issued unquoted, double-quoted, and single-quoted against a real pushed job -- byte-identical output all three ways; gap 4 confirmed via a hand-built `.replay` fixture (`RESOURCES_PUSH` defining a `XCE_RESF_SHARED` resource `licenses` and a host-based resource `mem`, `RESVALS_FULL` giving `licenses=8.0`) -- real captured output: `info resource licenses` → metadata + `value=8.00` read from `core->sharedInst`; `info resource mem` → metadata + "host-based resource -- see 'info worker <name>' for per-host values", no fabricated value; `info resource all` → lists `licenses` only, never `mem`. **(3) Direct source inspection** confirmed all five fixes present and correct in current on-disk code: `xce_repl_on_replay_complete()`/`repl_dispatch()`'s `replay_complete` guard (repl.c), `repl_print_event_summary()` called before the breakpoint check in `xce_repl_on_event()` (repl_break.c), the GLOBAL/SHARED filter + `sharedInst` read in `repl_info_resource_one()` (repl_info.c), and the single-pair quote-strip in `repl_split_tokens()` (repl.c). First verification pass returned NOT_VERIFIED solely because this checkbox/line hadn't been written yet and one live check (gap 4) was still test-only; both closed by this pass. - -- [x] 13. Run full cross-phase HA soak regression - - Run sequential and chaotic HA soak rounds against the complete feature (all three phases integrated), confirming zero job loss, divergence, or liveness stall attributable to this feature across the full run - - Observable: the soak run completes with a clean pass across all rounds, matching the zero-regression bar set by the Phase 1 and Phase 2 soak runs - - _Requirements: 4.3_ - - _Depends: 12.4, 10.6_ - - _Verified: 2026-08-07 — Full clean rebuild first (`nix develop ./src -- bash src/tools/xce-build.sh all`, profile=asan): 41/41 meson tests, 15/15 manual scheduler tests, all Go unit tests (apiserver/xcecli), all xceadmin pytest — zero failures, fresh `xce_controller`/`xce_test` binaries (mtime 13:12). Ran `/xce-test-ha` (`.claude/skills/xce-test-ha/run-ha-soak.sh`) against the complete integrated feature (all three phases — picker/core-driver/replay+REPL+simulation — live in one binary) in both required modes, 15 rounds each, fresh deploy dirs, `/home/claude/tmp` (non-overlay fs, dodges bug-924's overlayfs fdatasync wedge): **sequential** — `PASS: 15 rounds, 48 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)`; **chaotic** (overlapping kills inside election/catch-up windows, the bug-851-family stress mode) — `VERDICT mode=chaotic rounds=15 jobs=48 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=0 excluded_bug924_fdatasync=0 result=PASS`. Zero job loss, zero committed-prefix divergence, zero liveness stall, zero excluded-bug incidents in either mode — matches the zero-regression bar of the sibling specs' Phase 1/Phase 2 soak runs (scheduler-allocation task 3, scheduler-preemption task 6). This closes out scheduler-core: all 45 tasks (1.1–13) complete. - -## Implementation Notes - -- Task 1.3 added `sched_set_default_job_pick_plugin()` (schedobj.c) but deliberately did NOT call it from `sched_init()` — task 2.4 owns replacing `scheduler.c:43`'s hardcoded `"xce_sched_fcfs"` literal and must also wire this defaulting call (or equivalent logic) into scheduler startup, or `sched->job_pick_plugin` stays NULL through a real boot. -- Background IDE diagnostics ("file not found: rtti.h", "unknown type name" for various xce_* types) have appeared consistently on every scheduler.h/plugin.h/types.h/schedobj.c change so far (tasks 1.1-1.3) and are confirmed stale clangd artifacts each time via independent real `meson compile`/`meson test` runs — not real build failures. Don't let these block a task; verify with the real build instead. -- Meson build dir is `src/xce.build`, not repo-root `xce.build` — build/test commands must run from `src/` (e.g. `cd src && nix develop . --command bash -c "meson test -C xce.build --suite scheduler"`). -- Scheduler testsuite convention: run test binaries directly with `ASAN_OPTIONS=detect_leaks=0` (fixtures aren't torn down, by design — matches existing `meson.build` test registration). -- Task 2.3's review found a real concurrency gap for task 2.4 to close: once builtin_fcfs is wired live, `next_job()` (picker thread) and `on_dispatch`/`on_pend` (allocator thread) can race on the same unsynchronized `fcfs_plugin_data_t` private state — `cbuf_get()` unblocks the picker's next `cbuf_put()` before the allocator finishes processing the dequeued job (jbuf capacity is 1), so the picker can re-enter `next_job()` while the allocator is still mutating `priv->queues` via `on_dispatch`/`on_pend`. Today's production path never hits this because the allocator thread calls a different plugin category (`limit.c`'s resource-limit hook), never `fcfs.c`'s own private state. Task 2.4 must add a mutex guarding `fcfs_plugin_data_t` across all five picker hooks (or an equivalent handoff redesign) before registering builtin_fcfs as the live default. -- Task 2.4 closed the above race with `fcfs_plugin_data_t.lock` (mylock_t), taken by exactly 7 hooks (`on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update`, `begin_session`, `next_job`, `on_dispatch`) — `on_pend`/`end_session`/`init`/`fini` correctly excluded (they never touch `priv->queues`/`cursor`/`job_cursor`), a verified refinement of this note's original "all five hooks" estimate, not a gap. `sched_job_picker()` now branches `job_elect_fn != NULL` (legacy plugins like `xce_sched_fcfs`, delegates unchanged to their own self-contained loop) vs NULL (native `xce_job_picker_iface_t` pickers, runs a new generic per-session loop) — this split exists because legacy `job_elect_fn` implementations (e.g. `fcfs_elect_job`) already contain their own internal `cbuf_get(&wbuf)`, so a single uniform outer loop would double-consume the wbuf signal for legacy pickers. `sched_allocator()` now calls `picker_iface->on_dispatch()`/`on_pend()` after each allocation attempt (Req 1 AC 4), which was previously called for no picker at all — required for builtin_fcfs to actually remove dispatched jobs from its private state once live. -- Two follow-ups surfaced by task 2.4's review, not yet fixed (outside its file boundary): (1) `main.c` discards `sched_init()`'s return value, so a bad `job_pick_plugin` config fails `sched_init()` correctly but the process doesn't yet halt on it — needs a fast follow-up before relying on "fails startup" in production. (2) `pthread_testcancel()` in `builtin_fcfs_next_job()` is only at the loop entry/exit, not scattered through the queue/job walk (the walk holds `fcfs_plugin_data_t.lock` throughout, and this codebase has no `pthread_cleanup_push` precedent to safely unlock on mid-walk cancellation) — behaviorally inert today since `sched_queue_is_ready`/`sched_job_is_ready` are `return 1` stubs, but becomes a real cancellation-latency gap once Phase 2 implements real filtering. -- Task 4.2's Req 5 AC 7 kvmgr iteration-order audit (scope: jobs/queues/users/projects, per design.md's bounded-scope note) found no unstable site — recorded durably as a comment in `core.c` above `xce_sched_core_init()`. One open gap surfaced but explicitly out of this task's scope: `core->workers` is iterated order-sensitively by `sched_collect_workers()` → `seqjob_allocator.c`'s first-fit `sched_try_alloc_on_hosts()`, and design.md itself says the full AC 7 guarantee must close "before Phase 3 simulation work depends on it" — needs a follow-up audit/fix (or an explicit documented exemption) before Phase 3 (tasks 7+) starts relying on scheduling determinism across worker selection. -- Task 4.3 split its implementation into a NEW file `core_events.c` (not `core.c`, which design.md's File Structure Plan literally names) because implementing `xce_core_push_events()` directly in `core.c` would have pushed it to 977 lines, over this project's 800-line hard cap. `core.h` (the public contract) matches design.md's intent; only the `.c` file location differs. Worth reflecting in design.md next time it's touched. Also: `test_core_isolation.c`'s nm(1)-based zero-controller-I/O boundary audit (Req 5 AC 3) covers only `core.c`, not `core_events.c` — manually verified clean (no netio/xceapi/client/delay/idgen usage) but not mechanically enforced; extending that test to cover `core_events.c` would be a reasonable low-risk follow-up before later tasks add more logic to this file. -- Same pattern compounded across tasks 4.4 (`core_session.c`) and 4.5 (`core_clock.c`, which also genuinely NEEDS to be outside the wall-clock-call scan since it's the one file allowed to call `clock_gettime`) — `test_core_isolation.c` now covers only `core.c`, while three sibling files (`core_events.c`/`core_session.c`/`core_clock.c`) sit outside its mechanical audit, each manually verified clean but not enforced by a test. Worth a dedicated follow-up task extending the forbidden-header/forbidden-symbol half of the audit (not the wall-clock half, which `core_clock.c` must stay exempt from) to all `core_*.c` files before Phase 3 adds more. -- Task 5.2's review surfaced a genuine, pre-existing bug in `plugin.c`'s `plugin_end_session()` (predates this entire spec — `git blame` traces it to commit `638173dec`, 2026-02-26; untouched by any scheduler-core task's diff): the per-plugin guard checks `plugin->begin_session_fn != NULL` but then calls `plugin->end_session_fn(plugin)` — a guard/call mismatch. Confirmed live impact: `plugins/fcfs.c` registers `.begin_session_fn = NULL`/`.end_session_fn = fcfs_end_session`, so `fcfs_end_session()` (resets the legacy `job_elect_fn`-style iteration cursor) is never invoked today. Only matters if a deployment's `job_pick_plugin` config selects the legacy `xce_sched_fcfs` name rather than the new default `builtin_fcfs` (both fields NULL on `builtin_fcfs`, unaffected). Out of scope to fix here — needs its own bug-log entry / small fix task, not folded into scheduler-core. -- Task 9.2's review found a non-blocking follow-up, not fixed in that task: `sched_record_start()`'s TOCTOU-hardened `open()` (`recording.c`, `O_WRONLY|O_CREAT|O_NOFOLLOW`) lacks `O_NONBLOCK`. A FIFO planted at the resolved path in the same race window the symlink/hard-link defenses already close (between `sched_validate_record_filename()` returning and `open()` running) would hang the calling thread indefinitely — a DoS (thread hang), not data corruption, since the existing `st_nlink`/`S_ISREG` checks still reject it once `open()` eventually returns. Adding `O_NONBLOCK` to the open flags is zero-risk for the regular-file case and closes this; worth a small follow-up before task 9.2's code sees production traffic. -- Task 9.3's review found a genuine task-breakdown gap, not fixable within 9.3's own boundary: `record_snapshot_interval` (the field task 9.3's periodic-checkpoint mechanism reads) has no CLI or admin-request wiring anywhere in the codebase — `--record-snapshot-interval` appears in AC 11's own normative text but in zero task's actual scope. Neither task 9.4 (admin request handler — dispatches `SchedAdminRequest` actions, which don't carry an interval field either, per controller.capnp's `SchedAdminRequest{action, recordFile}`) nor group 10 (controller-side dispatch) owns scheduler startup CLI flags; only `main.c`'s argp table does, following task 9.1's own `--record-dir` precedent. As currently scoped, AC 11's mechanism is fully implemented and tested (via the struct field directly) but has no path to ever be configured non-zero in a real deployment. Needs a small follow-up task before this spec is considered complete: add a `--record-snapshot-interval N` CLI flag to `main.c`, wired to `sched.record_snapshot_interval`, mirroring `--record-dir`'s exact pattern (task 9.1). -- Task 9.3's review also flagged `test_record_lifecycle.c`'s size (826→1232 lines across tasks 9.2+9.3) as past this suite's established over-800 precedent (prior max `test_driver_translate.c`≈867) — the content itself is genuine, non-duplicative coverage (not padding), and a natural split point exists along the AC10 (lifecycle/TOCTOU)/AC11 (periodic-checkpoint/error-handling) seam by extracting shared fixture helpers into a common header. Non-blocking; worth a future cleanup pass, not urgent. -- Cross-task interaction, flagged before task 9.4 exists — resolution decided (not a fail/retry, a deferral): task 9.2's `sched_record_start()` calls `xce_sched_core_save_snapshot()` as its first action (AC 10 step 1) — the same function task 8.2 gated to return `-EBUSY` whenever `session_active != 0`. `main.c`'s driver loop is single-threaded (`while(1) { xce_sched_driver_run_normal_mode_round(&sched); ...; sleep(1); }`); incoming requests (including the future `SchedAdminRequest`) arrive asynchronously via `scheduler_req_handler` (handler.c) on the I/O side, same as job/queue/worker updates. **Decision**: task 9.4 must NOT call `sched_record_start()` directly from the request-handling thread (that both risks racing `session_active` and violates this codebase's async-I/O rule against blocking/file-I/O in a request-handler callback). Instead, `recordStart` must be enqueued the same way job/queue/worker updates already are (`pend_updates` dataq or an equivalent dedicated pending-admin-request slot) and picked up by `xce_sched_driver_translate_pending_events()` — which already runs at the top of every round, strictly before `begin_session()`, so it is guaranteed between-sessions by the same construction argument design.md uses for `core_push_events()`. This means recording "takes effect" at the next scheduling round, not synchronously on request arrival. Consequence for task 9.4: since AC 10 specifies a synchronous request-reply carrying the resolved filename, the actual `SchedAdminReply` send must be deferred until the driver thread performs the queued start (not sent eagerly from the handler thread) — needs a completion signal back to the waiting request context (e.g. a condition variable, or replying once `xce_sched_driver_translate_pending_events()` reports the deferred start applied). Given this deferral makes the direct `-EBUSY` race effectively unreachable from the admin path, the AC 10 error-codes table doesn't need a new "busy" entry for this case — `sched_record_start()` keeps its `-EBUSY` return as a defensive/fail-loud contract check (task 9.2's job, no torn state), it just should no longer be a path task 9.4's dispatch can actually trigger by construction. -- Task 12.1's review found a licensing gap (fixed): the readline()-based REPL input loop originally linked GNU readline (GPL) directly into `xce_scheduler`, a shipped production daemon (unlike `xce_test`, a debug-only binary controller's own readline use was already confined to). Switched project-wide to libedit (BSD) — `nixSupport/xce.nix`, both `meson.build`s, `repl.c`, and `src/controller/testsuite/test.c` now use `editline/readline.h` (API-compatible: `readline()`/`add_history()`), no hand-written accessor changes needed anywhere. -- Task 12.1's review also surfaced bug-1504 (fixed) and bug-1505 (deferred, out of boundary) — see `.wolf/buglog.json`. Verifying the licensing fix required a full clean rebuild (new nix dependency), which exposed a genuine, pre-existing, deterministic SIGSEGV in `write_SnapshotData` unrelated to task 12.1's own repl.c: `SnapshotData.internalState` was a capnp `Data` field, and this codebase's capnpc-c backend silently no-ops Data-field encoding in generated `encode_XXX()` helpers without zero-initializing the caller's stack struct first — a real, reproducible UB crash, not a flaky test. Fixed (bug-1504) by switching `internalState` to `List(UInt8)` (controller.capnp, typemap.h, test_e2e_capnp.c) — this same capnpc-c backend DOES correctly auto-generate real encode/decode for `List(UInt8)` (proven precedent: `FetchFileReply.content`/`bytesRead`), eliminating the whole no-op class for this field with no hand-written accessor code. Re-verified with a full clean rebuild: scheduler suite 34/34, no regressions. While chasing this, also hit a **second, unrelated instance of the same bug class** in the controller's HA/WAL-catchup subsystem (`WalCatchupEntry.payload`, bug-847's own territory) — confirmed systemic (affects any `:Data` field in `controller.capnp` fed through the generic codec, not a one-off) but deliberately **left unfixed** (bug-1505) as out of scheduler-core's boundary; needs its own project-wide `:Data`-field audit as a separate task. -- bug-1505 fixed (2026-08-07, follow-up outside any specific task's boundary — controller/HA subsystem, not scheduler-core): before fixing, confirmed zero production exposure — `xce_wal_batch_encode_raw()`/`wal_entry_to_wire()` (`request.c`) never call the generic `encode_WalCatchupEntry()` at all; they `memset()` their own struct and call `write_WalCatchupEntry()` directly with every field hand-set, so the uninitialized-`capn_ptr` crash was 100% test-only exposure (only `test_e2e_capnp.c` calls the generic path). Fixed the same way as bug-1504: `WalCatchupEntry.payload` (@2) and `batchPayloads` (@10) switched from `Data` to `List(UInt8)` (new `batchPayloadsLen` @11 companion; `payload` reuses the existing `wirePayloadLen` @9), `typemap.h`'s `wal_catchup_entry_t` field types updated, `request.c`'s `extract_data_bytes()` widened to accept a raw `capn_ptr` (it already only used `.p` internally) so its 4 call sites work regardless of which Data-shaped wrapper the accessor returns now. Verified `batchPayloads`' single-top-level-blob shape means this does NOT reintroduce bug-847's per-element cross-segment far-pointer loss (that issue is inherent to PER-ELEMENT data regardless of `Data` vs `List(UInt8)` spelling — `payload`, the per-element field, was already legacy/unpopulated in production before this fix). Strengthened `test_wal_catchup_append_term_fields_round_trip` with real payload/batchPayloads content assertions (previously never exercised — the test's `src.payload`/`batch_payloads` were left NULL, so the crash occurred on content the test didn't even care about). Verified: `controller_unit_test` no longer crashes (3/3 controller suite), scheduler suite unaffected (37/37, unrelated schema area). -- **Feature-level validation (`/kiro-validate-impl`, 2026-08-07) — GO, with 2 new non-blocking follow-ups found:** - 1. **Req 5 AC 3's tlog logging-integration bullet is built but never activated.** `core_clock.c`'s `xce_core_register_tlog_format()`/`xce_core_set_active_for_logging()` (task 4.5) are unit-tested but have zero call sites in `main.c`/`sim.c`/`driver.c` — task 4.5's own header comment explicitly deferred this ("this task does not wire it into mylog_init()/main.c; that remains a later task's job"), but no later task in this spec ever picked it up. Root-caused where it *should* wire in: NOT `main.c`'s normal-mode path — normal mode seeds `virtual_clock_base=0, real_clock_base=0` (`core.c:139-140`), so `xce_core_now()` there returns raw `CLOCK_MONOTONIC` nanoseconds-since-boot, not epoch time; activating it for normal-mode logging would make log timestamps *worse*, not better. The correct site is `sim.c`'s `sched_sim_driver_start()`, right after `sim_set_virtual_clock()` (line 394) — `virtual_clock_base` is seeded to `snapshot_ts` there (a real epoch-ns value from the `.replay` file), so `xce_core_now()` becomes a meaningful simulated timestamp. Left unfixed here rather than patched under validation-gate time pressure because activation needs a matching deactivation: `g_active_core_for_logging` is a single process-wide pointer, `sched_sim_driver_run()`'s own header comment already documents "no sim-driver stop/fini function exists" (deliberate, accepted gap) with multiple early-return paths, and several test binaries (e.g. `test_repl_equivalence.c`) construct more than one stack-allocated `xce_sched_sim_driver_t` per process — activating without a paired deactivation risks a real ASAN stack-use-after-scope hazard the very next time an unrelated log call fires in that process. Needs a small dedicated follow-up task: wire activation into `sched_sim_driver_start()` AND deactivation into every `sched_sim_driver_run()` return path (or add the fini function its own comment says doesn't exist yet). - 2. **`plugins/fcfs.c` vs `plugins/builtin_fcfs.c` — functionally correct, undocumented.** design.md (component description) says `fcfs.c` is "rewritten (Phase 1) to `xce_job_picker_iface_t`, drops `job_elect_fn`." The actual implementation instead added a new parallel file, `plugins/builtin_fcfs.c`, and left `fcfs.c` untouched still registering the legacy `job_elect_fn` (`fcfs.c:353`) — confirmed deliberate, not an oversight: `scheduler.c:466`'s own comment says the new picker is registered "alongside the untouched legacy `xce_sched_fcfs`", and task 1.2's compatibility-wrapper requirement needs exactly this kind of legacy `job_elect_fn`-only plugin to exist as its live exercise target. `builtin_fcfs` is confirmed the registered default (`schedobj.c:364`). Functionally sound; the only gap is that tasks 2.1-2.6 carry no `_Verified:` annotations (unlike every other task group) and design.md's literal text was never updated to record this file-layout decision — a documentation/traceability gap, not a defect. Worth a follow-up: add `_Verified:` notes to 2.1-2.6 and correct design.md's component description. - - Both items are cosmetic/traceability, not correctness or determinism defects — the soak, equivalence, and integration evidence for this validation pass (15-round sequential + 15-round chaotic HA soak, both zero-loss/zero-divergence; full rebuild 41/41 meson + 15/15 manual scheduler + all Go/xceadmin tests green; independent requirements-coverage/design-alignment/cross-task-integration subagent passes) is unaffected by either. - - **Status update (2026-08-19)**: Follow-up 1 (tlog activation) is now **FIXED** — see design.md's Open Questions/Risks #6. Follow-up 2 is **FULLY CLOSED**: design.md's component description was corrected the same day (2026-08-07, see design.md's own Open Questions/Risks #7), and tasks 2.1-2.6 were backfilled with `_Verified:` lines the next day (commit `d7b49690`, 2026-08-08) — an earlier revision of this status-update line incorrectly claimed the backfill was still open without checking; corrected after fresh verification confirmed both the `_Verified:` lines and the underlying `builtin_fcfs.c` implementation they describe are present and accurate (60/60 picker/fcfs-related tests green on a fresh rebuild, 2026-08-19). diff --git a/.kiro/specs/scheduler-dispatch-commit/design.md b/.kiro/specs/scheduler-dispatch-commit/design.md deleted file mode 100644 index 607025ed..00000000 --- a/.kiro/specs/scheduler-dispatch-commit/design.md +++ /dev/null @@ -1,344 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: Move a job's transition to a running state from worker-confirmation time to dispatch-decision-commit time, on both the scheduler and the controller, and build the dispatch-failure retry/terminal-resolution machinery this requires — closing bug-1624 (resource-in-use totals silently drop a dispatched-but-unconfirmed job at the next round boundary) and bug-1625 (the scheduler discards the controller's existing dispatch-failure notification). - -**Users**: Cluster operators who rely on accurate live resource-in-use accounting and correct job status; scheduler/controller maintainers who need dispatch failures to resolve deterministically instead of leaking stale allocation state. - -**Impact**: `sched_commit_decision()` (scheduler) and `dispatch_a_job_()` (controller) each gain a state transition alongside the resource-allocation commit they already perform. A new scheduler-side notify handler and a new controller-side classification step are added. No existing component is replaced; both integration points are extended in place. - -### Goals -- A job's state becomes RUN at the moment its dispatch is committed, on both scheduler and controller, with no window where the two disagree. -- `sched_account_jobs()`'s existing RUN-state gate — unmodified — naturally includes a dispatched job from the round it was dispatched. -- A dispatch failure resolves deterministically: transient/environmental failures retry the job back to PEND (bounded), local/invalid-decision failures fail the job permanently. Both outcomes release the stale allocation/commitment data from the failed attempt, on both sides. -- The worker's actual confirmed-start fact is still durably recorded, unchanged, for `start_time` and audit purposes. - -### Non-Goals -- Changing simulation-mode's own dispatch/commit path — `sched_commit_decision()` is not called by the simulation driver at all (confirmed in Research Log); no code change is needed there. -- Implementing preemption delivery or victim selection — `scheduler-preemption` owns that; this design only leaves its cross-reference note intact. -- Changing `scheduler-allocation`'s own allocation-plan logic — this design only changes when a job reaches RUN, not how a plan is chosen. -- Redesigning the WAL entry header format or the raft-apply-decoupling deferred-apply mechanism — this design adds one new WAL fact type using the existing `wal_jobstate_t` shape and the existing synchronous-mutation discipline `job.allocs` already follows, not a new mechanism. - -## Boundary Commitments - -### This Spec Owns -- The scheduler-side state transition inside `sched_commit_decision()`'s DISPATCH branch. -- The controller-side state transition inside `dispatch_a_job_()`, gated on `xce_startjob_req()` success, correctly synchronized under `RTTI_LOCK_WRITE`, and the new WAL fact type recording it durably. -- The controller-side dispatch-failure classification (retryable vs. terminal, including the bounded-retry check) and its two resolution paths. -- The new scheduler-side `NotifyData_data_jobDispatchFail` handler (the actual fix for bug-1625) — a single, narrow case that applies `JOB_STAT_PEND`. - -### Out of Boundary -- `scheduler.c`'s `sched_update()`/`sched_job_state_change()` — confirmed dead in the live driver (Research Log); not modified, not reused, not removed (removal is a separate cleanup, not required for this feature to be correct). -- `core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` case — already correct and already idempotent for a repeated RUN application; unmodified. -- `builtin_fcfs`'s picker hooks (`on_dispatch`, `on_job_state_change`) — unmodified; their existing idempotency is relied upon, not extended. -- The `limit` plugin's own usage accounting — confirmed to not share bug-1624's class of gate (Research Log); not investigated further here. -- `mark_job_fail()`'s own mutate-then-notify body and its existing `HOOK_DETAIL_JOB_FAIL` → `schedFailJob` chain — reused as-is for the terminal resolution path; not modified. -- Any change to `xcecli`/REST API response shapes — job status already reflects `job.state` directly; no new field is added there. -- Fixing bug-1626 at `job_ctrl()`'s own pre-existing (and unrelated to dispatch) call to `mark_job_fail()` (`job.c:1083`) — this design's new call sites correctly take `RTTI_LOCK_WRITE` themselves, but `job_ctrl()`'s own gap is a separate, pre-existing condition this spec does not fix as a drive-by patch. - -### Allowed Dependencies -- `sched_commit_decision()` (scheduler.c) and `sched_account_jobs()`'s existing `JOB_IS_RUN` gate (scheduler.c) — read, not modified, except for the one new transition line. -- `mark_job_fail()` (job.c) and its existing hook chain — called, not modified, under a caller-supplied `RTTI_LOCK_WRITE` (its own documented precondition). -- `notify_dispatch()` (job.c) — called, not modified (its meaning narrows by convention, not by signature change). -- `persist_submit_job_fact_ex()` (persist.c) and `wal_jobstate_t` (wal_types.h) — extended with one new WAL entry type, following the existing `_pad`-discriminated shape. -- `job_t`'s in-memory-only wrapper fields (`xce/types.h`) — extended with one new in-memory-only counter, following `submit_lsn`'s existing precedent. -- `job_lock`/`job_unlock` (`src/lib/jobj.c`) via the existing `lock(j, RTTI_LOCK_WRITE)`/`unlock(j)` calling convention already used correctly at most other job-mutation sites in `job.c` — this design's new mutation sites must follow the same convention, not the unlocked pattern `dispatch_a_job_()` and `job_ctrl()`'s existing `mark_job_fail()` call site currently use (bug-1626, pre-existing — see `research.md`'s Addendum). - -### Revalidation Triggers -- Any change to `sched_account_jobs()`'s RUN-state gate itself (not just what feeds it). -- Any change to `dispatch_a_job_()`'s failure-point ordering or return codes — the classification table (Components, below) is keyed to today's exact failure points. -- Any change to `mark_job_fail()`'s hook chain or to `HOOK_DETAIL_JOB_STOP`/`HOOK_DETAIL_JOB_FAIL`'s wiring to `sched.c`'s notify translation. -- `scheduler-preemption` moving to design — its AC6/AC9 pending-revision note depends on this design's RUN-timing conclusion being final. -- Any future fix to bug-1626 (the pre-existing `job_ctrl()`/`mark_job_fail()` unlocked call site, out of this spec's boundary) that changes `mark_job_fail()`'s locking contract itself, as opposed to fixing individual callers. - -## Architecture - -### Existing Architecture Analysis - -Two independently-maintained job records exist for the same logical job: the scheduler's `xce_sched_job_t` (`sched/types.h`, decision-making working copy) and the controller's `job_t`/`xce_job_t` (`xce/types.h`, the durable, user-facing record). They are today synchronized by two one-directional channels: scheduler → controller decision publish (`sched_commit_decision()` → `xcecli_jobdispatch()`), and controller → scheduler notify (`sched_handle_notify()`'s switch, `handler.c`). This design adds a state transition to each side's own record at its own commit point, rather than introducing a third synchronization channel — the two sides reach the same conclusion (RUN) independently, from the same triggering event (the dispatch decision), with the notify channel used only for the asymmetric case where the controller's downstream attempt fails after the scheduler already committed. - -A second, load-bearing discovery (Research Log): `scheduler.c`'s notify-apply path (`sched_update()`/`sched_job_state_change()`) is dead code in the current driver — the live path is `core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` case, reached via `xce_sched_driver_translate_pending_events()`. All references below to "the existing RUN-confirmation path" mean this live path. - -### Architecture Pattern & Boundary Map - -```mermaid -sequenceDiagram - participant Picker as Scheduler picker/allocator - participant SchedCommit as sched_commit_decision - participant Ctrl as Controller dispatch_a_job_ (owns resolution + notify) - participant Worker as Worker - participant Notify as Controller notify (existing) - participant SchedNotify as Scheduler sched_handle_notify - - Picker->>SchedCommit: DISPATCH decision - SchedCommit->>SchedCommit: set job-accounts, job-allocs, state=RUN - SchedCommit-->>Ctrl: publish decision - Ctrl->>Worker: xce_startjob_req - alt RPC succeeds - Worker-->>Ctrl: ok - Ctrl->>Ctrl: lock j write, commit job-allocs, state=RUN, append WAL_JOB_DISPATCH fact, unlock - Note over Ctrl: worker later confirms actual start<br/>WAL_JOB_START still recorded separately, unchanged - else RPC fails, transient - Ctrl->>Ctrl: increment dispatch_fail_count, stay PEND - Ctrl->>Notify: jobDispatchFail - Notify->>SchedNotify: NotifyData_data_jobDispatchFail - SchedNotify->>SchedNotify: apply JOB_STAT_PEND (new case) - else RPC fails, terminal or retry bound exceeded - Ctrl->>Ctrl: lock j write, mark_job_fail (existing), unlock - Ctrl->>Notify: schedFailJob (existing, unmodified path) - Notify->>SchedNotify: NotifyData_data_schedFailJob - SchedNotify->>SchedNotify: apply JOB_STAT_EXIT (existing, unmodified) - end -``` - -**Architecture Integration**: -- Selected pattern: extend two existing, independent commit points rather than add a synchronization protocol. Reuses the existing asymmetric notify channel only for the failure case, where it is unavoidable (the scheduler cannot know the controller's downstream outcome any other way). -- Domain/feature boundaries: scheduler-side accounting correctness and controller-side dispatch/failure handling remain separately owned, matching the existing component split; neither gains knowledge of the other's internals beyond the existing decision-publish/notify contracts. -- Existing patterns preserved: `sched_account_jobs()`'s RUN-state gate, `mark_job_fail()`'s mutate-then-notify shape, `wal_jobstate_t`'s `_pad`-discriminated WAL fact shape, `job_t`'s in-memory-only wrapper-field convention. -- New components rationale: a classification step and a bounded counter are new because no equivalent existed; a new WAL fact type is new because the transition now needs a durable record that `job.allocs`'s own (non-durable) precedent doesn't provide. -- Steering compliance: no controller self-termination path is introduced; the terminal-failure path stays entirely inside the existing "fail loud, mutate, notify" discipline `mark_job_fail()` already follows. - -## File Structure Plan - -### Modified Files -- `src/scheduler/src/scheduler.c` — `sched_commit_decision()`'s DISPATCH branch gains `job->state = JOB_STAT_RUN;`, alongside the existing `job->allocs`/`job->accounts` assignment. No new field. -- `src/scheduler/src/handler.c` — `sched_handle_notify()`'s switch gains one new `case NotifyData_data_jobDispatchFail:`, applying `JOB_STAT_PEND` through the existing `XCE_SCHED_OP_JSTATE_CHANGE` enqueue path (same shape as the adjacent `schedExitJob`/`schedFailJob` cases). -- `src/controller/job.c` — two coupled changes: - - `dispatch_a_job_()`'s success/failure handling is restructured: `job.allocs`'s commit block (today unconditional, `job.c:947-954`, run regardless of `xce_startjob_req()`'s `rc`) moves inside an `if (rc >= 0)` branch. On success, the function takes `lock(j, RTTI_LOCK_WRITE)`, commits `job.allocs`/`job.state`, appends the `WAL_JOB_DISPATCH` fact, then `unlock(j)` — closing bug-1626 for this call site as a direct consequence of restructuring it. On `xce_startjob_req()` failure (`XCE_DISPATCH_FAIL_SITE_RPC_FAILURE`, one of the six sites Controller Dispatch-Failure Classification enumerates), the function follows that component's same classify-resolve procedure. At each of the six post-lookup failure points, the function classifies and resolves inline (under its own `lock(j, RTTI_LOCK_WRITE)`): retryable (four sites, including RPC failure) → increment `dispatch_fail_count`, `unlock(j)`, `dereference(j)` (closing bug-1627's leak on these four paths), call `notify_dispatch()`; terminal (two sites) → call `mark_job_fail()` under the write lock, `unlock(j)` after (these two already correctly `dereference(j)` today, unaffected by bug-1627). - - `handle_jobdispatch()`'s loop drops its existing per-job `notify_dispatch(c, ep, alloc->jobid, ...)` calls (both the success-path and failure-path ones) — notification is now entirely owned by `dispatch_a_job_()`, preventing the double-notify race found in round 2 of this review (a terminal failure previously would have produced both `jobDispatchFail` and `schedFailJob`). -- `src/include/xce/types.h` — `xce_job_t`'s wrapper `job_t` (not the wire type itself) gains `uint32_t dispatch_fail_count;`, following `submit_lsn`'s existing "in-memory only, reset on replay" precedent. -- `src/include/wal_types.h` — one new `wal_entry_type` value, `WAL_JOB_DISPATCH = 12`, reusing the existing `wal_jobstate_t{jobid, state, _pad=0}` payload shape (the "legacy: state is absolute" discipline, matching `job.allocs`'s own synchronous-commit precedent — see Data Models). -- `src/controller/persist.c` — `WAL_JOB_DISPATCH` is added as a new `case` label to two existing fall-through groups: `apply_wal_entry()`'s (`persist.c:3452-3466`; currently has zero production call sites — see the Components section correction below) and `apply_committed_entry()`'s (the actual live-apply-and-recovery path, `persist.c:3700-3706`) — both already apply `wal_jobstate_t.state` as an absolute value for every type in their respective groups; no new function or case body. -- `src/controller/job.c` (same file, no new header) — `xce_classify_dispatch_fail()`, `xce_dispatch_fail_site_t`, and `xce_dispatch_fail_class_t` are declared `static`/file-local next to `dispatch_a_job_()`, its only caller — not in a shared header, since nothing outside this file consumes them (see Controller Dispatch-Failure Classification's placement-correction note). - -### New/Modified Test Files -- `src/scheduler/testsuite/test_scheduler.c` (or a new focused file) — `sched_commit_decision()` DISPATCH-branch RUN-transition. -- `src/scheduler/testsuite/test_handler.c` (or equivalent) — new `jobDispatchFail` case applies `JOB_STAT_PEND`, frees stale `accounts`/`allocs` via the existing shared path. -- `src/controller/testsuite/test_job_dispatch.c` (or equivalent existing dispatch test file) — gated commit on RPC success/failure for all seven failure returns (the jobid-lookup case's two variants unchanged, the six post-lookup cases newly classified); retryable vs. terminal classification; bounded-retry escalation; WAL fact durability across a simulated restart/replay. -- HA soak (`/xce-test-ha`) — no new file; existing sequential/chaotic regression re-run per Requirement 5.4. - -## System Flows - -See the sequence diagram above (Architecture Pattern & Boundary Map) for the end-to-end flow across both success and both failure branches. No additional flow diagrams are needed — the state-transition logic itself is a simple two-branch decision per side, not a multi-step process. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1, 1.2, 1.3 | Scheduler RUN transition at commit time | Scheduler Dispatch Commit | `sched_commit_decision` | Sequence: SchedCommit branch | -| 2.1, 2.3 | Controller RUN transition at dispatch success | Controller Dispatch Commit | `dispatch_a_job_` | Sequence: RPC succeeds branch | -| 2.2 | Confirmed-start fact still recorded separately | Controller Dispatch Commit (unmodified sibling) | `record_job_start_fact` (unmodified) | n/a | -| 2.4 | Durability of the new transition | WAL Dispatch Fact | `persist_submit_job_fact_ex`, `WAL_JOB_DISPATCH` apply cases (live + recovery) | Sequence: RPC succeeds branch | -| 3.1, 3.2 | Scheduler processes dispatch-failure notify | Scheduler Dispatch-Failure Handler | `sched_handle_notify` new case | Sequence: transient branch | -| 4.1, 4.2 | Retryable vs. terminal classification | Controller Dispatch-Failure Classification | classification table | Sequence: both failure branches | -| 4.3 | Bounded retry count | Controller Dispatch-Failure Classification | `dispatch_fail_count` | Sequence: transient branch | -| 4.4, 4.5 | Release stale allocation/commitment data | Scheduler: Scheduler Dispatch-Failure Handler (retry), `mark_job_fail` (terminal, unmodified). Controller: satisfied vacuously — Controller Dispatch Commit never commits on any failure path, so there is nothing to release | `core_events.c`'s existing free-on-non-RUN logic (scheduler side) | Sequence: both failure branches | -| 5.1 | Confirmed-start fact recorded unchanged | Controller Dispatch Commit (unmodified sibling) | n/a | n/a | -| 5.2 | Simulation mode unaffected | n/a (no simulation code touched) | n/a | Research Log | -| 5.3, 5.4 | Regression / HA soak | n/a (test-only) | n/a | n/a | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|---------------|--------|---------------|---------------------------|-----------| -| Scheduler Dispatch Commit | Scheduler / scheduler.c | Transition a job to RUN in the same operation that commits its dispatch decision | 1.1-1.3 | `sched_get_job` (P0), `sched_charge_account` (P0) | State | -| Scheduler Dispatch-Failure Handler | Scheduler / handler.c | Apply PEND to a job whose dispatch was reported as retryably failed | 3.1, 3.2, 4.4 | `XCE_SCHED_OP_JSTATE_CHANGE` enqueue (P0), `core_events.c`'s existing apply case (P0) | Event | -| Controller Dispatch Commit | Controller / job.c | Transition a job to RUN, and durably record it, only once dispatch to the worker actually succeeds | 2.1-2.4, 5.1 | `xce_startjob_req` (P0), WAL Dispatch Fact (P0) | State, Batch | -| Controller Dispatch-Failure Classification | Controller / job.c | Decide retryable-vs-terminal for a dispatch failure, including the bounded-retry escalation | 4.1-4.3 | `job_t.dispatch_fail_count` (P0), `mark_job_fail` (P0), `notify_dispatch` (P0) | Service | -| WAL Dispatch Fact | Controller / persist.c, wal_types.h | Durable record of the dispatch-time RUN transition, replayed on recovery | 2.4 | `persist_submit_job_fact_ex` (P0), `wal_jobstate_t` (P0) | Batch, State | - -### Scheduler (scheduler.c, handler.c) - -#### Scheduler Dispatch Commit -| Field | Detail | -|-------|--------| -| Intent | Set `job->state = JOB_STAT_RUN` in the same operation that commits `job->allocs`/`job->accounts` | -| Requirements | 1.1, 1.2, 1.3 | - -**Responsibilities & Constraints** -- Runs inside `sched_commit_decision()`'s existing `XCE_DECISION_DISPATCH` branch, after `job->accounts` is built and before `sched_commit_job_alloc()` charges it — ordering among these three lines is not load-bearing to each other, but all three must complete before this function returns. -- Does not call `plugin_job_state_change()`/`on_job_state_change` — the picker is already notified of this dispatch via the existing, unchanged `on_dispatch()` call in `sched_allocator()`. Firing a second hook is unnecessary and was confirmed unneeded via `builtin_fcfs_on_dispatch()`'s own pendJobs-removal behavior (Research Log). -- The later, now-redundant `schedStartJob`-triggered application of `JOB_STAT_RUN` (via the existing, unmodified `core_events.c` path) is a confirmed-safe no-op against an already-RUN job — no guard is added here to suppress it; suppressing it would be unnecessary defensive code against an already-idempotent path. - -**Contracts**: State [x] - -##### State Management -- State model: `job->state` transitions PEND → RUN synchronously, in-process, no deferred-apply (the scheduler has no WAL/durability concept — `xce_sched_job_t` is a working copy, not a system of record). -- Persistence & consistency: none — a scheduler restart already rebuilds `xce_sched_job_t` from a fresh controller push, which will already reflect whatever the controller's own (durable) state is by then. -- Concurrency strategy: unchanged — `sched_commit_decision()` already runs single-threaded on the driver's decision-drain loop; no new locking is introduced. - -#### Scheduler Dispatch-Failure Handler -| Field | Detail | -|-------|--------| -| Intent | The actual fix for bug-1625: process `NotifyData_data_jobDispatchFail` instead of falling to `default: -EINVAL` | -| Requirements | 3.1, 3.2, 4.4 | - -**Responsibilities & Constraints** -- New `case NotifyData_data_jobDispatchFail:` in `sched_handle_notify()`'s switch (`handler.c`), positioned alongside the existing `schedStartJob`/`schedExitJob`/`schedFailJob` cases. -- Calls the same enqueue helper those cases already use (`sched_job_state_change(sched, notify->data.job_dispatch_fail->jobid, JOB_STAT_PEND)` — the `handler.c`-local queuing wrapper, not the dead `scheduler.c` mutator of the same name), so the actual state application flows through the live `xce_sched_driver_translate_pending_events()` → `core_events.c` path identically to every other notify-driven state change. -- No new classification logic here — `jobDispatchFail`'s meaning is narrowed (Research Log) to exclusively mean "return this job to pending"; the controller has already decided this is the right resolution before sending it. -- The existing `core_events.c` `XCE_CORE_EVENT_JOB_STATE_CHANGE` case's `!JOB_IS_RUN(new_state)` branch frees `job->accounts`/`job->allocs` automatically — this component does not duplicate that logic (Requirement 4.4 is satisfied by the existing, unmodified free-on-non-RUN path, reused rather than reimplemented). - -**Contracts**: Event [x] - -##### Event Contract -- Subscribed events: `NotifyData_data_jobDispatchFail` (existing capnp/notify type, already sent by `notify_dispatch()`, currently unhandled). -- Ordering / delivery guarantees: same as every other notify — delivered in-order on the same connection, applied on the next round's `xce_sched_driver_translate_pending_events()` drain (same latency characteristics as `schedStartJob`/`schedExitJob`/`schedFailJob` today). - -### Controller (job.c, persist.c) - -#### Controller Dispatch Commit -| Field | Detail | -|-------|--------| -| Intent | Commit `job.allocs` and transition to RUN only once `xce_startjob_req()` succeeds; append a durable fact recording the transition | -| Requirements | 2.1, 2.3, 2.4, 5.1 | - -**Responsibilities & Constraints** -- `dispatch_a_job_()`'s existing unconditional `j->job.allocs = ...` assignment (today placed *after* the `xce_startjob_req()` call but *not* gated on its return code) becomes conditional: only committed when `rc >= 0`. `j->job.state = JOB_STAT_RUN` is set in the same conditional block. -- This ordering choice supersedes requirements.md 2.1's literal "before attempting to notify the worker" phrasing (Research Log: gating on RPC success removes an entire reversal-path category that gating on decision-receipt would require). The observable guarantee requirements.md 2.1/2.3 actually asks for — the transition and the allocation commit happen as one atomic operation, and status queries reflect RUN once that operation completes — is preserved exactly; only the specific instant within `dispatch_a_job_()` moves later, to right after a successful RPC instead of right before the RPC attempt. -- On success, appends one `WAL_JOB_DISPATCH` fact (see WAL Dispatch Fact, below) recording `{jobid, state=JOB_STAT_RUN}`, in addition to the synchronous in-memory mutation — mirrors `job.allocs`'s existing synchronous-commit style, not `record_job_start_fact()`'s deferred-apply style (Research Log: no same-job racing-fact concern exists for the first post-PEND transition). -- Requirement 2.2's worker-confirmed-start fact (`record_job_start_fact()`/`WAL_JOB_START`) is entirely unmodified — it continues to fire later, from the existing worker-confirmation code path, and continues to set `start_time`. -- Requirement 2.3 (status queries reflect RUN) requires no new code — `job.state` is read directly by every existing status/listing path today; committing it earlier is sufficient. - -**Contracts**: State [x] / Batch [x] - -##### State Management -- State model: PEND → RUN, gated on RPC success; on RPC failure, state is untouched (still PEND — never entered RUN, so no reversal is needed, per the Research Log finding). -- Persistence & consistency: the in-memory mutation and the WAL fact append happen back-to-back under `lock(j, RTTI_LOCK_WRITE)` (acquired explicitly by this component, held across both the `job.allocs`/`job.state` mutation and the `persist_submit_job_fact_ex()` call, released via `unlock(j)` immediately after) — `_pad = 0` (legacy/absolute), matching `wal_jobstate_t`'s documented convention for a mutation with no staleness window. -- Concurrency strategy: **corrects a pre-existing gap (bug-1626), does not merely reuse it.** `dispatch_a_job_()`'s existing locking, traced directly (`job.c:831-963`), takes only `RTTI_LOCK_READ` on `c->jobmgr`/`c->workermgr`/the worker `w` — it never takes any lock on the job object `j` itself, despite `j->job.allocs` already being mutated today. `RTTI_LOCK_READ` is a genuine `pthread`-backed reader lock (`my_rwlock_rlock()`, confirmed in `src/lib/jobj.c:708-725`), not a liveness/reference guard, so this is a real gap, not a documentation inaccuracy. This component's mutation (and the pre-existing `job.allocs` mutation it sits beside, which this restructuring already touches) is brought under `RTTI_LOCK_WRITE`, matching the convention most other job-mutating call sites in `job.c` already correctly follow (e.g. `job.c:768,1687,1795,2058`). `job_ctrl()`'s own separate, unrelated call to `mark_job_fail()` (`job.c:1083`) has the same gap and is intentionally left unfixed here (Out of Boundary) — only this feature's own new/restructured call sites are corrected. - -##### Batch / Job Contract -- Trigger: successful `xce_startjob_req()` return within `dispatch_a_job_()`. -- Input / validation: `jobid`, target state (`JOB_STAT_RUN`) — no new validation beyond what `dispatch_a_job_()` already performs before reaching this point. -- Output / destination: one `WAL_JOB_DISPATCH` WAL entry, replicated and applied through the existing commit/apply pipeline. -- Idempotency & recovery: `WAL_JOB_DISPATCH`'s case in both `apply_committed_entry()` (live) and `apply_wal_entry()` (recovery replay) applies `state` as an absolute value (matching every `_pad == 0` job-state fact today) — replaying it any number of times is a no-op after the first, identical to how every other absolute job-state fact already behaves. - -#### Controller Dispatch-Failure Classification -| Field | Detail | -|-------|--------| -| Intent | Decide retryable vs. terminal for one of `dispatch_a_job_()`'s six post-lookup failure points, including the bounded-retry escalation, and route to the corresponding resolution mechanism — fully inside `dispatch_a_job_()`, not deferred to its caller | -| Requirements | 4.1, 4.2, 4.3 | - -**Responsibilities & Constraints** -- A small classification table maps each of `dispatch_a_job_()`'s six post-lookup failure sites to one of two classes: - - **Retryable** (transient/environmental): worker lookup failure, worker not found, worker-not-ready (`E_XCE_BAD_WORKER`), the `xce_startjob_req()` RPC call itself failing. - - **Terminal** (local/invariant): invalid allocation (`nc <= 0`), queue-exec-info lookup failure. - - The seventh failure point (jobid lookup failure, `j` never resolved) is not part of this table at all — see the dedicated bullet below. -- **Placement correction, found in the same round as the site-identity fix above**: this table has no consumer outside `job.c` — nothing on the scheduler side classifies (that assumption was superseded when classification moved to be controller-authoritative). It is declared `static` in `job.c`, not in the shared `src/include/xce/errnox.h` — a shared header is for cross-component contracts, and this classification is an internal implementation detail of one function's restructuring. -- `xce_startjob_req()`'s own RPC failure (`job.c:956-960`, the function's final failure return) is the sixth site, `XCE_DISPATCH_FAIL_SITE_RPC_FAILURE` — see Controller Dispatch Commit for the accompanying `job.allocs`-gating and `dereference(j)` fixes at that exact call site. -- **Ownership correction found during `/kiro-validate-design` round 2**: classification and resolution happen *inside* `dispatch_a_job_()`, at each of its six post-lookup failure points, using the `j`/`job_t*` pointer already held there — not in its caller `handle_jobdispatch()`. Two independent reasons: (1) `handle_jobdispatch()` only has `alloc->jobid` after `dispatch_a_job_()` returns (`alloc` is freed immediately after), so resolving there would require a second `key_find()` lookup and reopen a TOCTOU gap between the two; (2) `handle_jobdispatch()`'s existing loop unconditionally calls `notify_dispatch()` for *every* `dispatch_a_job_()` return, success (`code=0`) or failure — `notify_dispatch()` itself always emits `NotifyData_data_jobDispatchFail` as the event type regardless of `code` (confirmed by reading its body, `job.c:1108-1125`). Left unchanged, a terminal failure would produce *both* a `jobDispatchFail` notify (from the existing unconditional call) *and* a `schedFailJob` notify (from `mark_job_fail()`'s hook chain) for the same event, racing to apply PEND and EXIT to the same job depending on arrival order. -- **Fix**: `handle_jobdispatch()`'s existing per-job `notify_dispatch()` calls (both the success-path `notify_dispatch(c, ep, alloc->jobid, 0)` and the failure-path `notify_dispatch(c, ep, alloc->jobid, rc1)`) are removed from that loop. `dispatch_a_job_()` now owns notification as part of resolution: it calls `notify_dispatch()` itself, only on the retryable-failure path, and never on success or on a terminal failure (`mark_job_fail()`'s own hook chain is the sole notification for that case). `handle_jobdispatch()`'s loop becomes a thin dispatch-and-free loop with no notification logic of its own. -- **Every one of the six post-lookup failure points calls `xce_classify_dispatch_fail(site)` with its own `xce_dispatch_fail_site_t` identity** (not the raw `rc`/`errno` value that point happened to return — see Service Interface for why) — classification is never implicit in which branch executed; the table is the single source of truth, actually called at every site, not merely documentation the branch structure happens to match. This also makes the function genuinely load-bearing (not dead weight the Unit Tests section would otherwise be testing in isolation). -- **Single critical section per failure point** — deliberately structured to avoid an unlock/relock gap between the count check and the resolution it drives (a `dispatch_fail_count` read separated from its acting-upon by a lock release would be a fresh race, on top of the one this design already closes). At each of the six post-lookup failure points, inside `dispatch_a_job_()`, before that point's existing `dereference(j)`/return (or, for the four sites bug-1627 identifies, where that `dereference(j)` is being newly added): - 1. `lock(j, RTTI_LOCK_WRITE)`. - 2. Call `xce_classify_dispatch_fail(site)` with this failure point's own `xce_dispatch_fail_site_t` value. If `XCE_DISPATCH_FAIL_TERMINAL`, skip to step 3's terminal branch immediately (no counter involved). If `XCE_DISPATCH_FAIL_RETRYABLE`, increment `job.dispatch_fail_count` and capture the resulting value into a local. - 3. **Still under the same lock**: for a retryable classification whose incremented count is within the configured bound (design default: 3 consecutive failures; exact value is a task-level config decision, not hardcoded here), nothing further needs the lock — this failure resolves retryable. For a retryable classification whose count exceeds the bound, or a classification that was terminal from step 2, this failure resolves terminal **without releasing and reacquiring the lock** — call `mark_job_fail()` directly here (already satisfying its `RTTI_LOCK_WRITE`-held precondition), with a `job_failinfo_t` describing the dispatch failure (`jobid`, `code` = the original error, `detail` = a short diagnostic string; `.stage` left at 0, which correctly routes `apply_job_fail_transition()`'s switch to its `default:` branch, `job->job.state = JOB_STAT_EXIT` — confirmed by reading that function directly; the two named `.stage` cases are for post-exec failures, not applicable to a job that never started). - 4. `unlock(j)`, then `dereference(j)` (closing bug-1627's leak on the four failure points where it exists today, found during rounds 3-4 of this review — the other two, `nc <= 0` and `get_queue_exec()` failure, already correctly `dereference(j)`, unaffected by that bug — this step is uniform across all six regardless). - 5. Only now, entirely outside the lock: if step 3 resolved retryable, call `notify_dispatch(c, ep, jobid, rc)` — now the *only* caller of this function for the dispatch-failure case, so `jobDispatchFail` unambiguously means "retry." If step 3 resolved terminal (whichever of the two ways it got there), `notify_dispatch()` is **not** called — `mark_job_fail()`'s hook chain (already fired, under the lock, in step 3) is the sole notification. -- `job.dispatch_fail_count` is reset to 0 by Controller Dispatch Commit's own success path (above, under that component's own `RTTI_LOCK_WRITE`) — a fresh successful dispatch clears any prior failure history for that job. -- **How the controller side satisfies Requirement 4.4/4.5's "release stale allocation/commitment data"**: not by cleaning up after a failed commit, but by never committing in the first place — `job.allocs`/`job.state` are only ever written inside the `rc >= 0` success branch (Controller Dispatch Commit), so on any of the six classified failure paths there is nothing to release; the requirement is satisfied vacuously by construction, not by an explicit cleanup step. This is the direct benefit of the RPC-gated commit ordering decision in `research.md`. -- `dispatch_a_job_()`'s earliest failure point (jobid lookup failure — two code variants, `key_find()` itself failing or returning a NULL `j`, before `j` is ever resolved) is a distinct case outside this table entirely: there is no `job_t*` to mark failed or count against. This is not a job-state resolution problem, it is "the job does not exist" — the function's existing `return rc`/`return -errno` is sufficient, unchanged. Requirement 4's classification and this component's locking/counting/notification logic apply only to the six failure points reached *after* `j` is successfully resolved and referenced. - -**Contracts**: Service [x] - -##### Service Interface -```c -/* - * Classifies one of dispatch_a_job_()'s six post-lookup failure points as - * retryable or terminal. Deliberately keyed on WHICH failure point was hit - * (an explicit site identity), not on the raw errno-style `int rc` that - * point returns -- key_find() (workermgr) and get_queue_exec() are two - * distinct, independently-implemented failure sources that are not - * guaranteed to return disjoint error codes (e.g. both could plausibly - * return a generic -ENOENT for unrelated reasons), so classifying by rc - * value alone would be ambiguous whenever two sites happen to collide. - * Does not itself mutate job state, take any lock, or send notifications -- - * called from within dispatch_a_job_() itself, at each site, which uses the - * result to choose between notify_dispatch() (retry) and mark_job_fail() - * (terminal). - */ -typedef enum { - XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP = 0, /* workermgr key_find() rc<0 */ - XCE_DISPATCH_FAIL_SITE_WORKER_NULL = 1, /* worker not found (w==NULL) */ - XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY = 2, /* E_XCE_BAD_WORKER */ - XCE_DISPATCH_FAIL_SITE_INVALID_ALLOC = 3, /* nc <= 0 */ - XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC = 4, /* get_queue_exec() failure */ - XCE_DISPATCH_FAIL_SITE_RPC_FAILURE = 5, /* xce_startjob_req() rc<0 */ -} xce_dispatch_fail_site_t; - -typedef enum { - XCE_DISPATCH_FAIL_RETRYABLE = 0, - XCE_DISPATCH_FAIL_TERMINAL = 1, -} xce_dispatch_fail_class_t; - -xce_dispatch_fail_class_t xce_classify_dispatch_fail(xce_dispatch_fail_site_t site); -``` -- Preconditions: `site` is one of the six named enumerators. **Corrected during task 1.1 implementation review**: a default-less `switch` does NOT compel a compile error in plain C without `-Werror` (this project sets none) — it only emits a non-fatal `-Wreturn-type` warning and falls through to undefined behavior on an unhandled value. The actual implementation instead follows this codebase's established fail-loud convention (matching `persist.c`'s `wal_classify()`, `persist.c:6030-6037`): a `default:` arm that ERROR-logs and returns a safe `XCE_DISPATCH_FAIL_TERMINAL`, with `assert(0 && ...)` gated to `#ifdef XCE_TESTING` only, so a future unhandled site fails loud in test builds and degrades safely (never silently misclassifies as retryable) in production. -- Postconditions: pure function, no side effects; the caller separately applies the bounded-retry override. `WORKER_LOOKUP`, `WORKER_NULL`, `WORKER_NOT_READY`, `RPC_FAILURE` classify `XCE_DISPATCH_FAIL_RETRYABLE`; `INVALID_ALLOC`, `QUEUE_EXEC` classify `XCE_DISPATCH_FAIL_TERMINAL`. -- Invariants: the same `site` always classifies the same way — no hidden state, no time-dependence, no dependence on the specific `rc`/`errno` value that accompanied it. - -### WAL Dispatch Fact (persist.c, wal_types.h) - -#### WAL Dispatch Fact -| Field | Detail | -|-------|--------| -| Intent | Durable record of the dispatch-time RUN transition, so a controller failover recovers the same state a live run would have reached | -| Requirements | 2.4 | - -**Responsibilities & Constraints** -- New `wal_entry_type` value `WAL_JOB_DISPATCH = 12` (next free value in the job-fact range, after `WAL_JOB_ARRAY_SUBMIT = 11`). -- Reuses the existing `wal_jobstate_t{jobid, state, _pad}` payload verbatim — no new struct. `_pad` is always `0` for this type (the "legacy: state is absolute value" convention `wal_jobstate_t`'s own header comment documents), matching the synchronous, non-deferred-apply commit style Controller Dispatch Commit uses. -- **Two separate functions, both needing the new case** — found by tracing precisely rather than assuming a single switch: `apply_wal_entry()` (`persist.c:3375+`) has its own fall-through group at `persist.c:3452-3466` (`WAL_JOB_START` through `WAL_JOB_RUSAGE`, including `WAL_JOB_FAIL` — applies every type as an absolute overwrite, unconditionally). `apply_committed_entry()` (`persist.c:3570+`, the live path invoked immediately when a WAL entry commits during normal operation — the one `raft-apply-decoupling` actually introduced the `_pad`-discriminated routing for) has its *own separate* fall-through group at `persist.c:3700-3706` (`WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_JOB_STOP`, `WAL_JOB_RESUME`, `WAL_JOB_RUSAGE`, `WAL_JOB_CKPT`, `WAL_JOB_PREASON` — deliberately excluding `WAL_JOB_FAIL`/`WAL_JOB_UPDATE`, which get their own `_pad`-aware handling immediately after). That group's own comment states the exact rationale this design already relies on: these types' payloads are "deterministic/state-independent... an absolute `state` baked at receipt time never goes stale" — `WAL_JOB_DISPATCH` fits this group by the same reasoning (Research Log: no same-job racing-fact concern for the first post-PEND transition). `WAL_JOB_DISPATCH` joins both fall-through lists — no new apply-time function or standalone case body is introduced in either. **Corrected during task 1.2 implementation review**: `apply_wal_entry()` is NOT actually "the recovery/replay-time path" in the current architecture — `persist_recover()`'s real boot-time replay (`replay_wal()`) and the standby catch-up path (post the 2026-07-10 `ha_apply_single_entry` refactor) both call `apply_committed_entry()` exclusively; `apply_wal_entry()` has zero production call sites today. Adding the case to both functions is still correct (matches every sibling `WAL_JOB_*` type's identical dual-registration, harmless dead-code parity, and costs nothing if `apply_wal_entry()` is ever revived) — only the prose description of what `apply_wal_entry()` currently *is* was wrong. -- Does not carry or restore `job.allocs`/`job.accounts` — those remain non-durable in-memory fields, unchanged from today's behavior (Boundary: Out of Boundary does not list a new durability guarantee for allocation data itself, only for the state transition). - -**Contracts**: Batch [x] / State [x] - -##### Batch / Job Contract -- Trigger: Controller Dispatch Commit's success path. -- Input / validation: `jobid`, `state = JOB_STAT_RUN`. -- Output / destination: standard WAL append → replication → commit pipeline, identical to every other job-state fact. -- Idempotency & recovery: absolute-value apply, safe to replay any number of times. - -## Data Models - -### Domain Model -- No new domain entities. `job_t.dispatch_fail_count` (via `xce_job_t`'s wrapper, not the wire type) is a new scalar field on an existing entity, not a new aggregate. This counter is controller-side only — the scheduler has no equivalent field, since only the controller's copy is read for classification (see Controller Dispatch-Failure Classification); a scheduler-side mirror would be dead weight with no consumer. - -### Logical Data Model -- `job_t` (`xce/types.h`, the `{xce_job_t job; uint64_t submit_lsn;}` wrapper — **not** `xce_job_t` itself): `+ uint32_t dispatch_fail_count` — in-memory only, following `submit_lsn`'s exact precedent ("recomputed... never serialized"). A controller failover resets this to 0, which is a safe, conservative degradation (more retries allowed post-failover, never fewer, and never a misrouted terminal/retry decision). -- `wal_entry_type` (`wal_types.h`): `+ WAL_JOB_DISPATCH = 12`, payload = existing `wal_jobstate_t`. - -## Error Handling - -### Error Strategy -Two independent failure surfaces, each already governed by an existing discipline this design does not change: the scheduler-side notify handler follows the same `-ENOENT`/`-EINVAL` conventions every sibling case in `sched_handle_notify()` already uses; the controller-side classification is keyed on an exhaustive, compiler-checked failure-site enum (no default/unrecognized branch is reachable) rather than on a possibly-ambiguous raw return code. - -### Error Categories and Responses -- **Unrecognized dispatch-fail notify jobid** (scheduler side): same as every existing case — `sched_job_state_change()`'s enqueue wrapper does not validate jobid existence at enqueue time; the live apply path's `sched_get_job() == NULL` branch (already present in `core_events.c`) logs a WARN and no-ops, unchanged. -- **A new post-lookup failure point added to `dispatch_a_job_()` in the future without updating the classification** (controller side): fails to compile — `xce_classify_dispatch_fail()`'s `switch` over `xce_dispatch_fail_site_t` has no `default` case, so a new enumerator with no matching `case` is a compiler error, not a silent runtime default. -- **Bounded-retry exhaustion**: not an error path — a deliberate, expected transition to terminal, using the exact same `mark_job_fail()` call as any other terminal classification. - -### Monitoring -No new logging infrastructure — `mark_job_fail()`'s existing hook-fired logging and `sched_handle_notify()`'s existing per-case `TRACE`/`WARN` conventions already cover both new code paths; only the log lines' content changes to reference the new case names. - -## Testing Strategy - -### Unit Tests -- `sched_commit_decision()`'s DISPATCH branch sets `job->state == JOB_STAT_RUN`, in addition to the existing `accounts`/`allocs` assertions. -- `sched_handle_notify()`'s new `jobDispatchFail` case enqueues an `XCE_SCHED_OP_JSTATE_CHANGE` targeting `JOB_STAT_PEND`; a follow-up drain via `xce_sched_driver_translate_pending_events()`/`core_events.c` confirms `job->accounts`/`job->allocs` are freed (reusing the existing free-on-non-RUN assertion pattern from other state-change tests). -- `dispatch_a_job_()`: for each of the seven failure returns (jobid lookup's two variants, plus the six post-lookup sites), confirm `job.state` remains PEND and `job.allocs` is not committed; for a successful `xce_startjob_req()`, confirm both are committed together with one `WAL_JOB_DISPATCH` fact appended. For the four retryable-class paths, additionally assert `j`'s reference count returns to its pre-call value (regression test for bug-1627's leak). -- `xce_classify_dispatch_fail()`: table-driven test covering all six `xce_dispatch_fail_site_t` enumerators against their documented classification (four retryable, two terminal). -- Bounded-retry escalation: three consecutive retryable failures for the same job route to `notify_dispatch()`; the fourth (exceeding the design-default bound of 3) routes to `mark_job_fail()` instead. - -### Integration Tests -- End-to-end: scheduler commits a DISPATCH decision → controller processes it successfully → scheduler's `sched_account_jobs()` includes the job's resource consumption from that same round onward, across a simulated multi-round gap before the worker's actual confirmation arrives (this is bug-1624's regression test). -- End-to-end retryable failure: controller's `xce_startjob_req()` fails (simulated worker-unreachable) → scheduler receives `jobDispatchFail` → job returns to PEND → scheduler's picker reconsiders it in a later round. -- End-to-end terminal failure: controller's jobid/allocation lookup fails → `mark_job_fail()` fires → scheduler receives the existing `schedFailJob` notify (unmodified path) → job resolves to EXIT with `accounts`/`allocs` freed. Assert the scheduler does **not** also receive a `jobDispatchFail` notify for this same failure (regression test for the round-2 double-notify finding) — `handle_jobdispatch()`'s loop must not call `notify_dispatch()` for a failure `dispatch_a_job_()` already resolved as terminal. -- WAL durability: append a `WAL_JOB_DISPATCH` fact, simulate a controller restart/replay, confirm `job.state == JOB_STAT_RUN` is recovered without requiring the (separate, unmodified) `WAL_JOB_START` fact to also be present. - -### Performance/Load -- Not applicable — this feature adds O(1) work per dispatch decision and per notify; no new per-round scan or O(n) pass is introduced (`sched_account_jobs()` itself is unmodified). - -## Supporting References -- `.kiro/specs/scheduler-dispatch-commit/research.md` — full discovery findings, the three design decisions this document builds on (dead-code path identification, RPC-gated commit ordering, controller-authoritative classification), and risk mitigations. diff --git a/.kiro/specs/scheduler-dispatch-commit/requirements.md b/.kiro/specs/scheduler-dispatch-commit/requirements.md deleted file mode 100644 index 81136c4e..00000000 --- a/.kiro/specs/scheduler-dispatch-commit/requirements.md +++ /dev/null @@ -1,71 +0,0 @@ -# Requirements Document - -## Project Description (Input) - -**Who has the problem:** scheduler and controller maintainers, and cluster operators who rely on accurate live resource accounting and (eventually) preemption. Surfaced while investigating bug-1611/1613-adjacent behavior and while designing the sibling `scheduler-sim-eager-reconciliation` spec. - -**Current situation:** `job->state`'s transition to RUN only happens later, via a separate worker-confirmation event — scheduler's `sched_job_state_change()` (`scheduler.c:963-988`) fires only on the controller's `NotifyData_data_schedStartJob` notify; controller's `job.state` flips to RUN only via `on_job_start()`/`record_job_start_fact()` (`job.c`), driven by the worker's own confirmation. Meanwhile, resource-commitment data is set much earlier and independently: scheduler's `job->accounts`/`job->allocs` are set synchronously inside `sched_commit_decision()`'s DISPATCH branch (`scheduler.c:1441-1446`), and controller's `job.allocs` is set synchronously inside `dispatch_a_job_()` (`job.c:831-963`) — both well before any worker confirmation. - -This split (intentional and, on its own, correct — "scheduled/dispatched" and "confirmed running" are legitimately different signals) has a concrete correctness consequence: every scheduler round, `xce_core_begin_session()` (`core_session.c:66-108`) unconditionally zeroes `core->sharedInUse`/`worker->resInUse`, and `sched_account_jobs()` (`scheduler.c:1258-1292`) re-derives those totals only from jobs where `JOB_IS_RUN(job->state)` is true — silently skipping a job that is dispatched (accounts already set) but still PEND (not yet confirmed). Normal-mode rounds run on a 1-second cadence (`main.c:453-460`), so any dispatch-to-confirmation latency over ~1s (ordinary for a real worker RPC + fork/exec) wipes that job's resource charge at the very next round and does not restore it until confirmation lands. The allocator's admission check (`sched_check_host_satisfy_`/`sched_check_shared_satisfy_`, `resource.c:252-370`, called from `seqjob_allocator.c:130`) reads exactly this same `resInUse`/`sharedInUse`, with no separate reserved/committed pool (`xce_sched_job_t` has only `state`/`allocs`/`accounts`, `sched/types.h:96-116`) — so the resource looks free again and can be double-granted to a second job. This is a genuine live over-commit/double-booking risk in normal-mode operation, logged as **bug-1624** (unfixed). - -Separately, and necessary for any fix to be safe: the controller already reports dispatch-RPC failure back to the scheduler via `NotifyData_data_jobDispatchFail` (`notify_dispatch()`, `job.c:1108-1125`), but the scheduler's `sched_handle_notify()` switch (`handler.c:499-544`) has zero case for it — it silently falls to `default: -EINVAL`. So today a dispatch failure leaves `job->accounts`/`job->allocs` (scheduler) and `j->job.allocs` (controller) stale forever with no cleanup, logged as **bug-1625** (unfixed). There is also currently no code path anywhere in `job.c` that ever sets a job's state back to `JOB_STAT_PEND` after dispatch — `mark_job_fail()`/`apply_job_fail_transition()` only ever resolves forward onto a terminal EXIT/DONE-ish state; a genuine retry-back-to-PEND path does not exist yet. - -**What should change:** -1. Redefine the RUN transition to occur synchronously at DISPATCH-decision-commit time rather than at worker-confirmation time, on BOTH scheduler (inside `sched_commit_decision()`'s DISPATCH branch, alongside where `accounts`/`allocs` already get set) and controller (inside `dispatch_a_job_()`, alongside where `job.allocs` already gets set) — so neither side's internal state diverges from the other for the same job, and `sched_account_jobs()`'s existing `JOB_IS_RUN` gate naturally includes a dispatched job from the round it was dispatched, fixing bug-1624 with no change to that gate's own logic. -2. Build the dispatch-failure handling this redefinition requires to be safe: scheduler-side handling for `NotifyData_data_jobDispatchFail` (fixing bug-1625), a retryable-vs-terminal failure classification (which failure codes/conditions resolve back to PEND for retry vs. forward to EXIT/FAIL permanently — no such classification exists today), and correct reversal of state/accounts/allocs/`job.allocs` on both scheduler and controller when a dispatch is retried back to PEND. -3. Document (not implement delivery for) that a job holding committed-but-unconfirmed resources is a legitimate future preemption victim candidate under the new RUN definition — `scheduler-preemption` (sibling spec, still at requirements-generated) currently assumes in its AC6/AC9 that every RUN victim has a live, signalable process; that assumption is worth revising before that spec proceeds to design. Actually implementing preemption delivery/victim selection remains `scheduler-preemption`'s own boundary. - -## Sequencing - -The user has directed that this spec land **before `scheduler-allocation`** begins implementation. `scheduler-allocation`'s new iterator-based `try_alloc_on_host()` will read the same `resInUse`/`sharedInUse` totals bug-1624 corrupts, and its planned `XCE_ALLOC_PREEMPT` borrowing mode interacts with what counts as committed/borrowable — building the new allocator on top of today's broken round-boundary accounting would compound rather than fix the problem. - -## Boundary Context - -- **In scope**: synchronous RUN-state transition at dispatch-decision-commit time on scheduler (`sched_commit_decision()`) and controller (`dispatch_a_job_()`); scheduler-side handling of `NotifyData_data_jobDispatchFail`; retryable-vs-terminal dispatch-failure classification; PEND-reversion of state/accounts/allocs on both scheduler and controller for retryable failures; a documented note (not implementation) that a RUN-but-unconfirmed job is a valid future preemption victim candidate. -- **Out of scope**: `scheduler-controller-msg-gap` (already implementation-complete — bug-1625 is thematically identical but a new, separately-discovered gap; not a reopening of that closed spec); `scheduler-sim-decision-parity` (implementation-complete, do not reopen); `scheduler-sim-eager-reconciliation` (sibling, in-flight, explicitly simulation-mode-only and explicitly excludes normal-mode behavior changes — this spec is the normal-mode counterpart but a distinct spec, not a merge); `scheduler-preemption`'s own preemption delivery/victim-selection mechanics (cross-referenced via a pending-revision note, not merged in); `scheduler-allocation`'s iterator/plan-generation work itself (unrelated axis — HOW allocation plans are computed, not WHEN state/accounting transitions happen); any change to how a job's state is displayed to end users beyond what naturally falls out of the state-transition-timing change; PEND job candidacy/picker-selection logic changes beyond what the retry-to-PEND path requires. -- **Adjacent expectations**: this spec extends `scheduler-sim-decision-parity`/`scheduler-sim-eager-reconciliation`'s and `scheduler-controller-msg-gap`'s existing mechanisms without reopening them. `scheduler-preemption`'s requirements.md received a cross-reference note (2026-08-23) flagging its AC6/AC9 as pending revision once this spec's design is settled. Because simulation mode replays whatever the controller actually recorded (it does not independently decide state-transition timing), a recording captured *after* this feature ships will naturally carry the dispatch decision and the running-state transition in the same round — narrowing, though not eliminating, the cross-round case `scheduler-sim-eager-reconciliation` handles for older or otherwise-delayed recordings. This is a beneficial side effect to note, not a dependency either spec should design around. - -## Requirements - -### Requirement 1: Synchronous running-state transition on dispatch commit (Scheduler) -**Objective:** As a scheduler maintainer relying on accurate live resource accounting, I want a job's state to become running as soon as the Scheduler commits its dispatch decision, so that per-round resource-in-use totals never lose track of a job's committed resources solely because worker confirmation has not yet arrived. - -#### Acceptance Criteria -1. When the Scheduler commits a dispatch decision for a job, the Scheduler shall transition that job to a running state in the same operation that records the job's resource allocation. -2. While a job's dispatch decision has been committed and no dispatch failure has since been reported for it, the Scheduler shall include that job's resource consumption in worker and shared resource-in-use totals from the round it was dispatched onward, regardless of how many rounds elapse before the worker confirms the job has started. -3. The Scheduler shall not change the existing rule that only jobs in a running state contribute to a round's resource-in-use totals; this feature changes only when a job reaches that state, not how totals are computed from it. - -### Requirement 2: Synchronous running-state transition on dispatch receipt (Controller), consistent with Scheduler -**Objective:** As a maintainer relying on scheduler-side and controller-side bookkeeping staying consistent for the same job, I want the Controller to mark a job running at the same moment it processes the Scheduler's dispatch decision, so the two components never disagree about whether a job has begun consuming its allocated resources. - -#### Acceptance Criteria -1. When the Controller's attempt to notify the assigned worker of a dispatch decision succeeds, the Controller shall transition that job to a running state in the same operation that commits the job's resource allocation. The transition and the allocation commit shall not depend on any later worker confirmation of actual process start. -2. The Controller shall continue to record the worker's confirmation that a job has actually started, including the job's actual start time, as a separate durable fact, independent of and in addition to the state transition performed under Criterion 1. -3. When a status query or job listing is served after a job's dispatch decision has been processed, the Controller shall report that job's state as running, whether or not the worker has yet confirmed the job started. -4. The Controller shall apply the state transition in Criterion 1 through the same durability and failover-consistency discipline it already applies to a job's other state transitions, so the transition is neither lost nor left divergent across a controller failover. - -### Requirement 3: Scheduler-side dispatch-failure handling -**Objective:** As a maintainer relying on the Scheduler not silently losing track of failed dispatches, I want the Scheduler to actually process the Controller's existing dispatch-failure notification, so a dispatch that never reaches a worker does not leave stale committed-resource bookkeeping behind indefinitely. - -#### Acceptance Criteria -1. When the Scheduler receives a dispatch-failure notification for a job from the Controller, the Scheduler shall process that notification rather than discard it as an unrecognized event. -2. When the Scheduler processes a dispatch-failure notification for a job, the Scheduler shall resolve that job according to the retryable-vs-terminal classification defined in Requirement 4. - -### Requirement 4: Retryable-vs-terminal dispatch-failure resolution -**Objective:** As a cluster operator, I want a dispatch failure caused by a transient or environmental condition to be retried automatically, and a dispatch failure caused by a locally-invalid decision to fail immediately without endless retrying, so failed dispatches resolve correctly in both directions. - -#### Acceptance Criteria -1. When a dispatch failure is caused by a transient or environmental condition — for example the assigned worker being unreachable or not in a ready state, or the dispatch request itself failing to be delivered — the Scheduler and Controller shall return the job to a pending state, making it eligible for the Scheduler to attempt dispatch again in a subsequent round. -2. When a dispatch failure is caused by a locally-invalid decision — for example the job or its allocation no longer being resolvable by the Controller — the Scheduler and Controller shall resolve the job to a terminal failed state without returning it to a pending state. -3. While a job has accumulated consecutive transient/environmental dispatch failures with no intervening successful dispatch, when the number of such failures reaches a bounded limit, the Scheduler and Controller shall resolve the job to a terminal failed state rather than returning it to pending again. The exact limit is a design-phase decision; this requirement only constrains it to be bounded, never infinite. -4. When a job is returned to a pending state following a dispatch failure, the Scheduler and Controller shall each release the resource-allocation and commitment data associated with the failed dispatch attempt, so a subsequent dispatch decision for that job is not blocked by stale data from the failed attempt. -5. When a job is resolved to a terminal failed state following a dispatch failure, the Scheduler and Controller shall each release the resource-allocation and commitment data associated with the failed dispatch attempt. - -### Requirement 5: No regression to existing guarantees -**Objective:** As a maintainer, I want this feature to preserve existing dispatch, accounting, and lifecycle guarantees outside the specific gap it closes. - -#### Acceptance Criteria -1. When a job's dispatch succeeds and the worker later confirms the job has started, the Controller shall continue to record the confirmed-start fact exactly as it does today, in addition to (not instead of) the earlier state transition performed under Requirement 2. -2. The Scheduler shall not apply this feature's dispatch-time state transition when operating in simulation/replay mode; simulation mode's own recorded-event replay path is unaffected by this feature. -3. Existing scheduler and controller job-lifecycle test suites shall continue to pass, except for assertions that directly depended on the state transition occurring only at worker-confirmation time, which shall be updated to reflect the new timing. -4. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, state divergence, or liveness stall attributable to this feature. diff --git a/.kiro/specs/scheduler-dispatch-commit/research.md b/.kiro/specs/scheduler-dispatch-commit/research.md deleted file mode 100644 index 34d4fc84..00000000 --- a/.kiro/specs/scheduler-dispatch-commit/research.md +++ /dev/null @@ -1,76 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `scheduler-dispatch-commit` -- **Discovery Scope**: Extension (existing scheduler + controller subsystems) -- **Key Findings**: - - `scheduler.c`'s `sched_update()`/`sched_job_state_change()` (the notify-driven state-change apply path) is **dead code in the live production driver** — `main.c`'s loop calls only `xce_sched_driver_run_normal_mode_round()`, which drains pending controller events via `xce_sched_driver_translate_pending_events()` → `xce_core_push_events()` → `core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` case, a parallel, independently-maintained mirror of the same logic. Design must target the live path, not the vestigial one. - - `builtin_fcfs_on_job_state_change()` is confirmed idempotent for a repeated RUN notification (comment: "a no-op, via `ordq_del`'s 'not found' return, if it wasn't a candidate already — e.g. two non-pending transitions in a row"). A later, now-redundant `schedStartJob` confirmation re-applying `JOB_STAT_RUN` to an already-RUN job is therefore safe with zero code change to that path. - - `sched_commit_decision()`'s DISPATCH branch (`scheduler.c:1441-1447`) is the correct, still-live integration point for the scheduler-side transition — it is the sole call site of `sched_charge_account()`/`job->accounts` assignment already, and is invoked from the decision callback (`xce_sched_driver_on_decision()`), a separate (and still-live) mechanism from the notify-apply path above. - - `dispatch_a_job_()` (`job.c:831-963`) currently sets `j->job.allocs` **unconditionally**, after attempting `xce_startjob_req()`, regardless of that call's return code — an existing, unrelated latent bug (stale allocation left behind on RPC failure) that Requirement 4's cleanup obligation subsumes rather than leaving as a separate patch. - - The `limit` plugin (`plugins/limit.c`) does not gate any of its own usage accounting on `JOB_IS_RUN`/`job->state` (`grep` found zero matches) — it is not a second instance of bug-1624's class of bug, and is correctly out of this spec's boundary. - - `commit_resolve_dispatch_t`/`commit_resolve_dispatch_entry` (`commit.c`) is the generic WAL-commit continuation dispatch primitive from `raft-apply-decoupling` — a naming coincidence ("dispatch" = vtask dispatch, not job dispatch), not a pre-built mechanism for this feature. The relevant precedent instead is `record_job_fail_fact()`'s "durable append only, no synchronous job mutation, apply-time recompute" pattern. - -## Research Log - -### Which driver code path actually applies a `JOB_STATE_CHANGE` in production -- **Context**: Requirement 1 requires the scheduler's dispatch-commit transition to interoperate correctly with the *existing* RUN-confirmation transition. Needed to find where that existing transition is actually applied today, not just where an equivalently-named function exists. -- **Sources Consulted**: `src/scheduler/src/main.c` (production loop), `src/scheduler/src/driver.c` (`xce_sched_driver_run_normal_mode_round`, `xce_sched_driver_translate_pending_events`), `src/scheduler/src/scheduler.c` (`sched_update`, `sched_job_state_change`), `src/scheduler/src/core_events.c` (`XCE_CORE_EVENT_JOB_STATE_CHANGE` case). -- **Findings**: `main.c`'s `while(1)` loop calls only `xce_sched_driver_run_normal_mode_round(&sched)`, whose body never calls `sched_update()`. `sched_update()` (and therefore `sched_job_state_change()`, only reachable through it) has no live caller in the current driver architecture. The actual apply path is `xce_sched_driver_translate_pending_events()` → `xce_core_push_events()` → `core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` case, documented there as mirroring `sched_job_state_change()` "logic-identical, signature-relocated." -- **Implications**: The design does not modify or reuse `scheduler.c`'s `sched_job_state_change()`/`sched_update()`. All discussion of "the existing RUN-confirmation path" in the design refers to `core_events.c`'s case, which needs **no code change** — it already behaves correctly (idempotently) when applied to a job whose state is already RUN. - -### Ordering of the controller-side state transition relative to the worker RPC -- **Context**: Requirement 2 AC1 (as drafted) says the transition happens "before attempting to notify the assigned worker." Tracing `dispatch_a_job_()`'s actual failure points revealed a simpler alternative. -- **Findings**: `dispatch_a_job_()` has five failure points that return before ever attempting `xce_startjob_req()` (jobid lookup, invalid alloc, worker lookup, worker-not-ready, queue-exec-info lookup) and one failure point at the RPC call itself. If the state transition (and `job.allocs` commitment) is gated on `xce_startjob_req()` returning success, none of these six failure modes ever produces a RUN transition that needs to be reversed — the job simply remains PEND, exactly where it already was. This eliminates an entire class of "revert RUN back to PEND" complexity on the controller side that would otherwise be needed to match Requirement 4. -- **Implications**: Design decision below (**Gate controller commit on RPC outcome**) supersedes the literal "before attempting to notify" phrasing in requirements.md AC1 — the *intent* (transition and allocation commit as one atomic, worker-latency-independent operation, not gated on separate confirmation) is preserved; only the specific ordering relative to the RPC changes, and only because it removes a whole reversal path rather than adding one. Flagged for requirements.md wording cleanup during task generation, not a scope change. - -## Design Decisions - -### Decision: Gate the controller's dispatch-time RUN transition and allocation commit on RPC success, not on decision receipt -- **Context**: Where exactly within `dispatch_a_job_()` to place the state transition, and whether dispatch-RPC failure needs a state-reversal path. -- **Alternatives Considered**: - 1. Transition to RUN immediately on receiving the decision, before any worker contact — matches the literal AC1 wording, but requires a full reversal path (RUN → PEND) for every one of `dispatch_a_job_()`'s six failure points. - 2. Transition to RUN (and commit `job.allocs`) only after `xce_startjob_req()` returns success — no reversal path needed for any failure point; the job simply never leaves PEND on failure. -- **Selected Approach**: Option 2. -- **Rationale**: Every one of `dispatch_a_job_()`'s failure points already returns before or at the RPC call. Gating on RPC success collapses "was RUN, must revert" into "never became RUN," which is strictly simpler and removes a whole category of possible bugs (a reversal that's forgotten on one of the five early-return paths). -- **Trade-offs**: A job dispatched to a worker that never replies at all (hangs, rather than erroring) would not reach RUN until the RPC layer's own timeout fires — acceptable, since that is already how `job.allocs` commitment behaves today, and no requirement asks for a stronger guarantee under a hung RPC. -- **Follow-up**: None — this closes the finding above rather than opening a new question. - -### Decision: Reuse `mark_job_fail()`/WAL-fact-append pattern for durability of the new transition, rather than the deferred-apply-first pattern `record_job_start_fact()` uses -- **Context**: Requirement 2 AC4 requires the transition to survive a controller failover. `record_job_start_fact()` uses a "durable append only, no synchronous mutation, recompute at apply time" pattern specifically because a JOB_START fact can race against other pending facts for the same job (merge-at-apply-time, per `wal_types.h`). The dispatch-commit transition has no such racing-fact concern: it is the *first* transition after a job leaves PEND, triggered synchronously by the one function that already holds the authoritative decision. -- **Alternatives Considered**: - 1. Full deferred-apply (submit fact, mutate only when the WAL entry is later applied) — correct but makes the transition NOT synchronously observable, conflicting with Requirement 2 AC1/AC3's "before/from" ordering language. - 2. Synchronous in-memory mutation (matching today's `j->job.allocs` assignment) plus a separate, best-effort durable WAL fact append recording the same transition for recovery purposes. -- **Selected Approach**: Option 2. -- **Rationale**: Matches this function's own existing precedent for `job.allocs` (synchronous, not deferred-apply) while still giving the transition a durable record for post-failover recovery, without introducing a same-job merge-ordering hazard `record_job_start_fact()`'s pattern exists to solve elsewhere. -- **Trade-offs**: A crash in the narrow window after the in-memory mutation but before the WAL fact durably commits would leave the pre-crash primary's own view momentarily ahead of what a promoted standby recovers to. Bounded by the existing commit-durability guarantee (`ha-commit-durability`) once the fact is submitted; the narrow pre-submission window is the same class of gap `job.allocs`'s own non-durable commit already accepts today, not a new one this feature introduces. -- **Follow-up**: Design phase specifies the new WAL fact's contract (extends `wal_jobstate_t`'s existing `_pad`-discriminated shape, precedent: `WAL_JOB_START`/`WAL_JOB_FAIL`/`WAL_JOB_UPDATE`). Exact bit layout is implementation-task detail. - -### Decision: Controller classifies once, authoritatively; `jobDispatchFail` becomes exclusively the retry signal, reusing the existing `schedFailJob` path for terminal resolution -- **Context**: Requirement 4 requires the same failure to resolve the same way (retry vs. terminal) regardless of which component acts on it. The controller observes a dispatch failure synchronously and directly (it makes the RPC); the scheduler only learns of it later, asynchronously. -- **Alternatives Considered**: - 1. Each side classifies independently against a shared error-code table — risks drift if the bounded-retry counter (Requirement 4 AC3) is tracked separately on each side. - 2. Controller classifies once (owns the freshest information and the durable retry counter) and sends the scheduler an explicit resolution field alongside the existing `jobDispatchFail` notify. - 3. Controller classifies once, and routes the two outcomes through two *already-distinct* existing channels instead of one overloaded one: terminal failures call the existing `mark_job_fail()` (`job.c:1844`), whose already-wired `HOOK_DETAIL_JOB_FAIL` → `sched.c` hook handler already sends `NotifyData_data_schedFailJob` — a notify the scheduler's `sched_handle_notify()` **already handles today**, unmodified, correctly freeing `job->accounts`/`job->allocs` on the transition away from RUN. Retryable failures keep using the existing (currently unhandled) `notify_dispatch()`/`NotifyData_data_jobDispatchFail`, whose meaning narrows to exclusively "return this job to pending." -- **Selected Approach**: Option 3. -- **Rationale**: Discovered while tracing `mark_job_fail()`'s hook chain (`job.c` → `HOOK_DETAIL_JOB_FAIL` → `sched.c:502` → `NotifyData_data_schedFailJob`) — the terminal path already has a fully-working, already-scheduler-handled notification mechanism with zero new code required. This reduces the scheduler-side new-code surface (bug-1625's actual fix) to a single new `case` in `sched_handle_notify()`'s switch that unconditionally applies `JOB_STAT_PEND` — no resolution field, no classification logic, no new payload shape needed on the scheduler side at all. -- **Trade-offs**: None identified — this is strictly less new surface than Option 2 for an equivalent guarantee. -- **Follow-up**: Design phase specifies exactly where the controller-side classification (including the bounded-retry check) is computed and where `dispatch_fail_count` is stored/reset. - -## Addendum (2026-08-23, found during `/kiro-validate-design`) - -- **`dispatch_a_job_()` and `job_ctrl()`'s existing call to `mark_job_fail()` never take `RTTI_LOCK_WRITE` on the job object, despite `apply_job_fail_transition()`'s own doc comment requiring it** (a genuine `pthread` rwlock, `my_rwlock_wlock()`, confirmed in `src/lib/jobj.c:708-725` — not a liveness/reference guard). Traced directly: `dispatch_a_job_()` (`job.c:831-963`) takes only `RTTI_LOCK_READ` on the containing maps and the worker, never on the job itself; `job_ctrl()` releases its own `RTTI_LOCK_READ` on the job (`job.c:1016`) well before calling `mark_job_fail()` (`job.c:1083`). Logged as bug-1626, pre-existing, independent of this feature. This directly affected the "no staleness window" locking claim originally drafted for design.md's Controller Dispatch Commit component — corrected there to explicitly acquire `RTTI_LOCK_WRITE` around this feature's own new/restructured mutations, closing the gap for those call sites specifically (`job_ctrl()`'s own separate call site is left as the still-open, out-of-boundary instance of bug-1626). -- **The scheduler-side `dispatch_fail_count` field originally planned for `xce_sched_job_t` was removed** — nothing in the design ever reads it (only the controller's copy drives classification, per the Decision above), so it was dead weight. Caught during the same validation pass. -- **`notify_dispatch()` (`job.c:1108-1125`) unconditionally emits `NotifyData_data_jobDispatchFail` regardless of whether `code` is 0 (success) or negative (failure) — confirmed by reading its body.** `handle_jobdispatch()`'s existing loop calls it for *every* `dispatch_a_job_()` return, success or failure alike. Combined with classification originally being planned for `handle_jobdispatch()` (the caller), this would have produced two conflicting notifies for a terminal failure — `jobDispatchFail` (apply PEND) racing against `schedFailJob` (apply EXIT) for the same event. Fixed by moving classification and all dispatch-outcome notification ownership fully inside `dispatch_a_job_()` (which also resolves the redundant-lookup/TOCTOU concern a caller-side classification would have had, since `handle_jobdispatch()` only retains `alloc->jobid` after `dispatch_a_job_()` returns) and removing `handle_jobdispatch()`'s own `notify_dispatch()` calls entirely. Found during round 2 of `/kiro-validate-design`. -- **`persist.c` has TWO separate functions that each need their own new `WAL_JOB_DISPATCH` case, not one** — `apply_wal_entry()` (`persist.c:3375+`, recovery/replay-from-disk path) and `apply_committed_entry()` (`persist.c:3570+`, live-apply-on-commit path, the one `raft-apply-decoupling` introduced `_pad`-discriminated routing for) each have their own independent fall-through group for simple absolute-value job-state facts (`persist.c:3452-3466` and `persist.c:3700-3706` respectively). An earlier draft of design.md pointed only at the first (recovery) group, which alone would satisfy replay-after-restart but not live application when the WAL entry actually commits during normal operation — the primary case Requirement 2.4 is actually about. Found by tracing both functions directly rather than assuming a single switch statement. -- **`dispatch_a_job_()` leaks one `reference(j)` on exactly its three retryable-class failure paths** (workermgr lookup failure, worker not found, worker not ready) — `reference(j)` is taken once at entry (`job.c:854`) but never released with a matching `dereference(j)` on these three returns, confirmed by direct comparison against the two terminal-class failure paths (`nc <= 0`, `get_queue_exec()` failure), both of which already correctly `dereference(j)`. Logged as bug-1627, pre-existing. Since this design already restructures exactly these three call sites (to add the retryable-path lock/counter/notify sequence), the missing `dereference(j)` is added there too — closed as a side effect of the restructuring already underway, not a separate patch. Found during round 3 of `/kiro-validate-design`. - -## Risks & Mitigations -- **Risk**: A dispatched-then-terminally-failed job's resource-allocation cleanup on the scheduler side must run through the same code that already frees `job->accounts`/`job->allocs` on any transition away from RUN (`core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` case) — if the new dispatch-failure handling bypasses this and frees ad hoc, it risks a double-free or a leak. Mitigation: route dispatch-failure resolution through the existing `XCE_SCHED_OP_JSTATE_CHANGE` → `XCE_CORE_EVENT_JOB_STATE_CHANGE` machinery (same enqueue path `schedStartJob`/`schedExitJob`/`schedFailJob` already use), not a new, parallel mutation path. -- **Risk**: The bounded-retry counter (Requirement 4 AC3) is new per-job state with no existing analog — if not reset on a fresh successful dispatch, a job that failed twice, succeeded, ran to completion, and was later resubmitted with the same in-memory job object (should not happen given job objects are per-jobid, but worth stating) could inherit a stale count. Mitigation: reset the counter in the same operation that commits a fresh dispatch decision (`sched_commit_decision()`'s DISPATCH branch), not only on success at the end of a retry sequence. -- **Risk**: `--sim`/simulation mode does not call `sched_commit_decision()` at all (confirmed in the sibling `scheduler-sim-eager-reconciliation` investigation) — no risk of driver divergence from this feature's scheduler-side change, but worth stating explicitly since `scheduler-driver-equivalence.md` is a hard project invariant. Mitigation: none needed; documented as a boundary note in requirements.md and design.md rather than a code change. - -## References -- `.kiro/steering/scheduler-driver-equivalence.md` — state-equivalence invariant between normal-mode and simulation-mode drivers; confirms this feature's scheduler-side change has no simulation-mode counterpart to keep in sync (see Risks). -- `.kiro/specs/scheduler-sim-decision-parity/design.md` — house style precedent for C-domain adaptation of the design template (Components as C modules/functions, Service Interface blocks as real C signatures with pre/post/invariants). -- `src/include/wal_types.h` — `wal_jobstate_t`'s `_pad`-discriminated shape, the precedent this feature's new WAL fact type follows. diff --git a/.kiro/specs/scheduler-dispatch-commit/spec.json b/.kiro/specs/scheduler-dispatch-commit/spec.json deleted file mode 100644 index a3342f8e..00000000 --- a/.kiro/specs/scheduler-dispatch-commit/spec.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "feature_name": "scheduler-dispatch-commit", - "created_at": "2026-08-23T03:25:54Z", - "updated_at": "2026-08-24T04:59:00Z", - "language": "en", - "phase": "implementation-complete", - "parent_spec": "scheduler-framework", - "description": "Move the RUN-state transition to dispatch-decision-commit time on both scheduler and controller, and build the currently-missing dispatch-failure retry/terminal-fail handling this requires -- fixes bug-1624 (resource-in-use round-boundary loss for dispatched-but-unconfirmed jobs) and bug-1625 (scheduler has no handler for NotifyData_data_jobDispatchFail).", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true, - "task_plan_validation": "2 rounds of independent task-graph sanity review. Round 1: NEEDS_FIXES (task 3.1 missing counter-reset wiring, task 3.2 undersized/overloaded across 6 failure sites with an asymmetric reference-leak fix, ambiguous boundary labels, undocumented non-parallel rationale) -- repaired: 3.1 now resets the counter with explicit _Depends:_, former 3.2 split into 3.2 (terminal, 2 sites) / 3.3 (retryable+bounded escalation, 4 sites, carries the reference-leak fix), boundary labels differentiated, non-parallel rationale stated on task 3's major line. Round 2: NEEDS_FIXES (task 3.2's requirements list incorrectly carried over 4.1/4.4, which belong only to the PEND-return path 3.3 performs) -- fixed directly (single-line correction), and the same leftover was independently found and fixed in task 5.3's mirrored requirements list. Final coverage check: all 18 requirement IDs and all 5 design components mapped, each exactly once except where a requirement genuinely spans tasks. GO.", - "sibling_specs": ["scheduler-core", "scheduler-allocation", "scheduler-preemption", "scheduler-controller-msg-gap", "scheduler-sim-eager-reconciliation"], - "blocks": ["scheduler-allocation"], - "related_bugs": ["bug-1624", "bug-1625", "bug-1626", "bug-1627"], - "design_validation": "5 rounds of /kiro-validate-design, all findings fixed in place (no NO-GO carried forward): RTTI_LOCK_WRITE gap (bug-1626), unused scheduler-side field removed, requirements.md AC 2.1 wording reconciled, double-notify race (notify_dispatch fires on success too) fixed by moving classification+notification inside dispatch_a_job_(), rc-value classification replaced with an exhaustive site-identity enum, a 6th failure site (RPC failure) found and added, dispatch_a_job_() reference leak found and fixed (bug-1627, 4 sites), WAL apply logic corrected to cover both apply_wal_entry() and apply_committed_entry(). Final pass found no further issues. GO.", - "sequencing_note": "User directive (2026-08-22): this spec must land before scheduler-allocation begins implementation -- scheduler-allocation's iterator-based try_alloc_on_host() reads the same resInUse/sharedInUse totals bug-1624 corrupts, and its planned XCE_ALLOC_PREEMPT borrowing mode interacts with what counts as committed/borrowable. scheduler-allocation's own spec.json should be updated to note this dependency." -} diff --git a/.kiro/specs/scheduler-dispatch-commit/tasks.md b/.kiro/specs/scheduler-dispatch-commit/tasks.md deleted file mode 100644 index ed2ab253..00000000 --- a/.kiro/specs/scheduler-dispatch-commit/tasks.md +++ /dev/null @@ -1,125 +0,0 @@ -# Implementation Plan - -- [ ] 1. Foundation: controller-side dispatch-failure classification and durable transition record -- [x] 1.1 (P) Define the dispatch-failure classification used to decide whether a failed dispatch attempt should retry or fail permanently - - Enumerate each distinct way a dispatch attempt to a worker can fail today (worker lookup, worker unavailable, worker not ready, invalid allocation, queue configuration lookup, the dispatch request itself failing), and classify each as either retryable (transient/environmental) or terminal (a locally invalid decision that retrying will not fix) - - The classification is a pure decision with no side effects, independently testable against every known failure case without needing a live dispatch attempt - - Observable completion: a standalone test exercises every classified failure case and asserts the expected retryable/terminal outcome - - _Requirements: 4.1, 4.2_ - - _Boundary: Controller Dispatch-Failure Classification (site enum + classify function)_ - -- [x] 1.2 (P) Add a durable record of a job's dispatch-time transition to a running state - - Introduce a new durable fact type recording that a job has transitioned to running state at dispatch-commit time, reusing the existing shape used for other job-state facts - - Wire this new fact type into both the normal live-commit apply path and the recovery/restart replay path, so the transition is recovered identically whether observed live or replayed after a restart - - Observable completion: appending this fact and then replaying it (simulating a restart) leaves the job's state as running, without requiring the separate worker-confirmed-start fact to also be present - - _Requirements: 2.4_ - - _Boundary: WAL Dispatch Fact_ - -- [x] 1.3 (P) Track how many consecutive times a job's dispatch has failed for a retryable reason - - Add an in-memory counter on the controller's job record that increments on each retryable dispatch failure and resets to zero on a successful dispatch - - This counter does not need to survive a controller failover — resetting to zero after a failover is an acceptable, safe degradation - - Observable completion: the counter can be incremented, read, and reset independently of any actual dispatch attempt - - _Requirements: 4.3_ - - _Boundary: Controller Dispatch-Failure Classification (retry-counter data model)_ - -- [ ] 2. Core: scheduler transitions a job to running state at dispatch-commit time - - No dependency on Task 1 — scheduler-side files only, independent of the controller-side foundation work above. -- [x] 2.1 (P) Scheduler marks a job as running in the same operation that commits its dispatch decision - - When the scheduler commits a decision to dispatch a job, the job's state becomes running immediately, in the same operation that already records the job's resource allocation — not waiting for any later confirmation - - The scheduler's existing resource-in-use accounting, which only counts jobs already in a running state, requires no changes — this task provides the input it was missing - - Observable completion: a job dispatched in one round appears in resource-in-use totals starting that same round, and continues to appear in every subsequent round regardless of how many rounds pass before any confirmation arrives - - _Requirements: 1.1, 1.2, 1.3_ - - _Boundary: Scheduler Dispatch Commit_ - -- [x] 2.2 (P) Scheduler processes a reported dispatch failure by returning the job to a pending state - - When the scheduler is told a job's dispatch failed, it returns that job to a pending state instead of silently ignoring the report as it does today - - Returning a job to pending correctly releases the resource-allocation data that was tentatively committed for it, reusing the scheduler's existing behavior for any other transition away from a running state - - Observable completion: a job whose dispatch failure is reported to the scheduler is again eligible for the scheduler's own job-selection process in a subsequent round, with no dangling resource-allocation data left behind - - _Requirements: 3.1, 3.2, 4.4_ - - _Boundary: Scheduler Dispatch-Failure Handler_ - -- [ ] 3. Core: controller transitions a job to running state only once dispatch to the worker succeeds, and resolves failures - - Sub-tasks 3.1-3.3 all restructure the same function and share its per-job lock scope, so none of them are marked parallel to each other, regardless of how disjoint their individual concerns look. -- [x] 3.1 Controller commits a job's resource allocation and running-state transition together, only on a successful dispatch - - The controller's existing (currently unconditional) commit of a job's resource-allocation data becomes conditional on the dispatch request to the worker actually succeeding; the job's state transition to running happens in that same successful outcome, protected against concurrent access to the job record - - Record the new durable dispatch-transition fact (Task 1.2) as part of this same successful outcome - - Reset the consecutive-retryable-failure counter (Task 1.3) to zero as part of this same successful outcome, so a job that later fails again starts counting from zero rather than retaining count from before its last success - - The worker's own later confirmation that the job actually started continues to be recorded exactly as it already is today, unaffected by this change - - Observable completion: querying a job's status immediately after a successful dispatch shows it as running, and that status is unaffected by whether the worker has separately confirmed the job started yet; a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once - - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1_ - - _Depends: 1.2, 1.3_ - -- [x] 3.2 Controller resolves a terminal (locally-invalid) dispatch failure - - When a dispatch attempt fails for a reason the classification (Task 1.1) marks terminal, the controller resolves the job to a permanently failed state immediately, protected against concurrent access to the job record, and releases any resource-allocation data tentatively associated with the attempt - - This covers the two failure sites that already release the job reference they hold today — confirm that behavior is preserved, not duplicated, by this new resolution step - - Observable completion: a terminally-classified dispatch failure leaves the job permanently failed, with no partial allocation data left behind and the job's reference count unchanged from before the attempt - - _Requirements: 4.2, 4.5_ - - _Depends: 1.1_ - -- [x] 3.3 Controller resolves a retryable (transient/environmental) dispatch failure, with bounded escalation - - When a dispatch attempt fails for a reason the classification (Task 1.1) marks retryable, the controller increments the consecutive-failure counter (Task 1.3) under the same protected access; if the incremented count is still within a bounded limit, it reports the failure so the job can be retried, otherwise it resolves the job to a permanently failed state exactly as Task 3.2 does for a terminal failure - - This covers the four failure sites that do not currently release the job reference they hold — fix each to release it as part of this same resolution step (Task 3.2's two sites already release theirs correctly and are unaffected) - - Observable completion: a retryable dispatch failure below the bound leaves the job eligible for another attempt with its reference count restored to the pre-attempt value; a retryable failure that pushes the count past the bound resolves the job to permanently failed instead - - _Requirements: 4.1, 4.3, 4.4, 4.5_ - - _Depends: 1.1, 1.3, 3.2_ - -- [ ] 4. Integration: caller no longer duplicates dispatch-outcome reporting -- [x] 4.1 Remove the caller's own dispatch-outcome reporting now that resolution owns it - - The function that invokes a dispatch attempt for each job in a batch no longer separately reports the outcome itself — reporting is now entirely owned by the dispatch resolution added in Task 3, preventing a failed dispatch from ever producing two different, conflicting outcome reports for the same job - - Observable completion: a dispatch failure that resolves as permanent produces exactly one outcome report, not two conflicting ones - - _Requirements: 3.1, 4.1, 4.2_ - - _Depends: 3.1, 3.2, 3.3_ - -- [ ] 5. Validation: end-to-end behavior and regression -- [x] 5.1 End-to-end test: resource-in-use accounting survives a multi-round gap between dispatch and confirmation - - Reproduce bug-1624 directly: dispatch a job, let several rounds pass before the worker confirms it started, and confirm resource-in-use totals reflect the job continuously from the round it was dispatched, matching what a same-round confirmation would have produced - - Observable completion: the regression scenario passes, and fails against the pre-change code path if temporarily reverted - - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.3_ - - _Depends: 2.1, 3.1_ - -- [x] 5.2 End-to-end test: retryable and bounded-retry dispatch failures resolve correctly - - Simulate a transient dispatch failure and confirm the job becomes eligible for another attempt; simulate enough consecutive transient failures to exceed the bound and confirm the job resolves to permanently failed instead of retrying indefinitely; simulate a failure, a success, then another failure and confirm the bound is evaluated as if only one failure had occurred - - Observable completion: all three scenarios produce the documented outcome, with the counter's bound value exercised at exactly the boundary (one under, exactly at, and one over) - - _Requirements: 3.1, 3.2, 4.1, 4.3, 4.4, 4.5_ - - _Depends: 2.2, 3.3_ - -- [x] 5.3 End-to-end test: terminal dispatch failure resolves without duplicate reporting or leaked references - - Simulate a locally-invalid dispatch decision and confirm the job resolves to permanently failed, that no retry-eligible report is also produced for the same failure, and that the job record's reference count returns to its pre-attempt value - - Observable completion: the assertions above all pass in the same test run - - _Requirements: 4.2, 4.5_ - - _Depends: 3.2, 4.1_ - -- [x] 5.4 Full regression: existing suites and HA soak - - Run the existing scheduler and controller test suites end to end, updating only the assertions that directly depended on the old confirmation-gated timing; run a full sequential and chaotic HA soak - - Observable completion: both test suites pass and the soak shows no new job loss, state divergence, or liveness stall attributable to this feature - - _Requirements: 5.2, 5.3, 5.4_ - - _Depends: 5.1, 5.2, 5.3_ - -## Implementation Notes -- Task 1.1: design.md's "Controller Dispatch-Failure Classification" Service Interface Preconditions text claims a default-less `switch` over `xce_dispatch_fail_site_t` gives "compiler-enforced exhaustiveness... fails to compile" for an unhandled case — verified false for plain C without `-Werror` (this project sets none, confirmed in `src/meson.build`/`src/controller/meson.build`). Implemented instead with a `default:` arm that ERROR-logs and returns a safe `XCE_DISPATCH_FAIL_TERMINAL`, with `assert(0 && ...)` gated to `#ifdef XCE_TESTING` only — mirrors the established, verified-matching precedent `persist.c`'s `wal_classify()` already uses (`persist.c:6030-6037`) for the same "never risk silent misbehavior on an unrecognized case" concern. Any later task/spec touching this same design.md section should correct its wording rather than re-trusting the "fails to compile" claim. -- Tasks 3.2/3.3 (retryable/terminal resolution, both call `xce_classify_dispatch_fail()`) should follow the same fail-loud `default:` convention if they add their own exhaustive switches (e.g. over `xce_dispatch_fail_class_t`'s two values). -- Operational note (not spec-relevant): implementer/reviewer subagents in this run consistently ended their turn mid-build (after starting `xce-build.sh controller`, before it finished) rather than blocking on it, requiring one resume-and-nudge each time. Future task dispatches in this run should explicitly instruct agents to poll/block synchronously until the build genuinely completes before composing their status/verdict block, to save round-trips. -- Task 1.2: `apply_wal_entry()` (persist.c) is confirmed to have ZERO production call sites today — `persist_recover()`'s actual boot-time replay path (`replay_wal()`) and the standby catch-up path (`ha_standby_advance_committed_and_apply()`, post the 2026-07-10 `ha_apply_single_entry` refactor) both go through `apply_committed_entry()` exclusively. design.md's "WAL Dispatch Fact" component (its "Preconditions"-adjacent prose) still characterizes `apply_wal_entry()` as "the recovery/replay-time path reading WAL segments from disk after restart" — that's stale. Adding `WAL_JOB_DISPATCH` to both functions' fall-through groups was still correct (matches every sibling `WAL_JOB_*` type's identical dual-registration, harmless dead-code parity) and is NOT something to undo — but design.md's wording should be corrected before any later task cites `apply_wal_entry()`'s role from that doc without re-verifying. -- Task 3.1: `xce_request()`/every `xce_*_req()` wrapper (`src/lib/request.c`) is genuinely non-blocking fire-and-forget (capnp-encode + `endpoint_enq()` onto `ep->outq`, returns immediately — never waits for a reply). `endpoint_enq()` succeeds unconditionally against a zeroed `endpoint_t` with `ep->iomgr == NULL`. Net: testing a controller-side `dispatch_a_job_()` success path needs no live peer/socketpair/round-trip thread — a zeroed static `endpoint_t` (`.fd = -1`, no iomgr) is sufficient, matching `test_commit_barrier_reply.c`'s own established `g_wkr_disp_ep` fixture precedent. Reused directly for `test_dispatch_commit.c`; any later task exercising `dispatch_a_job_()`/`xce_startjob_req()` in a test should use the same pattern rather than building real network mocking. -- Task 3.1: **`test_commit_barrier_reply.c`'s `test_jobdispatch_receipt_ack_no_wal_append`** (a different, `ha-commit-barrier-rollout`-owned pinned test) asserts a successful dispatch cycle appends ZERO WAL entries — now false, since a successful dispatch appends exactly one `WAL_JOB_DISPATCH` fact (Req 2.4). Confirmed by both task 3.1's implementer and its independent reviewer; this is explicitly the kind of stale assertion Requirement 5.3 / task 5.4 exists to update, deliberately left unfixed here (out of 3.1's boundary, and that file's tests were unreachable in a full-suite run anyway due to the bug-1429 hang below, registered earlier in `controller_test.c`'s `main()`). **Task 5.4 must update this assertion** before claiming full-suite regression-clean. -- Task 3.1: reconfirmed **bug-1429** (`controller_unit_test` TIMEOUT/hang, pre-existing, unrooted) a 3rd time, now hanging inside `test_commit_barrier_reply.c`'s `test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs` — a 3rd distinct hang location across 3 sessions, confirmed unrelated to `dispatch_a_job_()` (that test never calls it) and independently reproduced twice more (standalone binary run past the meson harness's 300s cap, and a fully separate rebuild). Because this hang sits earlier in `controller_test.c`'s registration order than most of this feature's own new test files, a full un-filtered `meson test` run will NOT reach them — **task 5.4's full-suite validation must run with the hanging group filtered out (`controller_unit_test -t <group>`) or budget for the 300s timeout**, not treat a bare `meson test` TIMEOUT as this feature's own failure. -- Task 3.1: confirmed pre-existing (not introduced by this task, not in this task's boundary to fix) — `dispatch_a_job_()`'s success path never calls `dereference(j)` on the reference taken at lookup; identical in the diff's before/after, and the sole caller `handle_jobdispatch()` never dereferences either. A candidate future bug-log entry, out of scope for this spec. -- Task 3.1: when validating a build against this suite's ASAN leak output, always replicate the project's own `ASAN_OPTIONS=detect_leaks=0` test env (`meson.build:229`) — a standalone binary run without it produces spurious leak reports (a red herring, not a regression) that can waste review time. -- Task 3.2: `job_t`'s real reference count lives in `jobj.c`'s file-local `job_private_t`, with no accessor exposed anywhere in the codebase — a test proving no reference-count regression can't do a direct numeric comparison. The working pattern (used by `test_dispatch_fail_terminal.c`) is a "canary reference": take one extra manual `reference()` on the job **immediately before** the call under test (not after — arming it after only relies incidentally on other assertions crashing first, which doesn't independently prove anything), then `dereference()` it after the call and confirm the job is still safely readable — an over-release during the call frees the job while the canary is still held, so the post-call touch is a real ASAN-caught heap-use-after-free. -- Task 3.2: reconfirmed bug-1429's hang again (still `test_commit_barrier_reply.c`, still unrelated to any change in this feature) — third+ reproduction, still not root-caused. Continues to require `controller_unit_test -t <group>` filtering for a clean isolated regression signal. -- Task 3.3: bug-1627 (the 4-site reference leak on `dispatch_a_job_()`'s retryable failure paths) is now FIXED — `dereference(j)` added at all four retryable sites, closing it as a direct consequence of this task's resolution restructuring. `.wolf/buglog.json` updated accordingly. -- Task 3.3: `notify_dispatch()` is `static` and defined AFTER `dispatch_a_job_()` in job.c with no forward declaration — needed one added near the top (mirroring `mark_job_fail`'s existing forward-decl precedent at job.c:24) so the new retryable-resolution code could call it. Any later task in this file needing to call a same-file function defined later in the file should check for/add a forward declaration the same way, rather than reordering definitions. -- Task 3.3: capturing a site's natural error code (`rc`/`errno`) into a stable local must happen BEFORE any intervening call once more logic is inserted before the eventual `return` — `errno` is not guaranteed stable across `lock()`/`ERROR()`/`free_launchinfo()` etc. This bit the worker-null and worker-not-ready sites specifically (both originally read `errno` directly at the `return` statement with nothing in between; task 3.3 had to capture it immediately after `SET_ERROR()`/at branch entry, before inserting the new lock/classify/resolve sequence). -- Task 3.3: `test_commit_barrier_reply.c` (a file otherwise owned by the `ha-commit-barrier-rollout` spec) is this binary's sole host of the `notify()` stub — any future task needing to observe `notify_dispatch()`'s calls in a controller unit test must add accessors there too (purely additive, read-only `_ext`-suffixed pass-throughs to its existing static capture globals — do not modify the stub body itself or its existing pinned tests). Confirmed as legitimate shared test infrastructure, not a boundary violation, by task 3.3's independent review. -- Task 3.3: the bound constant is `XCE_DISPATCH_FAIL_RETRY_BOUND = 3` (design.md's stated default), defined once near job.c's top and used at all four retryable sites — no magic numbers. 3 consecutive failures still retry; the 4th resolves terminal. -- Task 4.1: **`test_commit_barrier_reply.c`'s `test_jobdispatch_receipt_ack_no_wal_append` is now stale for a SECOND, independent reason**, on top of task 3.1's already-documented WAL-append-count staleness: its 2-job batch (one real-worker success, one `nhosts==0` terminal failure — no retryable job) now produces ZERO `notify_dispatch()` calls total post-task-4.1 (the success path never called it even before task 4.1, per task 3.1's design; the terminal path resolves via `mark_job_fail()`'s `schedFailJob` hook chain, not `notify_dispatch()`/`jobDispatchFail`, per task 3.2's design) — breaking its pinned `g_notify_count == 2` / both-`jobDispatchFail`-event assertions (lines ~1991-2003). **Task 5.4 must update BOTH the WAL-count assertion (task 3.1's finding) and the notify-count/event assertions (this finding) in the same test** before claiming full-suite regression-clean; the test's own scenario (1 success + 1 terminal, zero retryable jobs) may need to change too since it can no longer meaningfully exercise `notify_dispatch()` at all with today's cast of outcomes — consider adding a retryable-classified job to the batch if the intent is still to prove `notify_dispatch()`'s wiring through `handle_jobdispatch()`. -- Task 4.1: confirmed `handle_jobdispatch()`'s only caller (`cluster.c:471`, inside `myreq_handler`) uses its return value only as the generic RPC-handler `rc`, and `handle_jobdispatch()` itself always returns 0 unconditionally regardless of any job's outcome — nothing depends on the removed per-job notification behavior at the caller level. -- Task 5.1: a "same-round confirmation" scenario for resource-in-use accounting is legitimately modeled as `nrounds==1` rather than by simulating a real `schedStartJob` notify event — design.md documents (and the reviewer independently re-verified against `core_events.c`) that re-applying `JOB_STAT_RUN` to an already-RUN job via the confirmation path is a confirmed no-op, so `resInUse` depends solely on `JOB_IS_RUN(job->state)`, never on a separate confirmation flag. Reviewer's own finding: the resulting same-round-vs-multi-round comparison test is honest and correctly fails on revert, but adds little independent detection power beyond a per-round-asserting loop test (which is the one genuinely load-bearing proof) — worth knowing if a future task considers deduplicating these two tests, but not something to act on now. -- Task 5.1: no controller-side test was added for this task — Requirements 2.1/2.3 (controller transitions to RUN independent of confirmation) are already covered by task 3.1's `test_dc_success_commits_alloc_state_and_resets_counter`. No live scheduler+controller wire-protocol integration harness exists anywhere in this codebase; building one was judged disproportionate to what this task verifies, and the reviewer agreed. -- Task 5.2: **bug-1631 (logged, NOT fixed, deliberately deferred)** — `dispatch_a_job_()`'s four retryable-failure branches (job.c, tasks 3.1/3.3) never revert `job.state`/`job.allocs` when a retryable failure occurs while the job's controller-side state is CURRENTLY RUN (they were only ever designed for a job whose every dispatch attempt starts from PEND, which is all that's reachable today). A kiro-debug investigation (HIGH confidence), independently re-verified even more exhaustively by this task's own reviewer (full trace of every `pendJobs`-reinsertion and every controller-side `job.state`-mutation call site), confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system: the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. The only mechanism design.md itself anticipates for that is `scheduler-preemption`, confirmed still unimplemented (`requirements-generated`, no approvals). **Revisit bug-1631 and add the state/allocs reversal to `dispatch_a_job_()`'s four retryable branches only once `scheduler-preemption` (or an equivalent mechanism) actually creates a real RUN→retryable-redispatch path** — not before, since there's no reachable caller to regression-test against today. `test_rpc_failure_bound_resets_after_intervening_success` (`test_dispatch_fail_retry.c`) proves Requirement 4.3's actual, literal ask (the counter genuinely resets on an intervening success) without asserting the unreachable state/allocs behavior. -- Task 5.2: design.md's "vacuously by construction" reasoning (`Controller Dispatch Commit` and `Controller Dispatch-Failure Classification` sections — "on any of the six classified failure paths there is nothing to release") silently assumes every `dispatch_a_job_()` invocation starts from a job that's never had anything committed (i.e., a first-ever dispatch attempt) — true today (see bug-1631 above) but not stated as a scoped assumption in the doc text itself. Worth correcting design.md's wording if it's ever revisited for the `scheduler-preemption` follow-up. -- Task 5.3: `test_dispatch_fail_terminal.c` (task 3.2) and `test_jobdispatch_no_double_notify.c` (task 4.1) each covered PART of this task but not the full combination in one run — 3.2's test bypasses `handle_jobdispatch()` entirely (via `dispatch_a_job_test()`), so it can't exercise the double-notify race at all; 4.1's test only proves an *aggregate* batch notify count, not a *direct* per-job zero-notify proof. `test_handle_jobdispatch_terminal_no_notify_no_leak` (extending 4.1's file, the one with the right entry point) closes this by combining EXIT resolution + a direct single-job zero-notify assertion + the canary-reference no-leak proof, all through the real `handle_jobdispatch()` entry point in one test. Pattern worth remembering: a batch-aggregate test and a bypassed-entry-point test can each look like they cover a scenario while neither actually does, in combination, what a later task's literal wording asks for ("all pass IN THE SAME TEST RUN"). -- Task 5.4 (final task — spec complete): fixed `test_commit_barrier_reply.c`'s `test_jobdispatch_receipt_ack_no_wal_append` (owned by `ha-commit-barrier-rollout`), the last remaining stale assertion from this feature's own changes. WAL-count delta corrected from "must be zero" to `lsn_before + 1` (the batch's successful dispatch now appends one `WAL_JOB_DISPATCH` fact; `mark_job_fail()` was traced and confirmed to append nothing via any path). Notify-count: rather than degrading to "must be zero" (which would also silently kill the test's own reply-before-notify ordering assertion, since there'd be nothing left to order against), the batch's terminal-failure job was replaced with a genuinely retryable one (`WORKER_NULL` site, unregistered host, new job id) — preserves the test's ability to meaningfully exercise `notify_dispatch()`'s wiring through `handle_jobdispatch()`. Independently reviewed and approved, including scrutiny of this specific choice. -- Task 5.4: discovered and logged **bug-1632** (scheduler suite) — a new sibling instance of the already-known bug-1598 pattern (unjoined `sim_picker_thread_fn` background thread reading a stack-local fixture after frame reuse), this time in `test_sim_verify.c`'s own separate `run_sim_and_capture_stdout()` helper (bug-1598's original fix only covered 3 stack-locals in `test_sim_event_loop.c`, not this one). Confirmed via `git log`/`git diff` as untouched by this feature and confirmed non-deterministic (multiple clean re-runs). Pre-existing, unrelated, NOT fixed — a candidate for whoever next touches `test_sim_verify.c`'s fixture pattern. -- Task 5.4: bug-1429's hang did NOT reproduce during this task's full-suite runs (multiple clean 1133-test passes) — consistent with its own buglog history describing it as non-deterministic/timing-dependent, not evidence it's resolved. Still treat any future `meson test` TIMEOUT on `test_commit_barrier_reply`/`test_commit_barrier_rollout` as this known, pre-existing, unrooted issue rather than a new regression, per all of tasks 3.1-5.3's own prior confirmations. -- **Feature complete**: all tasks (1.1-1.3, 2.1-2.2, 3.1-3.3, 4.1, 5.1-5.4) implemented, independently reviewed and approved, and committed. Final regression: scheduler suite at documented baseline (51/52, 1 pre-existing unrelated failure), controller suite clean (1133 tests, only 2 pre-existing bug-1561 failures), sequential HA soak PASS (10 rounds, zero job loss, all 3 nodes led), chaotic HA soak PASS (10 rounds, `committed_divergence=0 committed_job_loss=0`) — independently re-verified by task 5.4's reviewer via its own rebuild plus a 5-round spot-check of both soak modes. bug-1624 and bug-1625 (this spec's originating bugs) are fixed by tasks 3.1/2.1 and 2.2 respectively; `.wolf/buglog.json` should be updated to reflect FIXED status as a follow-up. diff --git a/.kiro/specs/scheduler-framework/brief.md b/.kiro/specs/scheduler-framework/brief.md deleted file mode 100644 index 82a7e105..00000000 --- a/.kiro/specs/scheduler-framework/brief.md +++ /dev/null @@ -1,384 +0,0 @@ -# Brief: scheduler-framework - -## Problem -The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework: -1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation. -2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation. -3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails. -4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources. -5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. -6. **Monolithic scheduler** — the scheduler is tightly coupled to controller I/O (capnp, `apictx`). No way to replay recorded events for debugging or test scheduling policies without a live cluster. - -Operators need configurable scheduling policies, parallel job support, preemption, and a way to debug/tune scheduling behavior without a live cluster. - -## Current State - -### What exists -- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn`. Plugins registered in key-value map by name. -- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via `jbuf` cbuf. -- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources → collects workers → tries one host → returns first match. -- **Parjob allocator stub** (`parjob_allocator.c`): registered but `my_alloc()` is empty. -- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds). -- **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC. -- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session. -- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next. -- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`. - -### What's missing -- Configurable job picker: the plugin name is a hardcoded string literal -- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching -- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans -- Preemption: no mechanism to preempt lower-priority running jobs -- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` -- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session -- Scheduler simulation/debugging: no way to replay recorded events, no virtual clock, no deterministic mode -- Core/driver separation: scheduling logic is tightly coupled to controller I/O - -## Desired Outcome - -### Three Architectural Foundations - -**1. Pluggable Job Picker** — the core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no `jbuf` cbuf. - -The picker plugin interface (`xce_job_picker_iface_t`): - -| Category | Hooks | Purpose | -|----------|-------|---------| -| Lifecycle | `init`, `fini` | Setup and teardown | -| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called by core when events arrive from driver | -| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle | -| Post-allocation | `on_dispatch`, `on_pend` | Notification after allocation attempt | -| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) | - -**2. Iterator-Driven Allocation** — lazy `xce_alloc_plan_iter_t` (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with lazy `xce_host_iter_t` (candidate hosts for a given plan). `parjob_allocator` implements real combinatorial multi-slot allocation. Session-level unsatisfiable resreq fingerprint cache avoids re-evaluating already-proven-unsatisfiable resreqs. - -**3. Core/Driver Separation** — the scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it to the outside world: - -| Mode | Driver | Input | Output | -|------|--------|-------|--------| -| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller | -| Simulation | Replay file | JSONL event file → core events | Core decisions → log (with `--sim-verify` diff against recorded) | - -- **Discrete event-driven virtual clock**: jumps directly to next event timestamp, no fixed tick, monotonic -- **Snapshot + delta replay**: load state snapshot, replay only subsequent delta events -- **Deterministic guarantee**: same initial state + same events = same decisions -- **Runtime recording control**: xceadmin `scheduler record start/stop` commands -- **JSONL replay format**: one event per line, debuggable, plain text -- **What-if branching**: load snapshot, replay alternative delta events - -### Queue vs Project — Roles - -| | Queue | Project | -|---|---|---| -| **Role** | Scheduling mechanism | Resource ownership | -| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? | -| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster | -| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy, members (future) | -| **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many | - -One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware). The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2). - -## Alternatives Considered - -### Approach A: Incremental Enhancement (rejected) - -| Aspect | Detail | -|--------|--------| -| **Job selection** | Add `job_pick_plugin` config field, same as chosen approach | -| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation | -| **Iterator** | No new abstraction. Simple for-loop over plan array | -| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state | - -**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost. - -### Approach B: Iterator-Driven Allocation + Picker Plugin ★ Chosen - -| Aspect | Detail | -|--------|--------| -| **Job selection** | Pluggable `xce_job_picker_iface_t` with `next_job()`. Single `builtin_fcfs` default. Per-queue picker override. Synchronous, no dedicated thread. | -| **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable | -| **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 | -| **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval | -| **Preemption** | `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP via controller→worker | -| **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` | -| **Core/driver split** | Passive event-driven core + dual-mode driver (normal/simulation) | -| **Simulation** | JSONL replay, snapshot + delta, discrete event-driven virtual clock, deterministic | - -**Chosen because:** Lazy generation avoids memory pressure. Single `next_job()` interface makes pickers truly pluggable — the core loop never changes. Core/driver separation enables simulation mode for debugging/tuning without a live cluster. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. - -### Approach C: Full Scheduler Framework (deferred) - -| Aspect | Detail | -|--------|--------| -| **Job selection** | Policy chain — multiple plugins score/veto jobs | -| **Iterator** | Triple composition: job iterator → plan iterator → host iterator | -| **Resreq registry** | Named primary resreqs registered globally, reference-counted | -| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin | -| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT | - -**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's picker interface is designed to compose upward — a future policy-chain picker implements the same `next_job()` interface. - -### Decision Rationale - -Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The picker plugin interface with `next_job()` keeps the core loop simple (5 lines) and makes pickers truly pluggable. Core/driver separation enables simulation mode — a capability that would be prohibitively expensive to retrofit later. - -**Design evolution (2026-07-10 → 2026-07-27):** The original design specified a four-level architecture with three separate builtin plugins (`builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`) and a three-level `project→queue→job` loop hardcoded in the core. Analysis of the existing FCFS plugin (`fcfs.c`) showed it already encapsulates all picking logic internally. The architecture was simplified to a single `next_job()` plugin interface with one builtin picker (`builtin_fcfs`) — the three-level decomposition is now an internal implementation detail of the plugin, not the core loop. Core/driver separation was added (Req 10) to enable simulation mode for policy debugging and tuning. - -## Approach - -**Iterator-Driven Allocation with Pluggable Picker + Core/Driver Separation**: - -### Core scheduling loop - -```c -void* sched_allocator(void* arg) { - while (1) { - picker->begin_session(picker); - - xce_sched_job_t* job; - while ((job = picker->next_job(picker)) != NULL) { - if (try_allocate(job) > 0) { - picker->on_dispatch(picker, job->jobid); - emit_decision(DISPATCH, job); - } else if (can_preempt(job)) { - emit_decision(PREEMPT, job, victims); - picker->on_pend(picker, job->jobid); - } else { - emit_decision(PEND, job); - picker->on_pend(picker, job->jobid); - } - } - - picker->end_session(picker); - sleep(1); - } -} -``` - -The core loop has zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`. - -### Picker plugin interface - -```c -typedef struct xce_job_picker { - // Lifecycle - int (*init)(void* picker, xce_sched_core_t* core); - void (*fini)(void* picker); - - // State-change hooks (called by core when events arrive) - int (*on_queue_update)(void* picker, xce_sched_queue_t* queue); - int (*on_job_push)(void* picker, xce_sched_job_t* job); - int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state); - int (*on_job_update)(void* picker, xce_sched_job_t* job); - - // Session lifecycle - int (*begin_session)(void* picker); - int (*end_session)(void* picker); - - // Job selection - xce_sched_job_t* (*next_job)(void* picker); - - // Post-allocation notification - int (*on_dispatch)(void* picker, uint64_t jobid); - int (*on_pend)(void* picker, uint64_t jobid); - - // Optional hooks (NULL = not implemented) - int (*on_config)(void* picker, xce_conf4sched_t* config); - int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); -} xce_job_picker_iface_t; -``` - -### builtin_fcfs internal design - -The default picker internally implements the equivalent of today's selection: - -``` -builtin_fcfs.next_job(): - Iterate queues by priority (descending) - Skip queues with no ready jobs - For each queue: - If queue has per-queue picker override: - Delegate to that picker - Else: - Iterate jobs by priority desc, queue_time asc (FCFS) - Skip jobs not ready - Return first ready job - Return NULL if all queues exhausted -``` - -This is an implementation detail of the plugin, not visible to the core loop. - -### Preemption flow - -``` -scheduler controller worker -─────────── ────────── ────── -alloc fails for high-prio job J - -preempt_policy.select_victims(J): - collect running jobs from - Q_aggressor.preemptible_queues - filter: victim.prio < J.prio - filter: victim.queue.preemptible - rank: lowest priority first - select minimum set satisfying J.resreq - - → [victim1, victim2] - -XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 - .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED - .signal = SIGSTP victim's worker release resources - .preempting_jobid = J - -J pended → next round → retry alloc -``` - -### Core/Driver Architecture - -``` -┌──────────────────────────────────────────────────────────────┐ -│ DRIVER LAYER │ -│ │ -│ Normal Mode Simulation Mode │ -│ controller capnp ──┐ replay.jsonl ──┐ │ -│ handler → events │ virtual clock │ │ -│ → publish decisions│ → record/log │ │ -│ │ ▲ │ │ ▲ │ │ -└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘ - │ │ decisions │ │ decisions - │ │ (callback) │ │ (callback) - ▼ │ ▼ │ -┌──────────────────────────────────────────────────────────────┐ -│ SCHEDULER CORE │ -│ │ -│ core_push_event() → picker hooks → picker->next_job() │ -│ → try_allocate() → DISPATCH / PREEMPT / PEND │ -│ │ -│ ZERO dependencies on: controller, capnp, network, wall-clock │ -└──────────────────────────────────────────────────────────────┘ -``` - -### What stays unchanged -- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved, but the picker no longer uses the dedicated `job_elect_fn` thread -- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot -- Select expression system — no changes -- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus) -- Controller communication (handler.c opcodes, decision cbuf in driver) — extended, not replaced -- `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications - -### What's new -- `xce_job_picker_iface_t` — pluggable picker interface with full hook specification -- `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy iterator abstractions -- `XCE_DECISION_PREEMPT` decision type + `xce_preempt_decision_t` payload -- `builtin_fcfs` — single builtin picker (replaces three separate builtins from original design) -- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t` — core/driver struct split -- `xce_core_event_t` — typed event interface between driver and core -- `xce_core_decision_cb_t` — decision callback (core → driver) -- JSONL replay file format + snapshot/restore -- Discrete event-driven virtual clock -- `project_name` on queue structs -- `job_pick_plugin` on scheduler + queue structs (per-queue override) -- `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct - -## Scope - -### In -- Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic -- Single builtin picker (`builtin_fcfs`) replicating today's behavior -- Configurable picker plugin with per-queue override (`queue->job_pick_plugin`) -- Picker hook specification: lifecycle, state-change, session, post-allocation, optional hooks -- `xce_alloc_plan_iter_t` + `xce_host_iter_t` interfaces (lazy, cancel-safe, composable) -- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1) -- Preemption: `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP mechanism -- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy` -- Session-level unsatisfiable resreq fingerprint cache -- Core/driver separation: passive event-driven core, normal-mode driver, simulation-mode driver -- Discrete event-driven virtual clock (monotonic, jumps to next event timestamp) -- Runtime recording control via xceadmin commands (`scheduler record start/stop`) -- JSONL replay file format (HEADER + SNAPSHOT + events + DECISIONS) -- State snapshot/restore for replay efficiency -- Deterministic replay guarantee for debugging/tuning -- What-if branching (load snapshot, replay alternative delta events) -- Data structure changes: `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`, `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t` -- Picker plugin interface contract documentation -- Unit tests: plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, event recording/replay, snapshot/restore, deterministic replay -- Integration test: multi-slot parallel job allocation and preemption flow end-to-end - -### Out -- New job selection policies beyond the builtin FCFS picker — the pluggable interface enables them, but none ship here -- Hierarchical project resource quotas — deferred to `project-resource-ownership` spec -- Resreq template registry — future enhancement -- Select expression language changes -- Controller-side scheduling state replication (commit barrier wiring) -- Compound/ALTER resreq allocation changes -- GPU/affinity-aware allocation (pluggable later via allocator plugins) -- Kill-and-requeue preemption escalation (v1 is SIGSTP only) -- Deadline field on `xce_sched_job_t` -- Event payload JSON schema (open design question — deferred to design phase) -- Distributed/cluster-mode simulation (v1 is single-scheduler replay only) - -## Future: Project Resource Ownership - -Deferred to a separate `project-resource-ownership` spec. When implemented: - -- Projects form a tree with resource quota at each node -- A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue -- Project quota enforcement: projects over quota are skipped by the picker -- The core scheduling loop requires zero code changes — only a new picker plugin replaces `builtin_fcfs` -- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes -- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends - -The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. - -## Boundary Candidates - -The work naturally splits into these implementation phases (single spec, sequential): - -1. **Core/driver separation** — split `xce_scheduler_t` into `xce_sched_core_t` + `xce_sched_driver_t`; define `xce_core_event_t` and `xce_core_decision_cb_t`; implement passive event-driven core interface; wire normal-mode driver to existing handler.c -2. **Picker plugin interface + builtin_fcfs** — define `xce_job_picker_iface_t`; implement `builtin_fcfs` with all MUST/SHOULD hooks; retire dedicated picker thread + `jbuf`; make core loop call `next_job()` synchronously; add `job_pick_plugin` config with per-queue override -3. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction -4. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator -5. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline, queue preemption config fields -6. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop -7. **Simulation driver** — implement simulation-mode driver with JSONL replay, virtual clock, snapshot/restore, deterministic verification -8. **Recording + xceadmin commands** — implement runtime recording control, periodic snapshots during recording -9. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, simulation replay, HA soak regression - -## Out of Boundary -- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that implement `xce_job_picker_iface_t` -- Resreq template sharing across jobs (future: `resreq_registry` spec) -- Commit barrier wiring for scheduler decisions (separate hardening item in roadmap) -- Controller-side changes (the scheduler runs independently, receives updates via existing opcodes) - -## Upstream / Downstream - -### Upstream (dependencies) -- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed -- Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed -- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed -- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — split, not replaced - -### Downstream (future consumers) -- **New job selection policies** — priority, fair-share, deadline, backfill plugins implement `xce_job_picker_iface_t` -- **Project-resource-ownership** — project-aware picker plugin consumes `queue->project_name` and `xce_sched_project_t` -- **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement -- **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item) -- **Resreq template registry** — named resreqs shared across jobs, reference-counted - -## Existing Spec Touchpoints - -### Extends -- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same scheduling state. No conflicts; the msg-gap spec fills handler gaps this spec reads from. - -### Adjacent -- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume. -- **worker-join-refactor** — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path. - -## Constraints -- **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads) -- **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns. -- **Cancel-safe** — `next_job()` must have `pthread_cancel` cancellation points -- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session` -- **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive. -- **Deterministic replay** — same initial state + same event sequence → same decisions. No wall-clock time, no unseeded randomness, stable iteration order. -- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded). diff --git a/.kiro/specs/scheduler-framework/requirements.md b/.kiro/specs/scheduler-framework/requirements.md deleted file mode 100644 index 891c6283..00000000 --- a/.kiro/specs/scheduler-framework/requirements.md +++ /dev/null @@ -1,886 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: `sched_job_picker()` (`scheduler.c:39`) sets `job_pick_plugin = "xce_sched_fcfs"` as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: `parjob_allocator.c:73`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` — any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()` at `scheduler.c:93-103`) produces no allocation at all. Third, allocation is eager and one-shot: the sibling `seqjob_allocator.c:107` `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops — there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails. - -This feature closes all three gaps and adds two architectural foundations: - -**Allocation**: A lazy `xce_alloc_plan_iter_t` (candidate `(nhosts, slots_per_host)` plans, most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session. - -**Job picking**: A pluggable `xce_job_picker_iface_t` — the core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected (FCFS, priority, fair-share, deadline, etc.). The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. Preemption fires at Level 3 when allocation fails for a preemptible job. - -**Core/driver separation**: The scheduler core is a passive, event-driven engine with zero controller I/O dependencies. A normal-mode driver connects it to a live controller; a simulation-mode driver replays recorded cluster events through a discrete event-driven virtual clock, enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. - -> Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in `.kiro/specs/scheduler-framework/brief.md`. - -## Queue vs Project — Roles - -These are fundamentally different concerns: - -| | Queue | Project | -|---|---|---| -| **Role** | Scheduling mechanism | Resource ownership | -| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? | -| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster | -| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) | -| **Cardinality to job** | 1 job → 1 queue | 1 job → 1 project | -| **Cardinality to each other** | Many-to-many | Many-to-many | - -- **Queue** = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, priority (for inter-queue ordering within the picker), per-queue picker override, and preemption config. -- **Project** = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. One project submits to many queues; one queue receives from many projects. The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2). - -## Boundary Context - -- **In scope**: Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic behind a single plugin; `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1) for `maxTasks > 1`; single builtin picker (`builtin_fcfs`) replicating today's behavior through the new interface; configurable picker plugin with per-queue override (`queue->job_pick_plugin`); preemption decision type (`XCE_DECISION_PREEMPT`) flowing scheduler→controller→worker with SIGSTP mechanism; cross-queue victim selection (LSF-style per-queue `preemptible_queues` list); session-level unsatisfiable-resreq fingerprint cache keyed on `{select, rusage, user, queue, project}`; **core/driver separation** with passive event-driven core interface, normal-mode driver (controller-connected), simulation-mode driver (replay recorded events with discrete event-driven virtual clock), runtime recording control via xceadmin commands, JSONL replay file format, state snapshot/restore for replay efficiency, and deterministic replay guarantee for debugging/tuning; documentation of picker plugin interface contract (hooks, cancel-safe, session-aware); unit tests for plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, **event recording/replay, snapshot/restore, and deterministic replay**; integration test for end-to-end multi-slot parallel job allocation and preemption flow. - -- **Out of scope**: New job selection policies beyond the builtin FCFS picker (priority, fair-share, deadline, backfill) — the pluggable picker interface enables them, but none ship in this spec beyond `builtin_fcfs`; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring for scheduling decisions (separate roadmap hardening item, tracked adjacent to `ha-consistency-invariants`); `COMPOUND`/`ALTER` resreq allocation changes beyond what composing over `PRIMARY` already provides; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins); hierarchical project resource quotas (deferred to `project-resource-ownership` spec — only `project_name` on queue and `xce_sched_project_t` scaffolding ship here); kill-and-requeue preemption escalation (deferred — v1 is SIGSTP only); deadline field on `xce_sched_job_t` (deferred — needed for deadline-based pickers); **event payload JSON schema** (open design question — deferred to design phase); **distributed/cluster-mode simulation** (v1 is single-scheduler replay only). - -- **Adjacent expectations**: depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking (`xce_scheduler_t.sharedInst`/`sharedInUse`) — none of these change. `scheduler-controller-msg-gap` (implemented) governs controller↔scheduler message handling this feature reads state through but does not modify. `ha-consistency-invariants`' commit barrier is a future consumer of scheduling decisions; this feature does not wire it. Controller hook `HOOK_DETAIL_JOB_STOP` (enum value 65 in `cluster_def.h`) already exists and is reused for preemption stop actions. - -## Allocation Architecture (Reqs 2–4) - -Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER): - -### Resreq Type Hierarchy - -``` -ALTER → OR-of-compounds: first compound that fully allocates wins - └─ COMPOUND → AND-of-primaries: ALL primaries must allocate for success - └─ PRIMARY → plan_iter → host_iter → try_alloc (leaf level) -``` - -The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today's `sched_make_compound_allocation()` and `sched_make_alter_allocation()` dispatch. - -### Iterator Composition (PRIMARY leaf) - -``` - ┌──────────────────────────┐ - │ xce_alloc_plan_iter_t │ Req 2 - │ │ - │ init(primary_resreq) │ - │ next() → plan │ generates (nhosts, slots_per_host) - │ fini() │ candidates, most-preferred first - └────────────┬───────────────┘ - │ plan - ▼ - ┌──────────────────────────┐ - │ xce_host_iter_t │ Req 3 - │ │ - │ init(plan, workers) │ for a given plan, iterate - │ next() → worker │ candidate hosts matching - │ fini() │ select expression + availability - └────────────┬───────────────┘ - │ worker - ▼ - ┌──────────────────────────┐ - │ try_alloc(worker, plan) │ Req 4 (parjob) or - │ │ existing seqjob logic - │ check slots, shared res, │ - │ commit or advance │ - └────────────────────────────┘ -``` - -### Composition Algorithm (All Three Types) - -```c -// === PRIMARY leaf: iterator composition === -// Returns >0 on success, 0 on failure -int allocate_primary(sched, primary_resreq, ctx) { - plan_iter = plan_iter_init(primary_resreq, sched); - while ((plan = plan_iter->next(plan_iter)) != NULL) { - host_iter = host_iter_init(plan, sched->workers); - while ((worker = host_iter->next(host_iter)) != NULL) { - if (try_alloc_on_host(worker, plan, ctx) > 0) { - host_iter->fini(host_iter); - plan_iter->fini(plan_iter); - return ALLOC_SUCCESS; - } - } - host_iter->fini(host_iter); - } - plan_iter->fini(plan_iter); - return ALLOC_FAIL; -} - -// === COMPOUND: AND of primaries === -// ALL primaries must allocate. On any failure, roll back prior allocations -// and return failure. Same logic as today's sched_make_compound_allocation(). -int allocate_compound(sched, compound_resreq, ctx) { - for (int i = 0; i < compound_resreq->primaries.count; i++) { - primary = compound_resreq->primaries[i]; - if (allocate_primary(sched, primary, ctx) <= 0) { - // Roll back all primaries allocated so far in this compound - rollback_compound_allocations(ctx, i); - return ALLOC_FAIL; - } - } - return ALLOC_SUCCESS; -} - -// === ALTER: OR of compounds === -// First compound that fully allocates wins. Same logic as today's -// sched_make_alter_allocation(). On failure, no rollback needed -// (prior compounds were never committed — they failed their own checks). -int allocate_alter(sched, alter_resreq, ctx) { - for (int i = 0; i < alter_resreq->compounds.count; i++) { - compound = alter_resreq->compounds[i]; - if (allocate_compound(sched, compound, ctx) > 0) { - return ALLOC_SUCCESS; - } - // This compound failed — advance to next alternative - } - return ALLOC_FAIL; -} - -// === Top-level dispatch === -// Unchanged from today's sched_make_allocation() dispatch on obj->tag. -int allocate_job(sched, job, ctx) { - switch (job->resreq_type) { - case PRIMARY: - return allocate_primary(sched, job->primary_resreq, ctx); - case COMPOUND: - return allocate_compound(sched, job->compound_resreq, ctx); - case ALTER: - return allocate_alter(sched, job->alter_resreq, ctx); - } -} -``` - -**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back — the compound is atomic. Today this is handled by `sched_make_compound_allocation()` tracking allocations in `ctx` and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract. - -### Who Uses This Composition - -| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior | -|-----------|------------|---------------|---------------|----------| -| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host × 1 slot) | Full: filters hosts by select expression | Unchanged from today — first matching host wins | -| `parjob_allocator` | `> 1` | Full combinatorial: 1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1 | Full: filters hosts with enough available slots | New — walks plans until one succeeds | -| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` | - -### Key Design Properties - -- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization. -- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points. -- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. `fini()` on one does not affect the other. -- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside `allocate_primary()`. -- **seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host × 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case. - -## Requirements - -### Requirement 1: Pluggable job picker interface - -**Objective:** As a maintainer of the scheduler, I want the core scheduling loop to call a single `next_job()` plugin interface with no knowledge of how jobs are selected, so that picking strategies (FCFS, priority, fair-share, deadline, etc.) are truly pluggable and the core loop never changes regardless of the picking algorithm. - -#### Acceptance Criteria -1. The Scheduler core loop shall call `picker->next_job(picker)` to get the next schedulable job (or NULL if none ready), and shall not contain any logic about project iteration, queue iteration, or job ordering — those concerns are entirely encapsulated inside the picker plugin. - -2. The job picker plugin interface (`xce_job_picker_iface_t`) shall define these operations, organized by category: - - **Lifecycle hooks** (MUST): - - | Operation | Signature | Semantics | - |-----------|-----------|-----------| - | `init` | `int (*init)(void* picker, xce_sched_core_t* core)` | One-time setup, receives core reference for state queries | - | `fini` | `void (*fini)(void* picker)` | Teardown and cleanup of all picker-owned resources | - - **State-change hooks** (MUST — called by core when events arrive from driver): - - | Operation | Signature | Semantics | Triggered by | - |-----------|-----------|-----------|-------------| - | `on_queue_update` | `int (*on_queue_update)(void* picker, xce_sched_queue_t* queue)` | Add/update queue in picker's internal ordering | `QUEUE_UPDATE` event | - | `on_job_push` | `int (*on_job_push)(void* picker, xce_sched_job_t* job)` | Add new pending job to picker's candidate set | `JOBS_PUSH` event | - | `on_job_state_change` | `int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t new_state)` | Remove job from candidates if no longer pending; re-add if returning to pending | `JOB_STATE_CHANGE` event | - | `on_job_update` | `int (*on_job_update)(void* picker, xce_sched_job_t* job)` | Reorder job in candidates if priority/queue changed | `JOB_UPDATE` event | - - **Session hooks** (MUST — called at round boundaries): - - | Operation | Signature | Semantics | - |-----------|-----------|-----------| - | `begin_session` | `int (*begin_session)(void* picker)` | Reset iteration cursor; called at start of each scheduling round | - | `next_job` | `xce_sched_job_t* (*next_job)(void* picker)` | Return next schedulable job per picker's ordering, or NULL when exhausted | - | `end_session` | `int (*end_session)(void* picker)` | End-of-round notification; called before core sleeps between rounds | - - **Post-allocation hooks** (MUST — called after allocation attempt): - - | Operation | Signature | Semantics | - |-----------|-----------|-----------| - | `on_dispatch` | `int (*on_dispatch)(void* picker, uint64_t jobid)` | Job was allocated; remove from candidates permanently | - | `on_pend` | `int (*on_pend)(void* picker, uint64_t jobid)` | Job was pended; keep for next round but may deprioritize | - - **Optional hooks** (MAY — NULL if not needed): - - | Operation | Signature | Semantics | When needed | - |-----------|-----------|-----------|------------| - | `on_config` | `int (*on_config)(void* picker, xce_conf4sched_t* config)` | Rebuild picker strategy from config | Config-driven pickers (deadline, fair-share weights) | - | `on_job_rusage` | `int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages)` | Track per-job resource usage | Resource-aware pickers (fair-share, backfill) | - - Hooks not defined on the picker interface (worker updates, resource updates, resval updates) are not forwarded to the picker — picking is about job ordering, not resource availability. - -3. The core shall call state-change hooks synchronously during `core_push_event()` processing, before the event returns. Session hooks and post-allocation hooks are called synchronously during the scheduling session (between `BEGIN_ROUND` and session exhaustion). The picker runs in the core's scheduling thread — no dedicated picker thread. - -4. The core scheduling loop shall: - - ```c - begin_session(); - while ((job = picker->next_job(picker)) != NULL) { - if (try_allocate(job) > 0) { - picker->on_dispatch(picker, job->jobid); - emit_decision(DISPATCH, job); - } else if (can_preempt(job)) { - emit_decision(PREEMPT, job, victims); - picker->on_pend(picker, job->jobid); // retry next round - } else { - emit_decision(PEND, job); - picker->on_pend(picker, job->jobid); - } - } - end_session(); - ``` - - The core loop shall contain zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`. - -5. The picker plugin shall be cancel-safe: `next_job()` shall have defined `pthread_cancel` cancellation points. The picker shall be session-aware: `begin_session()` resets iteration state, `end_session()` is called before sleep-between-rounds. - -6. The hardcoded `"xce_sched_fcfs"` literal at `scheduler.c:43` (in `sched_job_picker()`) shall be replaced by loading the configured `sched->job_pick_plugin` via the existing `plugin_register`/`key_find` mechanism. If the configured plugin name is not registered, the Scheduler shall fail with a clear configuration error (same pattern as Req 5 AC 5). - -7. `sched_job_picker()`'s dedicated thread and `jbuf` cbuf communication shall be retired. The picker runs synchronously within the core's scheduling thread — `next_job()` is called directly, not via a separate thread + cbuf. This eliminates one thread and simplifies the session lifecycle for simulation mode. - -### Requirement 2: Lazy allocation plan iterator - -**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front. - -#### Acceptance Criteria -1. The Scheduler shall define an `xce_alloc_plan_iter_t` interface with `init(resreq, sched)`, `next() → plan`, and `fini()` operations. -2. Each call to `next()` shall generate exactly one candidate `(nhosts, slots_per_host, host_filter)` plan without generating or materializing any other plan in the same call. -3. The Scheduler shall order plans returned by successive `next()` calls from most-preferred to least-preferred, per the owning allocator's preference policy. -4. `next()` shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion. -5. Calling `fini()` shall release all resources held by the iterator, and the interface shall define no other required cleanup step. - -### Requirement 3: Lazy host candidate iterator - -**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering. - -#### Acceptance Criteria -1. The Scheduler shall define an `xce_host_iter_t` interface with `init(plan, workers)`, `next() → worker`, and `fini()` operations. -2. Each call to `next()` shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator. -3. `next()` shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call `next()` again after receiving NULL for that plan. -4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()`. -5. Calling `fini()` shall release all resources held by the host iterator, independent of the plan iterator's lifecycle. - -### Requirement 4: Parallel job combinatorial allocation - -**Objective:** As a cluster operator submitting a job that needs multiple task slots, I want `parjob_allocator` to actually allocate across hosts, so that jobs with `maxTasks > 1` are no longer silently unallocatable. - -#### Acceptance Criteria -1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host × N slots, 2 hosts × ⌈N/2⌉ slots, …, N hosts × 1 slot, using Requirement 2's plan iterator. -2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan. -3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job. -4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected. -5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents — the current stub behavior — for any job with `maxTasks > 1`. - -### Requirement 5: Configurable picker plugin - -**Objective:** As a cluster operator, I want to configure which job-picking policy the scheduler uses without recompiling, and override it per-queue, so that FCFS, priority, or custom pickers can be swapped operationally. - -#### Acceptance Criteria -1. The Scheduler shall add a `job_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_fcfs"` when unset — this is the global picker plugin name. - -2. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_queue_t`, defaulting to NULL (inherit from global). When non-NULL, the configured global picker plugin shall use this per-queue override for jobs within that queue. How the picker plugin uses per-queue overrides is an implementation detail of the plugin, not the core loop. - -3. The Scheduler shall add a `project_name` field to `xce_sched_queue_t`, set from queue configuration, establishing the queue→project relationship. The picker plugin may use this field internally for project-scoped iteration. - -4. The Scheduler shall accept the picker plugin name from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild. - -5. When the configured plugin name is not registered via the existing `plugin_register`/`key_find` mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default. - -6. The picker plugin interface contract (`xce_job_picker_iface_t`) shall be documented: each picker must be cancel-safe (responds to `pthread_cancel` at defined points in `next_job()`) and session-aware (resets its iteration state on `begin_session()`). - -### Requirement 6: Preemption policy and execution - -**Objective:** As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention. - -#### Acceptance Criteria -1. The Scheduler shall add a `XCE_DECISION_PREEMPT` opcode to `xce_decision_op_t` in `decision.h`, with a payload type `xce_preempt_decision_t` carrying `victim_jobids[]`, `n_victims`, `signal` (SIGSTP in v1), and `preempting_jobid`. -2. When allocation fails for a job J whose queue has a non-NULL `preemptible_queues` list, the Scheduler shall invoke the queue's `preempt_policy` plugin to select victim jobs from the target queues. -3. The default preemption policy `"lowest_priority_first"` shall: collect running jobs from target queues, filter to those with `priority < J.priority` and `queue->preemptible == true`, rank by priority ascending, and select the minimum subset whose freed resources satisfy J's resource request. -4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via `sched_commit_decision(XCE_DECISION_PREEMPT)`, publish it through the controller→worker pipeline, and pend job J for the next scheduling round (async preemption — the job is picked up after resources are freed). -5. When no sufficient victim set exists, the Scheduler shall pend job J with a resource-shortage reason without invoking preemption. -6. The worker receiving a preemption action shall send the specified signal (SIGSTP) to the victim job's process groups, and the controller shall transition the victim job's state to `JOB_SUSPENDED`. -7. The Scheduler shall add `preemptible_queues` (list of queue names this queue CAN preempt from), `preemptible` (bool — CAN this queue's jobs BE preempted by others), and `preempt_policy` (victim-selection plugin name, NULL = default) fields to `xce_sched_queue_t`. -8. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum shall be reused for preemption stop notifications. -9. Preemption victim selection shall never select a job whose priority is greater than or equal to the preempting job's priority, nor a job whose queue has `preemptible == false`. - -### Requirement 7: Session-level unsatisfiable resreq cache - -**Objective:** As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost. - -#### Acceptance Criteria -1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by a fingerprint of `{select_string, rusage_string, user, queue, project}`. -2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`). -3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation AND preemption (Requirement 6) also fails or is inapplicable, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. -4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption. -5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped. - -### Architectural Support for Future Object Sharing (Deferred) - -Full object sharing (resreq template registry, pend reason dedup pool) is deferred to a follow-up spec. However, the architecture designed in this spec must not block sharing from being added later. The following design invariants apply to all resreq, pend-reason, and related objects touched by this spec: - -1. **Resreq immutability**: `xce_primary_resreq_t`, `xce_compound_resreq_t`, and `xce_alter_resreq_t` objects shall be treated as immutable once created and validated. No code path shall mutate a resreq in place. Any modification (e.g., expanding a job-array template into per-element resreqs) shall create a new copy. This ensures that sharing via `const` pointer is safe to introduce later without auditing every consumer for mutation. - -2. **Pend reason factory**: All pend reason allocation shall go through `make_pendreason()` (already the single factory in `lib/preason.c`). Callers never `calloc(sizeof(xce_pendreason_t))` directly. This ensures a session-level dedup pool can be added inside the factory later without changing any caller. - -3. **Pointer-identity fast path**: Code that compares two resreqs or two pend reasons shall use pointer equality (`a == b`) as a fast path before falling back to deep comparison. When a resreq registry is introduced, identical resreqs will have the same pointer — the fast path avoids repeated deep comparison for no runtime cost today. - -4. **Select expression memoization**: The existing `select_eval_ctx_t` per-session memoization of parsed select-expression ASTs shall be preserved and extended to the core's internal resreq objects. The parsed/evaluated form of a select expression shall be attached to the resreq on first evaluation and reused for subsequent evaluations (same session, same resreq pointer). - -5. **Snapshot forward-compatibility**: The snapshot format shall not preclude object sharing. Shared objects in the snapshot shall be serializable by identity (an object table with IDs) rather than requiring inline embedding at every reference site. This is a design-phase concern but constrains the JSON schema choice. - -These invariants impose zero overhead in the current implementation — immutability is already the pattern, pointer-identity fast paths are already valid C, and the factory already exists. They simply rule out design choices that would make sharing expensive or impossible to retrofit. - -### Requirement 8: Builtin default picker - -**Objective:** As a maintainer, I want the new architecture to ship with a builtin picker plugin that replicates today's exact scheduling behavior, so that operators who don't configure a custom picker see no behavioral change. - -#### Acceptance Criteria - -1. **Registration**: `builtin_fcfs` shall be registered automatically at scheduler startup as `"builtin_fcfs"` and set as the default `sched->job_pick_plugin`. - -2. **Hook implementation** — `builtin_fcfs` shall implement every MUST and SHOULD hook from `xce_job_picker_iface_t` (Req 1 AC 2) as follows: - - | Hook | FCFS Implementation | - |------|-------------------| - | `init` | Allocate `fcfs_plugin_data_t`. Initialize `ordq_t queues` (ordered by `queue->priority desc`). Set iteration cursor to NULL. | - | `fini` | Free all per-queue `ordq_t pendJobs`, the `ordq_t queues`, and plugin data. | - | `on_queue_update` | Upsert queue into `ordq_t queues`. If queue exists, update its priority and reorder. If new, allocate `fcfs_queue_t` with empty `ordq_t pendJobs`. | - | `on_job_push` | If `job->state & JOB_STAT_PEND`: find the job's queue in `ordq_t queues`, add `fcfs_job_t{job}` to that queue's `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). | - | `on_job_state_change` | If `new_state` is not pending (EXIT/FAIL/DONE): remove job from its queue's `pendJobs` by jobid. If `new_state` returns to PEND: re-add to `pendJobs`. | - | `on_job_update` | If job's `priority` or `queue` field changed: remove from old position in `pendJobs`, re-insert at correct position per ordering. | - | `begin_session` | Reset iteration cursor to head of `ordq_t queues`. | - | `next_job` | Walk `ordq_t queues` from cursor. For each queue, skip if `sched_queue_is_ready()` returns false. Within queue, walk `ordq_t pendJobs`. Return first job where `sched_job_is_ready()` returns true. Advance cursor. Return NULL when all queues and jobs exhausted. | - | `end_session` | No-op for FCFS (iteration state reset in `begin_session`). | - | `on_dispatch` | Remove job from its queue's `ordq_t pendJobs` by jobid. | - | `on_pend` | No-op for FCFS — job remains in `pendJobs`, will be retried next round in same position. | - | `on_config` | NULL — FCFS is config-agnostic. | - | `on_job_rusage` | NULL — FCFS is not resource-aware. | - -3. **Internal data structures** — `builtin_fcfs` shall maintain its own private state, not query the core's `sched->queues`/`sched->jobs` during `next_job()`. All candidate management happens in the state-change hooks. This ensures: - - `next_job()` is fast — no core lock acquisition, no hash table scans - - The picker is independently testable — feed hooks, verify `next_job()` order - - Simulation snapshots capture picker state naturally via the hooks already recorded in the event stream - -4. **Ordering equivalence** — `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`: inter-queue by descending `queue->priority`, intra-queue by `priority desc, queue_time asc`. Behavior with per-queue picker overrides is as specified in Req 5 AC 2. - -5. **No dedicated thread** — `builtin_fcfs` shall implement `next_job()` as a synchronous call, not a dedicated thread + `jbuf` cbuf. The picker runs in the core's scheduling thread. `pthread_cancel` cancellation points shall exist in the `next_job()` iteration loop. - -6. **Session state** — `end_session()` is a no-op. Iteration state is reset in `begin_session()`. Undispatched jobs remain in `pendJobs` across sessions (they carry over to the next round). - -### Requirement 9: No regression to existing allocators, job picker, or HA soak - -**Objective:** As a maintainer, I want the iterator-driven rework and picker architecture scoped so that existing single-slot allocation and FCFS behavior are unaffected, so that this framework change is safe to ship independently of any new policy or allocator. - -#### Acceptance Criteria -1. `seqjob_allocator`'s externally observable allocation behavior (which host a `maxTasks == 1` job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the `xce_host_iter_t` interface. -2. `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`, given the same queue and job state — same inter-queue priority ordering and same intra-queue (priority desc, queue_time asc) ordering. -3. The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today. -4. Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string, dedicated picker thread, `jbuf` cbuf, or eager-one-shot behavior this feature replaces. -5. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. - -### Requirement 10: Core/Driver Separation with Dual-Mode Operation - -**Objective:** As a scheduler developer, I want the scheduling core separated from all external I/O so that the same core logic runs deterministically in production (driven by a live controller) and in simulation (driven by recorded cluster events with a discrete event-driven virtual clock), enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. - -#### Acceptance Criteria - -**Core Interface Contract** - -1. **Mode-agnostic core**: The Scheduler core shall have no concept of which mode it runs in. It processes events identically whether the driver is in normal mode (controller-connected, wall-clock time) or simulation mode (replay file, virtual clock). Specifically: - - There shall be no mode flag (`bool simulation_mode` or equivalent) anywhere in the core - - There shall be no `if (simulation) { ... } else { ... }` code path in the core - - `xce_sched_core_init()` shall take identical parameters regardless of mode (no mode-specific init arguments) - - The core's decision callback shall be the same function pointer type in both modes — the driver decides what to do with decisions (publish to controller vs. log to file) - - The core's `save_snapshot()` and `load_snapshot()` shall be identical in both modes — they serialize/deserialize core state, not driver state - -2. **Passive event-driven interface**: The core shall expose a pure event-driven interface — it receives typed events from the driver, updates internal scheduling state, and emits decisions through a driver-registered callback. The core shall have zero direct dependency on: controller communication, capnp serialization, network I/O, filesystem access, or wall-clock time sources. - -3. **Virtualized clock access**: The core shall access time exclusively through `xce_core_now()`. The core shall never call `time(NULL)`, `clock_gettime()`, `gettimeofday()`, or any other wall-clock time function directly. - - **Clock virtualization mechanism — two-base + speed formula:** - - The core stores three fields (atomically updated by the driver as a group): - - | Field | Type | Description | - |-------|------|-------------| - | `virtual_clock_base` | `_Atomic uint64_t` | Virtual-time anchor (nanoseconds) | - | `real_clock_base` | `_Atomic uint64_t` | Wall-clock anchor captured when `virtual_clock_base` was set (nanoseconds, `CLOCK_MONOTONIC`) | - | `speed_num` / `speed_den` | `_Atomic uint64_t` | Speed multiplier as rational fraction | - - `xce_core_now(core)` computes: - ```c - uint64_t wall_ns = clock_gettime(CLOCK_MONOTONIC); // only wall-clock call in the core - uint64_t elapsed = wall_ns - core->real_clock_base; - return core->virtual_clock_base + (elapsed * core->speed_num) / core->speed_den; - ``` - - | Mode | `virtual_clock_base` | `real_clock_base` | `speed` | Behavior | - |------|----------------------|-------------------|---------|----------| - | Normal | 0 | 0 | 1/1 | `xce_core_now()` = wall clock (degraded: `0 + (wall - 0) × 1`) | - | Simulation discrete | event `ts` | wall at load | 0/1 | `xce_core_now()` = frozen at event `ts` (time does not advance between events) | - | Simulation real-time | snapshot `ts` | wall at load | 1/1 | `xce_core_now()` tracks wall clock from virtual anchor | - | Simulation fast/slow | snapshot `ts` | wall at load | N/1 or 1/N | `xce_core_now()` runs N× faster or slower than wall clock | - - The driver atomically updates all three fields before each event dispatch. In simulation discrete mode (speed=0/1), the clock jumps directly from one event `ts` to the next — no continuous advancement. In normal mode (bases=0, speed=1/1), the formula degrades to returning wall-clock time directly, with zero overhead beyond the single `clock_gettime()` call. - - **Multi-thread safety**: `xce_core_now()` reads atomic fields lock-free. Both picker and allocator threads independently compute the same virtual time from the shared bases — no mutex needed. The driver updates all three fields as an atomic group (via a single mutex or sequenced atomic stores with a version counter). - - - All internal time uses go through `xce_core_now()`: scheduling logic (job queue-time ordering, session boundaries, pend-reason timestamps, job aging), **logging** (log message timestamps via tlog format function), **error messages**, **debug traces** — any code path that needs the current time calls `xce_core_now()` - - Job `sub_time` and `queue_time` are set by the driver (from `xce_core_now()` in normal mode, from event `ts` in simulation mode) before pushing the `JOBS_PUSH` event — the core never generates timestamps for externally-visible fields - - This virtualization is what enables deterministic replay: the same event file produces the same `xce_core_now()` return values (and therefore same log timestamps, same scheduling decisions) regardless of wall-clock speed - - **Third-party library constraint**: any library linked into the scheduler core that needs the current time must obtain it through `xce_core_now()` or a core-provided wrapper — never by calling `time()`, `clock_gettime()`, `gettimeofday()`, or similar directly. Specifically: - - **Logging (tlog)**: The `tlog` library internally captures `struct tlog_time` from wall-clock time before calling the registered format callback (`my_tlog_format`). The core shall register its own format function via `tlog_reg_format_func(fn, core_ptr)` where `core_ptr` is the `xce_sched_core_t*`. This format function shall ignore `info->time` entirely and instead read `xce_core_now(core_ptr)`, converting nanoseconds to `struct tm` fields via `gmtime_r()` (deterministic — pure computation from an integer, no system clock access). The `userptr` parameter of `tlog_reg_format_func()` carries the core pointer into the format callback. - - **libxce subset**: The core shall link only against the minimal data-structure subset of libxce (`ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti`) — not against `netio`, `delay`, `xceapi`, or `idgen` which call wall-clock time internally - - **pthread timed-blocking**: The core is multi-threaded (picker + allocator threads). In simulation mode, thread interleaving is made deterministic per AC 6 — `pthread_cond_timedwait()` and `pthread_mutex_timedlock()` are not needed in the deterministic execution path - - If a library cannot avoid calling wall-clock time directly (e.g., a third-party compression or serialization library that internally timestamps), it must not be linked into the core; such concerns stay in the driver layer - -4. **Core event types**: The core interface shall accept at minimum these input events: - - | Event Type | Payload | Description | - |------------|---------|-------------| - | `CONFIG_PUSH` | `xce_conf4sched_t*` | Full configuration push (startup or config change) | - | `JOBS_PUSH` | `int njobs, xce_job_t**` | Newly submitted jobs | - | `WORKERS_PUSH` | `int nworkers, xce_worker_t**` | Worker list update | - | `RESOURCES_PUSH` | `int n, xce_resource_t**` | Resource definitions | - | `RESVALS_FULL` | `int n, xce_resval_t**` | Full resource value snapshot | - | `RESVALS_DELTA` | `xce_resval_list_t*` | Incremental resource value update | - | `JOB_STATE_CHANGE` | `uint64_t jobid, uint32_t state` | Job started / exited / failed | - | `JOB_UPDATE` | `xce_job_t*` | In-place job field update (notify path) | - | `QUEUE_UPDATE` | `xce_queue_t*` | Queue definition update | - | `JOB_RUSAGE` | `xce_job_rusages_t*` | Job resource usage update | - | `BEGIN_ROUND` | `uint64_t timestamp_ns` | Trigger scheduling round at given logical time | - -5. **Decision output callback**: The core shall emit all scheduling decisions (`DISPATCH`, `PEND`, `PREEMPT`) through a `xce_core_decision_cb_t` callback function pointer registered by the driver at core init time, replacing the current internal `decisions` cbuf + `main.c` poll loop pattern. The callback signature shall be `int (*on_decision)(void* driver_ctx, xce_decision_t* decision)`. - -6. **Deterministic guarantee**: Given identical initial state and identical event sequences with identical timestamps, the core shall produce identical decision sequences. The core is multi-threaded (picker thread + allocator thread, matching today's architecture). To achieve deterministic replay: - - All hash table / key-value store iteration that affects scheduling output order shall be stabilized (sorted by key before iterating) - - Any required randomness (e.g., tie-breaking) shall use a PRNG seeded by the driver at init time - - In simulation mode, thread interleaving shall either be made deterministic (e.g., the simulation driver serializes event dispatch and decision collection through a single-threaded core path), or the core's threads shall synchronize on a simulation barrier that advances in lock-step with the virtual clock - -7. **Driver event queue**: The driver shall retain an internal pending-event queue (the existing `xce_sched_update_request_t` / `pend_updates` mechanism, or equivalent) to buffer events received from the controller between scheduling rounds. Events accumulate in the driver between `BEGIN_ROUND` invocations. When the core receives `BEGIN_ROUND`, it drains all buffered events, processes a scheduling session, and emits decisions. - -**Normal-Mode Driver** - -8. **Controller integration**: The normal-mode driver shall receive capnp push/notify messages from the controller (via the existing `scheduler_req_handler` callback), translate each to the corresponding core event type, and invoke the core's event push interface. The driver shall publish core-emitted decisions back to the controller via the existing `xcecli_ctx_t` API (`sched_publish_dispatch` / `sched_publish_pend_reason` semantics, generalized to include `PREEMPT`). - -9. **Runtime recording control via xceadmin**: The scheduler shall support two new xceadmin commands: - - `scheduler record start [FILE]` — take a full state snapshot, write it to the replay file, then begin recording all subsequent core input events and output decisions to the file - - `scheduler record stop` — stop recording, close the replay file, resume normal operation - - Recording shall have zero impact on scheduling correctness — the core processes identically whether recording is active or not. - -10. **Periodic snapshot during recording**: While recording is active, the driver shall periodically append fresh snapshots to the replay file (default: every `N` events, configurable via `--record-snapshot-interval N`, where N=0 disables periodic snapshots and records only the initial snapshot). This bounds replay time for any desired start point. - -**Simulation-Mode Driver** - -11. **Simulation mode startup**: The scheduler shall support a simulation mode activated by `--sim FILE` (replay file path) and `--sim-start-ts TIMESTAMP` (logical time to begin replay, default: start of file). In simulation mode, the driver loads the most recent snapshot with `ts <= --sim-start-ts`, then replays all delta events from that point forward. - -12. **Discrete event-driven virtual clock**: The simulation driver shall implement a virtual clock that advances by jumping directly to the timestamp of the next queued event — no fixed tick interval, no wall-clock waiting between events. The clock shall be strictly monotonically increasing. When multiple events share the same timestamp, they shall be delivered in file order. The driver shall support: - - `--sim-speed N` — rate limit: N=0 means "as fast as possible" (no delay), N=1 means real-time, N>1 means N× real-time - - `--sim-pause-at TIMESTAMP` — pause simulation at a specific logical time for interactive debugging - -13. **Deterministic replay verification**: In simulation mode with `--sim-verify`, the driver shall compare each core-emitted decision against the recorded decisions in the replay file. Any mismatch (missing, extra, or different decision) shall be reported as a regression with exact logical-time and decision-detail context. - -14. **What-if branching**: The simulation driver shall accept `--sim-alt-events FILE` (alternative delta event file). When both a base replay file (for snapshot loading) and an alt-events file are provided, the driver loads the snapshot from the base file, then replays delta events from the alt-events file — enabling "what-if" analysis (e.g., "what if job X had been submitted with higher priority?"). - -**Replay File Format** - -15. **JSONL format**: The replay file shall be newline-delimited JSON (JSONL), one event per line: - - ```jsonl - {"type":"HEADER","version":"x.y.z","config_hash":"sha256...","start_ts":1234567890000000000,"start_wall":"2026-07-27T12:00:00Z"} - {"type":"SNAPSHOT","ts":1234567890000000000,"data":{...full core state...}} - {"type":"CONFIG_PUSH","ts":1234568000000000000,"payload":{...}} - {"type":"JOBS_PUSH","ts":1234568100000000000,"payload":{"njobs":2,"jobs":[...]}} - {"type":"BEGIN_ROUND","ts":1234568200000000000,"payload":{}} - {"type":"DECISION","ts":1234568200000000000,"payload":{"op":"DISPATCH","jobid":"..."}} - ``` - - - `type`: event type name string - - `ts`: nanosecond logical timestamp (monotonic, unsigned 64-bit integer) - - `payload`: event-type-specific object - - `HEADER` line: first line of every file, carries scheduler version and configuration hash for compatibility checking - - `SNAPSHOT` lines: embed full core state in `data` (plain JSON, not base64-encoded — debuggable and directly extractable) - - `DECISION` lines: recorded decisions (output), present only in files produced by the normal-mode driver's recording feature — the simulation driver ignores them unless `--sim-verify` is active - - The event payload JSON schema is an open design question deferred to the design phase - -**State Snapshot** - -16. **Snapshot serialization**: The core shall support `save_snapshot()` producing a full state dump (plain JSON) containing all config, queues (with per-queue picker/preemption settings), jobs (with state, priority, queue-time, resreq), workers (with available/used resources), resources, resvals, users, projects, shared-resource tracking state, unsatisfiable-resreq cache, and session state. The core shall support `load_snapshot()` restoring to that exact state, producing a core that is indistinguishable from one that reached that state by replaying all events up to that timestamp. - -**Interaction with Existing Requirements** - -Reqs 1–9 describe logic **inside** the core — the picker loop, iterators, parjob allocator, preemption policy, resreq cache, and builtin plugins all operate within the core's event-driven boundary. Requirement 10 defines the boundary itself. Specifically: - -- Req 5 (picker slots from config): config arrives via `CONFIG_PUSH` event, not via capnp decode inside the core -- Req 6 (preemption → controller): `XCE_DECISION_PREEMPT` is emitted through the decision callback, not sent directly to controller -- Req 7 (resreq cache, session lifecycle): `BEGIN_ROUND` replaces the implicit `sleep(1)`-delimited session boundary in `main.c`; the cache still lives on `xce_sched_core_t` and is cleared at session start -- Req 9 (HA soak regression): the same core code can be validated in simulation mode against recorded HA failover traces, catching regressions without a live cluster - -## Data Structure Changes Summary - -### `xce_scheduler_t` additions -```c -char* job_pick_plugin; // default: "builtin_fcfs" (Req 5) -// unsatisfiable resreq cache (Req 7) -``` - -### `xce_sched_queue_t` additions -```c -char* project_name; // owning project (picker uses internally) -char* job_pick_plugin; // per-queue picker override (NULL = use global) -char** preemptible_queues; // queues this queue CAN preempt from -int n_preemptible_queues; -bool preemptible; // CAN this queue's jobs BE preempted? -char* preempt_policy; // victim selection plugin (NULL = default) -``` - -### `xce_sched_project_t` additions -```c -int32_t priority; // inter-project ordering (future spec) -// future: resource_quota, resource_used, parent_project (project-resource-ownership spec) -``` - -### `xce_job_picker_iface_t` (new, Req 1) -```c -typedef struct { - int (*init)(void* picker, xce_sched_core_t* core); - void (*fini)(void* picker); - int (*begin_session)(void* picker); - xce_sched_job_t* (*next_job)(void* picker); - int (*end_session)(void* picker); - int (*on_dispatch)(void* picker, uint64_t jobid); - int (*on_pend)(void* picker, uint64_t jobid); -} xce_job_picker_iface_t; -``` - -### `decision.h` additions -```c -typedef enum { - XCE_DECISION_NONE = 0, - XCE_DECISION_DISPATCH, - XCE_DECISION_PEND, - XCE_DECISION_PREEMPT, // NEW - XCE_DECISION_NEW_SESSION -} xce_decision_op_t; - -typedef struct { - uint64_t* victim_jobids; // jobs to preempt - int n_victims; - int signal; // SIGSTP (v1), SIGKILL (future) - uint64_t preempting_jobid; // job that triggered preemption -} xce_preempt_decision_t; -``` - -### Core/Driver Separation (Req 10) - -```c -// Core event (driver → core) -typedef enum { - XCE_CORE_EVENT_CONFIG_PUSH = 0, - XCE_CORE_EVENT_JOBS_PUSH, - XCE_CORE_EVENT_WORKERS_PUSH, - XCE_CORE_EVENT_RESOURCES_PUSH, - XCE_CORE_EVENT_RESVALS_FULL, - XCE_CORE_EVENT_RESVALS_DELTA, - XCE_CORE_EVENT_JOB_STATE_CHANGE, - XCE_CORE_EVENT_JOB_UPDATE, - XCE_CORE_EVENT_QUEUE_UPDATE, - XCE_CORE_EVENT_JOB_RUSAGE, - XCE_CORE_EVENT_BEGIN_ROUND, -} xce_core_event_type_t; - -typedef struct { - xce_core_event_type_t type; - uint64_t ts; // driver-supplied logical time (ns) - union { - xce_conf4sched_t* config; - struct { int njobs; xce_job_t** jobs; } job_push; - struct { int nworkers; xce_worker_t** workers; } worker_push; - struct { int n; xce_resource_t** resources; } resource_push; - struct { int n; xce_resval_t** vals; } resval_full; - xce_resval_list_t* resval_delta; - struct { uint64_t jobid; uint32_t state; } job_state; - xce_job_t* job_update; - xce_queue_t* queue_update; - xce_job_rusages_t* job_rusage; - // BEGIN_ROUND has no payload beyond ts - } payload; -} xce_core_event_t; - -// Decision callback (core → driver) -typedef int (*xce_core_decision_cb_t)(void* driver_ctx, xce_decision_t* decision); - -// Core struct (pure scheduling state, no I/O) -typedef struct { - // Scheduling state (moved from xce_scheduler_t) - xce_conf4sched_t* config; - uint64_t configver; - var plugins; - void* seqjob_allocator; - void* parjob_allocator; - void* affinity_allocator; - void* gpu_allocator; - var resmgr, ugmgr, qgmgr, pgmgr, hgmgr; - zarray_t sharedInst, sharedInUse; - var jobs, queues, workers, users, projects; - uint64_t nextid; - - // New fields for core/driver boundary - void* driver_ctx; // opaque driver context for callback - xce_core_decision_cb_t on_decision; // decision output callback - // Clock virtualization (Req 10 AC 3) — atomically updated by driver - _Atomic uint64_t virtual_clock_base; // virtual-time anchor (ns) - _Atomic uint64_t real_clock_base; // wall-clock anchor (ns, CLOCK_MONOTONIC) - _Atomic uint64_t speed_num; // speed numerator - _Atomic uint64_t speed_den; // speed denominator - uint64_t prng_seed; // seeded by driver at init - - // Picker plugin slot (Req 5) - char* job_pick_plugin; // default: "builtin_fcfs" - // per-queue overrides stored on xce_sched_queue_t.job_pick_plugin, - // read by the picker internally — not separate core slots - - // Unsatisfiable resreq cache (Req 7) - // ... (session-level fingerprint cache) - - // Session state - _Atomic int session_active; - _Atomic int eval_in_progress; - select_eval_ctx_t* sel_eval_ctx; -} xce_sched_core_t; - -// Normal-mode driver -typedef struct { - xce_sched_core_t core; - xcecli_ctx_t apictx; // controller API client - xce_cluster_config_t cluster_cfg; // HA candidate list - dataq_t* pend_updates[2]; // event buffer between rounds - // Recording state - FILE* record_file; - bool recording; - uint64_t record_snapshot_interval; - uint64_t events_since_snapshot; -} xce_sched_driver_t; - -// Simulation-mode driver -typedef struct { - xce_sched_core_t core; - FILE* replay_file; - uint64_t virtual_clock; // current logical time (ns) - uint64_t sim_speed; // 0=unlimited, 1=real-time, N=N× - uint64_t pause_at_ts; // 0=don't pause - bool verify; // compare decisions vs recorded - FILE* alt_events_file; // for what-if branching -} xce_sched_sim_driver_t; - -// Core interface -int xce_sched_core_init(xce_sched_core_t* core, xce_core_decision_cb_t cb, - void* driver_ctx, uint64_t prng_seed); -int xce_sched_core_push_event(xce_sched_core_t* core, xce_core_event_t* event); -uint64_t xce_core_now(xce_sched_core_t* core); // virtualized clock accessor -int xce_sched_core_save_snapshot(xce_sched_core_t* core, FILE* out); -int xce_sched_core_load_snapshot(xce_sched_core_t* core, FILE* in); -void xce_sched_core_fini(xce_sched_core_t* core); -``` - -## Architecture Diagram - -### Core Scheduling Loop - -``` -┌──────────────────────────────────────────────────────────┐ -│ SCHEDULER CORE LOOP │ -│ │ -│ begin_session() │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────┐ │ -│ │ picker->next_job() │ ←─ plugin decides HOW │ -│ │ returns job or NULL │ (FCFS, priority, fair-share, │ -│ └──────────┬───────────┘ deadline, ...) │ -│ │ │ -│ job │ NULL (exhausted) │ -│ ▼ ▼ │ -│ ┌──────────────────┐ end_session() → sleep → next round │ -│ │ try_allocate(job) │ │ -│ └────────┬─────────┘ │ -│ │ │ -│ success │ fail │ -│ ▼ ▼ │ -│ ┌────────────┐ ┌────────────────┐ │ -│ │ DISPATCH │ │ can_preempt? │ │ -│ │ on_dispatch│ │ yes → PREEMPT │ │ -│ └────────────┘ │ no → PEND │ │ -│ │ on_pend │ │ -│ └────────────────┘ │ -│ │ │ -│ ▼ │ -│ next_job() (loop continues) │ -└──────────────────────────────────────────────────────────┘ -``` - -### `builtin_fcfs` Internal Design - -The default picker plugin internally implements the equivalent of today's selection logic. This is **not** part of the core loop — it's the plugin's private implementation: - -``` -builtin_fcfs.next_job(): - ┌────────────────────────────────────────────┐ - │ Iterate queues by priority (descending) │ - │ Skip queues with no ready jobs │ - │ │ - │ For each queue: │ - │ If queue has per-queue picker override: │ - │ Delegate to that picker │ - │ Else: │ - │ Iterate jobs by priority desc, │ - │ queue_time asc (FCFS) │ - │ Skip jobs not ready │ - │ Return first ready job │ - │ │ - │ Return NULL if all queues exhausted │ - └────────────────────────────────────────────┘ -``` - -## Core/Driver Architecture (Req 10) - -``` -┌──────────────────────────────────────────────────────────────────┐ -│ DRIVER LAYER │ -│ │ -│ ┌──────────────────────┐ ┌────────────────────────────┐ │ -│ │ NORMAL MODE │ │ SIMULATION MODE │ │ -│ │ │ │ │ │ -│ │ controller capnp ────┐│ │ replay.jsonl ──────────┐ │ │ -│ │ (scheduler_req_ ││ │ (event file) │ │ │ -│ │ handler) ││ │ │ │ │ -│ │ │ ││ │ │ │ │ │ -│ │ ▼ ││ │ ▼ │ │ │ -│ │ ┌──────────────┐ ││ │ ┌──────────────┐ │ │ │ -│ │ │ event queue │ ││ │ │ virtual clock│ │ │ │ -│ │ │ (pend_updates)│ ││ │ │ (discrete │ │ │ │ -│ │ └──────┬───────┘ ││ │ │ event-driven)│ │ │ │ -│ │ │ ││ │ └──────┬───────┘ │ │ │ -│ │ │ events ││ │ │ events │ │ │ -│ │ │ + BEGIN_ROUND││ │ │ + BEGIN_ROUND │ │ │ -│ │ ▼ ││ │ ▼ │ │ │ -│ │ ┌──────────────┐ ││ │ ┌──────────────┐ │ │ │ -│ │ │ publish │ ││ │ │ record/log │ │ │ │ -│ │ │ decisions to │ ││ │ │ decisions │ │ │ │ -│ │ │ controller │ ││ │ │ (--sim-verify)│ │ │ │ -│ │ └──────▲───────┘ ││ │ └──────▲───────┘ │ │ │ -│ │ │ decisions ││ │ │ decisions │ │ │ -│ └─────────┼─────────────┘│ └─────────┼───────────────┘ │ -│ │ │ │ │ -│ │ xce_core_decision_cb_t │ │ -│ │ (callback) │ │ -└────────────┼──────────────────────────────────┼───────────────────┘ - │ │ - ▼ ▼ -┌──────────────────────────────────────────────────────────────────┐ -│ SCHEDULER CORE │ -│ (xce_sched_core_t) │ -│ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ xce_sched_core_push_event(core, event) │ │ -│ │ │ │ -│ │ Event types: CONFIG_PUSH, JOBS_PUSH, WORKERS_PUSH, │ │ -│ │ RESOURCES_PUSH, RESVALS_FULL, RESVALS_DELTA, │ │ -│ │ JOB_STATE_CHANGE, JOB_UPDATE, QUEUE_UPDATE, │ │ -│ │ JOB_RUSAGE, BEGIN_ROUND │ │ -│ └──────────────────────────┬──────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ Internal Scheduling Logic (Reqs 1–9) │ │ -│ │ │ │ -│ │ picker->next_job() → try_allocate(job) │ │ -│ │ │ │ │ │ -│ │ │ xce_alloc_plan_iter_t │ │ │ -│ │ │ → xce_host_iter_t │ │ │ -│ │ │ → try alloc │ │ │ -│ │ │ │ │ │ -│ │ │ success → DISPATCH │ │ │ -│ │ │ fail → PREEMPT or PEND │ │ │ -│ │ │ → picker->on_dispatch() │ │ │ -│ │ │ → picker->on_pend() │ │ │ -│ │ │ │ │ │ -│ │ Preemption (if alloc fails): │ │ -│ │ select victims → SIGSTP → pend job → retry next round │ │ -│ │ Resreq fingerprint cache (per-session) │ │ -│ └──────────────────────────┬──────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ on_decision(driver_ctx, decision) │ │ -│ │ DISPATCH / PEND / PREEMPT │ │ -│ └─────────────────────────────────────────────────────────────┘ │ -│ │ -│ ZERO dependencies on: controller, capnp, network, wall-clock │ -└──────────────────────────────────────────────────────────────────┘ -``` - -### Simulation Replay Sequence - -``` - load snapshot replay delta events - │ │ - ▼ ▼ - ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ - │ HEADER │───▶│ SNAPSHOT │───▶│ EVENT @t1 │───▶│ EVENT @t2 │───▶ ... - └──────────┘ └──────────┘ └──────────┘ └──────────┘ - │ │ │ - ▼ ▼ ▼ - restore core push + BEGIN_ push + BEGIN_ - state via ROUND(t1) → ROUND(t2) → - load_snapshot emit decisions emit decisions - - Virtual clock: t_snap ────────▶ t1 ───────────▶ t2 ───────────▶ - (discrete jumps, no wall-clock delay, monotonic) -``` - -## Preemption Flow - -``` -scheduler controller worker -─────────── ────────── ────── -alloc fails for high-prio job J - -preempt_policy.select_victims(J): - collect running jobs from - Q_aggressor.preemptible_queues - filter: victim.prio < J.prio - filter: victim.queue.preemptible - rank: lowest priority first - select minimum set satisfying J.resreq - - → [victim1, victim2] - -XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 - .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED - .signal = SIGSTP victim's worker release resources - .preempting_jobid = J - -J pended → next scheduling round → allocation retries on freed resources -``` - -## Future: Project Resource Ownership - -Deferred to a separate `project-resource-ownership` spec. When implemented: - -- Projects form a tree with resource quota at each node -- A project-aware picker plugin selects which project gets resources (fair-share, priority, proportional), then iterates queues scoped to that project, then jobs within each queue -- Project quota enforcement: projects over quota are skipped by the picker -- The core scheduling loop (Requirement 1) requires zero code changes — only a new picker plugin replaces `builtin_fcfs`. The core still calls `picker->next_job()` with no knowledge of projects. -- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes -- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends with quota/usage fields - -The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. The only difference is which picker plugin is configured. diff --git a/.kiro/specs/scheduler-framework/spec.json b/.kiro/specs/scheduler-framework/spec.json deleted file mode 100644 index 8c5192d2..00000000 --- a/.kiro/specs/scheduler-framework/spec.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "feature_name": "scheduler-framework", - "created_at": "2026-07-09T21:20:00Z", - "updated_at": "2026-07-27T00:00:00Z", - "language": "en", - "phase": "decomposed", - "description": "DECOMPOSED into scheduler-core, scheduler-allocation, scheduler-preemption. This spec is retained for discovery context; see child specs for active requirements.", - "approvals": { - "requirements": { - "generated": true, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "blocked_on": [], - "child_specs": ["scheduler-core", "scheduler-allocation", "scheduler-preemption"] -} diff --git a/.kiro/specs/scheduler-preemption/brief.md b/.kiro/specs/scheduler-preemption/brief.md deleted file mode 100644 index 25025ed1..00000000 --- a/.kiro/specs/scheduler-preemption/brief.md +++ /dev/null @@ -1,183 +0,0 @@ -# Brief: scheduler-preemption - -## Problem - -The XCE scheduler has no preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the scheduler has no way to free those resources. The job is simply pended, and it waits indefinitely — regardless of its priority — until running jobs naturally complete. - -Three specific gaps exist: - -1. **No preemption decision type** — the `xce_decision_op_t` enum has `DISPATCH` and `PEND` but no `PREEMPT`. The scheduler cannot communicate a preemption intent to the controller. -2. **No victim selection mechanism** — even if a preemption decision type existed, there is no logic to select which running jobs should be preempted. Cross-queue victim selection (a high-priority job in queue A preempting a low-priority job in queue B) is entirely absent. -3. **No SIGSTP pipeline** — the controller and worker have no path to deliver a stop signal to a running job and transition its state to `JOB_SUSPENDED`, freeing its resources for the preempting job. - -Operators with mixed-priority workloads need preemption to ensure critical work proceeds without manual intervention — the operator should not have to manually kill low-priority jobs to make room. - -## Current State - -### What exists -- **Decision framework** (`decision.h`): `xce_decision_op_t` with `DISPATCH`, `PEND`, and `NONE` ops. Core/decision callback flows to driver/controller. -- **Queue structure** (`xce_sched_queue_t`): has `priority` field for inter-queue ordering within the picker, but no preemption-related fields. -- **HOOK_DETAIL_JOB_STOP** (enum value 65 in `cluster_def.h`): existing hook detail enum, reused for preemption stop notifications. -- **Controller→worker messaging**: existing message pipeline for dispatch/pend decisions; the same path can carry preemption actions to workers. - -### What's missing -- `XCE_DECISION_PREEMPT` opcode and `xce_preempt_decision_t` payload in `decision.h` -- Victim selection policy: which running jobs get preempted for a given high-priority job -- Per-queue preemption configuration: which queues can preempt from which other queues, which queues are immune to preemption -- SIGSTP delivery pipeline: scheduler → controller → worker → process group -- Job state transition to `JOB_SUSPENDED` triggered by preemption -- Integration of preemption into the scheduling loop: when allocation fails, attempt preemption before pended - -## Desired Outcome - -A job preemption system with three layers: - -**1. Decision type** — `XCE_DECISION_PREEMPT` carries victim jobids, signal type, and preempting job id through the core's decision callback → normal-mode driver → controller → worker pipeline, reusing the same callback infrastructure (`xce_core_decision_cb_t`) used by `DISPATCH` and `PEND`. - -**2. Victim selection** — a `preempt_policy` plugin on each queue selects which running jobs to preempt. The default policy `"lowest_priority_first"` implements LSF-style cross-queue victim selection: -- Collect running jobs from the preempting job's `preemptible_queues` list -- Filter: victim priority < preempting job priority AND victim's queue has `preemptible == true` -- Rank by priority ascending (lowest priority first) -- Select the minimum subset whose freed resources satisfy the preempting job's resreq - -**3. SIGSTP pipeline** — the controller receives `XCE_DECISION_PREEMPT`, delivers `HOOK_DETAIL_JOB_STOP` to each victim's worker, the worker sends SIGSTP to the job's process group, the controller transitions the job to `JOB_SUSPENDED`, and resources are released. The preempting job is pended and retried in the next scheduling round (async preemption). - -### Queue Preemption Configuration - -Each queue carries three new fields: - -| Field | Type | Purpose | -|-------|------|---------| -| `preemptible_queues` | `char**` | List of queue names this queue CAN preempt from | -| `preemptible` | `bool` | Whether this queue's jobs CAN BE preempted by other queues | -| `preempt_policy` | `char*` | Victim-selection plugin name (NULL = use default) | - -## Alternatives Considered - -### Approach A: Kill-and-requeue only (rejected) - -| Aspect | Detail | -|--------|--------| -| **Mechanism** | SIGKILL victims, requeue from scratch | -| **Pros** | Simple, no suspended state, no resume logic | -| **Cons** | Destroys progress, lose all intermediate computation. Requeued jobs restart from zero. Not viable for long-running HPC workloads where checkpointing is explicit. | - -**Rejected because:** HPC jobs can run for hours or days. Killing a job that is 90% complete to make room for a higher-priority job wastes all accumulated work. SIGSTP + resume preserves progress when the preempted job is later resumed. - -### Approach B: SIGSTP with suspend/resume (chosen) - -| Aspect | Detail | -|--------|--------| -| **Mechanism** | SIGSTP to process group → job → `JOB_SUSPENDED` → resources freed → preempting job runs → preempted job optionally resumed later | -| **Victim selection** | Policy plugin per queue, default `"lowest_priority_first"` | -| **Cross-queue** | Per-queue `preemptible_queues` list enables cross-queue preemption | -| **Scope** | v1 is SIGSTP only; kill-and-requeue escalation deferred | - -**Chosen because:** Preserves victim job progress. Enables later resume (outside this spec's scope). Aligns with LSF-style preemption semantics familiar to HPC operators. The async pattern (pend preempting job, retry next round after SIGSTP) keeps the scheduling loop simple. - -## Approach - -### Preemption Flow - -``` -scheduler controller worker -─────────── ────────── ────── -alloc fails for high-prio job J - -preempt_policy.select_victims(J): - collect running jobs from - Q_aggressor.preemptible_queues - filter: victim.prio < J.prio - filter: victim.queue.preemptible - rank: lowest priority first - select minimum set satisfying J.resreq - - → [victim1, victim2] - -XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 - .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED - .signal = SIGSTP victim's worker release resources - .preempting_jobid = J - -J pended → next round → retry alloc -``` - -### Integration into Scheduling Loop - -Preemption fires at Level 3 in the core scheduling loop — after allocation failure, before pending: - -```c -while ((job = picker->next_job(picker)) != NULL) { - if (try_allocate(job) > 0) { - emit_decision(DISPATCH, job); - picker->on_dispatch(picker, job->jobid); - } else if (can_preempt(job)) { - victims = select_victims(job); - emit_decision(PREEMPT, job, victims); - picker->on_pend(picker, job->jobid); // retry next round - } else { - emit_decision(PEND, job); - picker->on_pend(picker, job->jobid); - } -} -``` - -The core loop itself contains no victim selection logic — that lives in the `preempt_policy` plugin. The core only checks `can_preempt(job)` (does the job's queue have a non-NULL `preemptible_queues` list?) and delegates to the policy plugin for victim selection. - -### What stays unchanged -- Core scheduling loop — preemption is an additional branch in the alloc-fail path (Level 3), not a restructure -- Decision callback (`xce_core_decision_cb_t`) — `PREEMPT` flows through the same callback as `DISPATCH` and `PEND` -- Existing plugin framework — `preempt_policy` is registered, loaded, and invoked through the existing `plugin_register`/`key_find` mechanism -- `HOOK_DETAIL_JOB_STOP` — existing enum value, reused without change - -### What's new -- `XCE_DECISION_PREEMPT` in `xce_decision_op_t` (`decision.h`) -- `xce_preempt_decision_t` payload carrying `victim_jobids[]`, `n_victims`, `signal`, `preempting_jobid` -- `preemptible_queues`, `preemptible`, `preempt_policy` fields on `xce_sched_queue_t` -- Default `"lowest_priority_first"` preemption policy plugin -- Controller handler for `XCE_DECISION_PREEMPT` → SIGSTP delivery to workers -- Worker-side SIGSTP handler → process group signal + `JOB_SUSPENDED` state transition - -## Scope - -### In -- `XCE_DECISION_PREEMPT` decision type and `xce_preempt_decision_t` payload -- `preempt_policy` plugin interface (victim selection) -- Default `"lowest_priority_first"` preemption policy -- LSF-style cross-queue victim selection via `preemptible_queues` list -- SIGSTP pipeline: scheduler → controller → worker → process group -- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy` -- Victim safety guards: never preempt same-or-higher priority, never preempt non-`preemptible` queues -- `JOB_SUSPENDED` state transition on victim jobs -- Reuse of `HOOK_DETAIL_JOB_STOP` for preemption stop notifications -- Unit tests: victim selection policy, safety guard enforcement, edge cases (empty target queues, no eligible victims, insufficient resources from any victim set) -- Integration test: end-to-end preemption flow — high-priority job submits, preemption fires, victims get SIGSTP, resources freed, preempting job dispatched in next round - -### Out -- Kill-and-requeue escalation (v1 is SIGSTP only) -- Deadline-based preemption (requires `deadline` field on jobs — deferred) -- Preempted job resume logic (outside this spec's scope — the `JOB_SUSPENDED` state is defined, but the resume mechanism is a separate feature) -- Preemption priority inversion guards beyond the priority comparison already specified -- Fair-share or proportional victim selection beyond the default `"lowest_priority_first"` policy (the plugin interface enables custom policies, but none beyond the default ship here) - -## Upstream / Downstream - -### Upstream (dependencies) -- **scheduler-core** (sibling spec): provides the decision callback (`xce_core_decision_cb_t`) through which `XCE_DECISION_PREEMPT` flows. The core scheduling loop and `xce_sched_core_t` are defined there. Preemption hooks into the core loop at Level 3 (alloc-fail branch). -- **scheduler-allocation** (sibling spec): provides `try_allocate()` — preemption fires only when allocation fails. The resreq fingerprint cache interacts with preemption (a cached-unsatisfiable resreq skips allocation AND preemption). -- **Controller message handling**: existing handler dispatch path for decisions — extended to handle `XCE_DECISION_PREEMPT`. -- **Worker job control**: existing job lifecycle management — extended to handle SIGSTP delivery and `JOB_SUSPENDED`. - -### Downstream (future consumers) -- **Resume mechanism** — a future spec for resuming `JOB_SUSPENDED` jobs, potentially with priority-based ordering -- **Kill-and-requeue escalation** — a future enhancement where repeated preemption failure escalates to SIGKILL -- **Custom preemption policies** — deadline-aware, fair-share, or resource-specific victim selection implemented as new `preempt_policy` plugins - -## Constraints -- **C (gnu17)** — no C++, no external libraries beyond existing (pthreads) -- **Async preemption** — the preempting job is pended and retried in the next scheduling round; preemption is not synchronous -- **Priority guard** — victim priority must be strictly less than preempting job priority (never preempt equal or higher) -- **Preemptible guard** — victim queue must have `preemptible == true` (opt-in — queues are not preemptible by default) -- **Minimal victim set** — select the minimum subset of victims whose freed resources satisfy the preempting job's resreq (avoid over-preemption) -- **No self-preemption** — the preempting job's own queue is implicitly excluded from victim selection -- **Cancel-safe** — `preempt_policy.select_victims()` must have `pthread_cancel` cancellation points diff --git a/.kiro/specs/scheduler-preemption/requirements.md b/.kiro/specs/scheduler-preemption/requirements.md deleted file mode 100644 index dcb20038..00000000 --- a/.kiro/specs/scheduler-preemption/requirements.md +++ /dev/null @@ -1,111 +0,0 @@ -# Requirements Document - -## Introduction - -The XCE scheduler currently has no job preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the only available action is to pend the job — it waits indefinitely until running jobs complete naturally, regardless of priority. - -This feature closes the preemption gap with three additions: - -1. A `XCE_DECISION_PREEMPT` decision type flowing through the same callback as `DISPATCH` and `PEND`, carrying victim job ids and signal type to the controller→worker pipeline. -2. A configurable `preempt_policy` plugin mechanism for victim selection, with a built-in `"lowest_priority_first"` default policy implementing LSF-style cross-queue victim selection. -3. A SIGSTP delivery pipeline: scheduler emits `XCE_DECISION_PREEMPT` → controller delivers `HOOK_DETAIL_JOB_STOP` to each victim's worker → worker sends SIGSTP to the job's process group → controller transitions the job to `JOB_SUSPENDED` and releases resources. - -The design uses async preemption: the preempting job is pended and retried in the next scheduling round after resources are freed, rather than blocking the scheduling loop waiting for victims to stop. - -> Full discovery context (problem, current state, alternatives considered, approach, boundary) is in `.kiro/specs/scheduler-preemption/brief.md`. -> -> This spec is a child of `scheduler-framework`. It depends on **scheduler-core** for the decision callback (`xce_core_decision_cb_t`) through which `XCE_DECISION_PREEMPT` flows. The preemption branch (Level 3) fires in the core scheduling loop after allocation failure, before pending — this integration point is defined in scheduler-core's requirements. - -## Boundary Context - -- **In scope**: `XCE_DECISION_PREEMPT` opcode and `xce_preempt_decision_t` payload in `decision.h`; `preempt_policy` plugin interface for victim selection; default `"lowest_priority_first"` preemption policy implementing LSF-style cross-queue victim selection; per-queue preemption configuration (`preemptible_queues`, `preemptible`, `preempt_policy` fields on `xce_sched_queue_t`); SIGSTP delivery pipeline through controller→worker; `JOB_SUSPENDED` state transition on victim jobs; reuse of `HOOK_DETAIL_JOB_STOP` for preemption stop notifications; victim safety guards (never preempt same-or-higher priority, never preempt non-`preemptible` queues, never over-preempt); integration with the core scheduling loop at Level 3 (alloc-fail branch). - -- **Out of scope**: Kill-and-requeue escalation beyond SIGSTP (deferred — v1 is SIGSTP only); deadline-based preemption (requires `deadline` field on `xce_sched_job_t`, deferred); preempted job resume logic (the `JOB_SUSPENDED` state is defined but resume is a separate feature); fair-share or proportional victim selection policies beyond the default (the plugin interface enables them, but no additional policies ship here); changes to the picker interface, iterator abstractions, core/driver separation, or the scheduling loop structure itself (those belong to sibling specs `scheduler-core` and `scheduler-allocation`). - -- **Adjacent expectations**: This spec depends on `scheduler-core` for the decision callback (`xce_core_decision_cb_t`) and the core scheduling loop into which the preemption branch integrates. The existing controller→worker message pipeline for dispatch/pend is extended to carry `XCE_DECISION_PREEMPT`. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum (value 65 in `cluster_def.h`) is reused for preemption stop actions. The existing `plugin_register`/`key_find` mechanism loads the `preempt_policy` plugin. - -## Data Structure Changes - -### `decision.h` additions -```c -typedef enum { - XCE_DECISION_NONE = 0, - XCE_DECISION_DISPATCH, - XCE_DECISION_PEND, - XCE_DECISION_PREEMPT, // NEW — preemption decision - XCE_DECISION_NEW_SESSION -} xce_decision_op_t; - -typedef struct { - uint64_t* victim_jobids; // jobs to preempt - int n_victims; // count of victims - int signal; // signal to send (SIGSTP in v1, SIGKILL in future) - uint64_t preempting_jobid; // job that triggered preemption -} xce_preempt_decision_t; -``` - -### `xce_sched_queue_t` additions -```c -char** preemptible_queues; // queue names this queue CAN preempt from (NULL = none) -int n_preemptible_queues; // count of preemptible_queues entries -bool preemptible; // CAN this queue's jobs BE preempted by other queues? -char* preempt_policy; // victim-selection plugin name (NULL = use default) -``` - -## Preemption Flow - -``` -scheduler controller worker -─────────── ────────── ────── -alloc fails for high-prio job J - -preempt_policy.select_victims(J): - collect running jobs from - Q_aggressor.preemptible_queues - filter: victim.prio < J.prio - filter: victim.queue.preemptible - rank: lowest priority first - select minimum set satisfying J.resreq - - → [victim1, victim2] - -XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 - .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED - .signal = SIGSTP victim's worker release resources - .preempting_jobid = J - -J pended → next scheduling round → allocation retries on freed resources -``` - -## Requirements - -### Requirement: Preemption policy and execution - -**Objective:** As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention. - -#### Acceptance Criteria - -1. The Scheduler shall add a `XCE_DECISION_PREEMPT` opcode to `xce_decision_op_t` in `decision.h`, with a payload type `xce_preempt_decision_t` carrying `victim_jobids[]`, `n_victims`, `signal` (SIGSTP in v1), and `preempting_jobid`. The `XCE_DECISION_PREEMPT` decision shall flow through the same `xce_core_decision_cb_t` callback as `DISPATCH` and `PEND`; the core scheduling loop shall contain zero knowledge of how the driver publishes preemption decisions. - -2. When allocation fails for a job J whose queue has a non-NULL `preemptible_queues` list, the Scheduler shall invoke the queue's `preempt_policy` plugin to select victim jobs from the target queues. When `preempt_policy` is NULL, the Scheduler shall use the default `"lowest_priority_first"` policy. - -3. The default preemption policy `"lowest_priority_first"` shall: WHILE the preempting job's resource request is not yet satisfied DO collect running jobs from target queues; filter to those with `priority < J.priority` AND `queue->preemptible == true`; rank by priority ascending (lowest priority first); select the next victim candidate and add to the victim set. The policy shall return the minimum subset of victims whose freed resources satisfy J's resource request. When multiple victim subsets of equal minimum size exist, the policy shall prefer the subset whose victims have the lowest summed priority. - -4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via the decision callback, and shall pend job J for the next scheduling round (async preemption — the job is retried after victim resources are freed). The Scheduler shall not wait synchronously for victim jobs to transition to `JOB_SUSPENDED`. - -5. When no sufficient victim set exists after exhausting all eligible running jobs from the target queues, the Scheduler shall pend job J with a resource-shortage pend reason without invoking preemption. The decision callback shall not be called for the preemption path in this case. - -6. The worker receiving a preemption action shall send the specified signal (SIGSTP in v1) to the victim job's process group. The controller shall transition the victim job's state to `JOB_SUSPENDED` after confirming the signal was delivered. Resources held by the victim job shall be released upon suspension. - -7. The Scheduler shall add `preemptible_queues` (list of queue names this queue CAN preempt from, NULL = none), `preemptible` (bool — CAN this queue's jobs BE preempted by other queues, default false), and `preempt_policy` (victim-selection plugin name, NULL = use default `"lowest_priority_first"`) fields to `xce_sched_queue_t`. These fields shall be populated from queue configuration at `QUEUE_UPDATE` time. - -8. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum (value 65 in `cluster_def.h`) shall be reused for preemption stop notifications, with no changes to the enum definition. The controller shall deliver this hook detail to each victim's worker as part of the preemption action. - -9. Preemption victim selection shall never select: - - A job whose priority is greater than or equal to the preempting job's priority. - - A job whose queue has `preemptible == false`. - - A job that is not in running state. - - A job from a queue not listed in the preempting job's `preemptible_queues` list. - - More victims than necessary to satisfy the preempting job's resource request (minimum sufficient victim set). - -> **Pending revision (2026-08-23):** the sibling spec `scheduler-dispatch-commit` (not yet designed) redefines "running state" to begin at dispatch-decision-commit time rather than worker-confirmation time — meaning a job could reach RUN with resources committed but no live, signalable process yet. AC6's SIGSTP-to-process-group delivery and AC9's "not in running state" exclusion both assume every RUN job has a live process today; once `scheduler-dispatch-commit` lands, this spec's design phase needs to add a second victim action (cancel an in-flight, unconfirmed dispatch) alongside SIGSTP, and decide whether a dispatched-but-unconfirmed job is eligible as a victim at all. Re-check this note before `scheduler-preemption` proceeds to `/kiro-spec-design`. diff --git a/.kiro/specs/scheduler-preemption/spec.json b/.kiro/specs/scheduler-preemption/spec.json deleted file mode 100644 index 3ffc24e2..00000000 --- a/.kiro/specs/scheduler-preemption/spec.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "feature_name": "scheduler-preemption", - "created_at": "2026-07-27T00:00:00Z", - "updated_at": "2026-07-27T00:00:00Z", - "language": "en", - "phase": "requirements-generated", - "parent_spec": "scheduler-framework", - "description": "Job preemption policy and execution for the XCE scheduler — SIGSTP-based victim selection and controller→worker pipeline", - "approvals": { - "requirements": { - "generated": true, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "blocked_on": ["scheduler-core"], - "sibling_specs": ["scheduler-core", "scheduler-allocation"] -} diff --git a/.kiro/specs/scheduler-sim-decision-parity/brief.md b/.kiro/specs/scheduler-sim-decision-parity/brief.md deleted file mode 100644 index 153323ab..00000000 --- a/.kiro/specs/scheduler-sim-decision-parity/brief.md +++ /dev/null @@ -1,44 +0,0 @@ -# Brief: scheduler-sim-decision-parity - -## Problem -Operators debugging scheduling behavior via `xce_scheduler -i -s <replay>` (or batch `--sim`) cannot trust two things about jobs that were already resolved (dispatched) in the *original* recorded run: (1) their resource usage is never reflected in `w->resInUse`/`core->sharedInUse` (`info worker` always shows `used=0.00` regardless of actual load — bug-1613), and (2) their `DISPATCH` decision is never visible in decision output, even under `step` (bug-1611). Both undermine the entire premise of simulation mode: that replaying a recording reproduces what actually happened, closely enough to debug and tune against. - -## Current State -- `scheduler-core` (implementation-complete, closed) built the core/driver split: `xce_sched_core_t` is mode-agnostic; `driver.c` (normal mode) and `sim_loop.c` (simulation mode) are separate driver implementations of the same session/decision-callback protocol. -- **Confirmed (bug-1613)**: `sched_account_jobs()` — re-derives `resInUse`/`sharedInUse` from currently-RUN-state jobs' already-built `job->accounts`, required after every session's resval reset — is called by `driver.c`'s `xce_sched_driver_run_normal_mode_round()` but never by `sim_loop.c`, which still uses the monolithic `xce_core_go()` (no interleaving point for it). The function is also currently signatured on `xce_scheduler_t*` (normal-mode-only struct), which `xce_sched_sim_driver_t` deliberately does not have (Req 5 AC 17 simulation-isolation boundary) — it cannot be called from `sim_loop.c` today without a signature change. -- **Confirmed (this session, new finding)**: `xce_sched_driver_run_normal_mode_round()`'s decision-drain loop shows every decision the normal-mode scheduler makes — including `DISPATCH`, with its full `xce_job_alloc_t` allocation payload — is written into the `.replay` file via `sched_record_session_add_decision()`. `sim_loop.c` already decodes this full `decisions` list every round (`decoded_session->decisions`, via `decode_ReplaySession_ptr()`), but only ever reads it when `sim->verify` is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct `job->allocs`/`job->accounts` for a job the live picker in *this* replay run doesn't independently re-decide — which is the normal case for any job already resolved before the replay began. This is very likely the single root cause behind both bug-1611 and bug-1613: the data needed already exists in every replay stream and is already decoded every round; it is simply never applied outside verify-mode comparison. -- `.kiro/steering/scheduler-driver-equivalence.md` (created this session) documents the constraint this spec restores: replaying a normal-mode recording must reproduce identical decisions AND identical resource-accounting state, not just matching final decisions on fixtures with no contention. - -## Desired Outcome -Replaying the exact recording of a normal-mode run reproduces, at every point in the replay: the same resource-accounting state (`resInUse`/`sharedInUse`/`job->accounts`) and the same decision visibility (a `DISPATCH`/`PEND`/`PREEMPT` line is observable for every job, whether the live picker re-derived it fresh this run or it was already resolved in the original recording) that the normal-mode run itself produced. - -## Approach -Two components, one spec: -1. **Accounting re-derivation parity**: relocate `sched_account_jobs()` and its callees (`sched_charge_account`/`sched_uncharge_account`/`sched_update_account`/`sched_update_res_accounts`) from `xce_scheduler_t*` to `xce_sched_core_t*` — every one of their bodies already only touches `sched->core`, so this is a signature change, not a logic change, following the exact precedent `sched_resolve_job_picker()` already established (task 4.3) for the identical reason. Then call it in `sim_loop.c` at the same relative position `driver.c` does: replace the monolithic `xce_core_go()` with the split `xce_core_begin_session()` → `sched_account_jobs(&(sim->core))` → `xce_core_signal_go()`. -2. **Recorded-decision application**: for a round where the live picker/allocator does not independently produce a decision for a job that the recording shows as already resolved, apply the corresponding recorded decision from `decoded_session->decisions` directly — reconstructing `job->allocs`/`job->accounts` (mirroring the field data `sim_verify_job_alloc_eq()` already knows how to compare) and feeding it through the same decision pipeline `sim_print_decision()`/`xce_repl_on_decision()` consume, so it is both charged and visible. - -## Scope -- **In**: Accounting-state re-derivation parity between the two drivers; applying recorded decisions for already-resolved jobs during replay (both interactive and batch `--sim`); avoiding double-application when a job is both recorded-resolved and somehow also independently re-decided; resource-contention test fixtures proving the fix (per `scheduler-driver-equivalence.md`'s testing discipline) since no-contention fixtures cannot distinguish correct from broken accounting. -- **Out**: Any change to REPL console output/formatting (that's `scheduler-sim-repl-verbosity`, a sibling spec, explicitly excluded from touching simulation state by its own Non-Goals). Any change to `--sim-verify`'s actual comparison semantics beyond ensuring recorded-decision application doesn't produce a spurious mismatch or double-count against it. Reopening or modifying `scheduler-core` itself. New scheduling/allocation policy. - -## Boundary Candidates -- Accounting relocation (`resource.c`/`scheduler.h`/`core.h`, signature-only change) — small, mechanical, low-risk, could land independently of the second piece. -- Recorded-decision application (`sim_loop.c`, and `sim_verify.c` for the double-count/verify-interaction question) — larger, the real design surface: what "already resolved, not independently re-decided" means precisely, and how it composes with `--sim-verify`'s existing comparison. - -## Out of Boundary -- `scheduler-sim-repl-verbosity`'s REPL output/formatting work. -- `scheduler-sim-diagnostics`'s future new inspection commands (limit usage, pend-reason) — this spec restores existing data's correctness, it does not add new inspectable data. -- Whether `sched_check_host_satisfy`/`sched_meet_host_selection` read `resInUse` for live capacity decisions — CONFIRMED during design (see `research.md`'s Research Log): they do, unconditionally. This is existing `scheduler-core` behavior this spec does not change, only relies on. - -## Upstream / Downstream -- **Upstream**: `scheduler-core`'s core/driver split and its now-documented equivalence invariant (`.kiro/steering/scheduler-driver-equivalence.md`). -- **Downstream**: `scheduler-sim-repl-verbosity`'s round-rollup dispatch counts (currently designed to read from the existing decision tally) become accurate once this lands, rather than permanently under-reporting dispatches for already-resolved jobs. `scheduler-allocation`/`scheduler-preemption` (both requirements-generated, not yet designed) inherit a scheduler core with a restored, tested equivalence guarantee to build on. - -## Existing Spec Touchpoints -- **Extends**: none formally — `scheduler-core` is closed; this spec restores its own documented invariant rather than reopening it. -- **Adjacent**: `scheduler-sim-repl-verbosity` (sibling, not yet implemented — benefits from this fix but does not depend on it to ship), `scheduler-sim-diagnostics` (sibling, brief only). - -## Constraints -- Must not regress REPL-vs-batch-sim decision equivalence (`test_repl_equivalence.c`) — that invariant is solid today and must stay solid. -- Must be validated against a resource-contention fixture (multiple jobs competing for insufficient capacity), per `scheduler-driver-equivalence.md`'s testing discipline — a no-contention fixture cannot prove this fix works. -- `sched_account_jobs()`'s relocation must be a pure signature change (no behavior change) to keep the accounting-parity piece low-risk and independently reviewable from the larger recorded-decision-application piece. diff --git a/.kiro/specs/scheduler-sim-decision-parity/design.md b/.kiro/specs/scheduler-sim-decision-parity/design.md deleted file mode 100644 index e23aa6a1..00000000 --- a/.kiro/specs/scheduler-sim-decision-parity/design.md +++ /dev/null @@ -1,273 +0,0 @@ -# Design Document - -## Overview -**Purpose**: Restore the scheduler's documented normal-mode/simulation-mode driver equivalence invariant (`.kiro/steering/scheduler-driver-equivalence.md`) for jobs already resolved before a given point in a replay — both their resource-accounting state and their decision visibility. - -**Users**: XCE scheduler engineers and operators debugging via `xce_scheduler -i -s <replay>` or batch `--sim`, and any future scheduler spec (`scheduler-allocation`, `scheduler-preemption`) relying on simulation mode reflecting real scheduling behavior. - -**Impact**: Changes `sim_loop.c`'s round-boundary handling (replacing the monolithic `xce_core_go()` with the split calls `driver.c` already uses, plus a new reconciliation step that also compacts `decoded_session->decisions` as it consumes entries — see Recorded-Decision Reconciliation's Implementation Notes), and the signature of five accounting functions currently in `resource.c` (relocated to operate on `xce_sched_core_t*`). `sim_verify.c`'s comparison predicates (`sim_decision_eq()` and its helpers) need no change — confirmed during design validation that `sim_loop.c`'s own array bookkeeping is sufficient to keep the existing positional comparison correct. No change to `core.c`/`core_events.c`'s event-application logic, to scheduling/allocation policy, or to REPL output formatting. - -### Goals -- Resource-in-use accounting (`resInUse`/`sharedInUse`) reflects every currently-running job in simulation mode, matching normal mode (Requirement 1). -- Simulation-mode capacity checks under contention produce the same outcome normal mode would (Requirement 2) — a direct consequence of Goal 1, since the capacity-check code path is unchanged and already reads this data correctly. -- Every job's decision (including ones resolved before this replay run started tracking them live) is visible through the existing decision-reporting channel (Requirement 3). -- Missing recorded data degrades gracefully, never fabricates or crashes (Requirement 4). -- Zero regression to REPL-vs-batch-sim equivalence, `--sim-verify`, or normal-mode behavior (Requirement 5). - -### Non-Goals -- Any change to REPL console output formatting (`scheduler-sim-repl-verbosity`'s boundary). -- Any new inspectable data or command (`scheduler-sim-diagnostics`'s boundary). -- Any change to scheduling/allocation policy or picker plugin behavior. -- Reopening `scheduler-core`. - -## Boundary Commitments - -### This Spec Owns -- The signatures of `sched_account_jobs()`/`sched_charge_account()`/`sched_uncharge_account()`/`sched_update_account()`/`sched_update_res_accounts()`/`sched_commit_job_alloc()` (currently `resource.c`, `xce_scheduler_t*`-coupled) and their relocation to `xce_sched_core_t*`. -- `sim_loop.c`'s round-boundary sequence (the `xce_core_go()` replacement and the new reconciliation step). -- The logic that reconstructs `job->allocs`/`job->accounts` from a recorded decision for a RUN-state job that has neither, and feeds that reconstructed decision through the existing decision-reporting channel. - -### Out of Boundary -- `driver.c`'s normal-mode round logic, beyond updating its one call site to the relocated accounting function signature. -- `core.c`/`core_events.c`'s event-application logic (JOB_PUSH/JOB_UPDATE/JOB_STATE_CHANGE handling) — unchanged. -- `repl_break.c`/`repl.c`/`repl_info.c` — unchanged; they consume whatever the decision-reporting channel produces, without needing to know whether a given decision was freshly computed or reconstructed. -- `--sim-verify`'s actual pass/fail semantics — this spec must not introduce a false mismatch, but does not redesign verification. - -### Allowed Dependencies -- `sim_verify.c`'s existing `sim_verify_job_alloc_eq()`/`sim_verify_host_alloc_eq()`/`sim_verify_alloc_eq()` field shapes as the reference for what a reconstructed `xce_job_alloc_t` must contain — not necessarily calling these functions, but matching the payload shape they already validate. -- `decoded_session->decisions`, already decoded every round by existing `sim_loop.c` code — no new decode path. -- The `sched_resolve_job_picker()` relocation (task 4.3) as the precedent for the accounting-function signature change. - -### Revalidation Triggers -- Any change to `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t`'s wire shape (`controller.capnp`) must be re-checked against the reconstruction logic here. -- Any change to how `--sim-verify` compares decisions must be re-checked against this spec's reconciliation step, and vice versa. -- If `scheduler-sim-diagnostics` later adds a "why did this job dispatch" trace, it can build on the now-correct `job->accounts` this spec establishes rather than re-deriving it. -- **When `scheduler-preemption` lands** (currently requirements-generated, not yet designed — `driver.h:96` presently states "no live allocator path produces this decision type"): Requirement 3.1 here covers "dispatched, pended, or preempted" jobs, but Recorded-Decision Reconciliation's trigger condition (RUN-state + `job->accounts == NULL`) only ever matches a recorded DISPATCH entry — a preempted job leaves RUN-state at the moment of preemption, so it cannot trigger this same mechanism, and PEND doesn't need reconciliation because the live picker already re-derives it fresh every round. This is not a gap today (no code path emits `XCE_DECISION_PREEMPT` yet, so no recorded replay can contain one), but once `scheduler-preemption` starts producing real PREEMPT decisions, revisit whether this spec's mechanism needs a second trigger condition for "job left RUN-state via preemption within this round, no reconciled visibility yet" or whether `scheduler-preemption`'s own design should own that instead. - -## Architecture - -### Existing Architecture Analysis -- `driver.c`'s `xce_sched_driver_run_normal_mode_round()` is the reference implementation for the exact sequence this spec ports to `sim_loop.c`: `xce_core_begin_session()` → driver-specific interleaved bookkeeping (`plugin_begin_session()`, `sched_account_jobs()`) → `xce_core_signal_go()` → decision-drain loop (`cbuf_get(&core->decisions)`, recording each via `sched_record_session_add_decision()` when recording, then `core->on_decision()`). -- `sim_loop.c` currently calls the older monolithic `xce_core_go()` (itself now just `xce_core_begin_session()`+`xce_core_signal_go()` back-to-back, per `core_session.c`) with no interleaving room, and decodes `decoded_session->decisions` every round but only consumes it under `sim->verify`. -- The accounting functions being relocated — `sched_account_jobs()` (`scheduler.c`) and its callees `sched_charge_account()`/`sched_uncharge_account()`/`sched_update_account()`/`sched_update_res_accounts()`/`sched_commit_job_alloc()` (`resource.c`) — already only touch `sched->core` in their bodies despite their `xce_scheduler_t*` signature. Confirmed by reading every function in this call chain, and independently re-confirmed with `grep "sched->" resource.c | grep -v "sched->core"` returning zero matches across that whole file (covers the five `resource.c` functions; `sched_account_jobs()` itself was separately read in full). - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TD - BeginSession[xce_core_begin_session] --> Reconcile[Reconcile RUN jobs with no accounts] - Reconcile --> Lookup{Recorded decision found in decoded_session} - Lookup -->|Yes| Rebuild[Rebuild job allocs and accounts from recorded decision] - Lookup -->|No| Unavailable[Report usage unavailable, continue] - Rebuild --> Compact[Free consumed entry, compact decoded_session.decisions] - Compact --> Report[Feed decision through existing reporting channel] - Report --> AccountJobs[sched_account_jobs on core] - Unavailable --> AccountJobs - AccountJobs --> SignalGo[xce_core_signal_go] - SignalGo --> Picker[Live picker and allocator run for still-pending jobs] - Picker --> DrainLoop[Decision drain loop -- same reporting channel for fresh decisions] -``` - -**Architecture Integration**: -- Selected pattern: extend the existing round-boundary interleaving slot (the same one `driver.c` already uses for its own bookkeeping) with one additional step, rather than a new hook or a second round-processing pass. -- Domain/feature boundaries: reconciliation logic lives in `sim_loop.c` only; the relocated accounting functions live in `resource.c`/`core.h` and are shared by both drivers; `driver.c` gets a one-line call-site update. -- Existing patterns preserved: the `job->accounts == NULL` check `sched_account_jobs()` already performs is reused as the reconciliation trigger and the double-application guard — no new state. -- New components rationale: reconciliation must run before `sched_account_jobs()` (so the same round's re-charge picks up newly-reconstructed jobs) and before `xce_core_signal_go()` (so the live picker/allocator's capacity checks for still-pending jobs see correct usage this round, satisfying Requirement 2). -- Steering compliance: `sim_loop.c` remains at its documented interleaving-slot discipline (no new call sites added to the core's own API surface); `resource.c`'s functions gain no new behavior, only a new valid signature. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Scheduler daemon (C, gnu17) | existing `src/scheduler/` toolchain | All logic in existing driver/core/verify translation units | No new dependency; matches `tech.md` | - -## File Structure Plan - -### Modified Files -- `src/scheduler/src/resource.c` — relocate `sched_charge_account()`, `sched_uncharge_account()`, `sched_update_account()`, `sched_update_res_accounts()`, `sched_commit_job_alloc()` from `xce_scheduler_t*` to `xce_sched_core_t*` parameters (body unchanged — every reference is already `sched->core.*`). -- `src/scheduler/include/scheduler.h` — remove/update the now-relocated declarations. -- `src/scheduler/include/core.h` — add the relocated declarations, mirroring `sched_resolve_job_picker()`'s existing placement and doc-comment precedent. -- `src/scheduler/src/scheduler.c` — relocate `sched_account_jobs()` itself the same way (it is defined here, not in `resource.c`; body likewise unchanged — every reference is already `sched->core.*`); update this file's own two call sites for the new signatures: `sched_commit_decision()`'s call to `sched_commit_job_alloc()` (pass `&(sched->core)`) and `sched_begin_session()`'s call to `sched_account_jobs()` likewise. -- `src/scheduler/src/driver.c` — update `xce_sched_driver_run_normal_mode_round()`'s `sched_account_jobs(sched)` call to `sched_account_jobs(&(sched->core))`. -- `src/scheduler/src/sim_loop.c` — replace the `xce_core_go(&(sim->core))` call with `xce_core_begin_session()` → reconciliation step (new static function(s)) → `sched_account_jobs(&(sim->core))` → `xce_core_signal_go()`; reconciled decisions feed through the same decision-reporting call the existing drain loop uses, and each consumed recorded decision is removed/compacted from `decoded_session->decisions` in the same pass so the existing `--sim-verify` positional walk (`di` against `cbuf_get(&(sim->core.decisions), ...)`, unchanged) stays correctly aligned. -- `src/scheduler/src/sim_verify.c` — **no change** (confirmed during design validation; see `sim_loop.c` entry above and the Recorded-Decision Reconciliation component's Implementation Notes). - -### New/Modified Test Files -- `src/scheduler/testsuite/test_driver_session_ordering.c` (or a new `test_sim_session_ordering.c` mirroring it) — proves `sim_loop.c` now upholds the same "picker never observes zeroed-but-not-yet-resummed accounting" property `driver.c`'s own test already proves. -- A new resource-contention fixture (multiple jobs competing for insufficient capacity) added to the scheduler's simulation testsuite, per `scheduler-driver-equivalence.md`'s testing discipline — required because existing fixtures evidently don't exercise this path. -- `src/scheduler/testsuite/test_sim_equivalence.c` (existing, decision-identity coverage) — extended with a case where a job is already resolved before the replay's live picker runs, verifying decision AND accounting-state identity, not just decision identity. - -## System Flows -See the Architecture Pattern & Boundary Map diagram above — it is the single non-trivial flow this feature introduces (the round-boundary reconciliation decision tree). No sequence diagram is needed beyond it; the decision-drain loop's own flow is unchanged. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|-------------|-----------|-------| -| 1.1, 1.2, 1.3, 1.4 | Accurate resource-usage reporting | Core-Level Resource Accounting, Sim Driver Round-Boundary Accounting | relocated `sched_account_jobs` | Round-boundary flow | -| 2.1, 2.2 | Correct capacity awareness under contention | Core-Level Resource Accounting (consequence of 1.x being correct) | `sched_check_host_satisfy_`/`sched_check_shared_satisfy_` (unchanged, already reads this data) | Round-boundary flow | -| 3.1, 3.2, 3.3 | Decision visibility for pre-resolved jobs | Recorded-Decision Reconciliation | decision-reporting channel | Round-boundary flow | -| 4.1, 4.2 | Graceful degradation | Recorded-Decision Reconciliation | n/a (error path) | Round-boundary flow | -| 5.1, 5.2, 5.3 | No regression | all components | n/a | n/a | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|---------------|--------|---------------|---------------------------|-----------| -| Core-Level Resource Accounting | Scheduler core / resource.c, core.h | Re-derive resource-in-use state from currently-running jobs' known allocations, callable by either driver | 1.1-1.4, 2.1-2.2 | `xce_sched_core_t` (P0) | Service | -| Sim Driver Round-Boundary Accounting | Sim driver / sim_loop.c | Call the core-level accounting function at the correct point in the round boundary, mirroring the normal-mode driver | 1.3, 1.4, 5.3 | Core-Level Resource Accounting (P0) | State | -| Recorded-Decision Reconciliation | Sim driver / sim_loop.c | Reconstruct allocation/accounting and decision visibility for a RUN-state job with no accounts, from this round's recorded decisions | 3.1-3.3, 4.1-4.2 | `decoded_session->decisions` (P0), decision-reporting channel (P0) | Service, State | - -### Scheduler Core (resource.c / core.h) - -#### Core-Level Resource Accounting -| Field | Detail | -|-------|--------| -| Intent | Re-derive `resInUse`/`sharedInUse` from every currently-running job's known resource allocation, usable identically by both drivers | -| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1, 2.2 | - -**Responsibilities & Constraints** -- Body unchanged from today's `sched_account_jobs()`/callees — this is a signature relocation (`xce_scheduler_t*` → `xce_sched_core_t*`), not a logic change, since every reference in the existing bodies is already `sched->core.*`. -- Both drivers call it at the identical relative position: after `xce_core_begin_session()`'s resval reset, before `xce_core_signal_go()` unblocks the picker/allocator. -- Requirement 2 (capacity awareness) is satisfied as a direct consequence — `sched_check_host_satisfy_()`/`sched_check_shared_satisfy_()` already read `resInUse`/`sharedInUse` unconditionally; once those fields are correctly populated in simulation mode, the existing (unmodified) capacity-check code path produces the same outcome normal mode would. - -**Contracts**: Service [x] / State [ ] - -##### Service Interface -```c -/* - * Re-charges every currently-running job's known resource allocation - * into core->sharedInUse and each worker's resInUse. Skips a job whose - * allocation is not yet known (job->accounts == NULL) -- reconciling - * that is the Recorded-Decision Reconciliation component's job, which - * must run first in the same round-boundary window for this call to - * see a fully up-to-date job set. - */ -int sched_account_jobs(xce_sched_core_t* core); -``` -- Preconditions: `core` non-NULL; called strictly after `xce_core_begin_session()`'s resval reset for the current round. -- Postconditions: every RUN-state job with known accounts is reflected in `resInUse`/`sharedInUse`; a RUN-state job with unknown accounts is silently skipped (not an error — see Recorded-Decision Reconciliation for why that's expected before this call runs, and Requirement 4 for the case where it stays unknown). -- Invariants: identical result given identical core state, regardless of which driver calls it. - -### Sim Driver (sim_loop.c) - -#### Sim Driver Round-Boundary Accounting -| Field | Detail | -|-------|--------| -| Intent | Call Core-Level Resource Accounting at the same relative round-boundary position `driver.c` already uses | -| Requirements | 1.3, 1.4, 5.3 | - -**Responsibilities & Constraints** -- Replaces the current single `xce_core_go(&(sim->core))` call with the split sequence, matching `xce_sched_driver_run_normal_mode_round()`'s documented ordering exactly. -- Must not change `xce_core_go()`'s existing error-handling contract (a failed `xce_core_begin_session()` still skips the round's decision drain, per today's `sim_loop.c` error path) — the split must preserve this, not simplify it away. -- Headless (non-interactive) and interactive (`-i`) simulation runs both go through this same call site — no divergence between them (Requirement 5.1 depends on this). - -**Contracts**: State [x] - -#### Recorded-Decision Reconciliation -| Field | Detail | -|-------|--------| -| Intent | For a RUN-state job with no known allocation, reconstruct it from this round's recorded decision and report it through the existing decision channel | -| Requirements | 3.1, 3.2, 3.3, 4.1, 4.2 | - -**Responsibilities & Constraints** -- Runs once per round, after `xce_core_begin_session()`, before `sched_account_jobs()`. -- Trigger condition: a job in `sched->core.jobs` (via the existing core.h iteration idiom) that is RUN-state and has `job->accounts == NULL`. -- On trigger, search this round's `decoded_session->decisions` for a matching jobid. If found, reconstruct `job->allocs`/`job->accounts` from the recorded payload (same field shape `sim_verify_job_alloc_eq()` already validates) and feed the reconstructed decision through the same reporting call the existing decision-drain loop uses for a freshly-drained decision (Requirement 3.1). -- If not found (Requirement 4): report the job's usage as unavailable — do not fabricate a value, do not treat this as fatal, continue processing the rest of the replay. This is the expected case for a job already running at the moment the replay's starting snapshot was taken (see Risks). -- Because `job->accounts` is set as a side effect, a job reconciled in round N is never reconsidered in round N+1 — the same NULL-check that identifies the trigger also prevents double-application (Requirement 3.3), including against a job that separately receives a fresh live decision in the same round (that path sets `job->accounts` too, via the unchanged `sched_commit_decision()`, so this component simply finds nothing to do for it). - -**Contracts**: Service [x] / State [x] - -##### Service Interface -```c -/* - * For every RUN-state job in `sim->core.jobs` with job->accounts == - * NULL, attempts to reconstruct its allocation from a matching - * decision in this round's decoded_session->decisions and reports it - * through the same channel a freshly-drained decision uses. A job with - * no matching recorded decision is left with job->accounts == NULL - * (Requirement 4 -- reported as unavailable elsewhere, not an error - * here) and is safe to call again next round. - * - * MUTATES *decoded_session: each consumed entry is freed and removed - * (shifted down, not swap-removed -- see Implementation Notes) from - * decisions[]/n_decisions, so decoded_session cannot be `const` here -- - * this is deliberate, not an oversight: the caller's own subsequent - * --sim-verify positional walk over the same decoded_session depends on - * consumed entries already being gone by the time it runs. - * - * record_ctx is nullable and mirrors the existing decision-drain loop's - * own `if (record_ctx != NULL)` gate (this reporting path must recording- - * capture identically to a freshly-drained decision, and recording is - * optional per run) -- CONFIRMED during implementation this third - * parameter is necessary, not avoidable: record_ctx exists only as a - * stack-local inside sched_sim_driver_run()'s per-ReplaySession block - * (no field on xce_sched_sim_driver_t can supply it), and the round- - * boundary call site this function is wired into (task 3) sits in that - * exact same lexical scope, so passing it through is the natural, - * unambiguous plumbing rather than a workaround. - */ -void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, - replay_session_t* decoded_session, - sched_record_session_ctx_t* record_ctx); -``` -- Preconditions: `sim`/`decoded_session` non-NULL (NULL is a safe no-op, matching this codebase's established "never load-bearing" REPL-hook convention). `record_ctx` may be NULL (recording not active for this run). -- Postconditions: every RUN-state job either has `job->accounts` set (reconstructed) or is left NULL (genuinely unavailable this round). `decoded_session->decisions`/`n_decisions` reflect the removal of every consumed entry (freed, not merely uncounted -- see memory-ownership note below), leaving the remaining entries in their original relative order. -- Invariants: never double-applies (idempotent per job, per the `job->accounts` gate); never fabricates allocation data not present in the recording; never leaves a freed pointer reachable through `job->allocs` or through the remaining `decoded_session->decisions` entries. - -**Implementation Notes** -- Integration with `--sim-verify` (CONFIRMED during design validation, resolving the prior "read the comparison driver at implementation time" placeholder): `sim_loop.c`'s verify branch (current code: the `while ((cbuf_get(&(sim->core.decisions), &d) == 0) ...)` loop) walks `decoded_session->decisions` *positionally* via an index `di`, comparing each live-drained decision `d` against `decoded_session->decisions[di]` and advancing `di` once per live decision. A recorded decision belonging to a job this round's reconciliation step consumes (i.e., a job whose allocation/decision visibility is being restored from the recording rather than freshly re-derived by the live picker) will **never** produce a corresponding live-drained `d` — the picker never re-decides an already-resolved job. Comparing positionally without accounting for this would misalign every subsequent entry and produce false mismatches (Requirement 5.2 violation), independent of any bug in the comparison logic itself. - - **Resolution**: the reconciliation step removes each recorded decision it consumes from `decoded_session->decisions` *before* the verify drain loop runs for that round — it already has the exact jobid-matched entry in hand from its own lookup, so this is a natural side effect of the same pass, not a second scan. This keeps the remaining (unconsumed) recorded entries positionally aligned with the live-drained stream exactly as they are today, when no reconciliation was needed for a job. No change to `sim_decision_eq()`/`sim_verify_job_alloc_eq()`/`sim_verify_job_preason_eq()` (the comparison predicates in `sim_verify.c`) is required — only `sim_loop.c`'s bookkeeping of which array entries reach that comparison. - - **Memory ownership (must not be overlooked)**: `decoded_session->decisions`/`n_decisions` (`replay_session_t`, `typemap.h`) are a plain heap-owned array — `free_ReplaySession_ptr()` frees it via `free_ReplayDecision_list(d->n_decisions, d->decisions)`, which walks and frees exactly the first `n_decisions` entries, then frees the array itself. Consequently, simply decrementing `n_decisions` to "drop" a consumed entry (without separately freeing that entry) is **not** safe compaction — it silently leaks that entry's payload every round a reconciliation happens, since the round-end free walk will never reach it again. The reconciliation step must instead: (1) extract the data it needs (see ownership sub-point immediately below — this is NOT a plain read for a DISPATCH entry), (2) call the existing `free_ReplayDecision(entry)` + `free(entry)` (the same two calls `free_ReplayDecision_list()` already makes per-element) on the one consumed entry, then (3) compact the array by shifting the remaining entries down over the removed slot (`memmove`), and decrement `n_decisions` by one. Shift-down, not swap-with-last: swap-with-last is the usual O(1) removal trick, but it would relocate the last entry into the removed slot and change its position relative to the other remaining entries — corrupting the very relative ordering the positional `di` comparison in the point above depends on. `decoded_session` is decoded fresh once per round regardless, so this compaction is confined to the current round with no cross-round effect. - - **Ownership hazard in step (1) above — a DISPATCH entry's `alloc` must be moved, not aliased, before step (2) frees the entry**: the established live-path precedent (`scheduler.c:1440`, `sched_commit_decision()`) is `job->allocs = decision->u.alloc;` — a pointer *takeover*, not a deep copy; `job->accounts` (`make_job_account(job->allocs)`) is then built directly on top of that same, now job-owned, memory. If reconciliation reconstructs `job->allocs` the same way from a recorded DISPATCH entry — `job->allocs = entry->body.dispatch->alloc;` (`sched_dispatch_decision_t.alloc`, an `xce_job_alloc_t*`, `typemap.h`) — that pointer is now aliased by BOTH `job->allocs` and the entry about to be freed in step (2): `free_ReplayDecision()` → `free_DispatchDecision()` → `free_JobAlloc_ptr(&d->alloc)` would free that exact `xce_job_alloc_t`, leaving `job->allocs` dangling immediately (and a double-free whenever the job's own allocation is later released through its normal lifecycle). **Required fix**: immediately after the takeover assignment, null out the source field — `entry->body.dispatch->alloc = NULL;` — before calling `free_ReplayDecision(entry)`; every `_ptr`-style free helper in this codebase (including `free_JobAlloc_ptr`) already no-ops on `NULL`, so this makes the subsequent free of the rest of the entry safe without a special case. This is DISPATCH-specific and, by construction, the *only* case this component needs to handle: the reconciliation trigger is a RUN-state job, and a job can only reach RUN-state via a recorded DISPATCH decision — a recorded PEND (or PREEMPT) entry for the same job in the same round is structurally impossible to match here, since PEND/PREEMPT are exactly the decisions that do *not* put a job into RUN-state. (Independently confirmed no `job->preason`/equivalent field exists to alias in the first place — `job->accounts`/`job->allocs` are DISPATCH-only concepts, consistent with this.) No PEND/PREEMPT ownership case needs to be designed or implemented. - - This also fixes a latent problem that exists independently of this feature: `--sim-verify` against any real replay containing already-resolved jobs likely already misaligns positionally today (the recorded stream has entries the live stream never independently reproduces), so this correction is a net improvement to verify-mode accuracy, not merely a regression guard. -- Validation: must be exercised by a resource-contention fixture, not only a fixture where every job fits, per `scheduler-driver-equivalence.md`. -- Snapshot-restored jobs (CONFIRMED during design validation): `xce_sched_core_load_snapshot()` (`snapshot_load.c`) restores `job->state` from snapshot data but never touches `job->accounts`/`job->allocs`. A job already RUN-state at snapshot-load time therefore always has `job->accounts == NULL` with no in-window recorded decision to reconcile from — this is unambiguously Requirement 4's graceful-degradation case, not a separate path requiring its own handling. - -## Data Models - -### Domain Model -No new persistent entity. `job->accounts`/`job->allocs` (existing fields on `xce_sched_job_t`, `xce/types.h`) become the single source of truth for "has this job's allocation been established, by either the live path or reconciliation" — reused, not extended. - -### Logical Data Model -No new fields. The reconciliation step reads `decoded_session->decisions` (already-decoded, existing structure) and writes `job->allocs`/`job->accounts` (existing fields) exactly as the live `sched_commit_decision()` path already does for a fresh dispatch — same target shape, different source. - -No Physical Data Model, API Data Transfer, or Event Schema sections apply — no persistent storage, no new wire format, no new wire-level contract. This stays inside the existing `.replay` stream shape. - -## Error Handling - -### Error Strategy -Consistent with this codebase's established "never load-bearing, degrade to unattended" discipline (`repl_state_ensure()`'s own precedent): a reconciliation failure for one job never blocks the rest of the replay. - -### Error Categories and Responses -- **No matching recorded decision for a RUN-state, unaccounted job** (Requirement 4.1, 4.2): report usage as unavailable, continue. Not an error condition — expected for jobs already running at snapshot time. -- **Allocation failure while reconstructing `job->accounts` from a recorded decision**: fall back to leaving the job unaccounted (same as "not found") rather than a partial/corrupt reconstruction — preserves Requirement 4's "never fabricate" guarantee. -- **`xce_core_begin_session()` failure**: unchanged from today's `sim_loop.c` behavior — the round's decision drain is skipped, matching the existing error path this feature's split-call migration must preserve exactly. - -### Monitoring -No new logging framework — existing `WARN`/`TRACE` conventions apply if reconciliation needs to report a skip; no daemon-level behavior change. - -## Testing Strategy - -### Unit Tests -- Core-Level Resource Accounting produces identical `resInUse`/`sharedInUse` output whether called with a `core` obtained from `xce_scheduler_t` or from `xce_sched_sim_driver_t` (Requirement 1.4). -- Recorded-Decision Reconciliation reconstructs `job->allocs`/`job->accounts` correctly from a synthetic recorded DISPATCH decision (Requirement 3.1, 3.2). -- Recorded-Decision Reconciliation is a no-op (does not double-apply) for a job whose `job->accounts` is already set (Requirement 3.3). -- Recorded-Decision Reconciliation leaves a job unaccounted, without fabricating data or erroring, when no matching recorded decision exists (Requirement 4.1, 4.2). -- Reconciling a DISPATCH entry, then later releasing the job's allocation through its normal end-of-life path, must be ASAN-clean (no double-free, no leak) — this directly exercises the move-then-null-then-free ownership handling in the Implementation Notes above, run under the project's standard ASAN build per `tech.md`/`build-deployment.md` rather than relying on a plain functional assertion to catch it. - -### Integration Tests -- **Resource-contention fixture** (new, required per `scheduler-driver-equivalence.md`): multiple jobs competing for insufficient capacity, replayed in simulation mode; assert `resInUse` matches what the equivalent normal-mode recording produced at every round, and that no job is dispatched in simulation mode that normal mode would have pended (Requirement 2.1, 2.2). -- `test_repl_equivalence.c` (existing) re-run unmodified to confirm no regression to REPL-vs-batch-sim equivalence (Requirement 5.1). -- `--sim-verify` run against a recording containing already-resolved jobs, confirming no false mismatch is reported as a result of this feature (Requirement 5.2). -- A replay starting from a snapshot with jobs already running at snapshot time, confirming graceful degradation (Requirement 4) rather than a crash or fabricated value. - -### E2E -Live-verify against the real 108-round replay capture that originally motivated bug-1611/bug-1613: after the fix, `info worker` shows nonzero `used=` for jobs confirmed running via `JOB_RUSAGE` ticks, and their `DISPATCH` decisions become visible in `step`/`continue` output. - -## Optional Sections -Security Considerations, Performance & Scalability, and Migration Strategy are omitted: this is an internal scheduler-daemon correctness fix with no new attack surface, no new persistent data, and no schema/data migration. Performance is bounded by existing per-round job iteration this codebase already tolerates (`sched_account_jobs()`'s own existing full-job-table walk). - -## Supporting References -None beyond `research.md`. diff --git a/.kiro/specs/scheduler-sim-decision-parity/requirements.md b/.kiro/specs/scheduler-sim-decision-parity/requirements.md deleted file mode 100644 index 2eb28dc4..00000000 --- a/.kiro/specs/scheduler-sim-decision-parity/requirements.md +++ /dev/null @@ -1,50 +0,0 @@ -# Requirements Document - -## Introduction -The scheduler's `.replay`-driven simulation mode (`--sim`/`--interactive`) exists so operators can debug and tune scheduling behavior against recorded production data, trusting that a replay reproduces what actually happened. That trust currently does not hold for any job already resolved (dispatched, pended, or preempted) *before* a given point in the replay: its resource usage is never reflected in worker/shared resource-in-use totals, and its decision is never visible in decision output — because the recorded decision stream, though fully decoded every round, is only ever used for optional `--sim-verify` comparison, never applied to reconstruct state or report the decision otherwise. This feature restores decision *and* resource-accounting state parity between the normal-mode and simulation-mode drivers, per the constraint documented in `.kiro/steering/scheduler-driver-equivalence.md`. - -## Boundary Context -- **In scope**: resource-in-use accounting correctness during simulation-mode execution (interactive and batch); capacity-awareness correctness of simulation-mode scheduling decisions under resource contention; decision visibility (through existing decision-reporting channels) for jobs already resolved in the original recording; graceful behavior when recorded decision data for a job is unavailable. -- **Out of scope**: any change to REPL console output formatting (owned by the sibling `scheduler-sim-repl-verbosity` spec); any new inspectable data or REPL command (owned by the sibling `scheduler-sim-diagnostics` brief); any change to `--sim-verify`'s comparison logic itself, beyond not producing a false mismatch as a side effect of this feature; any change to scheduling/allocation policy; any change to normal-mode (non-simulation) behavior. -- **Adjacent expectations**: this feature restores an invariant `scheduler-core` (implementation-complete) was already expected to uphold; it does not reopen or modify that spec. `scheduler-sim-repl-verbosity`'s round-summary decision counts depend on decisions being reported through the same channel this feature restores, but that spec does not depend on this one to ship. - -## Requirements - -### Requirement 1: Accurate resource-usage reporting in simulation mode -**Objective:** As an operator inspecting simulation-mode resource state, I want per-worker and shared resource usage to reflect every currently-running job, so that what I inspect during a replay is trustworthy. - -#### Acceptance Criteria -1. While a job is in a running state during simulation-mode execution, the scheduler shall include that job's resource consumption in its worker's resource-in-use total. -2. While a job is in a running state during simulation-mode execution, the scheduler shall include that job's resource consumption in any shared resource's resource-in-use total it consumes from. -3. When a scheduling round completes, the scheduler shall refresh resource-in-use totals to reflect every currently-running job, not only jobs whose resource consumption was already reflected in an earlier round. -4. When replaying a recorded run in simulation mode, the scheduler shall report the same resource-in-use totals as normal-mode execution reported at the equivalent point in the original run. - -### Requirement 2: Correct capacity awareness under resource contention in simulation mode -**Objective:** As an operator relying on simulation mode to reflect realistic scheduling outcomes, I want the scheduler's capacity checks during simulation-mode execution to account for resources already consumed by running jobs, so that simulated scheduling behavior matches live behavior under contention. - -#### Acceptance Criteria -1. While resource capacity is already consumed by running jobs, when the scheduler evaluates whether a job fits on a worker during simulation-mode execution, the scheduler shall treat that already-consumed capacity as unavailable, the same way normal-mode execution does. -2. The scheduler shall not make a simulation-mode scheduling decision that normal-mode execution of the same recorded state would not also make. - -### Requirement 3: Decision visibility for jobs resolved before replay began -**Objective:** As an operator watching simulation-mode decision output, I want to see a decision for every job the replay reaches, including jobs already resolved in the original run, so I have a complete picture of scheduling activity. - -#### Acceptance Criteria -1. When a job was dispatched, pended, or preempted in the original recorded run and simulation-mode replay reaches the point that event occurred, the scheduler shall make that decision observable through the same decision-reporting channel used for a freshly-computed decision. -2. The scheduler shall preserve the original decision's type and resource-allocation detail when reporting it during replay. -3. When simulation-mode independently computes a fresh decision for a job in a given round, the scheduler shall report only that one decision for the job in that round, not also a duplicate from the recorded stream. - -### Requirement 4: Graceful degradation when recorded decision data is unavailable -**Objective:** As an operator running simulation mode against an incomplete or older recording, I want the scheduler to degrade gracefully rather than fail, so a data gap in one job's history doesn't block inspection of the rest of the replay. - -#### Acceptance Criteria -1. If a job's originating decision cannot be found in the recorded data when simulation-mode needs it to report or account for that job, the scheduler shall report that job's resource usage as unavailable rather than fabricating a value. -2. If a job's originating decision cannot be found in the recorded data, the scheduler shall continue processing the remainder of the replay without terminating. - -### Requirement 5: No regression to existing driver-equivalence and verification guarantees -**Objective:** As a maintainer relying on existing scheduler guarantees, I want this fix to preserve them, so existing verification workflows and tests continue to hold. - -#### Acceptance Criteria -1. The scheduler shall continue to produce identical decisions between interactive (step-by-step) and batch simulation execution modes. -2. The scheduler shall continue to support decision verification mode (comparing freshly-computed decisions against recorded ones) without reporting a mismatch caused by this feature's own behavior. -3. While simulation mode is not enabled, the scheduler shall not change its existing resource-accounting or decision-reporting behavior. diff --git a/.kiro/specs/scheduler-sim-decision-parity/research.md b/.kiro/specs/scheduler-sim-decision-parity/research.md deleted file mode 100644 index c77aaedf..00000000 --- a/.kiro/specs/scheduler-sim-decision-parity/research.md +++ /dev/null @@ -1,62 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `scheduler-sim-decision-parity` -- **Discovery Scope**: Extension (existing system) — light discovery -- **Key Findings**: - - `sched_check_host_satisfy_()`/`sched_check_shared_satisfy_()` (resource.c) read `w->resInUse`/`core->sharedInUse` directly as inputs to the live allocator's own capacity check — bug-1613's accounting gap is **confirmed** to affect simulation-mode scheduling decisions under contention, not just displayed data. This resolves the open question from the brief and raises this spec's priority. - - Every decision the normal-mode scheduler makes (including `DISPATCH` with its full `xce_job_alloc_t` payload) is written into the `.replay` stream via `sched_record_session_add_decision()` (`xce_sched_driver_run_normal_mode_round()`, driver.c). `sim_loop.c` already decodes this per round (`decoded_session->decisions`) but only reads it under `sim->verify`, purely for comparison. - - `job->accounts == NULL` is already the exact signal `sched_account_jobs()` uses to skip a job it can't re-charge — the same signal cleanly identifies "this RUN-state job has never been reconstructed" and, once set, naturally prevents double-application from a later pass (idempotent by construction, no new bookkeeping field needed). - - `sim_verify_job_alloc_eq()`/`sim_verify_host_alloc_eq()`/`sim_verify_alloc_eq()` (sim_verify.c) already implement field-by-field equality for exactly the `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t` payload shapes this feature needs to reconstruct `job->allocs` from — reusable, not something to reinvent. - - The actual round-level comparison driver in sim_verify.c (as opposed to the eq-helpers read this session) was not read — how `--sim-verify` currently handles a recorded decision with no live-drained counterpart (which is exactly the case this feature changes the meaning of) needs to be confirmed at implementation time, flagged as a risk below. - -## Research Log - -### Does resInUse feed actual scheduling decisions, or only display data? -- **Context**: The brief flagged this as unconfirmed and consequential — determines whether bug-1613 is a display bug or a decision-correctness bug. -- **Sources Consulted**: `resource.c` (`sched_check_host_satisfy_`, `sched_check_shared_satisfy_`, `sched_try_alloc_on_host_`). -- **Findings**: `sched_check_host_satisfy_()` reads `w->resvals` (total) and `w->resInUse` (in-use) and passes both to `unsatisfy_()`, whose result determines whether `sched_try_alloc_on_host_()` (the live allocator's per-host evaluation, called from the picker/allocator's job-election path) treats a host as viable. `sched_check_shared_satisfy_()` does the identical thing against `core->sharedInUse` for shared resources. -- **Implications**: Confirmed, not hypothetical. Requirement 2 exists because this mechanism is real: with `resInUse` permanently zero, simulation-mode capacity checks can systematically pass where normal mode would fail, meaning simulated scheduling under contention can diverge from what actually happened. - -### Where does the recorded decision stream carry allocation data, and what does the normal-mode driver already do with decisions? -- **Context**: Confirming decoded_session->decisions is a complete, reusable data source rather than a partial one. -- **Sources Consulted**: `driver.c` (`xce_sched_driver_run_normal_mode_round`'s decision-drain loop), `sim_verify.c` (equality helpers). -- **Findings**: Every decision drained from `sched->core.decisions` in a normal-mode round — before being handed to `sched->core.on_decision` — is recorded via `sched_record_session_add_decision()` if recording is active, with its full payload (`xce_job_alloc_t` for DISPATCH, including per-host and shared-resource allocation lists). `sim_verify.c` already has mature, tested field-by-field equality functions for this exact payload shape. -- **Implications**: `decoded_session->decisions` is a complete historical record of every decision (not just PEND, which is the only op the live sim-mode picker currently re-derives for already-resolved jobs). Reconstructing `job->allocs` from a recorded `DispatchDecision` is a straightforward field copy, not a reverse-engineering exercise — and the equality-checking precedent in `sim_verify.c` shows the exact shape to build against. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Reconstruct from recorded decisions, keyed on `job->accounts == NULL` (chosen) | At the round-boundary accounting point, for every RUN-state job with no accounts, look up and apply its recorded decision from this round's `decoded_session->decisions` | Reuses data already decoded every round; reuses `sim_verify.c`'s existing payload-equality precedent for the reconstruction shape; the NULL-accounts check is naturally idempotent (no new dedup bookkeeping) | Requires care around `--sim-verify`'s existing comparison semantics (see Risks) | Directly extends the point Requirement 1's accounting fix already establishes — one round-boundary pass, not a second parallel mechanism | -| Reconstruct from `JOB_UPDATE`'s wire `allocs` field | Extend `core_events.c`'s `XCE_CORE_EVENT_JOB_UPDATE` handler to check `updated->allocs != NULL`, mirroring `core_build_job()`'s existing JOBS_PUSH pattern | Simpler, localized to one file | **Rejected as primary mechanism** — confirmed this session that allocation data does not travel via `JOB_UPDATE` events at all; it travels via the recorded decision stream exclusively (see Research Log). This approach has no data to act on for the dominant case. | Not viable as the primary fix; not pursued further | -| Re-run the live picker/allocator against every job every round, discard non-matching results | Force full re-derivation regardless of recording | Would not depend on recorded-decision structure at all | Breaks the entire premise of replay (a deterministic capture-and-replay, not a live re-simulation) and risks non-determinism/performance cost re-running allocation logic that already ran once | Rejected — contradicts `scheduler-core`'s own design intent for what "replay" means | - -## Design Decisions - -### Decision: Key reconstruction and dedup on `job->accounts == NULL`, not a separate tracking set -- **Context**: Need to (a) know which RUN-state jobs need reconstruction from a recorded decision, and (b) avoid double-applying when a job also gets a fresh live decision in the same round. -- **Alternatives Considered**: (1) a separate per-job "already reconciled" flag/set; (2) reuse the existing `job->accounts == NULL` check `sched_account_jobs()` already performs. -- **Selected Approach**: (2). -- **Rationale**: `job->accounts` is set exactly once, at the moment a job's allocation becomes known — either via a fresh `sched_commit_decision()` (live path, unchanged by this feature) or via this feature's reconstruction pass. Once set, both `sched_account_jobs()`'s re-charge and this feature's own reconstruction pass naturally skip the job on every subsequent round — no new state, no drift risk between two independently-maintained bookkeeping mechanisms. -- **Trade-offs**: None significant — this is a strict simplification over introducing new state. -- **Follow-up**: none. - -### Decision: Feed a reconstructed decision through the same reporting path as a fresh one -- **Context**: Requirement 3 needs decision *visibility*, not just accounting correctness — an operator must see the decision, not only have `resInUse` quietly become correct. -- **Alternatives Considered**: (1) only fix accounting, leave decision visibility as a separate/future concern; (2) synthesize a decision object from the recorded data and route it through the exact same callback (`on_decision`/`xce_repl_check_decision`) a freshly-drained decision uses. -- **Selected Approach**: (2). -- **Rationale**: Requirement 3.1 explicitly requires "the same decision-reporting channel used for a freshly-computed decision" — a parallel/different reporting path would be a second thing to keep in sync, exactly the class of bug this whole spec exists to close. -- **Trade-offs**: The reconstructed decision must be a genuine `xce_decision_t` (or converted to one), not a display-only string — slightly more construction work than a printf, but necessary for it to be indistinguishable from a live decision to any consumer (REPL, `scheduler-sim-repl-verbosity`'s round rollup, etc.). -- **Follow-up**: none. - -## Risks & Mitigations -- **`--sim-verify` interaction — RESOLVED during design validation** (see `design.md`'s Recorded-Decision Reconciliation Implementation Notes for the full mechanism). The round-level comparison driver (`sim_loop.c`'s `while (cbuf_get(&(sim->core.decisions), &d) ...)` loop) walks `decoded_session->decisions` positionally via an index `di`. Confirmed: a job this feature reconciles never produces a live-drained counterpart, so the reconciliation step must remove (properly freed, not just index-dropped — see design.md's memory-ownership note) each consumed recorded decision from `decoded_session->decisions` before the verify loop runs, keeping the remaining entries positionally aligned by shifting down (not swap-with-last, which would reorder them). `sim_verify.c`'s comparison predicates need no change. -- **Snapshot-restored jobs already running at replay start — RESOLVED**: confirmed by reading `snapshot_load.c` that `xce_sched_core_load_snapshot()` restores `job->state` but never `job->accounts`/`job->allocs`. Such a job is therefore always `job->accounts == NULL` with no in-window recorded decision — unambiguously Requirement 4's graceful-degradation case, no separate handling needed. -- **Relocating `sched_account_jobs()`'s signature touches a function with production callers** — mitigated by the fact the relocation is signature-only (body unchanged, per the `sched_resolve_job_picker()` precedent already proven safe for the identical pattern) and both call sites (`driver.c`, `sim_loop.c`) are updated in the same change. Independently re-confirmed during design validation: `grep "sched->" resource.c | grep -v "sched->core"` returns zero matches across the entire file — every one of the six relocated functions' bodies (not just `sched_account_jobs()` itself) genuinely touches only `sched->core`. -- **New risk surfaced while resolving the `--sim-verify` risk above**: naively freeing a consumed DISPATCH entry to avoid the leak would double-free, because the live path (`scheduler.c:1440`, `job->allocs = decision->u.alloc;`) takes ownership of `alloc` by pointer move, not copy — reconciliation reconstructing `job->allocs` the same way would alias memory about to be freed. Mitigation: null out the source entry's `alloc` field immediately after the takeover assignment, before freeing the rest of the entry (every `_ptr`-style free helper in this codebase already no-ops on `NULL`). See design.md for the full writeup. - -## References -- `.wolf/buglog.json` bug-1611, bug-1613 — the originating findings. -- `.kiro/steering/scheduler-driver-equivalence.md` — the invariant this spec restores, including its testing discipline (resource-contention fixtures required). -- `.kiro/specs/scheduler-sim-repl-verbosity/` — sibling spec; its round-rollup dispatch counts benefit from this fix once implemented but do not depend on it. diff --git a/.kiro/specs/scheduler-sim-decision-parity/spec.json b/.kiro/specs/scheduler-sim-decision-parity/spec.json deleted file mode 100644 index 1ffc7eb4..00000000 --- a/.kiro/specs/scheduler-sim-decision-parity/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "scheduler-sim-decision-parity", - "created_at": "2026-08-21T08:30:00Z", - "updated_at": "2026-08-22T13:10:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/scheduler-sim-decision-parity/tasks.md b/.kiro/specs/scheduler-sim-decision-parity/tasks.md deleted file mode 100644 index 8bbc5084..00000000 --- a/.kiro/specs/scheduler-sim-decision-parity/tasks.md +++ /dev/null @@ -1,104 +0,0 @@ -# Implementation Plan - -- [x] 1. Foundation: shared accounting surface and contention test data -- [x] 1.1 Make resource-in-use accounting callable from either driver - - Relocate the resource re-accounting service (currently tied to the normal-mode-only driver struct) so it operates on the mode-agnostic core state shared by both drivers, mirroring the same relocation already done for the job-picker resolution service - - Update all three existing call sites to the new signature: the normal-mode round handler's accounting call, the scheduler's own session-begin path's accounting call, and the scheduler's decision-commit path's call into the job-allocation-commit function that is part of this same relocated group - - Body of the accounting logic itself is unchanged — this is a pure capability-relocation step, not new behavior - - Observable completion: the full scheduler test suite still builds and passes unchanged (zero behavior change), and the simulation driver's build now has visibility to call the same accounting service the normal-mode driver already uses - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 5.3_ - - _Boundary: Core-Level Resource Accounting_ - -- [x] 1.2 (P) Build a resource-contention replay fixture for testing - - Create a recorded-session test fixture with multiple jobs competing for capacity insufficient to run all of them at once (existing fixtures only ever have enough capacity for every job, which cannot exercise or prove capacity-check correctness) - - Fixture must be usable both as a normal-mode recording source and as a simulation-mode replay input, so the same scenario can be run through both drivers and compared - - Observable completion: a scheduling round exists in the fixture where at least one job is denied capacity that a currently-running job is consuming - - _Requirements: 2.1, 2.2_ - - _Boundary: Scheduler test suite (contention fixture)_ - -- [x] 2. Core: round-boundary accounting and decision reconciliation -- [x] 2.1 Give simulation mode the same round-boundary accounting sequence normal mode uses - - Replace simulation mode's single combined session-start call with the same two-step sequence (reset accounting, then unblock the picker) the normal-mode driver already uses, opening an interleaving point between them - - Call the relocated resource re-accounting service (task 1.1) in that interleaving point, in the same relative position the normal-mode driver already calls it - - Preserve the existing error-handling behavior for a failed session start exactly as it works today - - Interactive (step-by-step) and batch simulation runs must go through this same call site with no divergence between them - - Observable completion: a job dispatched fresh within a simulation run (not already resolved before the replay reached it) now shows correct worker and shared resource-in-use totals immediately after the round it was dispatched in, matching what the equivalent normal-mode recording shows at the same point - - _Requirements: 1.1, 1.2, 1.3, 1.4, 5.1, 5.3_ - - _Depends: 1.1_ - - _Boundary: Sim Driver Round-Boundary Accounting_ - -- [x] 2.2 Detect and locate already-resolved jobs needing reconciliation - - Identify, once per round, every currently-running job that has no known resource allocation yet (this is the signature of a job that was already resolved earlier in the original recording, before this replay run's own live decision-making ever considered it) - - For each such job, search this round's decoded recorded-decision data for the matching originating decision by job identity - - When no matching recorded decision is found, leave the job's allocation unknown, report its usage as unavailable rather than fabricating a value, and continue processing the rest of the replay without treating this as an error - - A job left unresolved this round must be safe to re-check again on a later round without any adverse effect - - Observable completion: given a fixture where a job is already running with no recorded decision available in the replay's own window (e.g. a job dispatched before the snapshot the replay started from), the job is reported as having unavailable usage and the replay continues to completion without error - - _Requirements: 4.1, 4.2_ - - _Boundary: Recorded-Decision Reconciliation_ - -- [x] 2.3 Reconstruct an already-resolved job's allocation from its recorded decision, safely - - When a matching recorded dispatch decision is found (task 2.2), take ownership of its resource-allocation data to become the job's own allocation and accounting record — the same ownership-transfer pattern the live dispatch path already uses, not a duplicate copy - - Once that data has been taken over, remove the now-fully-consumed recorded decision entry from this round's decoded data, releasing only the parts of it that were not taken over, and close the resulting gap so the remaining recorded entries keep their original relative order - - A job whose allocation is reconstructed this way must never be reconsidered by this same mechanism again on a later round - - Observable completion: reconciling a job's allocation and then later releasing that job's allocation through its normal end-of-life path is clean under the project's standard memory-safety build (no double-release, no leaked allocation) — this is the task's primary correctness bar, not a peripheral check - - _Requirements: 1.1, 1.2, 3.2_ - - _Depends: 2.2_ - - _Boundary: Recorded-Decision Reconciliation_ - -- [x] 2.4 Make a reconciled job's decision visible the same way a fresh one is - - Feed a reconstructed decision (task 2.3) through the exact same decision-reporting path a freshly-computed decision already uses, preserving its original decision type and resource-allocation detail - - If simulation mode's own live decision-making independently produces a fresh decision for a job in the same round it would otherwise be reconciled, report only the fresh one — never both - - Observable completion: replaying a fixture where a job was already dispatched before the replay's live decision-making reaches it now shows that job's dispatch decision in step/continue output, exactly once, with the same resource-allocation detail the original recording captured - - _Requirements: 3.1, 3.3_ - - _Depends: 2.3_ - - _Boundary: Recorded-Decision Reconciliation_ - -- [x] 3. Integrate reconciliation into the round boundary and confirm decision-verification stays correct - - Call the reconciliation mechanism (tasks 2.2-2.4) inside the round-boundary sequence (task 2.1), positioned after the session-start reset and before the resource re-accounting call, so the same round's re-accounting picks up every newly-reconstructed job and the still-pending live decision-making sees correct capacity - - Confirm the round's decision-verification comparison (which walks the same recorded-decision data positionally) is unaffected by reconciliation having already consumed and removed some of those entries — no special-casing should be needed in the comparison logic itself, only in what data reaches it - - Observable completion: running decision verification against a recording containing already-resolved jobs reports no mismatch attributable to this feature, and a job whose allocation was reconciled this round is reflected in resource-in-use totals before the live picker/allocator evaluates capacity for any other job in that same round - - _Requirements: 1.3, 1.4, 2.1, 2.2, 5.2_ - - _Depends: 2.1, 2.4_ - - _Boundary: Sim Driver Round-Boundary Accounting, Recorded-Decision Reconciliation_ - -- [x] 4. Validation -- [x] 4.1 (P) Prove capacity-aware parity under contention - - Using the resource-contention fixture (task 1.2), replay it in simulation mode and compare worker and shared resource-in-use totals against the equivalent normal-mode recording at every round - - Assert simulation mode never makes a scheduling decision (dispatch where normal mode pended, or vice versa) that the normal-mode recording of the same state did not also make - - Observable completion: the contention fixture replay produces byte-for-byte identical resource-in-use totals and scheduling outcomes to its normal-mode recording, at every round, not just the final one - - _Requirements: 1.4, 2.1, 2.2_ - - _Depends: 3_ - - _Boundary: Scheduler test suite (contention parity)_ - -- [x] 4.2 (P) Regression suite: existing guarantees still hold - - Re-run the existing interactive-vs-batch decision-equivalence coverage unmodified and confirm it still passes - - Extend decision-verification coverage with a recording containing already-resolved jobs and confirm no mismatch is reported as a result of this feature - - Confirm a replay starting from a snapshot with jobs already running at snapshot time (no recorded decision available in-window) degrades gracefully rather than crashing or fabricating a value - - Confirm that with simulation mode not enabled, resource-accounting and decision-reporting behavior is byte-for-byte unchanged from before this feature - - Observable completion: all four checks above pass in the same test run, with the pre-existing equivalence test's pass/fail behavior unchanged from before this feature - - _Requirements: 4.1, 4.2, 5.1, 5.2, 5.3_ - - _Depends: 3_ - - _Boundary: Scheduler test suite (regression/verify coverage)_ - -- [x] 4.3 Live-verify against the original motivating recording - - Replay the real recorded session that originally surfaced the resource-accounting and decision-visibility gaps this feature fixes, through to completion - - Observable completion: inspecting a worker mid-replay now shows nonzero resource usage for jobs independently confirmed running via their ongoing usage updates, and their dispatch decisions are now visible in step/continue output where they previously never appeared - - _Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3_ - - _Depends: 4.1, 4.2_ - - _Boundary: Scheduler test suite_ - -## Implementation Notes -- (task 1.2) Whole-scenario white-box test files in `src/scheduler/testsuite/` (fixture construction + real driver/allocator plumbing) routinely exceed the project's 800-line guidance (`test_sim_verify.c` is 934 lines, uncorrected; the new `test_sim_resource_contention.c` is 883) — the steering's "don't use as precedent" caveat names only non-test files, so this is an accepted pattern for this class of test, not a defect to fix per task. Tasks 4.1-4.3 will likely add similarly-sized files; don't flag file size alone as blocking for them. -- (task 2.1) Confirmed (by reading `sim_picker_thread_fn()`): simulation mode's live picker thread unconditionally PENDs every job today, and `sched_commit_decision()` (the only real setter of `job->accounts`) is normal-mode-only. So after 2.1 alone, `resInUse` is still zero end-to-end in sim mode -- 2.1 only wires the call site/interleaving slot. Tasks 2.2-2.4 (Recorded-Decision Reconciliation) are what actually populate `job->accounts` and make `resInUse` correct. Don't expect nonzero `resInUse` from a sim run until 2.2-2.4 land -- this is expected, not a regression. -- (task 2.1) White-box test files that `#include "../src/sim_loop.c"` directly (test_sim_event_loop.c, test_sim_verify.c, test_sim_alt_events.c, test_sim_record_output.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c, test_repl_execution_control.c, test_repl_equivalence.c) need a link-satisfying stub added whenever sim_loop.c gains a new extern reference (here: `sched_account_jobs()`). Tasks 2.2-2.4 will likely add more such references (e.g. into the recorded-decision/reporting machinery) -- expect to touch these same 8 files again for stub updates, and check for linker errors rather than guessing which files need it. -- (task 2.3) Do NOT check "not found" via `i == decoded_session->n_decisions` after the lookup loop once the loop's array can be mutated mid-iteration (e.g. once `n_decisions` gets decremented by a compaction) -- that check goes stale for a last-index match. Use an explicit `found` flag instead. This bug would only have surfaced later (task 2.3 introduced the mutation that makes the old check wrong) but was caught and fixed here. -- (task 2.3) `src/lib/misc.c`'s `free_host_account()` never frees the `strdup`'d `host` field `make_host_account()` (`src/lib/xcealloc.c:119`) allocates -- a real, pre-existing leak, equally present on the live `scheduler.c` dispatch/release path, previously masked because this test suite is registered with `ASAN_OPTIONS=detect_leaks=0`. Logged as bug-1621, intentionally NOT fixed here (out of this feature's boundary -- misc.c isn't owned by scheduler-sim-decision-parity). If a later task's ASAN run surfaces this same leak, it's not a new regression -- don't waste time re-diagnosing it, it's bug-1621. -- (task 2.3) `make_job_account()`'s return isn't NULL-checked before use, matching the live-path precedent (`scheduler.c`'s `sched_commit_decision()` has the same gap) -- informational only, not a defect to fix in this feature unless the live path itself gets hardened first. -- (task 2.4) The reconciliation function ended up needing a third parameter (a nullable recording-context handle) beyond the two design.md originally specified, because that handle only exists as a stack-local inside the round-loop and nothing on the driver struct can supply it -- design.md's Service Interface has been updated to match. Task 3 (wiring): the call site sits in the exact same lexical scope where that recording-context variable is already declared, so pass it straight through -- no new plumbing needed. -- (task 2.4) `xce_decision_t` objects built via `make_dispatch_decision()`/`make_pend_decision()` are never freed anywhere in this codebase on any live path (verified by grep across `scheduler/src/*.c`) -- this is an established, accepted pattern here, not something to "fix" if you see it under `detect_leaks=1`. Don't add a free for it; that would deviate from the codebase's own convention without a broader cleanup pass. -- (task 3) `get_resvals()` (resource.c) returns `nres` (the count filled in) on success, NOT `0` -- a test-authoring bug (bug-1622, not production) came from assuming the latter. If you write a test against `get_resvals()`, check its actual return-value contract first. -- (task 3) Feature is now end-to-end wired and independently confirmed working (reviewer removed the call site itself and watched both integration tests fail for the right reason, then restored and watched them pass). Tasks 4.1-4.3 (Validation) can now exercise the real, complete path -- no more "not fully wired yet" caveats apply. -- (task 4.1) `sim_picker_thread_fn()` is confirmed (by direct read, independently re-confirmed by the reviewer) to be structurally PEND-only -- it has no capacity-check/dispatch path for a job it evaluates fresh. This means task 1.2's fixture (or any single-round, freshly-arriving-jobs fixture) can NEVER be literally replayed-and-diffed for full dispatch/PEND parity -- sim's picker always PENDs a fresh job regardless of whether resource accounting is correct, so that comparison would be tautological for freshly-arriving jobs (though still meaningful for reconciled/already-resolved jobs). If a later task needs "real recording vs. sim replay" parity proof, build it the way task 4.1 did: a real driven normal-mode recording plus a hand-crafted sim-mode fixture modeling the identical facts (including at least one already-resolved-before-replay job to exercise reconciliation), not a literal single-file replay-and-diff. This is a real architectural limit, not a testing shortcut -- true dispatch-decision parity for freshly-arriving jobs is scheduler-allocation's (a separate, not-yet-designed spec) job, out of this feature's scope. -- (task 4.2) `builtin_fcfs`'s pend-handling (`builtin_fcfs_on_pend`/`begin_session`) never clears a job from `pendJobs` once it's added -- a still-genuinely-PEND job recurs as a fresh PEND decision in the recorded stream EVERY round/session, not just once. This is pre-existing, unrelated-to-this-feature behavior (also visible in `test_repl_equivalence.c`'s own existing fixture). If you build a multi-round `.replay` fixture with a job that stays PEND across rounds, expect and account for that job reappearing in every round's decoded decisions -- don't assume a PEND is a one-time event. -- (task 4.2) Not every reconciliation-adjacent test needs to be discriminating via the "disable the reconciliation call site" toggle to be worth keeping -- a test exercising the "not found, nothing to reconcile" branch through the REAL `sched_sim_driver_run()`/`xce_sched_core_load_snapshot()` path can be identical whether reconciliation runs or not, and still catch real bugs (fabrication, snapshot-load corruption, crashes) that a hand-built-core unit test (task 2.2's) can't reach. Judge a test's value by what regression classes it actually covers, not solely by one discriminating-power toggle. -- **(task 4.3 -- IMPORTANT SCOPE FINDING, confirmed structurally by code reading, not speculation)**: Recorded-Decision Reconciliation only succeeds when a job's recorded DISPATCH decision and its recorded JOB_STATE_CHANGE-to-RUN event land in the SAME replay round (reconciliation searches only that round's own freshly-decoded `decoded_session->decisions`; each round's decode is discarded once that round finishes processing). Every test in this feature that exercises the successful-match path (`test_sim_round_boundary_reconciliation.c` task 3, `test_sim_contention_parity.c` task 4.1 "Part B", `test_sim_e2e_original_motivation.c` task 4.3's `ANCHOR_ID`) had to hand-engineer this exact coincidence -- none occurs from organic, continuously-driven scheduling, because a job cannot be "freshly PEND for this round's live picker" and "already RUN before this round's picker runs" at the same time (`builtin_fcfs`'s dispatch/state-change hooks make these mutually exclusive by construction). In any real deployment with genuine dispatch-to-worker-confirmation latency (the normal case for an async, network-distributed scheduler), a job's DISPATCH decision and its RUN confirmation will typically land in DIFFERENT rounds -- meaning this feature, as designed and implemented, likely does NOT fix bug-1611/bug-1613's dominant real-world manifestation ("a job dispatched at round 32, confirmed running by some later round, still shows resInUse=0/no visible decision when inspected at round 100"). It only fixes the narrower case of same-round dispatch+confirmation coincidence. If bug-1611/1613's common case needs fixing, the follow-up would need reconciliation to search backward across MULTIPLE prior rounds (or maintain a persistent jobid-keyed index of not-yet-consumed recorded decisions across the whole replay, not just the current round) -- a materially different, larger design than what this feature built. Flagged for the user/future spec, not something task 4.3 could fix within its validation-only boundary. diff --git a/.kiro/specs/scheduler-sim-diagnostics/brief.md b/.kiro/specs/scheduler-sim-diagnostics/brief.md deleted file mode 100644 index 7a39b8df..00000000 --- a/.kiro/specs/scheduler-sim-diagnostics/brief.md +++ /dev/null @@ -1,58 +0,0 @@ -# Brief: scheduler-sim-diagnostics - -## Problem - -Operators/developers investigating scheduler behavior in simulation mode (the `--sim`/`--interactive` REPL shipped by `scheduler-core`) can inspect job/queue/worker/resource state (`info job/queue/worker/resource/summary`) but cannot answer two common investigation questions: - -1. **What limits are configured, and what is their current usage?** `xce_limit_t`/`xce_vres_t` (limit config + live per-user/queue/project/host usage instances) exist only inside the `limit` plugin's private `kvmgr` tables (`plugin.c`'s `limit_plugin_data_t`) — invisible to the REPL and to `sched_get_*()` accessors. -2. **Why did a job pend, or why did a job that was expected to pend actually run?** `xce_pendreason_t` is attached to pend decisions and to `xce_sched_obj_t`, but the REPL never prints it, and for a job that *did* dispatch there is no recorded trail of which limit/resource checks passed — only the final DISPATCH outcome. - -## Current State - -- `scheduler-core` (implementation-complete) shipped the REPL, breakpoints, and state-inspection commands (Req 5 AC 22) — this is the platform this work extends. -- No limit accessor exists outside the `limit` plugin's internal tables. -- No decision-reasoning trace exists for the dispatch path; only pend decisions carry a reason, and it isn't surfaced. - -## Desired Outcome - -Simulation-mode REPL (and, over time, other diagnostic surfaces added under this spec) lets an investigator answer "what is configured / in use" and "why did the scheduler decide X" without reading source or adding printf debugging. This is an **open-ended, continuously-extended spec** — new debug/troubleshoot capabilities get added as requirements over time, driven by discussion or by needs surfacing from other in-flight specs. Not all scope is known upfront. - -## Approach - -Extend the existing REPL/simulation-driver surface (not a new component). Each new capability is its own Requirement added when its scope is settled — this brief captures only the two seed capabilities identified so far; more will be appended. - -**Seed capability 1 — Limit inspection**: expose limit config (name, per_user/queue/project/host scope, value/percent) and live usage (per-vres-instance current value vs limit) via a new `info limit <name|all>` REPL command. Requires plumbing a read path from the `limit` plugin's private kvmgr tables to the REPL (core has no existing accessor for plugin-private state). - -**Seed capability 2 — Decision-reasoning trace**: capture *why* each scheduling decision was made — not just the outcome. For PEND, surface the existing `xce_pendreason_t` (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (`sched_try_alloc_on_host_`, `limit_prepare_allocation`) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision. - -## Scope - -- **In**: Simulation-mode / REPL-facing investigation tooling only. Read-only inspection — no state mutation (consistent with `scheduler-core` Req 5's existing "no write-side mutations" boundary). Capabilities are added incrementally as separate ACs/Requirements. -- **Out**: Live production inspection against a controller-connected scheduler (no `OPCode_schedAdmin`-style wire work) — explicitly deferred; this spec is simulation-mode only per user decision. Any write/mutation commands. New allocation or picker logic — this spec only observes decisions made by existing logic. - -## Boundary Candidates - -- Limit inspection (read path + REPL command) — self-contained, touches `limit.c` (expose accessor) + `repl_info.c` (new command). -- Decision-reasoning trace (new instrumentation + REPL command) — larger: touches the allocation hot path (`resource.c`, `plugin.c`) to record trace data, plus decision/REPL plumbing to surface it. -- Future capabilities: unscoped — to be added as separate requirements when identified. - -## Out of Boundary - -- Live/production (controller-connected) diagnostics. -- Any REPL command that mutates scheduler state. -- New scheduling policy, allocator, or picker behavior. - -## Upstream / Downstream - -- **Upstream**: `scheduler-core` (REPL, simulation driver, `info` command family this extends). `scheduler-allocation` and `scheduler-preemption` (siblings, currently blocked on scheduler-core, now unblocked) are likely future sources of new diagnostic requirements once they land, since this spec's scope is meant to absorb debug needs "according to the requirement of other projects." -- **Downstream**: none yet — open-ended. - -## Existing Spec Touchpoints - -- **Extends**: `scheduler-core` (REPL/simulation driver architecture, Req 5's `info`/breakpoint/snapshot command family). -- **Adjacent**: `scheduler-allocation`, `scheduler-preemption` — not yet implemented; expect this spec to gain requirements once those introduce new decision types/state worth inspecting. - -## Constraints - -- Must not alter deterministic replay guarantees (`scheduler-core` Req 5's normal↔sim equivalence invariant) — inspection commands are read-only and must not perturb core state or event ordering. -- Not starting immediately — requirements will be added/refined per-capability over time rather than fully speced upfront. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md deleted file mode 100644 index 12dcbbfc..00000000 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md +++ /dev/null @@ -1,212 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: This feature closes the residual gap in `scheduler-sim-decision-parity`'s Recorded-Decision Reconciliation mechanism (`sim_reconcile_unaccounted_jobs()`, `sim_loop.c`) so a simulation-mode replay correctly captures a job's resource allocation regardless of how many rounds separate its recorded DISPATCH decision from its RUN-state confirmation — the realistic case in any real deployment, and the dominant real-world manifestation of bug-1611/bug-1613. - -**Users**: Scheduler operators debugging or tuning via `xce_scheduler -i -s <replay>` (interactive) or batch `--sim` replay. - -**Impact**: Changes when `sim_reconcile_unaccounted_jobs()` is willing to search a round's recorded decision stream for a job — from "only if the job is already RUN this round" to "whenever the job has no captured allocation yet, regardless of its current state." No other function changes. - -### Goals -- Capture a job's recorded DISPATCH allocation at the round it actually occurred in the original recording, independent of the job's state at that moment. -- Preserve correct, state-gated resource-in-use accounting: a captured-but-not-yet-running job still contributes zero to `resInUse` until it genuinely transitions to RUN. -- Make the dispatch decision observable (printed, recorded, REPL-checked) at the historically accurate round. - -### Non-Goals -- Any change to `job->state` transition timing or triggers. -- Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per `scheduler-allocation`). -- Any change to `--sim-verify`'s comparison semantics beyond not introducing a new false mismatch. -- Reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks, or any normal-mode (`driver.c`) behavior. - -## Boundary Commitments - -### This Spec Owns -- The trigger condition of `sim_reconcile_unaccounted_jobs()` (`sim_loop.c`) — which jobs are eligible to be searched for a matching recorded DISPATCH decision each round. -- The test coverage proving capture now works across round separation, still respects the existing no-fabrication/no-double-capture guarantees, and does not regress `--sim-verify` or interactive/batch parity. - -### Out of Boundary -- `sched_account_jobs()`'s own `JOB_IS_RUN` charging gate (`scheduler.c:1258`) — unchanged; it is exactly the correct deferral mechanism this feature relies on, not something it modifies. -- The match/capture mechanics once a job is eligible (jobid matching, ownership transfer of `xce_job_alloc_t`, `make_job_account()`, shift-down compaction, decision reporting via `make_dispatch_decision()`/`sim_print_decision()`/`xce_repl_check_decision()`) — all unchanged, all already correct per `scheduler-sim-decision-parity`. -- Normal-mode (`driver.c`) behavior, simulation mode's live picker, `--sim-verify`'s comparison algorithm itself, and any `job->state` transition logic. - -### Allowed Dependencies -- `sim->core.jobs` iteration and `decoded_session->decisions` (both already owned/populated by `sim_reconcile_unaccounted_jobs()`'s existing implementation) — P0. -- `sched_account_jobs()`'s existing `JOB_IS_RUN` gate as the sole mechanism deferring actual resource charging — P0, must remain unmodified for this design to be correct. -- `make_job_account()`/`make_dispatch_decision()` (existing, unmodified) — P0. - -### Revalidation Triggers -- Any future change to `sched_account_jobs()`'s charging gate (e.g. adding a new precondition beyond `JOB_IS_RUN`) must re-verify this feature's Requirement 1.2/1.3 still hold, since this design relies on that gate — not a new one of its own — for correct deferral. -- Any future change that makes simulation mode's live picker emit a DISPATCH decision (currently PEND-only, `scheduler-allocation`'s stated scope) must re-verify Requirement 3.3's dedup assumption, which currently holds structurally (no live DISPATCH ever competes with a reconciled capture) rather than via an explicit check in this feature's own code. -- Any change to `decoded_session`'s per-round decode/free lifecycle (`sim_loop.c:871`/`:1006`) must re-verify this feature's core premise (every round's own decisions are only ever visible during that round). - -## Architecture - -### Existing Architecture Analysis -`sim_reconcile_unaccounted_jobs()` runs unconditionally once per simulation round, strictly between `xce_core_begin_session()` (which zeroes `resInUse`/`sharedInUse`) and `sched_account_jobs()` (which re-sums them from every `JOB_IS_RUN` job with `job->accounts != NULL`) — see `sim_loop.c:912-957`. Its current trigger gate additionally requires `JOB_IS_RUN(job->state)` before considering a job at all: - -```c -if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) { - continue; -} -``` - -`decoded_session` (this round's decoded recorded-decision stream) is freed at the end of the same round (`sim_loop.c:1006`) and never persisted — so a job skipped at round N because it was still PEND has no later opportunity to be matched against round N's data, even after it reaches RUN at round N+k. - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TD - RoundStart[Round begins] --> Decode[decode_ReplaySession this round] - Decode --> Begin[xce_core_begin_session resets resInUse] - Begin --> Reconcile[sim_reconcile_unaccounted_jobs] - Reconcile --> Gate{job accounts is NULL} - Gate -- no --> NextJob[skip already-captured job] - Gate -- yes --> Search[search this rounds decoded decisions for jobid] - Search --> Found{matching DISPATCH found} - Found -- yes --> Capture[capture allocs and accounts, report decision] - Found -- no --> Retry[leave accounts NULL, retry next round] - Capture --> Charge[sched_account_jobs] - Retry --> Charge - NextJob --> Charge - Charge --> ChargeGate{job state is RUN and accounts set} - ChargeGate -- yes --> ResInUse[include in resInUse] - ChargeGate -- no --> Defer[exclude from resInUse] -``` - -**Architecture Integration**: -- Selected pattern: single-condition removal from an existing, already-correct-in-every-other-respect mechanism — not a new component. -- Domain/feature boundaries: capture (this feature, `Gate`/`Search`/`Capture`/`Retry` above) stays fully decoupled from charging (`sched_account_jobs`, `ChargeGate` above, untouched) — the same separation normal mode already has between `sched_commit_decision()` (capture) and `sched_account_jobs()` (charge). -- Existing patterns preserved: per-round re-invocation, jobid-scoped matching, ownership-transfer capture, shift-down compaction, three-call decision reporting — all untouched. -- New components rationale: none — this is a boundary-widening change to one existing condition. -- Steering compliance: directly closes the gap `.kiro/steering/scheduler-driver-equivalence.md` documents as bug-1611's residual; testing discipline (contention fixture) from that same steering doc drives the Testing Strategy below. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| Scheduler core (C, gnu17) | Existing `sim_loop.c`/`scheduler.c` | Trigger-condition change only | No new dependency; no schema/wire-format change | - -## File Structure Plan - -### Modified Files -- `src/scheduler/src/sim_loop.c` — remove `!JOB_IS_RUN(job->state)` from `sim_reconcile_unaccounted_jobs()`'s trigger condition (`sim_loop.c:592`); update the function's header doc comment (`sim_loop.c:~505-565`) to describe the new state-independent trigger and reframe the "double-application guard" description to also cover "this is now the sole capture-eligibility gate." -- `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` — replace `test_reconcile_skips_non_run_state_job` (asserts the bug: a PEND job with a matching recorded entry must stay uncaptured) with an inverted-expectation test proving a PEND job **is** now captured; add new tests for cross-round separation, `job->state` transitioning after capture, the no-recorded-origin-ever case, and the "captured-but-not-RUN contributes zero to resInUse" case (Requirement 1.2) driven through `sched_account_jobs()`. -- `src/scheduler/testsuite/test_sim_contention_parity.c` and/or a new contention-focused test file — add a resource-contention fixture (capacity insufficient for all candidate jobs) exercising cross-round dispatch-to-confirmation separation, per `scheduler-driver-equivalence.md`'s explicit testing discipline that a no-contention fixture cannot prove accounting correctness. -- A `--sim-verify` regression test (existing `test_sim_*` verify-mode file, or a new one if none fits) spanning a multi-round-separated dispatch — proves Requirement 4.2. - -### No New Files -This feature introduces no new production source file — see Simplification decision in `research.md`. - -## System Flows - -```mermaid -sequenceDiagram - participant Replay as Replay stream - participant Recon as sim_reconcile_unaccounted_jobs - participant Job as job in core jobs - participant Acct as sched_account_jobs - - Replay->>Recon: round N decoded decisions include DISPATCH for job X - Note over Job: job X state is PEND at round N - Recon->>Job: capture allocs and accounts from DISPATCH - Recon->>Recon: report decision at round N - Acct->>Job: check state is RUN - Note over Acct: state is PEND, skip charging - Replay->>Job: round N+k replays job X state change to RUN - Recon->>Job: accounts already set, skip re-search - Acct->>Job: check state is RUN and accounts set - Acct->>Acct: charge resInUse from round N+k onward -``` - -**Key decisions not obvious from the diagram**: the decision is reported once, at round N (the historically accurate round, Requirement 2.1) — not at round N+k when charging actually begins. Charging timing (round N+k) and reporting timing (round N) are intentionally decoupled, mirroring how normal mode's own `job->state` display and `resInUse` accounting are already decoupled from `sched_commit_decision()`'s synchronous capture. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | Capture at originating round regardless of state | `sim_reconcile_unaccounted_jobs` | trigger gate | Search/Capture | -| 1.2 | Captured-but-not-RUN excludes from resInUse | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/Defer | -| 1.3 | RUN transition includes in resInUse from that point | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/ResInUse | -| 1.4 | Matches normal-mode's equivalent-point reporting | `sched_account_jobs` (shared with normal mode) | n/a | n/a | -| 2.1 | Decision visible at historically accurate round | `sim_reconcile_unaccounted_jobs` (decision reporting, unchanged) | `make_dispatch_decision`/`sim_print_decision`/`xce_repl_check_decision` | Capture step | -| 2.2 | Original decision type/detail preserved | `make_dispatch_decision` (unchanged) | n/a | Capture step | -| 2.3 | No duplicate report when live decision also exists | live picker (unchanged, PEND-only — structural invariant) | n/a | n/a | -| 3.1 | Never capture from a non-originating decision | jobid match (unchanged) | `entry->body.dispatch->alloc->jobid` check | Search | -| 3.2 | Never re-capture once captured | `job->accounts != NULL` guard (unchanged, now the sole gate) | trigger gate | Gate | -| 3.3 | Already-captured job not overwritten by a later live decision | structural (live picker never emits DISPATCH) | n/a | n/a | -| 4.1 | Same-round case still works | trigger gate (RUN is a subset of "any state") | trigger gate | Gate/Search | -| 4.2 | No new `--sim-verify` mismatch | compaction ordering (unchanged), verified by new test | decoded_session compaction | Capture step | -| 4.3 | Interactive/batch parity preserved | shared `sched_sim_driver_run()` call site (unchanged) | n/a | n/a | -| 4.4 | No recorded origin anywhere: report unavailable, keep going | "not found" branch (unchanged) | TRACE log, `job->accounts` stays NULL | Retry | -| 4.5 | Normal mode unaffected | `driver.c` untouched | n/a | n/a | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|--------------|--------|---------------|---------------------------|-----------| -| `sim_reconcile_unaccounted_jobs` | Scheduler sim driver | Widen capture eligibility to any not-yet-captured job, regardless of state | 1.1, 2.1, 3.1, 3.2, 4.1, 4.4 | `sched_account_jobs` (P0, relied on unmodified) | Service | - -### Scheduler sim driver - -#### `sim_reconcile_unaccounted_jobs` - -| Field | Detail | -|-------|--------| -| Intent | Capture a job's recorded DISPATCH allocation as soon as the replay reaches the round it occurred in, independent of the job's current state | -| Requirements | 1.1, 2.1, 3.1, 3.2, 4.1, 4.4 | - -**Responsibilities & Constraints** -- Owns only the capture-eligibility decision (which jobs are searched each round) and the search/capture/report mechanics already delivered by `scheduler-sim-decision-parity` (unchanged). -- Has no knowledge of resource charging — that remains `sched_account_jobs()`'s exclusive responsibility. -- Never fabricates data: a job with no matching recorded decision in a given round is left uncaptured and re-checked the following round (Requirement 4.4). - -**Dependencies** -- Inbound: `sched_sim_driver_run()`'s round loop — calls this once per round, unchanged call site and unchanged position relative to `xce_core_begin_session()`/`sched_account_jobs()`/`xce_core_signal_go()` (P0). -- Outbound: `sched_account_jobs()` — this feature supplies `job->accounts` earlier than before; `sched_account_jobs()`'s own unmodified `JOB_IS_RUN` gate determines when that data starts being charged (P0). - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -/* Unchanged signature. Behavior change: a job is now eligible for capture - * whenever job->accounts == NULL, regardless of job->state (previously - * additionally required JOB_IS_RUN(job->state)). Capturing a job's - * allocation while it is not yet RUN is inert until sched_account_jobs()'s - * own JOB_IS_RUN gate later includes it -- this function performs no - * charging itself. */ -static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, - replay_session_t* decoded_session, - sched_record_session_ctx_t* record_ctx); -``` -- Preconditions: unchanged — `sim`/`decoded_session` NULL is a safe no-op; `record_ctx` may legitimately be NULL. -- Postconditions: unchanged — a matched job has `job->allocs`/`job->accounts` set and its entry removed from `decoded_session->decisions`; an unmatched job is left exactly as it was. -- Invariants (new): a job is captured **at most once**, at the first round (in replay order) whose decoded stream contains its originating DISPATCH decision — independent of that round's job state. - -**Implementation Notes** -- Integration: delete `!JOB_IS_RUN(job->state)` from the trigger condition at `sim_loop.c:592`; no other line in the match/capture/report path changes. -- Validation: see Testing Strategy — contention fixture, cross-round separation, `--sim-verify`, interactive/batch parity, and the inverted `test_reconcile_skips_non_run_state_job` replacement are all required, not optional. -- Risks: repeated per-round "not found" TRACE logging for a job with no recorded origin anywhere in the replay (Requirement 4.4 case) — accepted, TRACE-level, see `research.md`'s Risks section. -- **Verified safe (early capture vs. cleanup)**: capturing `job->accounts` while a job is still PEND is safe against every existing job-cleanup path. `sched_uncharge_account()` has zero production callers anywhere in the scheduler — production resource accounting is a reset-and-resum model (`xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round; `sched_account_jobs()` re-sums it from scratch from currently-`JOB_IS_RUN` jobs only). Both places `job->accounts` is freed on a non-RUN transition (`scheduler.c:974-977`, `core_events.c:915-918`) release memory only, with no `resInUse` mutation. A job captured while PEND that later transitions directly to a terminal state without ever passing through RUN is therefore never summed by `sched_account_jobs()` at any round and its accounts are freed with zero accounting impact — early capture cannot under-count `resInUse`. See `research.md`'s corresponding Research Log entry for the full trace. - -## Error Handling - -### Error Strategy -- **No matching recorded decision this round**: not an error — `job->accounts`/`job->allocs` stay `NULL`, a TRACE line is logged, the job is re-checked next round (unchanged behavior, now reached by more jobs per round since the gate widened). -- **No recorded origin anywhere in the replay** (Requirement 4.4): the same "not found" path applies every round for the rest of the replay; the scheduler continues processing without terminating, and that job's resource usage is reported as unavailable (never fabricated) wherever it is inspected (e.g. `info worker`). -- **`decoded_session` decode failure**: unchanged — already handled upstream of this function (`sim_loop.c:874-879`) by skipping verification for that round; this feature does not alter that path. - -## Testing Strategy - -### Unit Tests (`test_sim_reconcile_unaccounted.c`) -- Replace `test_reconcile_skips_non_run_state_job` with `test_reconcile_captures_non_run_state_job_when_dispatch_recorded` — a `JOB_STAT_PEND` job with a matching recorded DISPATCH entry now gets `job->allocs`/`job->accounts` captured and the entry consumed (Requirement 1.1, 3.1). -- New: `test_reconcile_still_skips_already_captured_job_regardless_of_state` — a job with `job->accounts` already set (any state) is never re-searched, proving the state-widened gate did not weaken the existing no-double-capture guard (Requirement 3.2). -- New: `test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries` — a PEND job with no matching entry this round stays uncaptured and is eligible again next round (Requirement 4.4, state-independent retry). -- New: `test_reconcile_multiple_jobs_captured_independently_across_different_rounds` — several PEND jobs simultaneously outstanding, each with its own recorded DISPATCH decision spread across different rounds (some matching this round, some not yet recorded, some already captured from an earlier round); assert each job's `job->accounts` reflects only its own jobid's allocation data, captured at its own correct round, with zero cross-job interference from the wider per-round scan (Requirement 1.1, 3.1 under concurrent load — the single-job tests above cannot distinguish "correct" from "correct by coincidence, only one candidate available"). -- Existing `test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch`, `test_reconcile_ignores_non_dispatch_entry_with_matching_jobid`, `test_reconcile_compacts_via_shift_down_preserving_order`, `test_reconcile_is_safe_to_recheck_on_a_later_round`, `test_reconcile_then_release_via_job_state_change_is_asan_clean`, `test_reconcile_null_args_are_a_safe_noop` — re-run unmodified as regression coverage (Requirement 4.1); no expected behavior change for an already-RUN job. - -### Integration Tests (round-boundary + charging interaction) -- New: `test_captured_pend_job_excluded_from_resinuse_until_run` — drive `sim_reconcile_unaccounted_jobs()` then `sched_account_jobs()` back to back for a PEND job with a freshly-captured account; assert `w->resInUse`/`core->sharedInUse` stay zero (Requirement 1.2), then transition the job to RUN and re-run `sched_account_jobs()`; assert `resInUse` now reflects it (Requirement 1.3). -- New (resource-contention fixture, per `scheduler-driver-equivalence.md`'s explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (`sched_check_host_satisfy_`) sees correct `resInUse` at each round, not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (`scheduler-driver-equivalence.md`). -- New: `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` — a `--sim-verify` run over a replay where dispatch and RUN confirmation are several rounds apart reports zero mismatches (Requirement 4.2). -- Existing `test_repl_equivalence.c`-style interactive/batch parity coverage re-run unmodified — this feature's call site is shared between both modes, so no new test is needed to prove Requirement 4.3, only confirmation the existing suite still passes. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md b/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md deleted file mode 100644 index 5a46bb4b..00000000 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md +++ /dev/null @@ -1,52 +0,0 @@ -# Requirements Document - -## Project Description (Input) -XCE scheduler operators debugging via simulation-mode replay (`xce_scheduler -i -s <replay>` / batch `--sim`) are the ones affected. Currently, Recorded-Decision Reconciliation (`sim_reconcile_unaccounted_jobs()`, `sim_loop.c`, delivered by the `scheduler-sim-decision-parity` spec, implementation-complete) only succeeds when a job's recorded DISPATCH decision and its `JOB_STATE_CHANGE`-to-RUN confirmation land in the exact same replay round -- it gates on `job->state == RUN` first, then searches that same round's decoded decisions for a match. In any real deployment with genuine dispatch-to-worker-confirmation latency, these two events land in different rounds, so reconciliation never fires for that job -- meaning `scheduler-sim-decision-parity` does not actually fix bug-1611/bug-1613's dominant real-world manifestation (a job dispatched at round N, confirmed running some rounds later, still shows `resInUse=0` / no visible decision when inspected much later in the replay). - -Investigation found the live/normal-mode path already gets this right: `scheduler.c`'s `sched_commit_decision()` sets `job->allocs`/`job->accounts` synchronously at DISPATCH-decision time, completely independent of `job->state` -- and `sched_account_jobs()`'s own charging logic separately gates on `JOB_IS_RUN(job->state)` before checking `job->accounts != NULL`, so a job with accounts already set but still PEND-state is simply not charged yet. This is the correct, already-proven pattern. - -What should change: simulation mode's recorded-decision capture should be decoupled from the RUN-state gate the same way, so it works regardless of how many rounds separate a job's DISPATCH decision from its RUN confirmation. Whenever a round's recorded decision stream contains a DISPATCH decision for a job that does not yet have `job->accounts` set, that decision's allocation data should be captured immediately -- regardless of the job's current state -- rather than waiting for `job->state` to become RUN before searching for a match. `sched_account_jobs()`'s existing RUN-state gate continues to defer actual resource charging until the job's state genuinely transitions to RUN (whenever that recorded event is later replayed), so behavior stays correct even though the underlying allocation data was captured earlier. This must not change `job->state` transition semantics themselves (used elsewhere for job status display, permissions, and preemption logic) -- only how/when the DISPATCH decision's allocation data is captured internally for accounting purposes. - -This is a follow-up to the `scheduler-sim-decision-parity` spec (implementation-complete, all 10 tasks landed and reviewed) -- do not reopen or modify that spec's own requirements/design/tasks. This is its own new spec, building on that feature's existing `sim_reconcile_unaccounted_jobs()` mechanism and round-boundary wiring. - -## Boundary Context -- **In scope**: capturing a job's recorded DISPATCH allocation data as soon as simulation-mode replay encounters it in a round's recorded decision stream, regardless of the job's current state at that point; preserving correct, state-gated resource-in-use accounting once capture no longer depends on same-round state coincidence; making the dispatch decision observable at the historically accurate round; preventing double-capture or fabricated data across this wider round span. -- **Out of scope**: any change to how or when a job's state itself transitions (PEND/RUN/DONE/etc.) -- state transition timing and triggers are unchanged, only when allocation data is captured for accounting purposes changes; any change to `--sim-verify`'s comparison semantics beyond not producing a new false mismatch as a side effect; any change to simulation mode's live picker (it remains PEND-only for freshly-arriving jobs, unchanged, out of this feature's scope per `scheduler-allocation`); any change to PEND or PREEMPT decision handling; reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks; any change to normal-mode (non-simulation) behavior. -- **Adjacent expectations**: this feature extends `scheduler-sim-decision-parity`'s existing Recorded-Decision Reconciliation mechanism and round-boundary wiring; it does not replace or duplicate that mechanism. It relies on that feature's decision-reporting channel and resource-accounting service remaining exactly as delivered. - -## Requirements - -### Requirement 1: Cross-round resource-accounting accuracy for dispatched jobs -**Objective:** As an operator inspecting simulation-mode resource state, I want a job's resource consumption reflected correctly regardless of how many rounds separate its dispatch from its running confirmation, so that what I inspect during a replay is trustworthy even under realistic dispatch-to-confirmation latency. - -#### Acceptance Criteria -1. When simulation-mode replay encounters a round containing a job's originating dispatch decision, the scheduler shall capture that decision's resource-allocation data for the job, regardless of the job's current state at that point. -2. While a job's resource-allocation data has been captured but the job has not yet transitioned to a running state, the scheduler shall not include that job's resource consumption in any worker or shared resource-in-use total. -3. When a job whose resource-allocation data was already captured subsequently transitions to a running state, the scheduler shall include that job's resource consumption in its worker's and any shared resource's resource-in-use totals from that point forward, regardless of how many rounds elapsed between capture and the transition. -4. The scheduler shall report the same resource-in-use totals for a dispatched-then-confirmed-later job as normal-mode execution reported at the equivalent point in the original run, regardless of the round separation between dispatch and confirmation. - -### Requirement 2: Decision visibility at the historically accurate round -**Objective:** As an operator watching simulation-mode decision output, I want a job's dispatch decision to become visible when the replay reaches the round it was actually decided in, so I have an accurate picture of when scheduling activity happened, not only that it eventually happened. - -#### Acceptance Criteria -1. When simulation-mode replay reaches the round containing a job's originating dispatch decision, the scheduler shall make that decision observable through the existing decision-reporting channel at that round, independent of when the job's state later transitions to running. -2. The scheduler shall preserve the original decision's type and resource-allocation detail when reporting it. -3. When simulation-mode independently computes a fresh decision for a job in a given round, the scheduler shall report only that one decision for the job, not also a duplicate from the recorded stream. - -### Requirement 3: No fabrication and no double-capture across the wider round span -**Objective:** As a maintainer relying on this feature's correctness, I want captured allocation data to never be fabricated or captured more than once, so that widening reconciliation's effective round span does not introduce a new class of correctness risk. - -#### Acceptance Criteria -1. The scheduler shall never capture resource-allocation data for a job from anything other than that job's own originating recorded decision. -2. Once a job's resource-allocation data has been captured, the scheduler shall not attempt to capture it again from a later round's recorded decisions, regardless of how many further rounds are replayed. -3. If a job's resource-allocation data was already captured, and simulation-mode's own live decision-making subsequently produces a fresh decision for that job, the scheduler shall treat the job as already resolved and not overwrite or duplicate its already-captured data. - -### Requirement 4: No regression to existing guarantees -**Objective:** As a maintainer relying on `scheduler-sim-decision-parity`'s already-delivered guarantees, I want this feature to preserve them, so existing verification workflows, tests, and behavior continue to hold. - -#### Acceptance Criteria -1. The scheduler shall continue to correctly reconcile the same-round dispatch-and-confirmation case that `scheduler-sim-decision-parity` already handles. -2. The scheduler shall continue to support decision verification mode (comparing freshly-computed decisions against recorded ones) without reporting a mismatch caused by this feature's own behavior. -3. The scheduler shall continue to produce identical decisions between interactive (step-by-step) and batch simulation execution modes. -4. While a job has no recorded originating decision available anywhere in the replay's own data (for example, a job already running before the replay's recording window began), the scheduler shall report that job's resource usage as unavailable rather than fabricating a value, and shall continue processing the remainder of the replay without terminating. -5. While simulation mode is not enabled, the scheduler shall not change its existing resource-accounting, decision-reporting, or job-state-transition behavior. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/research.md b/.kiro/specs/scheduler-sim-eager-reconciliation/research.md deleted file mode 100644 index 6b01e7d2..00000000 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/research.md +++ /dev/null @@ -1,90 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `scheduler-sim-eager-reconciliation` -- **Discovery Scope**: Extension (widens an existing, already-shipped mechanism's trigger condition; no new subsystem, no external dependency) -- **Key Findings**: - - The fix is a single-condition removal in `sim_reconcile_unaccounted_jobs()`'s trigger gate (`sim_loop.c:592`), not a new cross-round data structure. The per-round `decoded_session` decode/reconcile/free cycle already runs every round regardless of any job's state; the only reason cross-round matching fails today is that the gate additionally requires `JOB_IS_RUN(job->state)` before a job is even considered for a search, so a job still PEND at the round its own DISPATCH decision appears is skipped outright and never re-checked once it later becomes RUN (by then that round's `decoded_session` is long freed). - - `sched_account_jobs()` (the function that actually turns `job->accounts` into `w->resInUse`/`core->sharedInUse`) already independently gates on `JOB_IS_RUN(job->state)` before charging. This means capturing `job->accounts` early (while a job is still PEND) is inert until the job's real state later transitions to RUN — the existing charging gate is exactly the state-appropriate deferral this feature needs, and it requires no change. - - One existing test, `test_reconcile_skips_non_run_state_job` (`test_sim_reconcile_unaccounted.c:507-531`), asserts the CURRENT (buggy) behavior by name and message ("a non-RUN-state job must never be reconciled here") — this test encodes the bug and must be replaced (inverted), not left alongside new coverage. - -## Research Log - -### Where does resInUse actually go to zero, and what re-populates it? -- **Context**: bug-1613's buglog root-cause trail (confirmed) plus `.kiro/steering/scheduler-driver-equivalence.md`. -- **Sources Consulted**: `src/scheduler/src/scheduler.c` (`sched_account_jobs`, `sched_commit_decision`), `src/scheduler/src/resource.c` (`sched_update_res_accounts`, `sched_charge_account`), `.wolf/buglog.json` bug-1611/bug-1613 full history. -- **Findings**: `xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round; `sched_account_jobs()` is the only re-summing mechanism, iterating `core->jobs` and charging every job that is both `JOB_IS_RUN` and has `job->accounts != NULL`. Normal mode's `sched_commit_decision()` sets `job->accounts` synchronously at DISPATCH-decision time, independent of `job->state` — so by the time a job's state reaches RUN, its accounts are already long-set. Simulation mode has no equivalent synchronous path (its own live picker never emits a DISPATCH — confirmed in the bug-1613 trail — it is PEND-only), so `job->accounts` for a replayed job can only come from `sim_reconcile_unaccounted_jobs()` reconstructing it from the recorded decision stream. -- **Implications**: the fix belongs entirely in how/when `sim_reconcile_unaccounted_jobs()` captures `job->accounts` — `sched_account_jobs()` itself is correct today and must not change. - -### Why does today's reconciliation only work same-round? -- **Context**: bug-1611's confirmed root-cause note and the `scheduler-sim-decision-parity` spec's task 4.3 implementation notes (both quoted in `.wolf/buglog.json`). -- **Sources Consulted**: `src/scheduler/src/sim_loop.c:566-709` (`sim_reconcile_unaccounted_jobs`, read in full), `src/scheduler/src/sim_loop.c:840-960` (round-boundary call sequence, read in full). -- **Findings**: `decoded_session` is decoded fresh from the replay stream once per round (`decode_ReplaySession_ptr`, `sim_loop.c:871`) and freed at the end of the same round (`free_ReplaySession_ptr`, `sim_loop.c:1006`) — it never survives past its own round. `sim_reconcile_unaccounted_jobs()` is called unconditionally every round (`sim_loop.c:948`, between `xce_core_begin_session()` and `sched_account_jobs()`), but its trigger gate (`sim_loop.c:592`) is: - ```c - if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) { - continue; - } - ``` - A job whose DISPATCH decision is recorded at round N but whose RUN-state confirmation is only replayed at round N+k (realistic dispatch-to-confirmation latency) is skipped at round N (state is still PEND) and is never re-considered once it reaches RUN at round N+k, because round N's `decoded_session` — the only place that DISPATCH decision ever existed — is already gone. -- **Implications**: no new persistent, cross-round data structure is needed. The function already re-runs every round with a live view of the round's own recorded decisions; it only needs to stop requiring `JOB_IS_RUN` before searching. Removing that one condition means every not-yet-captured job is (re-)checked against every round's own decoded stream until either found (captured, permanently, via the existing `job->accounts != NULL` guard) or the replay ends. - -### Does removing the RUN-state gate risk fabricating or double-capturing? -- **Context**: Requirement 3's explicit no-fabrication / no-double-capture acceptance criteria. -- **Sources Consulted**: `sim_loop.c:592-693` (full match/capture logic, read in full). -- **Findings**: The match loop already restricts a capture to the job's own jobid (`entry->body.dispatch->alloc->jobid != job->jobid`, `sim_loop.c:605`) — this is state-independent and unaffected by the gate change. The `job->accounts != NULL` continue-guard (unchanged) already prevents a second capture on any later round, for any job, regardless of state. Widening the state condition therefore only widens *which jobs are eligible to be searched for*, never *what counts as a match* or *whether a second capture can happen*. -- **Implications**: Requirement 3 (no fabrication, no double-capture) is satisfied by logic that already exists and needs no modification — only new test coverage proving it still holds once the gate widens. - -### Interaction with `--sim-verify` -- **Context**: Requirement 4.2 (`--sim-verify` must not report a new mismatch caused by this feature). -- **Sources Consulted**: `sim_loop.c:925-1004` (reconciliation call site through the verify positional-walk loop, read in full). -- **Findings**: Reconciliation runs and compacts `decoded_session->decisions` *before* the `--sim-verify` positional walk (`sim_loop.c:970-1004`) reads that same array. Today, a job whose confirmation lands in a later round leaves its DISPATCH entry sitting **unconsumed** in the round-N `decoded_session` (reconciliation skipped it), which `--sim-verify`'s walk would then see as a recorded entry with no matching live decision — a latent false-mismatch risk in `--sim-verify` mode that already exists independent of this fix. Widening the gate causes reconciliation to consume that entry at the correct round instead, which removes this latent risk rather than introducing a new one. -- **Implications**: this fix is verify-neutral-to-positive, not verify-risky. Still needs an explicit test per Requirement 4.2's own wording (a `--sim-verify` run across a multi-round-separated dispatch must not report a new mismatch). - -### Does early capture risk an incorrect uncharge when a captured-but-not-yet-RUN job later leaves the job set entirely? -- **Context**: capturing `job->accounts` while a job is still PEND (this feature's whole point) means, for the first time, a job can carry non-NULL `accounts` without ever having been charged. If any cleanup path unconditionally treats "accounts non-NULL" as "this job was charged, so subtract it," early capture would cause `resInUse` to under-count (a phantom decrement for a charge that was never added) for any captured-but-never-actually-run job (e.g. cancelled while still PEND). -- **Sources Consulted**: every production (non-test) reference to `job->accounts`/`sched_uncharge_account` across `src/scheduler/src/*.c`, via grep: `resource.c:591-602` (`sched_uncharge_account`/`sched_commit_job_alloc` definitions), `scheduler.c:964-980` (`sched_job_state_change`), `core_events.c:898-921` (`XCE_CORE_EVENT_JOB_STATE_CHANGE` handler) — the two places a job's `job->accounts` gets freed outside of normal completion. -- **Findings**: `sched_uncharge_account()` has **zero production callers** anywhere in the scheduler (confirmed by grep across every `.c` file in `src/scheduler/src/` — its only callers are test files exercising it in isolation). Both places `job->accounts` gets freed on a non-RUN state transition (`scheduler.c:974-977`, `core_events.c:915-918`) do exactly one thing: `free_job_account(job->accounts); job->accounts = NULL;` — memory release only, no `resInUse` mutation, no call to `sched_uncharge_account`. Production accounting is entirely a **reset-and-resum** model, not incremental charge/uncharge: `xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round, and `sched_account_jobs()` re-sums it from scratch by summing every *currently* `JOB_IS_RUN` job with non-NULL `accounts`. A job never counts toward `resInUse` until a round where it is both RUN and has accounts — regardless of how early those accounts were captured. -- **Implications**: a job captured while still PEND that later transitions directly to a terminal state (cancelled, failed) **without ever passing through RUN** is simply never summed by `sched_account_jobs()` at any round (it never satisfies `JOB_IS_RUN`), and its `job->accounts` is freed harmlessly with zero `resInUse` impact when its terminal state is applied. Early capture cannot cause an under-count. This is not a risk this design needs to mitigate — it is a consequence of an accounting model (reset+resum, no incremental uncharge) that was already correct before this feature and remains correct with a wider capture-eligibility gate. - -### Does bug-1624's normal-mode fix (job->state = RUN set synchronously at dispatch-commit) already close this gap for current recordings? -- **Context**: `bug-1624` (fixed 2026-08-23, `scheduler-dispatch-commit` tasks 2.1/3.1) made `sched_commit_decision()` set `job->state = JOB_STAT_RUN` synchronously in the same operation as `job->accounts`/`job->allocs`, specifically to close a *live* normal-mode over-commit window. Since this predates the current investigation, it's worth confirming it doesn't also incidentally collapse the round-separation this spec fixes for any recording made after 2026-08-23. -- **Sources Consulted**: `grep -n "job->state\|JOB_STAT_RUN" src/scheduler/src/sim_loop.c` (only hit: the trigger gate itself, `sim_loop.c:592`); `src/scheduler/src/scheduler.c:1441-1447` (`sched_commit_decision`, read in full earlier); `.wolf/buglog.json` bug-1624's full fix note. -- **Findings**: `sim_reconcile_unaccounted_jobs()` never sets `job->state` anywhere — replay-mode state transitions come exclusively from the separately-recorded `XCE_CORE_EVENT_JOB_STATE_CHANGE` event stream, applied via `core_events.c`, independent of whatever the decision stream contains. `sched_commit_decision()`'s `job->state = JOB_STAT_RUN` (bug-1624's fix) is a direct in-memory field write in the *live* scheduler process — not a call into the core's event-push pipeline (`xce_core_push_events()`) — so it is not, by itself, a recordable core event. The recorded `JOB_STATE_CHANGE` event a replay later applies still originates from whatever asynchronous signal produces it for recording purposes (the controller's own worker-confirmation path), independent of bug-1624's live in-memory synchronization. -- **Implications**: bug-1624 fixed a live-mode admission-safety window; it does not change what ends up in a recording's event stream or its round timing. This spec's premise — a recorded DISPATCH decision and its recorded RUN-state-change event can land in different replay rounds — holds for recordings made before *and* after bug-1624, and is not something a future reader should mistake as already resolved by that unrelated fix. - -### Existing test locking in the bug -- **Context**: identifying what existing coverage this feature must invalidate, not just extend. -- **Sources Consulted**: `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` (grepped test names, read `test_reconcile_skips_non_run_state_job` in full at lines 507-531). -- **Findings**: `test_reconcile_skips_non_run_state_job` adds a `JOB_STAT_PEND` job with a matching recorded DISPATCH entry and asserts `job->accounts` stays `NULL`, with the message "a non-RUN-state job must never be reconciled here, regardless of what recorded data exists for it this round." This is the literal bug this spec fixes, encoded as a passing test. -- **Implications**: this test must be replaced (inverted expectation, renamed), not left in place alongside new coverage — the design's File Structure Plan and the eventual task list must call this out explicitly, or the fix will be blocked by its own existing test suite. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Widen existing gate (chosen) | Remove `!JOB_IS_RUN(job->state)` from `sim_reconcile_unaccounted_jobs()`'s trigger condition; rely on the function's existing every-round re-invocation and the existing `job->accounts != NULL` guard for persistence/idempotency | Minimal surface area (one condition), reuses every existing invariant (dedup, ownership transfer, decision reporting) untouched, symmetric with normal mode's own state-independent capture | Every not-yet-captured job is now re-searched against every round's decoded stream until found — O(jobs × rounds) worst case for a job with no recorded origin at all (Req 4.4 case) | Chosen: smallest design satisfying all requirements (design-synthesis Simplification lens) | -| New persistent cross-round pending-capture table | Maintain a separate `jobid -> pending decision` map populated across rounds, independent of `sim->core.jobs` iteration | Could in principle capture before a job even exists in `core->jobs` | Solves a problem that does not exist here — `decoded_session` already only ever contains decisions for jobs `sim->core.jobs` already knows about at that round; adds a new owned lifetime/lookup structure and a new invalidation surface for no behavioral gain | Rejected — violates the Simplification lens; not pursued further | - -## Design Decisions - -### Decision: Remove the `JOB_IS_RUN` pre-filter instead of adding a new capture-tracking structure -- **Context**: bug-1611/1613's residual gap is that reconciliation only matches within the same round a job happens to already be RUN. -- **Alternatives Considered**: - 1. New persistent `jobid -> captured allocation` structure spanning rounds, decoupled entirely from `sim->core.jobs` iteration. - 2. Widen the existing trigger gate on the existing per-round iteration (chosen). -- **Selected Approach**: Delete `!JOB_IS_RUN(job->state)` from the trigger condition at `sim_loop.c:592`; leave every other line of the function (match search, ownership transfer, decision reporting, compaction) unchanged. -- **Rationale**: The function already iterates `sim->core.jobs` every round and already searches that round's own decoded stream; the only defect is an overly-narrow precondition on which jobs are eligible. `sched_account_jobs()`'s own unchanged RUN-state gate already provides the correct "don't charge until actually running" deferral this feature's Requirement 1.2 needs — building a second deferral mechanism would duplicate an already-correct one. -- **Trade-offs**: A job that never has a recorded origin anywhere in the replay (Requirement 4.4) is now re-searched, and its "not found" TRACE-level log line re-emitted, every round for the rest of the replay instead of never being searched again after one failed same-round attempt. Accepted: TRACE-level (filtered by default log level in normal operation) and structurally required by Requirement 4.4's "shall continue processing... without terminating" wording, which implies retrying, not giving up. -- **Follow-up**: Verify during implementation that no other caller or test assumes today's "same-round-only" contract as load-bearing behavior beyond the one test identified above. - -## Risks & Mitigations -- **Existing test asserts the bug as correct behavior** (`test_reconcile_skips_non_run_state_job`) — Mitigation: replace it with an inverted-expectation test in the same task that changes the gate, not as a follow-up; do not let old and new expectations coexist even briefly. -- **Resource-contention blind spot** (`scheduler-driver-equivalence.md`'s own documented testing discipline: a no-contention fixture cannot distinguish "accounting correct" from "accounting wrong but unread") — Mitigation: Testing Strategy must include at least one fixture where a worker's capacity is insufficient for all candidate jobs, so a wrongly-zero `resInUse` would visibly change a scheduling decision, not just a REPL display value. -- **`--sim-verify` false-mismatch risk already latent today** — Mitigation: add an explicit `--sim-verify` regression test spanning a multi-round dispatch-to-confirmation gap; confirm it reports zero mismatches after the fix (and, ideally, confirm it currently free of new mismatches — the fix is expected to strictly improve this, not just avoid regressing it). -- **Log noise from repeated "not found" TRACE lines** across many rounds for a job with no recorded origin — Mitigation: accepted as documented trade-off (see Design Decision above); TRACE level already filtered in default operation, not escalated further by this spec. -- **Wider per-round search cost**: every not-yet-captured job (previously only RUN-state ones) is now linearly searched against that round's decoded decision list every round until captured or the replay ends — a round with a large PEND backlog does more search work than before. Mitigation: the per-job, per-round search cost itself is unchanged (same linear scan the existing mechanism already performs); only the set of jobs subjected to it widens. Replay is a debugging/tuning tool, not a real-time or production hot path, and `--sim-speed` already exists to control replay pacing — accepted without further mitigation; revisit only if a real replay is observed to become impractically slow. - -## References -- `.kiro/steering/scheduler-driver-equivalence.md` — the enforced state-equivalence invariant and its testing discipline this feature closes the remaining gap in. -- `.kiro/specs/scheduler-sim-decision-parity/` — the parent spec whose `sim_reconcile_unaccounted_jobs()` mechanism this feature extends (implementation-complete, not reopened). -- `.wolf/buglog.json` bug-1611, bug-1613 — full confirmed root-cause and partial-fix history. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json b/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json deleted file mode 100644 index 4bb1de00..00000000 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "scheduler-sim-eager-reconciliation", - "created_at": "2026-08-22T14:50:13Z", - "updated_at": "2026-08-22T14:58:00Z", - "language": "en", - "phase": "tasks-generated", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md b/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md deleted file mode 100644 index 58414a45..00000000 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md +++ /dev/null @@ -1,58 +0,0 @@ -# Implementation Plan - -- [ ] 1. Core: widen reconciliation's capture eligibility to any not-yet-captured job -- [x] 1.1 Prove and implement state-independent capture of a job's recorded dispatch allocation - - Replace the existing test that locks in the bug (asserting a not-yet-running job must never be reconciled) with one asserting the opposite: a job whose recorded dispatch decision is found this round gets its allocation captured regardless of whether it has reached a running state yet - - Run the new/updated test against the current code first to confirm it fails for the expected reason (the old state-gated behavior), then remove the state precondition from the capture-eligibility check so only "not yet captured" remains - - Update the function's own explanatory comments so they describe the new state-independent trigger instead of the old state-gated one - - Observable: the updated unit test suite shows the not-yet-running job's allocation is captured and its recorded entry consumed, where it previously stayed uncaptured - - _Requirements: 1.1, 3.1_ - - _Boundary: sim_reconcile_unaccounted_jobs_ - -- [x] 1.2 Extend unit coverage for capture-safety guarantees under the widened eligibility - - Add a test proving a job whose allocation was already captured is never re-searched or re-captured on a later round, regardless of its state - - Add a test proving a job with no matching recorded decision this round stays uncaptured and remains eligible to be found on a later round - - Add a test with several simultaneously outstanding jobs whose recorded dispatch decisions land in different rounds, proving each job's captured allocation reflects only its own recorded decision with no cross-job interference - - Observable: the unit test suite demonstrates no-double-capture, retry-on-no-match, and per-job isolation all hold under the widened eligibility check - - _Requirements: 3.1, 3.2, 4.1, 4.4_ - - _Boundary: sim_reconcile_unaccounted_jobs_ - -- [ ] 2. Integration: prove captured-but-not-running allocations stay excluded from resource-in-use accounting until the job actually runs -- [x] 2.1 Verify resource-in-use accounting defers to the job's real running state, not its capture time - - Drive a captured, not-yet-running job's allocation through the resource-in-use recomputation and confirm it contributes nothing to worker or shared resource usage - - Transition that same job to a running state and drive resource-in-use recomputation again, confirming its allocation now contributes from that point forward - - Observable: a test demonstrates the same captured job's resource usage is zero before its running-state transition and correctly non-zero after, using the exact allocation captured earlier - - _Requirements: 1.2, 1.3, 1.4_ - - _Boundary: sim_reconcile_unaccounted_jobs, sched_account_jobs_ - - _Depends: 1.1_ - -- [ ] 3. Validation: end-to-end proof under contention, decision-verification mode, and full regression -- [x] 3.1 (P) Verify correct capacity accounting under resource contention with realistic dispatch-to-confirmation separation - - Build a replay fixture where a worker's capacity is insufficient for every candidate job, with dispatch decisions and their running-state confirmations recorded several rounds apart - - Confirm the live capacity check sees the correct, non-zero resource-in-use total at each round rather than treating capacity as permanently free - - Observable: a resource-contention test demonstrates the capacity check's view of resource usage matches what actually happened in the original recording at each round, not a permanently-zero reading - - _Requirements: 1.1, 1.2, 1.3, 1.4_ - - _Boundary: sim_reconcile_unaccounted_jobs, sched_account_jobs_ - - _Depends: 1.1_ - -- [x] 3.2 (P) Verify decision-verification mode reports no new mismatch across a round-separated dispatch - - Run decision-verification mode over a replay where a job's dispatch decision and its running-state confirmation are recorded several rounds apart - - Confirm the verification pass reports zero mismatches for that job's decision - - Observable: a decision-verification test run over a multi-round-separated dispatch scenario reports zero mismatches - - _Requirements: 2.1, 2.2, 2.3, 4.2_ - - _Boundary: sim_reconcile_unaccounted_jobs_ - - _Depends: 1.1_ - -- [x] 3.3 Confirm full regression: existing reconciliation, interactive/batch parity, and the full scheduler suite are unaffected - - Re-run the existing reconciliation unit tests that were not modified by this feature and confirm their already-running-job scenarios still pass unchanged - - Re-run the existing interactive-versus-batch simulation parity coverage and confirm it still passes unchanged - - Run the full scheduler test suite and confirm no new failures beyond any already-tracked, unrelated pre-existing ones - - Observable: a full scheduler test suite run shows the same pass/fail baseline as before this feature, with only the new tests added by this feature as net-new coverage - - _Requirements: 3.3, 4.1, 4.3, 4.5_ - - _Depends: 3.1, 3.2_ - -## Implementation Notes - -- Task 1.2: the pre-existing `test_recordstart_symlink_escape_replies_error` failure (bug-1560, `test_admin_request.c`) is unrelated to this spec and appears in every task's full-suite run — do not treat it as a regression. -- Task 3.1: any NEW `xce_sched_sim_driver_t` fixture passed to `sched_sim_driver_run()` in a scheduler test file MUST be declared `static`, not stack-local — `sched_sim_driver_run()` spawns a picker thread that is never joined and keeps reading the struct for the life of the process, so a stack-local instance is a latent stack-use-after-return once the test function returns (already-documented class, bug-1598/bug-1632; this spec's own task 3.1 round-1 review caught a live recurrence). `sched_sim_driver_start()` unconditionally `memset()`s the struct first, so `static` storage is safe as long as each fixture is written by exactly one call site. Caveat (task 3.2): this `static` fix is NOT safe for a helper called multiple times per process (e.g. `run_sim_and_capture_stdout()`) — an earlier call's never-joined zombie picker thread could still be alive when a later call's `memset()` reuses the same static storage. Leave multiply-invoked fixtures stack-local; only single-invocation fixtures should convert. -- Task 3.2: sim mode's live picker (`sim_picker_thread_fn` + `builtin_fcfs`) never calls `on_dispatch()` — it unconditionally PENDs every candidate, including a job whose allocation `sim_reconcile_unaccounted_jobs()` already captured. Such a job stays in the picker's `pendJobs` and gets re-PENDed every round until its recorded `JOB_STATE_CHANGE` event actually removes it. Any `--sim-verify`/decision-stream fixture spanning a captured-but-not-yet-RUN job's intervening rounds must include a matching `PEND(jobid)` entry at each of those rounds, or the fixture will produce a false mismatch (or mask a real one) unrelated to the behavior under test. diff --git a/.kiro/specs/scheduler-sim-repl-verbosity/design.md b/.kiro/specs/scheduler-sim-repl-verbosity/design.md deleted file mode 100644 index a5099b3f..00000000 --- a/.kiro/specs/scheduler-sim-repl-verbosity/design.md +++ /dev/null @@ -1,291 +0,0 @@ -# Design Document - -## Overview -**Purpose**: This feature reduces and reorganizes the console output the `xce_scheduler --interactive` simulation REPL prints while executing `continue`, so an operator can watch a long replay's progress without wading through repetitive, low-information lines. - -**Users**: XCE scheduler engineers and operators debugging scheduling behavior via `xce_scheduler -i -s <replay>`, a capability shipped by the `scheduler-core` spec. - -**Impact**: Changes only the text `xce_repl_on_event()` (repl_break.c) and `sim_print_decision()` (sim_loop.c) print while `sim->repl_state->mode == XCE_REPL_MODE_CONTINUE`. No change to scheduling decisions, replay determinism, `step`-mode output, or any other REPL command. - -### Goals -- Suppress routine per-event echo during `continue` execution, while still surfacing structural changes (Requirement 1). -- Replace unconditional per-decision echo with one summary line per round during `continue` (Requirement 2). -- Collapse consecutive idle or unchanged rounds into a single range notification (Requirement 3). -- Compress array-job identifier lists into index ranges everywhere the REPL displays multiple job identifiers together (Requirement 4). -- Leave `step` mode and every other existing REPL command byte-for-byte unchanged (Requirement 5). - -### Non-Goals -- New REPL commands or newly inspectable data (limit usage, pend-reason detail) — tracked separately under the `scheduler-sim-diagnostics` brief. -- Any change to scheduling/allocation logic or to the deterministic outcome of a replay. -- Non-interactive (headless, no `-i`) scheduler output. -- Resolving bug-1611 (whether `DISPATCH` decisions are structurally invisible in replay mode) — this spec only guarantees Requirement 2.4 holds regardless of that investigation's outcome (see Design Decisions, `research.md`). - -## Boundary Commitments - -### This Spec Owns -- Console output formatting/verbosity logic inside `xce_repl_on_event()` (repl_break.c), the `continue`-mode gate on `sim_print_decision()`'s call site (sim_loop.c), and the new round-boundary rollup/collapse logic added at `xce_repl_post_session_check()` (repl.c). -- The new jobid-range-compression helper and every call site that adopts it. -- New `xce_sched_repl_state_s` fields required to track collapse state across rounds. - -### Out of Boundary -- Any new `info`/`break`/`set` subcommand or newly inspectable scheduler data. -- Scheduling/allocation/decision logic itself (`scheduler.c`, `src/plugins/`). -- `sim_print_decision()`'s behavior when `sim->interactive` is false (headless mode) — unchanged. -- Root-causing bug-1611 — a separate investigation; this spec only ensures its own requirements hold regardless of the answer. - -### Allowed Dependencies -- Existing `xce_sched_repl_state_s` fields (`mode`, `last_n_dispatch`/`last_n_pend`/`last_n_preempt`, `session_count`) — repl_internal.h. -- The existing `core->jobs` iteration idiom (`iter_make`/`iter_next`, `JOB_IS_PEND`) already used by `repl_info_jobs()`. -- Existing helpers `fmt_jobid_ex()` (lib/utils.c) and `repl_format_ts_human()` (repl.c). -- Must not add a new call site to `sim_loop.c`'s main loop beyond gating the two calls that already exist there (`sim_print_decision`, the `xce_repl_check_event` wrapper) — that file is documented at its line-budget ceiling (repl.h's own header comment). - -### Revalidation Triggers -- If `scheduler-sim-diagnostics` adds pend-reason data, the round-rollup collapse fingerprint (Design Decisions, `research.md`) may need to incorporate it — revisit Requirement 3.2's implementation then. -- If bug-1611 is confirmed a real dispatch-visibility gap and separately fixed, the rollup's dispatch *count* source should switch from the decision tally to the same state-derived approach already used for the pend-jobid set. -- Any change to `xce_sched_repl_state_s`'s existing `mode`/tally fields by another spec should re-check this feature's rollup logic. - -## Architecture - -### Existing Architecture Analysis -- `repl.c`/`repl_break.c`/`repl_info.c`/`repl_params.c` form the REPL command family under `scheduler-core`. `repl.h` folds the `sim->interactive` gate into `static inline` wrapper functions so `sim_loop.c`'s own diff stays to single-line, brace-free call expressions — this feature's `sim_loop.c` change must follow that same convention rather than adding a bare `if` block to the loop body. -- `xce_sched_repl_state_s` (repl_internal.h) is the REPL's own session-scoped state struct, zero-initialized via `calloc` in `repl_state_ensure()` — new fields need no separate lifecycle. -- `xce_repl_post_session_check()` (repl.c) already runs exactly once per completed round, immediately after that round's decisions are drained, purely to check the `step`-budget pause condition today. - -### Architecture Pattern & Boundary Map - -**Correction (post-review)**: breakpoint-hit detection and structural-event classification both live in `repl_break.c` (`xce_repl_on_event`, `xce_repl_on_begin_round`, `xce_repl_on_decision` — each calls `repl_command_loop()` directly on a match), not in `repl.c`. Requirement 3.5 ("a structural event ends the active range") and Requirement 3.6 ("a breakpoint hit ends the active range") therefore cannot be satisfied by round-boundary logic in `repl.c` alone — `repl_break.c` needs a way to force-flush the Round Rollup's active range from its own call sites. The Round Rollup component now exposes an explicit flush entry point for this, called from both files. - -```mermaid -flowchart TD - RoundEnd[Round decisions drained] --> Query[Query core jobs for PEND state] - Query --> Fingerprint[Build fingerprint: dispatch count, pend jobid set, preempt count] - Fingerprint --> Compare{Matches active collapsed range} - Compare -->|Yes| Extend[Extend active range silently] - Compare -->|No| FlushCall[Call round rollup flush] - FlushCall --> Begin[Begin new collapsed range with this round] - Extend --> NextRound[Advance to next round] - Begin --> NextRound - StructuralEvent[Structural event classified in repl_break.c] --> FlushCall - BreakpointHit[Breakpoint hit detected in repl_break.c] --> FlushCall - ReplayComplete[Replay completion in repl.c] --> FlushCall - FlushCall --> PrintRange[Print the flushed range, if one was open] -``` - -**Architecture Integration**: -- Selected pattern: a state accumulator attached to the existing round-boundary hook (`xce_repl_post_session_check`), exposing one shared flush entry point that `repl_break.c`'s event/breakpoint/decision hooks and `repl.c`'s replay-completion hook all call — no new hook call sites added to `sim_loop.c`. -- Domain/feature boundaries: the Round Rollup's state and flush/print logic live in `repl.c`; `repl_break.c` calls the flush entry point but does not own or mutate the collapse state directly; `sim_loop.c` only gains a mode gate on its two existing calls. -- Existing patterns preserved: `repl.h`'s `sim->interactive`-gated inline-wrapper convention; `repl.c`'s shared-helper convention (`repl_format_ts_human`, `repl_split_tokens`) extended with two new shared helpers (range compression, and the round-rollup flush entry point declared in `repl_internal.h` for `repl_break.c` to call). -- New components rationale: deciding "print now vs. extend silently" requires comparing against the *previous* round's fingerprint, which must persist across separate `xce_repl_post_session_check()` invocations — hence new persistent fields on `xce_sched_repl_state_s` rather than function-local state. The flush entry point exists because three independent call sites (round boundary, breakpoint hit, replay completion) across two files all need to end an active range the same way. -- Steering compliance: one domain per file preserved; no new production `.c` file needed; `snake_case`/`repl_*` naming conventions followed throughout. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Scheduler daemon (C, gnu17) | existing `src/scheduler/` toolchain, no new dependency | All logic lives in existing REPL/sim-driver translation units | Matches `tech.md`; no new library, no CGO/Go involvement | - -## File Structure Plan - -### Modified Files -- `src/scheduler/src/repl_internal.h` — extend `xce_sched_repl_state_s` with round-collapse state fields (see Data Models) and declare the new shared jobid-range-compression helper. -- `src/scheduler/src/repl_break.c` — `xce_repl_on_event()` gains routine-vs-structural event classification gated on `rs->mode` (Requirement 1); `repl_print_event_summary()`'s `JOB_PUSH`-style cases adopt the jobid-range-compression helper (Requirement 4); `xce_repl_on_event()`, `xce_repl_on_begin_round()`, and `xce_repl_on_decision()` each call the Round Rollup's flush entry point before printing a structural-event line or before re-entering `repl_command_loop()` on a breakpoint hit (Requirement 3.5, 3.6). -- `src/scheduler/src/repl.c` — `xce_repl_post_session_check()` gains the round-boundary fingerprint/collapse logic (Requirements 2, 3); implements the new shared jobid-range-compression helper (Requirement 4) and the Round Rollup's flush entry point (Requirement 3.5, 3.6), following the file's existing shared-helper convention; `xce_repl_on_replay_complete()` calls the same flush entry point before printing the existing replay-complete banner. -- `src/scheduler/src/sim_loop.c` — the existing call site invoking `sim_print_decision()` gains a `continue`-mode gate, mirroring `repl.h`'s inline-wrapper convention (Requirements 2, 5). -- `src/scheduler/include/repl.h` — no functional change expected; existing inline wrappers already cover the gating points this feature needs. - -### New/Modified Test Files -- `src/scheduler/testsuite/test_repl_execution_control.c` — extended with child-process output-capture tests covering Requirements 1, 2, 3, and the Requirement 5 no-regression checks (this file already drives `sched_sim_driver_run()` end-to-end and captures real process output via `child_continue_then_inspect`-style helpers). -- `src/scheduler/testsuite/test_repl_output_compression.c` (new) — focused unit tests for the jobid-range-compression helper (Requirement 4): a pure function, tested without the process harness. - -## System Flows -See the Architecture Pattern & Boundary Map diagram above — it is the only non-trivial flow this feature introduces (a per-round branching decision with two exit points: silent extension, or flush-and-restart). Per-event classification (Requirement 1) is a simple, non-branching gate and does not warrant a separate diagram. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|-------------|-----------|-------| -| 1.1, 1.2, 1.3, 1.4 | Mode-aware event echo | Event Echo Gate | `xce_repl_on_event` | n/a (per-event, upstream of round boundary) | -| 2.1, 2.2, 2.3, 2.4 | Per-round decision summary | Round Rollup | `xce_repl_post_session_check`, state-derived pend query | Round-boundary flow | -| 3.1, 3.2, 3.3, 3.4 | Collapsing repetitive rounds (idle/unchanged detection, range content) | Round Rollup (collapse state) | fingerprint compare / flush | Round-boundary flow | -| 3.5, 3.6 | Ending an active collapsed range on outcome change, structural event, breakpoint hit, or replay completion | Round Rollup (flush entry point, and its own call from `xce_repl_on_replay_complete`), Event Echo Gate (calls it on structural events/breakpoint hits) | `repl_round_collapse_flush` | Round-boundary flow | -| 4.1, 4.2, 4.3, 4.4, 4.5 | Job identifier range compression | Jobid Range Compressor | `repl_fmt_jobid_range` | n/a (pure formatting) | -| 5.1, 5.2, 5.3, 5.4 | No regression to step mode / other commands | Event Echo Gate, `sim_print_decision` gate | mode checks | n/a | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|---------------|--------|---------------|---------------------------|-----------| -| Event Echo Gate | REPL / repl_break.c | Classify each event as routine or structural and gate the per-event echo by mode | 1.1-1.4 | `rs->mode` (P0), Round Rollup flush entry point (P0) | State | -| Round Rollup | REPL / repl.c | Build a per-round fingerprint, print it, or extend/flush a collapsed range; expose a flush entry point callable from breakpoint/structural-event/replay-complete sites | 2.1-2.4, 3.1-3.6 | `core->jobs` query (P0), existing decision tally (P0) | State, Service | -| Jobid Range Compressor | REPL / repl.c (shared helper) | Compress array-job identifier lists into index ranges | 4.1-4.5 | `fmt_jobid_ex` decomposition (P0) | Service | -| `sim_print_decision` gate | Sim driver / sim_loop.c | Suppress the direct per-decision print when the Round Rollup will report it instead | 2.3, 5.1 | REPL mode state (P0) | State | - -### REPL Output (repl.c / repl_break.c / sim_loop.c) - -#### Event Echo Gate -| Field | Detail | -|-------|--------| -| Intent | Classify each decoded event and suppress its individual echo line under `continue`, while breakpoint matching still evaluates every event | -| Requirements | 1.1, 1.2, 1.3, 1.4 | - -**Responsibilities & Constraints** -- Structural events (always echoed, any mode): `CONFIG_PUSH`, `JOBS_PUSH`, `WORKERS_PUSH`, `RESOURCES_PUSH`, `RESVALS_FULL` — population/config arrivals an operator needs to see even while `continue` is otherwise quiet. Before printing a structural event's line, this gate calls the Round Rollup's flush entry point so any active collapsed range is closed and reported first (Requirement 3.5) — a structural event and a stale collapsed range must never interleave out of order. -- Routine events (echoed only in `step` mode): `RESVALS_DELTA`, `JOB_STATE_CHANGE`, `JOB_UPDATE`, `QUEUE_UPDATE`, `JOB_RUSAGE`. `JOB_STATE_CHANGE` is classified routine deliberately — the Round Rollup (Requirement 2.4) already surfaces job status transitions at round granularity, so an individual `JOB_STATE_CHANGE` line under `continue` would duplicate that information. -- Breakpoint matching (`repl_find_event_breakpoint`, plus the equivalent checks in `xce_repl_on_begin_round`/`xce_repl_on_decision`) runs on every event regardless of echo suppression — Requirement 1.4 is unaffected by this gate. On a match, before re-entering `repl_command_loop()`, the matching hook calls the Round Rollup's flush entry point so the operator sees the collapsed range's final state before the prompt returns (Requirement 3.6). - -**Dependencies** -- Inbound: `xce_repl_on_event` — calls this gate before its existing breakpoint check (P0) -- Outbound: `repl_print_event_summary` — invoked only when the gate allows it (P0); Round Rollup's flush entry point — called before a structural-event line or a breakpoint-triggered command-loop re-entry (P0) - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x] - -##### State Management -- State model: reads `sim->repl_state->mode` (existing field); no new state owned by this component. -- Persistence & consistency: none — a per-call decision. -- Concurrency strategy: none required — single-threaded sim driver, same as every other REPL hook. - -#### Round Rollup -| Field | Detail | -|-------|--------| -| Intent | Once per completed round in `continue` mode, compute a fingerprint and either extend an active collapsed range or flush it and begin reporting the new state | -| Requirements | 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 | - -**Responsibilities & Constraints** -- Active only while `sim->interactive && rs->mode == XCE_REPL_MODE_CONTINUE`; `step` mode keeps today's full per-decision path unchanged (Requirement 2.3, 5.1). -- Pend-jobid set is sourced by directly querying `core->jobs` for `JOB_IS_PEND` state (same iteration idiom as `repl_info_jobs()`) — not solely from the decision tally, so Requirement 2.4 holds independent of bug-1611 (see `research.md` Design Decisions). -- Dispatch/preempt counts are sourced from the existing `last_n_dispatch`/`last_n_preempt` tally. -- Must flush (print) any active collapsed range before: a breakpoint-triggered re-entry into `repl_command_loop()`, replay completion, or a fingerprint change from the active range (Requirement 3.5, 3.6). Because the breakpoint and structural-event triggers originate in `repl_break.c`, not `repl.c`, this component exposes a flush entry point other files call rather than owning every trigger site itself. - -**Dependencies** -- Inbound: `xce_repl_post_session_check` — the round-boundary call site the fingerprint/collapse logic attaches to (P0); the Event Echo Gate (`xce_repl_on_event`) and the breakpoint checks in `xce_repl_on_begin_round`/`xce_repl_on_decision` (all repl_break.c) — call the flush entry point (P0); `xce_repl_on_replay_complete` (repl.c) — calls the flush entry point before its existing banner (P0) -- Outbound: `core->jobs` iterator (P0), `xce_repl_on_decision`'s existing tally fields (P0), Jobid Range Compressor for rendering the pend-jobid list (P0) - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] - -##### Service Interface -```c -/* - * Ends the active collapsed round range, if one is open, printing its - * summary line immediately. Safe to call when no range is active (a - * no-op). Called from the round boundary on a fingerprint change, and - * from repl_break.c before a structural-event line or a breakpoint- - * triggered command-loop re-entry, and from xce_repl_on_replay_complete - * before its existing banner. - */ -void repl_round_collapse_flush(xce_sched_sim_driver_t* sim); -``` -- Preconditions: `sim`/`sim->repl_state` non-NULL (same "safe no-op on NULL" discipline every other REPL hook in this file family already follows). -- Postconditions: no collapsed range is active after the call; if one was active, its summary line has been printed exactly once. -- Invariants: idempotent — calling it again immediately afterward with no intervening round is a no-op, never a duplicate print. - -##### State Management -- State model: new fields on `xce_sched_repl_state_s` hold the active collapsed range (see Data Models) — whether one is open, its round/timestamp bounds, and its cached fingerprint. -- Persistence & consistency: in-memory, scoped to one REPL session; zero-initialized for free by `repl_state_ensure()`'s existing `calloc`. -- Concurrency strategy: none required — single-threaded sim driver. - -#### Jobid Range Compressor -| Field | Detail | -|-------|--------| -| Intent | Group a list of jobids by array base and render contiguous index runs as compressed ranges | -| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5 | - -**Responsibilities & Constraints** -- Pure, read-only formatting function — never mutates its input list or any scheduler/job state. -- Shared by `repl_print_event_summary()`'s job-arrival listings and the Round Rollup's pend-jobid display (Requirement 4.5). - -**Dependencies** -- Inbound: `repl_print_event_summary` (P0), Round Rollup (P0) -- Outbound: `fmt_jobid_ex` (lib/utils.c) — reuses its existing `(cluster, apiserver, timestamp, job[, idx])` decomposition (P0) - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -/* - * Formats `njobids` raw jobids into `buf` (bufsz-bounded): groups by - * array base (cluster, apiserver, timestamp, job) and compresses - * contiguous [idx] runs into [lo-hi] form, comma-joining non-contiguous - * runs within one base; non-array jobids are listed individually. - * Returns the number of bytes written (excluding the NUL terminator), - * or a negative value if buf was too small to hold the full listing. - */ -int repl_fmt_jobid_range(char* buf, size_t bufsz, - const uint64_t* jobids, int njobids); -``` -- Preconditions: `jobids` non-NULL whenever `njobids > 0`; `buf`/`bufsz` describe a valid, writable buffer. -- Postconditions: `buf` holds a NUL-terminated, comma-separated listing; distinct array bases appear in input order, indices ascend within each base's compressed ranges. -- Invariants: never fabricates or silently drops a jobid — the compressed listing always represents exactly the same id set the uncompressed listing would show. - -#### `sim_print_decision` Gate -| Field | Detail | -|-------|--------| -| Intent | Suppress `sim_print_decision()`'s direct per-decision stdout line when the Round Rollup will report the round instead | -| Requirements | 2.3, 5.1 | - -**Responsibilities & Constraints** -- Skip the direct print only when `sim->interactive && sim->repl_state->mode == XCE_REPL_MODE_CONTINUE`; `step` mode and non-interactive (headless) runs are unaffected (Requirement 5.1, 5.4). -- Implemented as an inline-wrapper-style gate at the existing call site in `sched_sim_driver_run()`, consistent with `repl.h`'s established convention, rather than a bare `if` block added to the loop body. - -**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x] - -##### State Management -- State model: reads `sim->interactive` and `sim->repl_state->mode` (both existing fields); owns no new state. - -## Data Models - -### Domain Model -This feature adds a "round collapse state" concept to the REPL's existing session-scoped state (`xce_sched_repl_state_s`, repl_internal.h) — not a new top-level entity. It follows the same pattern prior REPL task revisions used to add `output_path`/`replay_complete` to that struct. - -### Logical Data Model -New fields on `xce_sched_repl_state_s` (illustrative — exact types finalized at implementation time): - -| Field | Type | Purpose | -|-------|------|---------| -| `collapse_active` | `bool` | Whether a collapsed range is currently open | -| `collapse_start_round` / `collapse_end_round` | `uint64_t` | Round-number bounds of the active range | -| `collapse_start_ts` / `collapse_end_ts` | `uint64_t` | Sim-time bounds, for the collapsed range's time span (Requirement 3.4) | -| `collapse_dispatch_count` / `collapse_preempt_count` | `int` | Cached fingerprint counts | -| `collapse_pend_jobids` / `collapse_n_pend_jobids` | pointer + count | Cached fingerprint pend-jobid set, in whatever list form the Jobid Range Compressor consumes | - -**Consistency & Integrity**: all fields reset together whenever a range flushes (print, then clear) or a new REPL session begins — `repl_state_ensure()`'s existing `calloc` zero-initializes new fields with no additional lifecycle code required. - -No Physical Data Model, API Data Transfer, or Event Schema subsections apply: this feature has no persistent storage, no network wire format, and adds no new wire-level contract — it stays inside `scheduler-core`'s existing "simulation isolation" boundary (repl.h's own header comment: no `scheduler.h`, no `client.h`, no capnp types). - -## Error Handling - -### Error Strategy -Consistent with existing `repl.c`/`repl_break.c` conventions: allocation failures print an existing-style `"[sim] ..."` message and degrade to the safest available behavior rather than crashing or losing information, matching `repl_state_ensure()`'s own "never load-bearing, degrade to unattended" precedent. - -### Error Categories and Responses -- **Allocation failure while building a round's fingerprint or growing the collapsed-range pend-jobid buffer**: fall back to printing that round's line directly (uncollapsed) rather than losing the round's information — preserves Requirement 2.1's obligation to report every round with decisions. -- **`repl_fmt_jobid_range()` buffer too small**: returns a negative value; the caller falls back to the existing uncompressed per-id listing for that call — never truncates silently, never crashes, preserves the "never drop a jobid" invariant even in a buffer-constrained edge case. - -### Monitoring -No new logging framework is introduced — the existing `[sim] ...` stdout convention remains the only user-facing signal; this is an interactive CLI tool, not a daemon, so no zap/WARN-level logging applies here. - -## Testing Strategy - -### Unit Tests (new `test_repl_output_compression.c`) -- Contiguous indices from one array base compress to a single `[lo-hi]` range (Requirement 4.1). -- Non-contiguous indices compress to comma-joined sub-ranges within one base (Requirement 4.2). -- Multiple array bases in one listing compress independently (Requirement 4.3). -- A mixed array + non-array listing shows non-array ids individually and compresses only the array ids (Requirement 4.4). -- A buffer too small to hold the full listing returns a negative value and writes nothing past `bufsz` (Error Handling). - -### Integration Tests (extend `test_repl_execution_control.c`, child-process output capture) -- `continue` across multiple idle rounds (zero decisions) produces one collapsed range line, not one line per round (Requirement 3.1). -- `continue` across rounds sharing an identical nonzero pend set produces one collapsed "unchanged" range line — the exact shape hand-validated against the real 108-round replay that motivated this spec (Requirement 3.2). -- A structural event mid-`continue` ends any active collapsed range and prints its own line before continuing (Requirement 1.3, 3.5). -- A breakpoint hit mid-collapse flushes the active range before entering the command loop (Requirement 3.6). -- Replay completion (EOF) flushes any active collapsed range before the existing "replay complete" banner (Requirement 3.6). -- `step`-mode output is unchanged versus this feature's pre-implementation baseline — regression check alongside the existing `test_should_pause_fires_on_exactly_the_nth_session`-style fixtures (Requirement 5.1, 5.3). -- A job's tracked status changing between rounds is reflected in the next round summary even under a fixture that never emits a synthetic `DISPATCH` decision — exercises the state-derived-pend-set design decision independent of bug-1611's resolution (Requirement 2.4). - -### E2E -Not applicable beyond the integration tests above — this feature has no HTTP/API surface; the child-process harness already exercises the real `xce_scheduler` binary end-to-end at the process level. - -## Optional Sections -Security Considerations, Performance & Scalability, and Migration Strategy are omitted: this is a local, single-operator interactive CLI output change with no new attack surface, no persistent or high-load data path (round counts are bounded by replay length, already tolerated by existing `step` output), and no schema or data migration. - -## Supporting References -None beyond `research.md` — no external API, vendor library, or long schema requires separate documentation. diff --git a/.kiro/specs/scheduler-sim-repl-verbosity/requirements.md b/.kiro/specs/scheduler-sim-repl-verbosity/requirements.md deleted file mode 100644 index dc26e357..00000000 --- a/.kiro/specs/scheduler-sim-repl-verbosity/requirements.md +++ /dev/null @@ -1,59 +0,0 @@ -# Requirements Document - -## Introduction -xce_scheduler's `--interactive` simulation REPL (shipped by the `scheduler-core` spec) lets an operator step or continue through a recorded replay while inspecting scheduler state. Under `continue`, the REPL currently prints the same volume of output as single-stepping every round — a real 108-round replay produced roughly 2,700 lines, most of it either routine per-event chatter that carries no new information under `continue`, or an unchanged pending-job decision repeated identically round after round. This feature reorganizes REPL output during `continue` execution so an operator can track a long replay's progress without wading through repetitive, low-information lines, while leaving `step` mode and every other REPL command unchanged. - -## Boundary Context -- **In scope**: console output shown by the `--interactive` simulation REPL during `continue` execution — which events are echoed, how scheduling-decision outcomes are summarized per round, how repetitive rounds are collapsed, and how multi-job identifier lists are displayed. -- **Out of scope**: any new REPL command or newly inspectable data (for example limit usage or pend-reason detail — tracked separately under `scheduler-sim-diagnostics`); any change to scheduling/allocation behavior or to the deterministic outcome of a replay; any change to `step` mode's existing output; non-interactive (headless, no `-i`) scheduler output. -- **Adjacent expectations**: this feature extends the existing REPL shipped by `scheduler-core` (implementation-complete) without changing its command surface. `scheduler-sim-diagnostics` (currently a discovery brief, not yet a formal spec) is expected to separately add pend-reason and limit-usage inspection; this feature's round summaries should remain compatible with, and not duplicate, that future work. - -## Requirements - -### Requirement 1: Mode-Aware Event Echo During Continuous Execution -**Objective:** As an operator running a simulation replay, I want `continue` to suppress routine per-event chatter while still alerting me to major state changes, so that I can watch a long replay without being flooded by lines carrying no new information. - -#### Acceptance Criteria -1. While the REPL is executing in continue mode, when a routine per-event notification (a job progress update, a job resource-usage update, or a resource value update) occurs, the REPL shall not print an individual line for that event. -2. While the REPL is executing in step mode, when any event occurs, the REPL shall print an individual line for that event. -3. When a structural change occurs (new jobs submitted, new workers added, new resources added, or configuration pushed), the REPL shall print a line describing that change regardless of execution mode. -4. When an event matches a defined breakpoint, the REPL shall interrupt continue-mode execution and enter the interactive command loop, regardless of whether the event is routine or structural. - -### Requirement 2: Per-Round Decision Summary -**Objective:** As an operator watching simulation replay progress, I want one summary line per round showing what scheduling decisions were made, so that I can track scheduling outcomes without reading a separate line per affected job. - -#### Acceptance Criteria -1. While the REPL is executing in continue mode, when a round completes with at least one scheduling decision (dispatch, pend, or preempt), the REPL shall print one summary line for that round showing the count of each decision type and the identifiers of the jobs affected. -2. When a round completes with zero scheduling decisions, the REPL shall not print an individual decision line for that round. -3. While the REPL is executing in step mode, the REPL shall print full per-decision detail for every decision. -4. When a job's tracked status changes from one round to the next, the REPL's round summary shall reflect that change. - -### Requirement 3: Collapsing of Repetitive Round Output -**Objective:** As an operator, I want consecutive rounds that produce no new information to be reported as a single range instead of one line each, so that long stretches of idle or unchanged simulation state don't dominate the output. - -#### Acceptance Criteria -1. When two or more consecutive rounds occur with zero scheduling decisions, the REPL shall report them as a single collapsed range instead of one line per round. -2. When two or more consecutive rounds occur with an identical decision outcome (same decision counts and the same affected job identifiers), the REPL shall report them as a single collapsed range instead of repeating the same summary line for each round. -3. The REPL shall include the first round number, the last round number, and the round count in every collapsed range notification. -4. Where round timestamps are displayed, the REPL shall include the covered time span in each collapsed range notification. -5. When a round's decision outcome differs from the immediately preceding reported round, or a structural event occurs, the REPL shall end the active collapsed range and report it before continuing. -6. When continue-mode execution stops for any reason (a breakpoint hit or replay completion), the REPL shall end and report any active collapsed range before the stop is reported to the operator. - -### Requirement 4: Job Identifier Range Compression -**Objective:** As an operator reading REPL output that lists many job identifiers from the same job array submission, I want those identifiers compressed into an index range, so that I can read the list without scrolling past dozens of nearly-identical entries. - -#### Acceptance Criteria -1. When the REPL displays two or more job identifiers that belong to the same job array submission and have contiguous array indices, the REPL shall display them as a single compressed index range. -2. When job identifiers from the same job array submission have non-contiguous indices, the REPL shall display each contiguous sub-range separately within the same listing. -3. When a listing contains job identifiers from more than one job array submission, the REPL shall group and compress each job array submission's identifiers independently. -4. When a listing mixes job-array and non-array job identifiers, the REPL shall display the non-array job identifiers individually while compressing the job-array identifiers as described above. -5. The REPL shall apply job identifier range compression consistently everywhere it displays multiple job identifiers together, including job-arrival notifications and round decision summaries. - -### Requirement 5: No Change to Step-Mode Detail or Existing Commands -**Objective:** As an operator relying on today's `step` behavior and `info`/`break`/`delete`/`set`/`snapshot` commands, I want this feature to leave them unchanged, so that existing debugging workflows keep working exactly as before. - -#### Acceptance Criteria -1. The REPL shall preserve existing `step` command output exactly as before this feature, with no reduction in per-event or per-decision detail. -2. The REPL shall preserve existing `info`, `break`, `delete`, `set`, and `snapshot` command behavior and output format unchanged. -3. The REPL shall introduce no new command as part of this feature. -4. The REPL shall not alter the scheduling decisions or final simulation state produced by a replay — only the console output describing them. diff --git a/.kiro/specs/scheduler-sim-repl-verbosity/research.md b/.kiro/specs/scheduler-sim-repl-verbosity/research.md deleted file mode 100644 index 76f4cc93..00000000 --- a/.kiro/specs/scheduler-sim-repl-verbosity/research.md +++ /dev/null @@ -1,77 +0,0 @@ -# Research & Design Decisions - -## Summary -- **Feature**: `scheduler-sim-repl-verbosity` -- **Discovery Scope**: Extension (existing system) — light discovery -- **Key Findings**: - - Two independent, unconditional stdout print paths cause `continue`-mode volume to match `step`-mode volume: `xce_repl_on_event()` (repl_break.c:708-731, gated only by `sim->interactive`, not by `rs->mode`) and `sim_print_decision()` (sim_loop.c:388-411, called directly from `sched_sim_driver_run()`'s main loop with no `sim->interactive`/mode gate at all). - - `xce_repl_post_session_check()` (repl.c), called once per completed round from `sched_sim_driver_run()` right after each round's decisions are drained, is the correct integration point for round-boundary rollup/collapse logic — it already runs exactly once per round when interactive, and already exists purely for the `step`-budget pause check. - - `xce_repl_on_decision()` (repl_break.c:776-799) already tallies `last_n_dispatch`/`last_n_pend`/`last_n_preempt` per round, but that tally is decision-event-driven — real replay data shows it may never observe a fresh `DISPATCH` decision in replay mode even when jobs demonstrably transition to running (bug-1611). The pend-jobid set for the round rollup must therefore be derived by directly querying `core->jobs` for `JOB_IS_PEND` state (the same iteration idiom `repl_info_jobs()` already uses), not solely from the decision tally, to satisfy Requirement 2.4 regardless of bug-1611's outcome. - - Existing test file `test_repl_execution_control.c` already drives `sched_sim_driver_run()` end-to-end via a child-process harness (`child_continue_then_inspect`, `child_continue_twice_after_complete`) that captures real process output — this is the correct home for behavioral tests of Requirements 1-3 (mode gating, rollup, collapsing), since it observes actual printed text, not just internal state. - - No new external dependency is needed — all formatting is plain C string/stdio work consistent with the rest of `repl.c`/`repl_break.c`. - -## Research Log - -### Extension point analysis: where does continue-mode output actually originate? -- **Context**: Requirements 1-3 need continue mode to suppress/summarize output that today prints unconditionally. -- **Sources Consulted**: `repl_break.c` (`xce_repl_on_event`, `xce_repl_on_decision`), `sim_loop.c` (`sim_print_decision`), `repl.c` (`xce_repl_post_session_check`, `repl_should_pause`), `repl.h` (interactive-gated inline wrappers), `repl_internal.h` (`xce_sched_repl_state_s`), CodeGraph call-graph for `sched_sim_driver_run`. -- **Findings**: - - `xce_repl_check_event()` (repl.h, `static inline`) gates `xce_repl_on_event()` on `sim->interactive` only — inside `xce_repl_on_event()`, `repl_print_event_summary()` fires unconditionally before any breakpoint check, with no reference to `rs->mode`. - - `sim_print_decision()` has no interactive or mode gate at all; `sched_sim_driver_run()` calls it directly for every drained decision, the same call site that also drives `xce_repl_check_decision()` (which does the tally) and `sched_record_session_add_decision()` (recording). - - `xce_repl_post_session_check()` fires once per completed round, strictly after that round's decisions are drained and before the next round's events begin — confirmed via call-graph (`sched_sim_driver_run → xce_repl_post_session_pause`). -- **Implications**: Requirement 1's gate belongs inside `xce_repl_on_event()`/`repl_print_event_summary()`. Requirement 2/3's rollup and collapsing logic belongs at the `xce_repl_post_session_check()` boundary, reading state that `xce_repl_on_decision()`'s existing tally and a direct `core->jobs` query both feed. Requirement 2/3 also requires suppressing `sim_print_decision()`'s direct per-decision print when `sim->interactive && rs->mode == XCE_REPL_MODE_CONTINUE`, since that print path is the one bug-1610 identified as unconditional. - -### Job identifier range compression: existing formatting conventions -- **Context**: Requirement 4 needs contiguous array-job indices displayed as ranges. -- **Sources Consulted**: `lib/utils.c` (`fmt_jobid`/`fmt_jobid_ex`), `repl_break.c`'s `repl_print_event_summary()` (current `JOB_PUSH` jobid list rendering). -- **Findings**: `fmt_jobid_ex()` already separates the packed 64-bit jobid into `(cluster, apiserver, timestamp, job[, idx])` components via `EXTRACT_*` macros; an array job's `idx` is already isolated during formatting. There is no existing range-compression helper anywhere in the REPL formatting code — every current jobid list (the `JOB_PUSH` line, and any future round-rollup pend list) is one `fmt_jobid()` call per id. -- **Implications**: A new helper is needed; the natural base for grouping is the already-decoded `(cluster, apiserver, timestamp, job)` tuple (identical for every element of one array submission), with only `idx` varying. No change to `fmt_jobid`/`fmt_jobid_ex` themselves — the new helper wraps a *list* of ids, not a single id. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Gate at existing hook boundaries (chosen) | Add mode/collapse logic inside `xce_repl_on_event`, `sim_print_decision`'s call site, and `xce_repl_post_session_check` | Reuses every existing per-round/per-event integration point; no new hook call sites needed (sim_loop.c already at its documented line-budget ceiling per repl.h's own header comment) | Requires touching two files (`repl_break.c`, `sim_loop.c`) instead of one | Matches this codebase's own precedent (repl.h's inline-wrapper gating) | -| New dedicated "output policy" module | Introduce a separate `repl_output.c` owning all print decisions, called from a single new hook | Cleaner separation of concerns | Adds a 6th call site into `sim_loop.c`'s round loop, which repl.h's own header comment says there is no line budget for; larger diff for a formatting-only feature | Rejected — over-engineered for the boundary this spec owns | - -## Design Decisions - -### Decision: Round rollup's pend-jobid set is state-derived, not decision-tally-derived -- **Context**: Requirement 2.4 requires the round summary to reflect real job status changes even though bug-1611 shows the decision tally may never register a fresh `DISPATCH` in replay mode. -- **Alternatives Considered**: - 1. Trust `xce_repl_on_decision()`'s existing `last_n_dispatch`/`last_n_pend`/`last_n_preempt` tally directly for the pend-jobid set — simplest, but inherits bug-1611's blind spot if it turns out to be a real gap rather than benign replay semantics. - 2. Derive the pend-jobid set each round by directly iterating `core->jobs` for `JOB_IS_PEND` state (same idiom as `repl_info_jobs()`), independent of the decision tally. -- **Selected Approach**: Option 2 for the pend-jobid set; the existing tally is still used for the dispatch/preempt *counts* shown in the rollup header, since bug-1611 only questions dispatch visibility and preempt is unaffected. -- **Rationale**: Requirement 2.4 must hold regardless of how bug-1611 resolves. A state query is authoritative by construction; a decision-tally is only as complete as whatever decisions the replay happens to (re-)generate. -- **Trade-offs**: One extra `core->jobs` iteration per round in continue mode (bounded by job count, not a hot path — replay/step already tolerates equivalent iteration cost in `repl_info_jobs()`/`repl_count_jobs()`). If bug-1611 is later confirmed a real gap, the dispatch *count* shown in the rollup may still under-report until that separate bug is fixed — flagged as a known limitation, not silently masked. -- **Follow-up**: bug-1611 should be investigated (separately from this spec) before or during implementation; if it proves to be a real visibility gap rather than benign replay semantics, a follow-up task should switch the dispatch count to the same state-diff approach used for the pend set. - -### Decision: Collapse fingerprint keys on (dispatch_count, pend_jobid_set, preempt_count), not raw decision objects -- **Context**: Requirement 3.2 needs to detect when consecutive rounds are "the same" to collapse them. -- **Alternatives Considered**: - 1. Compare full decision-object lists round to round. - 2. Compare a compact fingerprint: dispatch count + preempt count + the state-derived pend-jobid set (as an ordered/canonical list). -- **Selected Approach**: Option 2. -- **Rationale**: The real replay validated this exact fingerprint shape — 65 consecutive rounds shared an identical `(0, {8 jobids from base A + 6 jobids from base B}, 0)` fingerprint and collapsed correctly by hand-simulation. Comparing raw decision objects would be more expensive and no more precise, since the pend-jobid set already captures every observable difference the operator cares about. -- **Trade-offs**: A change in *why* a job is pending (a future pend-reason feature under `scheduler-sim-diagnostics`) would not by itself break the collapse, since the fingerprint doesn't include pend reason. Acceptable for this spec's boundary — pend-reason display is explicitly out of scope here. -- **Follow-up**: If `scheduler-sim-diagnostics` later adds pend-reason display, revisit whether the fingerprint needs to include it (a revalidation trigger, see design.md Boundary Commitments). - -### Decision: Round Rollup exposes a shared flush entry point instead of owning every flush trigger itself -- **Context**: An independent task-graph sanity review (run before task generation) found that breakpoint-hit detection and structural-event classification both live in `repl_break.c` (`xce_repl_on_event`, `xce_repl_on_begin_round`, `xce_repl_on_decision`, each calling `repl_command_loop()` directly on a match) — not in `repl.c` where the initial design placed all Round Rollup logic. Requirements 3.5 and 3.6 cannot be satisfied by round-boundary logic alone. -- **Alternatives Considered**: - 1. Duplicate collapse-flush logic inside `repl_break.c` as well as `repl.c` — rejected, violates single-ownership of the collapse state and risks divergent flush behavior. - 2. Move all breakpoint/structural-event detection into `repl.c` so Round Rollup owns every trigger directly — rejected, a much larger and riskier change to existing, already-tested breakpoint-matching code with no benefit to this spec's boundary. - 3. Round Rollup exposes one flush entry point (`repl_round_collapse_flush`); `repl_break.c`'s existing hooks call it before their existing structural-event print / breakpoint-triggered command-loop entry. -- **Selected Approach**: Option 3. -- **Rationale**: Keeps collapse-state ownership entirely inside `repl.c` (single writer, no duplicated logic) while letting the two other legitimate trigger sites (`repl_break.c`'s breakpoint/structural-event detection, and `repl.c`'s own `xce_repl_on_replay_complete`) call a small, idempotent, safe-on-NULL interface — consistent with this codebase's existing "safe no-op on NULL/not-yet-allocated state" convention already documented on every other REPL hook. -- **Trade-offs**: Event Echo Gate (repl_break.c) now has a real outbound dependency on Round Rollup's interface, removing the parallel-safety the initial task draft assumed between those two components' implementation work — tests for both can still be written independently, but implementation must be sequenced. -- **Follow-up**: none — resolved before task generation; `design.md` and the task graph both reflect this. - -## Risks & Mitigations -- **bug-1611 (dispatch visibility in replay mode) is unresolved** — Mitigation: pend-set correctness (Requirement 2.4) does not depend on its resolution (see Design Decision above); dispatch-count accuracy is called out as a known, separately-tracked limitation rather than silently assumed correct. -- **`sim_print_decision()`'s exact call site inside `sched_sim_driver_run()` was confirmed via call-graph but its surrounding loop body was not fully read** — Mitigation: implementation task must re-confirm the precise insertion point and existing control flow around it before editing; this is a normal implementation-time verification, not a design ambiguity (the gating *contract* — suppress under `sim->interactive && continue mode` — is unambiguous regardless of exact line placement). -- **No existing test coverage on `xce_repl_on_event`/`sim_print_decision` text output** (per CodeGraph) — Mitigation: Requirement-driven tests added in `test_repl_execution_control.c` (child-process output capture) before implementation, per this repo's TDD mandate. - -## References -- `.wolf/buglog.json` bug-1609, bug-1610, bug-1611 — the three findings that motivated this spec, each with file:line root causes already recorded. -- `.kiro/specs/scheduler-sim-diagnostics/brief.md` — sibling brief; Requirement 2.4's "why did it pend" is explicitly out of this spec's boundary and belongs to that brief's seed capability 2 if formalized later. diff --git a/.kiro/specs/scheduler-sim-repl-verbosity/spec.json b/.kiro/specs/scheduler-sim-repl-verbosity/spec.json deleted file mode 100644 index 1273a615..00000000 --- a/.kiro/specs/scheduler-sim-repl-verbosity/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "scheduler-sim-repl-verbosity", - "created_at": "2026-08-21T06:50:37Z", - "updated_at": "2026-08-21T07:10:00Z", - "language": "en", - "phase": "tasks-generated", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/scheduler-sim-repl-verbosity/tasks.md b/.kiro/specs/scheduler-sim-repl-verbosity/tasks.md deleted file mode 100644 index 72973746..00000000 --- a/.kiro/specs/scheduler-sim-repl-verbosity/tasks.md +++ /dev/null @@ -1,69 +0,0 @@ -# Implementation Plan - -- [ ] 1. Foundation: job identifier range compression -- [ ] 1.1 Write failing unit tests for compressing job-array identifier listings into index ranges - - Cover a single contiguous range, non-contiguous indices within one array submission, multiple distinct array submissions in one listing, a mix of array and non-array identifiers, and an output buffer too small to hold the full listing - - Observable: the new test cases execute and fail against the current codebase, which has no compression behavior yet - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_ - -- [ ] 1.2 Implement job identifier range compression so the new tests pass - - Group identifiers by their shared array submission and render contiguous index runs as a single compressed range, comma-joining non-contiguous runs - - Never drop or fabricate an identifier, and fail safely (without truncating or crashing) when the output buffer is too small - - Observable: all tests from 1.1 pass; the helper is ready for the event-echo and round-summary work to adopt - - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_ - -- [ ] 2. Core: per-round decision summary, repetitive-round collapsing, and the shared range-flush capability -- [ ] 2.1 Write failing tests for round-level decision summaries, collapsing, and the shared flush capability - - Cover: a round with at least one decision produces one summary line naming decision counts and affected job identifiers; a round with zero decisions produces no individual line; two or more consecutive idle rounds collapse into a single range notification; two or more consecutive rounds with an identical decision outcome collapse into a single range notification; a collapsed range notification states its round bounds, round count, and covered time span; a change in outcome ends an active collapsed range; a job's status change between rounds is reflected in the following round's summary even without a dedicated dispatch event to observe; the shared flush capability safely does nothing when no range is active and is safe to call more than once in a row - - Observable: the new test cases fail against today's unconditional per-decision echo, which has no summarizing, collapsing, or flush behavior - - _Requirements: 2.1, 2.2, 2.4, 3.1, 3.2, 3.3, 3.4_ - - _Boundary: Round Rollup_ - - _Depends: 1.2_ - -- [ ] 2.2 Implement the round-boundary decision summary, collapsing logic, and the shared range-flush capability - - Add the state needed to track an active collapsed range across rounds - - At each round boundary during continuous execution, determine the round's decision outcome directly from current job status (not solely from decision events) together with the existing per-round decision tally, then extend an active matching range or end it and begin reporting the new outcome - - Expose the range-ending behavior so other parts of the REPL can trigger it safely, including when nothing is active; wire it into the existing replay-completion notification so a range open at end-of-replay is closed and reported before that notification prints - - Observable: all tests from 2.1 pass; replaying the real 108-round capture that motivated this feature under continuous execution now produces the hand-validated collapsed shape (an idle range before the first job arrival, one line per job arrival, and one collapsed range per stable pending stretch, with any range still open at end-of-replay closed out before the completion notice) instead of thousands of repeated lines - - _Requirements: 2.1, 2.2, 2.4, 3.1, 3.2, 3.3, 3.4, 3.6_ - - _Boundary: Round Rollup_ - -- [ ] 3. (P) Core: mode-aware event echo, including ending an active collapsed range on structural events or breakpoint hits — Depends: 2.2 -- [ ] 3.1 (P) Write failing tests for suppressing routine event chatter and for ending an active collapsed range at the right moments - - Cover: a routine per-tick event (job progress, resource usage, or resource value update) produces no line under continuous execution; the same event still produces a line while single-stepping; a structural change (new jobs, workers, resources, or configuration arriving) still produces a line under continuous execution and ends any active collapsed range first; an event matching a breakpoint still interrupts continuous execution regardless of its classification, and ends any active collapsed range before the interruption is shown - - Observable: the new test cases fail against today's unconditional per-event echo, which has no classification or range-ending behavior - - _Requirements: 1.1, 1.2, 1.3, 1.4, 3.5, 3.6_ - - _Boundary: Event Echo Gate_ - - _Depends: 1.2, 2.2_ - -- [ ] 3.2 (P) Implement the mode-aware event echo, range compression adoption, and range-ending wiring - - Suppress routine per-event lines during continuous execution while preserving full detail while single-stepping; keep structural-change lines and breakpoint interruption behavior unaffected by the classification - - Render job-arrival listings using the compressed range format - - Before printing a structural-change line, and before a breakpoint interruption is shown, end any active collapsed range first so output never interleaves out of order - - Observable: all tests from 3.1 pass; a continuous-execution run over a replay with only routine activity produces no per-event lines, a job arrival still prints one compressed line and correctly closes out any prior collapsed range, and a breakpoint hit shows the collapsed range's final state before the interactive prompt returns - - _Requirements: 1.1, 1.2, 1.3, 1.4, 3.5, 3.6, 4.5_ - - _Boundary: Event Echo Gate_ - -- [ ] 4. (P) Integration: retire the redundant per-decision echo during continuous execution — Depends: 2.2 -- [ ] 4.1 (P) Write failing tests confirming the old per-decision echo no longer duplicates the new round summary - - Cover: during continuous execution, no bare per-decision line appears outside the round summary; while single-stepping, full per-decision detail still prints exactly as before; a non-interactive (headless) replay's decision output is unaffected - - Observable: the new test cases fail while the old unconditional per-decision print path still runs alongside the new round summary - - _Requirements: 2.3, 5.1, 5.4_ - - _Boundary: sim_print_decision gate_ - - _Depends: 2.2_ - -- [ ] 4.2 (P) Suppress the old per-decision echo during continuous execution so the round summary is the sole source of decision output - - Leave single-stepping and non-interactive output untouched - - Observable: all tests from 4.1 pass; a full continuous-execution run over a replay shows only round-summary lines, with no duplicate bare decision lines - - _Requirements: 2.3, 5.1, 5.4_ - - _Boundary: sim_print_decision gate_ - -- [ ] 5. Validation: regression and real-data verification — Depends: 3.2, 4.2 -- [ ] 5.1 Verify single-stepping and every other existing REPL command remain unchanged - - Confirm single-stepping output, and the inspection/breakpoint/parameter/snapshot commands' behavior and output, match their pre-feature baseline exactly; confirm no new command was introduced - - Observable: the existing REPL test suite's pre-existing assertions all still pass unmodified, and a step-mode transcript comparison shows no textual difference from the pre-feature baseline - - _Requirements: 5.1, 5.2, 5.3_ - -- [ ] 5.2 Run the full scheduler test suite and re-verify against the real replay capture that originally motivated this feature - - Observable: the full test suite passes with no regressions, and re-running the interactive REPL's continuous execution against the real capture reproduces the collapsed output shape validated during design, cutting a several-thousand-line transcript down to a handful of lines while preserving every state change - - _Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 3.4_ diff --git a/.kiro/specs/scheduler-snapshot-job-allocation/brief.md b/.kiro/specs/scheduler-snapshot-job-allocation/brief.md deleted file mode 100644 index 803a5c72..00000000 --- a/.kiro/specs/scheduler-snapshot-job-allocation/brief.md +++ /dev/null @@ -1,64 +0,0 @@ -# Brief: scheduler-snapshot-job-allocation - -## Problem -Simulation-mode capacity/contention analysis is untrustworthy for any scenario seeded from a snapshot -- the common case, since real investigations start from a live snapshot + replay (`xce_scheduler -i -s <replay>`). Reported live: after loading a snapshot with 22 running jobs and replaying 79 sessions forward, `info summary` correctly counts 22 running jobs, but `info worker`'s "used" resource values only ever reflect 2 of them -- the rest permanently show zero contribution, making any capacity check against a snapshot-seeded scenario silently wrong. - -A second, more severe defect in the same load path was found during this session's cross-component persistence survey (see Current State): every job restored from a snapshot -- PENDING or RUNNING -- gets a NULL `job->schedobj`, a field the live allocator dereferences unconditionally. This is a latent crash (same class as the already-fixed bug-1569 SIGSEGV), not just a wrong-value bug, and is folded into this spec since it's the same load-side reconstruction step and the same root cause pattern (a field silently missing from restoration, not from the wire). - -## Current State -- `xce_sched_core_load_snapshot()` (`snapshot_load.c`) restores a job's `state` correctly (so a snapshot-loaded job shows as RUN) but never restores `job->allocs`/`job->accounts` -- confirmed: `build_wire_job()` (`snapshot_wire.c:167`) carries an explicit `/* job->allocs/accounts: not captured */` comment. -- This was a deliberate, documented scope decision from `scheduler-core` task 8.1 (`snapshot.c`'s own "Scope decisions" header comment), not an oversight -- there was no deep-copy helper for `xce_job_alloc_t` anywhere in the codebase, and aliasing it (instead of deep-copying) across two independently-alive cores would create a genuine double-free hazard the moment either core's normal job-lifecycle path frees it. The header explicitly flags the consequence: a restored RUNNING job "forgets what it is currently dispatched on." -- `scheduler-sim-eager-reconciliation` (this session, already shipped) added `sim_reconcile_unaccounted_jobs()`, which DOES reconstruct `job->allocs`/`job->accounts` -- but only by finding a matching DISPATCH decision within the replay event stream currently being processed. A job already running as of the snapshot had its dispatch decision happen before the snapshot was taken, so that event can never appear in the subsequent replay -- this mechanism structurally cannot recover it. It correctly handles the complementary case (dispatch decision discoverable later in replay); this spec closes the other half. -- `sched_account_jobs()` recomputes every worker's `resInUse` fresh each round purely by walking jobs where `JOB_IS_RUN(state) && accounts != NULL` and charging their accounts -- confirmed via source read. Once `job->accounts` is correctly populated from any source, "used" accounting works automatically; no separate accounting-side change is needed. -- Viability check (this session, subagent-verified) found the fix is smaller than first assumed: the wire type already has an `allocs` field -- `Job.allocs`/`SchedJob.allocs @7: JobAlloc` in the capnp schema map directly onto `xce_job_t.allocs` (same C struct, no translation layer), and `JobAlloc`/`HostAlloc`/`ResAlloc` already fully mirror `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t`. It already flows through the existing generic `encode_SnapshotEntry_ptr()`/`decode_SnapshotEntry()` calls with zero schema changes needed -- no version-gating required (that pattern is only needed for the opaque `internal_state` byte blob elsewhere in the same file, not for a real typed field). -- No existing deep-copy helper for `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t` anywhere in the codebase (confirmed via grep for `dup_`/`copy_`/`clone_` conventions). `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` deliberately contain no capnp code (an existing, intentional architectural boundary) -- the deep-copy should be composed from the existing `make_alloc`/`make_host_alloc`/`make_job_alloc` constructors (`src/lib/xcealloc.c`), not the capnp-generated codec, and paired with the existing `free_JobAlloc_ptr` for cleanup. -- Side-finding, already fixed and committed this session ahead of this spec (bug-1722): `make_job_account()`/`make_host_allocs()` in `xcealloc.c` had a live heap buffer overflow for any job spanning more than one host (`calloc(1, ...)` sized for one host pointer, loop wrote up to `nhosts-1`). Fixed independently before this spec starts, since this spec's own tests will exercise multi-host allocations through `make_job_account()` more than any existing code path does today. -- **Direct precedent, same bug class, sibling persistence layer, already fixed (bug-1678, Gitea #43):** the controller's WAL had the identical gap for the exact same data -- after restarting all daemons, a previously-running job's `STATUS=RUN` recovered correctly on replay but `EXEC_HOST` came back permanently blank, because `WAL_JOB_DISPATCH` never persisted the job's allocation. Part of a larger campaign (bug-1687/1689/1690/1691/1692, all already committed before this session, plus bug-1697 start_time, committed this session) fixing WAL replay silently dropping various job fields on restart. Bug-1678's own writeup confirms `JobAlloc`/`xce_job_alloc_t` was already a full top-level capnp struct requiring **no schema change** -- the fix was pure save/load glue (`capnp_encode_joballoc()`/`decode_joballoc()` in `persist.c`, mirroring existing `capnp_encode_worker()`/`restore_job()` patterns), landed as `db142d41 fix(controller): persist dispatch-time resource allocation into WAL_JOB_DISPATCH`. This independently confirms the viability check's finding for the scheduler side (existing wire field, no schema change, pure glue) via a working, shipped implementation of the identical shape in a sibling persistence mechanism -- the controller's WAL and the scheduler's snapshot are separate, non-sharing persistence layers, so fixing one did not fix the other. -- **`job->schedobj` -- undocumented, likely-live crash risk, found and confirmed viable-to-fix this session:** the live JOBS_PUSH path (`core_build_job()`, `core_events.c:199-266`) unconditionally builds `job->schedobj` for every job via `core_build_schedobj()` (`core_events.c:174-197`) -- required scaffolding the allocator dereferences with no NULL check (`sched_make_allocation()`/`sched_make_primary_allocation()`/`sched_make_compound_allocation()`, `scheduler.c:179-267`; also `plugins/limit.c`). `snapshot_load.c`'s job-reconstruction loop never builds it, so every snapshot-restored job (PENDING or RUNNING) gets `job->schedobj == NULL` -- the same failure shape as the already-fixed bug-1569 SIGSEGV, reproducible the moment a restored PENDING job reaches the allocator. Not mentioned in `snapshot.c`'s own "Scope decisions" header, and violates that file's own stated intent to keep job-reconstruction "field-for-field identical" to `core_build_job()`. -- **`schedobj` is fully derivable from data this spec already restores -- confirmed, no new wire field needed.** `core_build_schedobj(obj, j, resreq)` (and the older, structurally identical `make_sched_obj()` in `schedobj.c`, used by `make_sched_job()`) builds the struct from exactly three inputs, all already present by the time job reconstruction reaches this step: (1) `job->user`/`queue`/`project` -- interned strings copied from `j->submit->user/queue/project_name`, already restored via `sched_get_user/queue/project`; (2) `job->resreq` -- already parsed from the snapshot's captured resreq string, its `tag`/`primary`/`compound`/`alter` fields just get wrapped in owner-linked `xce_{primary,compound}_schedobj_t` structs; (3) freshly-initialized empty `glimits`/`hlimits` (session-scoped limit-tracking bookkeeping, correctly starts empty for a restored job exactly as it does for a freshly-submitted one). `configver=0` and `obj->alloc`/`obj->reason` (and the primary/compound variant's own `reason`) are zero/NULL at construction in the live path too -- traced `obj->alloc`'s only writers (`scheduler.c:194/228/257`, inside `sched_make_allocation()`) and confirmed it is pure per-scheduling-round scratch state for an active PENDING-job allocation attempt, never persisted job state, so NULL-after-restore is correct for both a RUNNING job (never re-enters that path) and a PENDING job (gets it populated on its next evaluation, same as a fresh submission). One implementation wrinkle: `core_build_schedobj()` is `static` to `core_events.c`, and per the existing no-shared-helper convention (see Existing Spec Touchpoints), `snapshot_load.c` cannot call it directly -- the fix replicates the same construction logic locally, mirroring how job/queue/worker reconstruction already works there, not a call into `core_events.c`. - -## Desired Outcome -A job that is already running as of a snapshot's save point survives a save/load round-trip with its full allocation intact: `job->allocs` populated, `job->accounts` reconstructed via the existing `make_job_account()`, and worker/shared resource "used" accounting correctly reflects it from the moment the snapshot loads -- no longer dependent on whether its dispatch decision happens to also appear later in a replay stream. A snapshot file saved before this change still loads cleanly (no crash, no misinterpretation) -- those jobs simply have nothing to restore, the same `used=0` behavior as today, a clean degrade rather than a regression. - -Every job restored from a snapshot -- PENDING or RUNNING -- also has a valid, non-NULL `job->schedobj`, rebuilt directly from its already-restored `resreq`/user/queue/project (no new data captured, no wire change), matching what a freshly-submitted job gets and eliminating the latent NULL-deref crash risk in the allocator for any restored PENDING job. - -## Approach -Extend snapshot save/load to populate/restore the wire type's already-existing `allocs` field, and locally reconstruct `job->schedobj` -- no schema change for either: -1. New deep-copy helper(s), composed from the existing `make_alloc`/`make_host_alloc`/`make_job_alloc` constructors, that build a fresh, independent `xce_job_alloc_t` from a source one -- paired with the existing `free_JobAlloc_ptr` for cleanup. -2. `build_wire_job()` (`snapshot_wire.c`) populates the deep-copied allocation when `job->allocs != NULL`. -3. `xce_sched_core_load_snapshot()` (`snapshot_load.c`) restores it via the same deep-copy (never aliasing the source snapshot's own copy) and calls `make_job_account()` to rebuild `job->accounts` -- mirroring the exact pattern `sim_reconcile_unaccounted_jobs()` already established for its own narrower case. -4. Backward compatibility is a plain NULL/`nhosts>0` guard, not the `internal_version` byte-blob pattern -- unnecessary for a real typed field. -5. In the same `xce_sched_core_load_snapshot()` job-reconstruction loop, locally replicate `core_build_schedobj()`'s exact field-for-field logic (user/queue/project strings, resreq-tag-driven primary/compound/alter wrapper, fresh empty `glimits`/`hlimits`, `configver=0`) to build `job->schedobj` for every restored job, regardless of state. No wire/schema addition -- every input it needs is already restored earlier in the same reconstruction path. Cannot call `core_build_schedobj()` directly (`static` to `core_events.c`); this is a local reimplementation, consistent with the existing no-shared-helper convention already followed for job/queue/worker reconstruction. - -## Scope -- **In**: the deep-copy helper; `build_wire_job()` save-side wiring; `xce_sched_core_load_snapshot()` load-side wiring + `job->accounts` reconstruction; local `job->schedobj` reconstruction for every restored job (PENDING and RUNNING); cross-core round-trip tests (multi-host, multi-resource, shared-resource allocations); a lightweight test proving a snapshot with no allocation/schedobj data (the natural NULL-field case, not an archived legacy file) still loads cleanly with today's `used=0` behavior; a sim-mode end-to-end test proving `info worker`'s "used" value is correct immediately after loading a snapshot with running jobs (closing the originally reported gap); a test proving a restored PENDING job can reach the allocator (`sched_make_allocation()`) without a NULL-deref (closing the schedobj gap). -- **Backward compatibility note (pre-release product):** since these are typed, optional wire fields rather than the `internal_version` byte-blob pattern, an absent field simply decodes as NULL -- correct handling is a NULL guard that falls out of writing the load logic correctly, not a dedicated compatibility effort. No archived pre-feature snapshot file needs to be preserved/maintained as a fixture; a freshly-generated snapshot with no allocation data exercises the same code path. -- **Out**: any change to `sim_reconcile_unaccounted_jobs()` itself (already correct for its own case, stays as-is); any capnp wire schema change (not needed); any change to `internal_state`/`internal_version` (unrelated field); the `job->priority` and queue `project_name`/`job_pick_plugin` gaps `snapshot.c`'s own "Scope decisions" also documents (separate, unrelated, pre-existing, not part of this fix); `job->rusages` (no wire slot exists at all -- would need an actual schema addition, unlike allocs/schedobj which need none -- tracked as a separate follow-on, not folded in here). - -## Boundary Candidates -- **Deep-copy helper** (`xcealloc.c`, or a new file if it grows past the codebase's ~800-line-per-file convention): the one piece of genuinely new logic for the allocation half. -- **Save-side wiring** (`build_wire_job`/`snapshot_wire.c`): small, mechanical once the helper exists. -- **Load-side wiring** (`xce_sched_core_load_snapshot`/`snapshot_load.c`): small, mirrors `sim_reconcile_unaccounted_jobs()`'s existing `job->accounts` reconstruction pattern. -- **`schedobj` local reconstruction** (`snapshot_load.c`): small, mechanical -- a local field-for-field mirror of `core_build_schedobj()`, no new data, no shared helper. -- **Test coverage**: cross-core round-trip (multi-host/multi-resource/shared), backward-compat (old snapshot file), the end-to-end sim-mode "used" accounting proof, and the restored-PENDING-job-reaches-the-allocator schedobj proof. - -## Out of Boundary -- `job->priority`, queue `project_name`/`job_pick_plugin` -- both already-documented, unrelated pre-existing snapshot gaps. -- `job->rusages` -- undocumented gap found this session, but needs a real wire schema addition; separate follow-on, not folded into this spec. -- Any change to the live (non-snapshot) dispatch/accounting path. -- `sim_reconcile_unaccounted_jobs()` itself. - -## Upstream / Downstream -- **Upstream**: `scheduler-core`'s existing snapshot save/load (task 8.1) -- this is a direct extension of it, not independently founded. `scheduler-sim-eager-reconciliation` (this session) -- the complementary reconciliation mechanism this fix does not replace. -- **Downstream**: any future `scheduler-sim-diagnostics` work inspecting resource accounting; the roadmap's existing backlog item for `xce_test`'s future `snapdump` tooling may want to report whether a snapshot carries job allocations, once this exists. - -## Existing Spec Touchpoints -- **Extends**: `scheduler-core` (owns `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c`) -- not reopened as a spec (matching this session's own precedent for `scheduler-sim-eager-reconciliation`), but this spec's design must respect its existing architecture/conventions: no capnp code in these 3 files, the field-copy-not-shared-helper convention `snapshot.c`'s own header already documents and justifies (avoiding `core_events.c`'s static, unexported `core_build_job`/`core_build_queue`/`core_build_worker`). -- **Adjacent**: `scheduler-sim-eager-reconciliation` -- complementary mechanism, same symptom class the user originally reported, different root-cause layer. `scheduler-dispatch-commit` (owns `WAL_JOB_DISPATCH`, the controller-side fact bug-1678 extended to carry the allocation) -- no code dependency, but its bug-1678 fix is the direct working precedent for this spec's approach. - -## Constraints -- No capnp code in `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` -- existing architectural boundary, `snapshot.c`'s own header explains why. -- Must never alias the deep-copied allocation across two independently-alive cores -- the exact hazard the original scope decision flagged. Every restore must be a genuine deep copy, correctly paired with `free_JobAlloc_ptr` on cleanup. -- `xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()` both already enforce `session_active==0` (-EBUSY guard) -- confirmed no concurrent job-list mutation race during the deep copy. This fix must preserve that guarantee, not weaken it. -- `job->schedobj` reconstruction must exactly mirror `core_build_schedobj()`'s field set (user/queue/project strings, resreq-tag wrapper, empty `glimits`/`hlimits`, `configver=0`) -- no additional serialized data required or permitted, since everything it needs is already present after `job->resreq`/`user`/`queue`/`project` restoration earlier in the same reconstruction path. Do not call into `core_events.c`'s static builders; replicate locally. -- This spec is the confirmed instance motivating a new standing invariant, formalized alongside this brief: `xce-audit-invariants`' scheduler dimension 4 ("Field-complete snapshot save/load parity") and `persistence.md`'s "Snapshot Save/Load Field-Completeness (Scheduler)" section. Tasks generated for this spec should include running `/xce-audit-invariants scheduler` (or at minimum manually re-checking dimension 4) as part of final validation, not just the feature's own unit tests -- proving THIS field round-trips doesn't by itself prove no OTHER field has the same gap. diff --git a/.kiro/specs/scheduler-snapshot-job-allocation/design.md b/.kiro/specs/scheduler-snapshot-job-allocation/design.md deleted file mode 100644 index b603ac85..00000000 --- a/.kiro/specs/scheduler-snapshot-job-allocation/design.md +++ /dev/null @@ -1,290 +0,0 @@ -# Design Document: scheduler-snapshot-job-allocation - -## Overview - -This feature extends the scheduler's existing snapshot save/load mechanism (`snapshot.c`/`snapshot_wire.c`/`snapshot_load.c`, `scheduler-core` task 8.1) so that a job's resource allocation and internal scheduling representation survive a save/load round-trip. Today, `xce_sched_core_load_snapshot()` restores a job's `state` correctly but silently drops two fields the job depends on: its resource allocation (`job->allocs`/`job->accounts`) and its internal scheduling representation (`job->schedobj`). The first causes `info worker`'s "used" resource values to permanently undercount snapshot-restored running jobs; the second is a latent NULL-dereference crash risk in the allocator for any restored PENDING job. - -**Users**: operators running simulation-mode capacity/contention analysis via `xce_scheduler -i -s <replay>`, and any future consumer of the scheduler's snapshot format (e.g. the roadmap's planned `xce_test` `snapdump` tooling). - -**Impact**: extends `build_wire_job()` (`snapshot_wire.c`) and `xce_sched_core_load_snapshot()` (`snapshot_load.c`); adds one new function to `xcealloc.c`/`xcealloc.h`. No wire schema change — both fixes restore data using capacity the wire format or the load path already has. - -### Goals -- Restore `job->allocs`/`job->accounts` for a job already RUNNING at snapshot save time, so worker/shared resource "used" accounting is correct immediately on load (Req 1). -- Construct a valid, non-NULL `job->schedobj` for every restored job — PENDING or RUNNING — so it can reach the allocator without crashing (Req 2). -- Preserve every existing snapshot/reconciliation guarantee, including clean loading of snapshot files that predate this feature (Req 3). - -### Non-Goals -- Restoring `job->priority` or a queue's `project_name`/`job_pick_plugin` — existing, separate, already-documented `snapshot.c` scope decisions, unchanged. -- Capturing `job->rusages` — no wire slot exists for it at all today; closing that gap needs an actual schema addition, tracked as a separate follow-on. -- Any change to `sim_reconcile_unaccounted_jobs()` (`scheduler-sim-eager-reconciliation`) — it already correctly handles the complementary case (a dispatch decision discoverable later in replay) and is not modified here. -- Fixing `kill_sched_job()`'s incomplete per-job cleanup (it frees `job->rusages` only, not `resreq`/`schedobj`/`allocs`/`accounts`) — a pre-existing gap affecting every job today, live or restored; an unrelated, broader job-lifecycle concern (see `research.md` Risks). -- Any change to live, non-snapshot dispatch or resource-accounting behavior. - -## Boundary Commitments - -### This Spec Owns -- `dup_job_alloc()`: a new deep-copy function (`xcealloc.c`/`xcealloc.h`) that builds an independent `xce_job_alloc_t` from a source one. -- `build_wire_job()` save-side wiring: populating the wire type's already-existing `allocs` field from `job->allocs` when non-NULL. -- `xce_sched_core_load_snapshot()` load-side wiring: restoring `job->allocs`/`job->accounts` via the deep copy, and constructing `job->schedobj` for every restored job via a new local, static helper. -- Keeping `snapshot.c`'s own "Scope decisions" header current: correcting the now-stale `allocs`/`accounts` bullet and adding `job->rusages` as a newly-documented gap, so the file's own source-level documentation and this spec's actual behavior never diverge. - -### Out of Boundary -- `job->rusages`, `job->priority`, queue `project_name`/`job_pick_plugin` — see Non-Goals. -- `sim_reconcile_unaccounted_jobs()` and the wider simulation replay/decision pipeline. -- The controller's WAL (`persist.c`, bug-1678) — a separate, non-sharing persistence mechanism already fixed independently. -- `kill_sched_job()`'s cleanup completeness. -- Any capnp schema change — `Job.allocs`/`SchedJob.allocs` already exist (`controller.capnp:326,341`). -- The pre-existing resreq-parse-failure path itself (a job whose `resreq` fails to parse on load still reaches `sched_add_job()`/the picker today, independent of this feature) — this spec's `schedobj` guard tolerates that path without crashing on it, but does not close the narrow residual case where such a job is also PENDING and later reaches the allocator (see Components and Interfaces → Load-Side Job Reconstruction → Known limitation). - -### Allowed Dependencies -- `make_alloc`/`make_host_alloc`/`make_job_alloc`/`make_job_account` (`xcealloc.c`) — existing, unmodified. -- `free_JobAlloc_ptr` (generated, `controller.capnp.h:5467`) — existing, NULL-safe. -- `sched_get_user`/`sched_get_queue`/`sched_get_project` (`snapshot_load.c` already uses these for job reconstruction). - -### Revalidation Triggers -- Any change to `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t`'s field set (would require both `dup_job_alloc()` and the `JobAlloc`/`HostAlloc`/`ResAlloc` capnp schema to move together). -- Any change to `core_build_schedobj()`'s field set (`core_events.c`) — `snapshot_load.c`'s local mirror must be updated in lockstep, since it is a deliberate duplication, not a shared call. -- Any change to `xce_sched_obj_t`'s struct shape (`sched/types.h`). -- Any change to `xce_sched_core_load_snapshot()`'s or `build_wire_job()`'s function signature. - -## Architecture - -### Existing Architecture Analysis - -**Save path** (`snapshot.c:487-509`, `xce_sched_core_save_snapshot()`): iterates `core->jobs`, calls `build_wire_job(job)` per job (`snapshot_wire.c:118-171`). `build_wire_job()` currently maps `jobid`/timestamps/`state`/queue-user-project names/`mintasks`/`maxtasks`/`resreq` (as a formatted string) onto the wire `xce_job_t`, but leaves the wire type's `allocs` field unset even when `job->allocs != NULL` (`snapshot_wire.c:167`'s own comment flags this). - -**Load path** (`snapshot_load.c:283-340`, the job-reconstruction loop inside `xce_sched_core_load_snapshot()`): for each wire job `wj`, allocates a fresh `xce_sched_job_t`, restores `jobid`/timestamps/`state`, resolves `user`/`queue`/`project` via `sched_get_*()`, parses `resreq` from the wire's resreq string, calls `sched_add_job()`, then fires the picker's `on_job_push` hook. It never touches `job->schedobj`, `job->allocs`, or `job->accounts` — all three are left at their `calloc()`-zeroed NULL. - -**Direct precedent for both halves of this fix, already shipped**: -- bug-1569 (thread B) is the exact same `job->schedobj`-NULL-deref bug class, already fixed for the JOBS_PUSH path — `core_build_job()` (`core_events.c:199-266`) now unconditionally calls `core_build_schedobj()` (`core_events.c:174-197`) before the job becomes visible to the allocator, precisely because the allocator (`sched_make_allocation()` and its callees, `scheduler.c:179-267`) dereferences `job->schedobj` with no NULL check. -- bug-1678 is the controller-side WAL analog of the `job->allocs` gap (`WAL_JOB_DISPATCH` never persisted the allocation), fixed with pure save/load glue and no schema change — direct precedent that this shape of fix is viable without touching the wire schema. - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - subgraph SaveSide[Save Path] - CoreJobs[core jobs list] --> BuildWireJob[build_wire_job] - BuildWireJob --> WireJob[wire xce_job_t incl allocs] - end - - subgraph WireFormat[Snapshot Wire Format] - WireJob --> SnapshotData[sched_snapshot_data_t] - end - - subgraph LoadSide[Load Path] - SnapshotData --> LoadLoop[job reconstruction loop] - LoadLoop --> ResreqParse[parse resreq] - ResreqParse --> SchedobjBuild[local schedobj builder] - ResreqParse --> AllocCopy[dup_job_alloc] - AllocCopy --> AccountBuild[make_job_account] - SchedobjBuild --> NewJob[restored xce_sched_job_t] - AccountBuild --> NewJob - NewJob --> AddJob[sched_add_job] - AddJob --> Allocator[allocator and accounting] - end - - DupHelper[dup_job_alloc in xcealloc.c] --> AllocCopy -``` - -**Architecture Integration**: -- **Selected pattern**: pure save/load glue on both sides — no new abstraction layer, matching the existing `build_wire_*()`/reconstruction-loop pattern already used for every other snapshot entity. -- **Domain boundaries**: `xcealloc.c` owns the deep-copy primitive (already owns the `make_*` family it composes from); `snapshot_wire.c` owns save-side field population; `snapshot_load.c` owns load-side reconstruction — unchanged ownership, just closing a gap in the existing seams. -- **Existing patterns preserved**: no capnp code added to `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` (existing architectural boundary); job/queue/worker reconstruction in `snapshot_load.c` continues to avoid calling `core_events.c`'s static builders. -- **New components rationale**: `dup_job_alloc()` is the one genuinely new function — no existing deep-copy helper for `xce_job_alloc_t` exists anywhere in the codebase. The schedobj builder is a local duplication, not a new abstraction (see Design Decisions in `research.md`). -- **Steering compliance**: closes the confirmed instance motivating `persistence.md`'s "Snapshot Save/Load Field-Completeness (Scheduler)" section and `xce-audit-invariants`' scheduler dimension 4 — both already cite this spec. - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|------------------|-------| -| Scheduler core (C, gnu17) | existing `libxce` + scheduler codebase | All new/changed code | No new dependency | -| Wire format | Cap'n Proto, existing `controller.capnp` schema | `Job.allocs`/`SchedJob.allocs @7: JobAlloc` already present | Zero schema changes | - -## File Structure Plan - -### Modified Files -- `src/include/xcealloc.h` — add `extern xce_job_alloc_t* dup_job_alloc(const xce_job_alloc_t*);` declaration. -- `src/lib/xcealloc.c` — add `dup_job_alloc()`: deep-copies a `xce_job_alloc_t` (all hosts, all per-host allocations, all shared-resource allocations) by composing `make_alloc()`/`make_host_alloc()`/`make_job_alloc()`, explicitly re-stamping each copied `xce_alloc_t`'s `from`/`duration` from the source (see `research.md` for why `make_alloc()`'s own `time(NULL)` stamp is insufficient here). 207-line file today, comfortably under the 800-line cap after this addition. -- `src/scheduler/src/snapshot_wire.c` — `build_wire_job()` (currently 118-171): populate the wire job's `allocs` field via `dup_job_alloc(job->allocs)` when `job->allocs != NULL`, leaving it unset otherwise (identical shape to how every other optional field in this function is handled). -- `src/scheduler/src/snapshot.c` — update the file's own "Scope decisions" header comment (currently lines 234-268): rewrite the `job->allocs`/`job->accounts` bullet, which currently states they "are NOT captured," to reflect that they now are (captured/restored by this spec); add a new bullet documenting `job->rusages` as a known, deliberate gap (no wire slot exists, would need a capnp schema addition, tracked as a separate follow-on) so a future `/xce-audit-invariants scheduler` dimension-4 pass sees it as documented rather than rediscovering it as undocumented. `job->schedobj` needs no bullet — it is fixed, not a gap. -- `src/scheduler/src/snapshot_load.c` — `xce_sched_core_load_snapshot()`'s job-reconstruction loop (currently 283-340): - - new static helper `snapshot_build_schedobj(xce_sched_obj_t*, xce_job_t*, xce_resreq_t*)`, a field-for-field local port of `core_build_schedobj()` (`core_events.c:174-197`), including its three tag-dispatch helpers (`primary`/`compound`/`alter`) ported the same way `core_build_job()`'s own helpers already were. - - call site: after the existing resreq-parsing block, before `sched_add_job()` — allocate `job->schedobj`, call the new helper only when `job->resreq != NULL` (guards the pre-existing, unrelated resreq-parse-failure path without changing its behavior). - - call site: restore `job->allocs` via `dup_job_alloc(wj->allocs)` when `wj->allocs != NULL`, then `job->accounts = make_job_account(job->allocs)` — mirrors the exact pattern `sim_reconcile_unaccounted_jobs()` (`sim_loop.c`) already established for its own narrower case. - -### Test Files (new/extended) -- `src/lib/testsuite/test_xcealloc.c` — new `dup_job_alloc()` unit tests (this file already exists from this session's unrelated bug-1722 fix, wired into `xcelib_test`). -- `src/scheduler/testsuite/test_snapshot_save_load.c` — extend with cross-core round-trip tests covering `allocs`/`accounts`/`schedobj`, and the backward-compatibility case. -- `src/scheduler/testsuite/test_sim_e2e_original_motivation.c` — extend `test_e2e_original_motivation_long_replay` (the test that already reproduces the originally-reported "info worker only reflects 2 of 22 running jobs" symptom) to assert the fix. - -## System Flows - -```mermaid -sequenceDiagram - participant Loader as xce_sched_core_load_snapshot - participant Job as restored xce_sched_job_t - participant Helper as snapshot_build_schedobj - participant Dup as dup_job_alloc - participant Acct as make_job_account - - Loader->>Job: allocate, restore jobid/state/timestamps/user/queue/project - Loader->>Job: parse resreq from wire string - alt resreq parsed successfully - Loader->>Helper: build schedobj(job->resreq, wj->submit->user/queue/project) - Helper-->>Job: job->schedobj populated - else resreq parse failed (pre-existing, unrelated gap) - Loader-->>Job: job->schedobj left NULL, unchanged from today - end - alt wj->allocs present - Loader->>Dup: dup_job_alloc(wj->allocs) - Dup-->>Job: job->allocs populated (independent copy) - Loader->>Acct: make_job_account(job->allocs) - Acct-->>Job: job->accounts populated - else wj->allocs absent (old snapshot or never-dispatched job) - Loader-->>Job: job->allocs/accounts left NULL, same as today - end - Loader->>Loader: sched_add_job(core, job) - Loader->>Loader: picker on_job_push hook -``` - -Both new construction steps run before `sched_add_job()` so no code can observe a job with a NULL `schedobj` (for a job with valid `resreq`) or a stale/aliased `allocs` after this feature lands — the ordering mirrors `core_build_job()`'s own (build fully, then register). - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1, 1.2, 1.3 | Restore a running job's allocation, reflected in accounting immediately | `dup_job_alloc`, load-side wiring | `dup_job_alloc()`, `make_job_account()` | Load sequence, "wj->allocs present" branch | -| 1.4 | Restored allocation independent of the source core | `dup_job_alloc` | `dup_job_alloc()` | Architecture: deep-copy pattern decision | -| 2.1, 2.2, 2.4 | Every restored job gets a usable scheduling representation | `snapshot_build_schedobj` | local static helper | Load sequence, schedobj branch | -| 2.3 | schedobj built from data the snapshot already captures, no new wire data | `snapshot_build_schedobj` | uses `job->resreq`, `wj->submit->*` | Architecture: Existing Architecture Analysis | -| 3.1, 3.2, 3.3, 3.4 | No regression to reconciliation, other fields, documented gaps, live path | (no new component — verified by leaving these paths untouched) | n/a | Non-Goals | -| 3.5 | Old/absent-allocation snapshots load cleanly | load-side wiring | `wj->allocs != NULL` guard | Load sequence, "wj->allocs absent" branch | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|--------------|--------|---------------|---------------------------|-----------| -| `dup_job_alloc` | `xcealloc.c` (libxce) | Deep-copy a job's resource allocation, independent of the source | 1.1, 1.4 | `make_alloc`/`make_host_alloc`/`make_job_alloc` (P0) | Service | -| Save-side wiring | `snapshot_wire.c` `build_wire_job()` | Populate the wire job's `allocs` field | 1.1 | `dup_job_alloc` (P0) | Service | -| Load-side allocation restore | `snapshot_load.c` `xce_sched_core_load_snapshot()` | Restore `job->allocs`/`job->accounts` | 1.1, 1.2, 1.3, 1.4, 3.5 | `dup_job_alloc` (P0), `make_job_account` (P0) | Service | -| `snapshot_build_schedobj` | `snapshot_load.c` (local, static) | Construct `job->schedobj` for every restored job | 2.1, 2.2, 2.3, 2.4 | `job->resreq` (P0, must already be parsed) | Service | - -### Persistence / Scheduler Snapshot Domain - -#### `dup_job_alloc` - -| Field | Detail | -|-------|--------| -| Intent | Build a fresh, independent `xce_job_alloc_t` from a source one, for safe cross-core snapshot restoration | -| Requirements | 1.1, 1.4 | - -**Responsibilities & Constraints** -- Deep-copies every host allocation and every per-host resource allocation, plus any shared-resource allocations (`nres_shared`/`allocs_shared`). -- Never aliases any pointer from the source — every string is independently duplicated (via the existing `make_*` constructors' own `strdup()` calls), every struct independently `calloc()`'d. -- Explicitly re-stamps each copied `xce_alloc_t`'s `from`/`duration` from the source, since `make_alloc()` itself stamps `from = time(NULL)` rather than accepting the caller's original value (see `research.md`). -- Returns `NULL` on any allocation failure partway through; caller is responsible for treating a `NULL` return the same as "no allocation data" (Req 3.5's degrade path), not as a fatal error. A rare OOM partway through the copy may leak the partially-built pieces — this matches this file's own existing convention (`make_host_account()`/`make_job_account()` do not unwind partial state on OOM either; `xce_sched_core_save_snapshot()`'s own `arr_append()` failure handling just `break`s without freeing prior entries). Full rollback-on-OOM is a deliberately rejected stronger guarantee: OOM-during-snapshot-load is not a scenario this codebase invests in recovering from cleanly anywhere else in this file family, and a leak on that path does not violate Req 1.4's independence/no-corruption guarantee (independence and freedom from leaks are separate properties). - -**Dependencies** -- Outbound: `make_alloc`, `make_host_alloc`, `make_job_alloc` (`xcealloc.c`, P0 — the copy is composed entirely from these). - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -xce_job_alloc_t* dup_job_alloc(const xce_job_alloc_t* src); -``` -- Preconditions: `src` may be `NULL` (returns `NULL`); if non-NULL, `src->nhosts >= 0` and `src->allocs` has exactly `nhosts` entries (the existing wire/constructor invariant). -- Postconditions: on success, returns a new `xce_job_alloc_t*` with no pointer shared with `src`, including `from`/`duration` on every copied `xce_alloc_t`. On failure (partial or total allocation exhaustion), returns `NULL`; a partially-built copy may leak on this path, matching this file's existing OOM-handling convention (see Responsibilities & Constraints) rather than promising full rollback. -- Invariants: the returned value, once passed to `free_JobAlloc_ptr()`, cleans up exactly as any other `xce_job_alloc_t` — no new free path is introduced. - -**Implementation Notes** -- Integration: called from both `build_wire_job()` (save side, copying a live job's allocation into the wire struct) and `xce_sched_core_load_snapshot()` (load side, copying the wire struct's allocation into the restored job) — same function, both directions, since both need the identical independence guarantee. -- Validation: covered by the persistence-completeness test matrix below (multi-host, multi-resource, shared-resource, and independence-after-mutation cases). -- Risks: none beyond the `from`/`duration` fidelity nuance already resolved in `research.md`. - -#### Load-Side Job Reconstruction (`snapshot_load.c`) - -| Field | Detail | -|-------|--------| -| Intent | Restore `job->allocs`/`job->accounts`/`job->schedobj` as part of the existing per-job reconstruction loop | -| Requirements | 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 2.4, 3.5 | - -**Responsibilities & Constraints** -- Must run before `sched_add_job()` so no other code ever observes a partially-reconstructed job (see System Flows ordering). -- Must guard `schedobj` construction on `job->resreq != NULL` — the pre-existing resreq-parse-failure path is unchanged, not fixed, by this feature. -- Must treat `wj->allocs == NULL` (old snapshot file, or a job that was never dispatched) as the normal, non-error case — no WARN, no behavior change from today. -- **Known limitation, explicitly out of boundary (not closed by this spec)**: a PENDING job whose `resreq` fails to parse on load (pre-existing gap — e.g. the target core's `resmgr` lacks a resource type the source core had, plausible specifically in the cross-core scenarios Req 1.4 exercises) still reaches `sched_add_job()`/the picker with `job->schedobj == NULL`, since this component's guard skips schedobj construction whenever `resreq` is NULL. If that job is later selected by the picker, it hits the same bug-1569-class NULL-deref this feature otherwise closes. This is not a new regression — every restored job hit this before this feature, unconditionally — but it means Requirement 2.2 is closed for the general case (successful resreq parse) and not for this narrow, already-broken sub-case. Fixing it requires changing the pre-existing resreq-parse-failure path itself (e.g. not registering such a job with the picker at all), which is a distinct, unrelated defect and explicitly out of this spec's boundary — tracked here rather than silently assumed away. - -**Dependencies** -- Inbound: `xce_sched_core_load_snapshot()`'s existing job loop (P0 — this is an extension of it, not a new call site). -- Outbound: `dup_job_alloc` (P0), `make_job_account` (P0), the new `snapshot_build_schedobj` local helper (P0). - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -static void snapshot_build_schedobj(xce_sched_obj_t* obj, xce_job_t* wj, - xce_resreq_t* resreq); -``` -- Preconditions: `obj` freshly `calloc()`'d (zeroed); `resreq` already successfully parsed; `wj` the wire job carrying `wj->submit->user/queue/project_name`. -- Postconditions: `obj->tag`/`obj->u.{primary,compound,alter}` set per `resreq->tag`; `obj->user`/`queue`/`project` are fresh string copies; `obj->glimits`/`hlimits` initialized empty; `obj->configver = 0`; `obj->alloc`/`obj->reason` left NULL (correct: both are per-scheduling-round scratch state, never persisted job state — see `research.md`). -- Invariants: field-for-field identical output to `core_build_schedobj()` given the same inputs — any future divergence is a bug, not an accepted difference (Revalidation Trigger above). - -**Implementation Notes** -- Integration: called once per restored job, immediately after resreq parsing succeeds. -- Validation: covered by the "restored PENDING job reaches the allocator without crashing" test (Testing Strategy below) — the direct regression test for bug-1569's failure shape. -- Risks: duplicated logic could drift from `core_build_schedobj()` over time; mitigated by the Revalidation Trigger and `/xce-audit-invariants scheduler` dimension 4. - -## Data Models - -### Snapshot Entity Field-Completeness (Persistence Review) - -Per-field disposition for every field this feature touches, per `persistence.md`'s "Snapshot Save/Load Field-Completeness (Scheduler)" standing invariant: - -| Entity | Field | Captured on save? | Restored on load? | Disposition | -|--------|-------|--------------------|--------------------|-------------| -| `xce_sched_job_t` | `allocs` | Yes (this feature, `dup_job_alloc` via `build_wire_job`) | Yes (this feature) | Fixed by this spec | -| `xce_sched_job_t` | `accounts` | N/A (derived, never itself serialized) | Yes (this feature, via `make_job_account(job->allocs)`) | Fixed by this spec | -| `xce_sched_job_t` | `schedobj` | N/A (derived from `resreq`/user/queue/project, never itself serialized) | Yes (this feature, `snapshot_build_schedobj`) | Fixed by this spec | -| `xce_alloc_t` | `from`, `duration` | Yes (wire round-trip, pre-existing) | Yes (this feature explicitly re-stamps both after `make_alloc()`) | Fixed by this spec (functionally inert today, but field-complete) | -| `xce_sched_job_t` | `priority` | No | No | Pre-existing, documented gap (`snapshot.c` Scope Decisions) — unchanged | -| `xce_sched_job_t` | `rusages` | No (no wire slot exists) | No | Pre-existing gap, undocumented at the source until this spec — explicitly out of scope, needs a schema addition (see Non-Goals); this spec adds it to `snapshot.c`'s own "Scope decisions" header so it reads as documented, not rediscovered, on the next audit (see File Structure Plan) | -| `xce_sched_queue_t` | `project_name`, `job_pick_plugin` | No | No | Pre-existing, documented gap — unchanged | - -### Domain Model -No new domain concepts — `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t`/`xce_job_account_t`/`xce_sched_obj_t` are all pre-existing structs (`sched/types.h`, `xce/types.h`). This feature restores existing relationships (job → allocation → per-host/per-resource accounts; job → scheduling representation) across a save/load boundary, it does not introduce new ones. - -## Error Handling - -### Error Strategy -This is an internal, in-process reconstruction path with no external I/O of its own — errors are allocation failures (OOM) or pre-existing malformed-input states, both handled by degrading to today's behavior rather than crashing or aborting the load. - -### Error Categories and Responses -- **Allocation failure inside `dup_job_alloc()`**: returns `NULL`; caller leaves `job->allocs`/`job->accounts` at NULL — identical to the "old snapshot, no allocation data" case (Req 3.5). Not treated as a load failure. -- **`job->resreq == NULL` at the schedobj-construction call site** (pre-existing, unrelated resreq-parse-failure path): `snapshot_build_schedobj()` is not called; `job->schedobj` stays NULL, identical to today's behavior for this already-existing edge case. Not a new regression — this feature does not change what happens to a job whose resreq fails to parse, only what happens to a job whose resreq parses successfully. **Known limitation**: if such a job is PENDING and later reaches the allocator, it still hits the bug-1569-class crash this feature otherwise closes — see Components and Interfaces → Load-Side Job Reconstruction for why fixing that sub-case is out of this spec's boundary. -- **`wj->allocs == NULL`** (old snapshot file, or a job that was pending/never dispatched at save time): no restoration attempted, no WARN — this is the normal case, not an error (Req 3.5). - -### Monitoring -No new logging beyond what's necessary to debug a `dup_job_alloc()` allocation failure (a `WARN`, consistent with how every other optional-field restoration failure in `xce_sched_core_load_snapshot()` is already logged, e.g. the existing worker-resval-restoration `WARN` at `snapshot_load.c:200-204`). - -## Testing Strategy - -- **Unit — `dup_job_alloc()` (`src/lib/testsuite/test_xcealloc.c`)**: single-host/single-resource copy; multi-host copy; multi-resource-per-host copy; shared-resource (`nres_shared`/`allocs_shared`) copy; independence proof (mutate/free the source after copying, confirm the copy is unaffected — the direct test of the Req 1.4 guarantee); `from`/`duration` fidelity (copy preserves the source's exact original values, not `make_alloc()`'s own `time(NULL)` stamp). -- **Integration — cross-core snapshot round-trip (`test_snapshot_save_load.c`)**: save a core with a mix of RUNNING jobs (multi-host, multi-resource, shared-resource allocations) and PENDING jobs, load into a second, independent core; assert `job->allocs`/`job->accounts` match the source for RUNNING jobs, `job->schedobj` is non-NULL for every job, and freeing either core's job list does not corrupt or crash the other (Req 1.4). Extend with a snapshot containing no allocation data at all (the natural NULL case, not an archived legacy file — see Requirement 3's backward-compat criterion) and confirm it loads cleanly with `used=0`. -- **Regression — restored job reaches the allocator (`test_snapshot_save_load.c` or a new case in the same file)**: load a snapshot containing a PENDING job, run it through `sched_make_allocation()` (or the picker→allocator path), assert no crash/UB — the direct regression test for the bug-1569-class failure this feature closes (Req 2.2). -- **E2E — sim-mode "used" accounting (`test_sim_e2e_original_motivation.c`, extending `test_e2e_original_motivation_long_replay`)**: reproduce the originally-reported scenario (a snapshot with multiple running jobs, replayed forward) and assert `info worker`'s "used" values correctly reflect every running job's contribution immediately after load, not just the ones whose dispatch also appears later in replay. -- **No regression — existing reconciliation (`test_sim_reconcile_unaccounted.c`, `test_sim_snapshot_reconcile_gap.c`)**: run unchanged; both must continue passing exactly as today, confirming `sim_reconcile_unaccounted_jobs()`'s complementary case is untouched (Req 3.1). - -## Constraints (Invariant Cross-Check) - -- **No capnp code in `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c`** — preserved; `dup_job_alloc()` is composed entirely from existing plain-C constructors, no capnp codec call added to these three files. -- **No aliasing across independently-alive cores** — preserved; `dup_job_alloc()` is a genuine deep copy, verified by the independence unit test above. -- **`session_active == 0` (`-EBUSY`) guard on both save and load** — unchanged; this feature adds no new mutation path outside the existing per-job reconstruction loop, which already runs entirely under this guard. - -**`xce-audit-invariants` scheduler dimension walkthrough** (all four dimensions checked against this design, per the standing dimension list in the `xce-audit-invariants` skill): - -1. **Driver state-equivalence** (`scheduler-driver-equivalence.md`) — not implicated. All call sites of `xce_sched_core_load_snapshot()` are in `sim.c`; normal-mode `driver.c` never loads a snapshot, so there is no normal-mode/sim-mode divergence to introduce. This fix arguably *strengthens* dimension 1: restored jobs now get their accounting via the same `make_job_account()` call the live dispatch path already uses, rather than diverging from it as they do today (no `accounts` at all). -2. **Core module I/O isolation** (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c`, mechanically enforced by `test_core_isolation.c`) — not implicated. Every changed/new file (`xcealloc.c`, `snapshot_wire.c`, `snapshot_load.c`, `snapshot.c`) is outside the isolated core-family set; no I/O is added to any of them, and none of them are added to or removed from `sched_core_isolation_lib`'s membership. -3. **Picker→allocator `jbuf` capacity-1 rendezvous** (`scheduler/CLAUDE.md`) — not implicated. Snapshot load runs before/outside any scheduling session round; nothing in this design touches `scheduler.c`'s handoff buffer, raises its capacity, or adds a second producer/consumer. -4. **Field-complete snapshot save/load parity** (`persistence.md`'s "Snapshot Save/Load Field-Completeness (Scheduler)"; this spec is dimension 4's own confirmed instance) — directly addressed. The Data Models field table above enumerates every field this spec touches or is aware of, correctly separating documented from undocumented gaps; `snapshot.c`'s own header is kept current (File Structure Plan) so the next audit reads `rusages` as documented rather than rediscovering it. A full re-audit of untouched entities (worker/user/project) is deliberately not duplicated here — that is what running `/xce-audit-invariants scheduler` (or at minimum re-checking dimension 4) as final task validation is for, per the brief's own Constraints section. diff --git a/.kiro/specs/scheduler-snapshot-job-allocation/requirements.md b/.kiro/specs/scheduler-snapshot-job-allocation/requirements.md deleted file mode 100644 index 2f5aefde..00000000 --- a/.kiro/specs/scheduler-snapshot-job-allocation/requirements.md +++ /dev/null @@ -1,41 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Engineers using `xce_scheduler -i -s <replay>` to run simulation-mode capacity/contention analysis from a live snapshot + replay -- the common investigation workflow -- currently get silently wrong results for any job that was already RUNNING as of the snapshot's save point. `xce_sched_core_load_snapshot()` restores such a job's `state` correctly (so it shows as RUN), but never restores `job->allocs`/`job->accounts` (`build_wire_job()` in `snapshot_wire.c` carries an explicit `/* job->allocs/accounts: not captured */` comment) -- this was a deliberate, documented scope decision from `scheduler-core` task 8.1, made because no deep-copy helper for `xce_job_alloc_t` existed and aliasing it across two independently-alive cores would create a double-free hazard. As a result, `sched_account_jobs()` (which recomputes every worker's `resInUse` each round by walking jobs with `JOB_IS_RUN(state) && accounts != NULL`) never charges these jobs' resources, so `info worker`'s "used" values permanently undercount snapshot-restored running jobs -- reported live as: after loading a snapshot with 22 running jobs and replaying 79 sessions forward, `info summary` correctly counts 22 running jobs but `info worker`'s "used" resource values only ever reflect 2 of them. This should change so that a job already running as of a snapshot's save point survives the save/load round-trip with its full allocation intact -- `job->allocs` populated via a new deep-copy helper (composed from the existing `make_alloc`/`make_host_alloc`/`make_job_alloc` constructors, paired with `free_JobAlloc_ptr` for cleanup, never aliasing the source snapshot's own copy), `job->accounts` reconstructed via the existing `make_job_account()`, and worker/shared resource "used" accounting correct from the moment the snapshot loads -- while a snapshot file saved before this change still loads cleanly with today's `used=0` behavior for its jobs (clean degrade, not a regression). The wire type already has an `allocs` field mirroring the C struct with no schema change needed, so this is confined to save-side wiring in `build_wire_job()` (`snapshot_wire.c`) and load-side wiring in `xce_sched_core_load_snapshot()` (`snapshot_load.c`), following the exact reconstruction pattern `sim_reconcile_unaccounted_jobs()` (`scheduler-sim-eager-reconciliation`) already established for the complementary case (a dispatch decision discoverable later in replay) -- this spec closes the other half, where the dispatch decision predates the snapshot and can never appear in the subsequent replay stream. - -A second, more severe defect in the same load path was found and folded into this spec's scope during a cross-component persistence survey: every job restored from a snapshot -- PENDING or RUNNING -- gets a NULL `job->schedobj`, a field the live allocator (`sched_make_allocation()` and its callees in `scheduler.c`, plus plugins like `limit.c`) dereferences unconditionally, the same failure shape as the already-fixed bug-1569 SIGSEGV, reproducible the moment a restored PENDING job reaches the allocator. Unlike `allocs`, this needs no new captured data at all -- `core_build_schedobj()` (the live construction path, `core_events.c`) builds `schedobj` purely from the job's `user`/`queue`/`project` names and its `resreq` (already parsed from the resreq string this spec's load path already restores), plus freshly-initialized, session-scoped empty bookkeeping (`glimits`/`hlimits`) and reset defaults (`configver=0`, a NULL `alloc`/`reason` that are themselves confirmed to be pure per-scheduling-round scratch state, never persisted job state, in the live path too). The fix is to locally replicate `core_build_schedobj()`'s exact field-for-field logic inside `xce_sched_core_load_snapshot()`'s job-reconstruction loop for every restored job -- `core_build_schedobj()` itself is `static` to `core_events.c` and cannot be called directly, consistent with the existing convention that `snapshot_load.c` never shares helpers with `core_events.c`'s builders. - -## Boundary Context -- **In scope**: capturing and restoring a running job's resource allocation across a snapshot save/load cycle, so worker and shared resource utilization is correct immediately after loading a snapshot; reconstructing every restored job's internal scheduling representation (regardless of whether the job is pending or running) so it can be evaluated by the scheduler's normal allocation logic without failure; loading snapshot files created before this feature existed without error or behavior change for the jobs they contain. -- **Out of scope**: restoring a job's priority, or a queue's owning-project/picker-plugin-override fields (pre-existing, separate, already-documented snapshot gaps, unchanged by this feature); capturing a job's latest controller-reported resource-usage report in a snapshot (a distinct, currently-uncaptured piece of job state that would require the snapshot format itself to gain new capacity, unlike the fields this feature restores) -- tracked as a separate future feature; any change to how simulation-mode reconciles a job's allocation from a dispatch decision appearing later in a replay stream (an existing, separate mechanism this feature does not modify); any change to live, non-snapshot job dispatch or resource-accounting behavior. -- **Adjacent expectations**: this feature is a direct extension of the scheduler's existing snapshot save/load capability -- it relies on that capability's existing file format and loading behavior remaining otherwise unchanged. It complements the existing replay-based reconciliation mechanism (which recovers a job's allocation when its dispatch decision appears later in a replay): together, a job's allocation should be recoverable regardless of whether its dispatch happened before the snapshot was taken or is discoverable later in replay. - -## Requirements - -### Requirement 1: Resource allocation restoration for jobs already running at snapshot time -**Objective:** As an operator running simulation-mode capacity/contention analysis from a snapshot, I want a job that was already running when the snapshot was taken to keep its resource allocation after the snapshot loads, so that worker and shared resource utilization I inspect is accurate from the moment the snapshot loads, not only for jobs whose dispatch happens to also appear later in replay. - -#### Acceptance Criteria -1. When the scheduler loads a snapshot containing a job in a running state that has recorded resource-allocation data, the scheduler shall restore that job's resource allocation as part of loading the snapshot. -2. When a running job's resource allocation is restored while loading a snapshot, the scheduler shall include that job's resource consumption in its assigned worker's and any shared resource's in-use totals immediately, without requiring any further replay activity. -3. The scheduler shall report a restored running job's resource-in-use contribution identically to how it reports the same contribution for a job whose allocation was established through normal dispatch. -4. While a snapshot has been loaded into a scheduler instance, the scheduler shall treat the restored resource-allocation data as independent of the snapshot-producing instance's own data, such that later activity or shutdown in either instance does not corrupt or crash the other. - -### Requirement 2: Valid scheduling representation for every restored job -**Objective:** As an operator running simulation-mode replay from a snapshot, I want every job loaded from a snapshot -- pending or running -- to be fully usable by the scheduler's normal allocation logic, so that replay does not crash or misbehave when the scheduler evaluates a restored pending job for placement. - -#### Acceptance Criteria -1. When the scheduler loads any job from a snapshot, the scheduler shall construct that job's internal scheduling representation so the job is usable by the scheduler's allocation logic. -2. When a pending job restored from a snapshot is evaluated for resource allocation, the scheduler shall process it without crashing or producing undefined behavior. -3. The scheduler shall reconstruct a restored job's scheduling representation entirely from information the snapshot format already captures (its queue, user, project, and resource request), without requiring any additional information to be present in the snapshot file. -4. The scheduler shall treat a job restored from a snapshot the same as a normally-submitted job with the same queue, user, project, and resource request, for the purposes of allocation evaluation. - -### Requirement 3: No regression to existing snapshot, reconciliation, and accounting behavior -**Objective:** As a maintainer relying on the scheduler's existing snapshot and reconciliation guarantees, I want this feature to leave them intact, so existing workflows, tests, and documented scope decisions continue to hold. - -#### Acceptance Criteria -1. The scheduler shall continue to correctly reconcile a job's resource allocation through the existing replay-based reconciliation mechanism for any job whose dispatch decision appears within the replay stream. -2. The scheduler shall continue to save and restore every other previously-supported job, queue, worker, and session field in a snapshot exactly as before, with no change in behavior. -3. The scheduler shall continue to not restore a job's priority or a queue's owning-project/picker-plugin-override fields, consistent with the scheduler's existing, separately-documented scope decisions. -4. While loading a snapshot, the scheduler shall not alter its live, non-snapshot job dispatch or resource-accounting behavior. -5. While a snapshot does not contain resource-allocation data for a given job (for example, one saved before this feature existed), the scheduler shall load it successfully and report that job's resource consumption as it does today (no contribution), rather than treating the absence as an error condition. diff --git a/.kiro/specs/scheduler-snapshot-job-allocation/research.md b/.kiro/specs/scheduler-snapshot-job-allocation/research.md deleted file mode 100644 index 57542fd1..00000000 --- a/.kiro/specs/scheduler-snapshot-job-allocation/research.md +++ /dev/null @@ -1,73 +0,0 @@ -## Summary -- **Feature**: `scheduler-snapshot-job-allocation` -- **Discovery Scope**: Extension (existing scheduler snapshot save/load subsystem, `scheduler-core` task 8.1) -- **Key Findings**: - - `core_build_schedobj()` (`core_events.c:174-197`) is a pure function of data `xce_sched_core_load_snapshot()`'s job loop already has in scope by the time this feature's insertion point runs — no new captured/wire data is needed for `job->schedobj`, only a local, mechanical port of the same 3-field construction (mirrors `scheduler-core`'s own no-shared-helper convention already used for job/queue/worker reconstruction). - - This exact bug class (allocator dereferences `job->schedobj` unconditionally, no NULL check) already happened once and was fixed: bug-1569 thread B, `core_events.c:86-109`'s own header comment documents it verbatim as the reason `core_build_schedobj()` exists at all. `snapshot_load.c` was never updated to carry the same fix — it is the one remaining code path with the identical gap. - - `xce_sched_core_load_snapshot()`'s `in` parameter is `const sched_snapshot_data_t*` (`snapshot_load.c:59`) and, per `sim.c`'s 4 call sites, the same decoded snapshot can plausibly be loaded into more than one core. This rules out the ownership-transfer/MOVE pattern `sim_reconcile_unaccounted_jobs()` uses (`sim_loop.c:635-637`, safe there only because its source is a single-use, about-to-be-freed replay decision) — the allocation restore here must be a genuine deep copy. - - `xce_alloc_t.from`/`.duration` (`xcealloc.c:30-31`) are write-only today — grepped the entire tree, confirmed the only reads are wire round-trip test assertions (`test_e2e_capnp.c`), no live scheduling/accounting logic consumes either field. `make_alloc()` itself stamps `from = time(NULL)` rather than accepting it as a parameter, so a naive deep-copy built by calling `make_alloc()` and stopping there would silently reset every restored allocation's `from` to snapshot-load time. Functionally inert today, but a genuine per-field fidelity gap the persistence review gate would flag — cheap to close (one extra assignment per copied `xce_alloc_t`), so the design closes it rather than accepting the drift. - - `kill_sched_job()` (`schedobj.c:17-28`), the only per-item destructor registered for `core->jobs` (`core.c:93`), frees `job->rusages` only — it does not free `job->resreq`, `job->schedobj` (no `reset_sched_obj()` call), `job->allocs`, or `job->accounts`. This is a pre-existing gap affecting every job today (live-dispatched or restored), not introduced by this feature. Out of scope: fixing job cleanup is a job-lifecycle concern independent of snapshot restoration, would affect all jobs not just restored ones, and Requirement 2.4 ("treat a restored job the same as a normally-submitted job") is satisfied by structural parity, leak included, not by this spec silently fixing an unrelated defect. - - capnp schema confirms both `Job.allocs @7: JobAlloc` and `SchedJob.allocs @7: JobAlloc` exist with matching field numbers (`src/schema/controller.capnp:326,341`), and `JobAlloc`/`HostAlloc`/`ResAlloc` map directly onto `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t` with no translation layer — zero schema changes required, confirming the brief's viability check. - - `xce_sched_core_load_snapshot()`'s existing job-reconstruction loop (`snapshot_load.c:288-340`) already lets a job with a resreq-parse failure proceed to `sched_add_job()` with `job->resreq == NULL` (pre-existing, unrelated gap, not fixed here). Both new construction steps (schedobj, allocs) must tolerate this without crashing — schedobj construction is naturally skipped when `job->resreq == NULL` (mirrors `core_build_schedobj()`'s own precondition), and allocs restoration is independent of `resreq` entirely. - -## Research Log - -### Exact insertion point and construction inputs for `job->schedobj` -- **Context**: Requirement 2 needs every restored job to get a valid `job->schedobj`. Needed to confirm it's genuinely derivable with no new captured data, and find the precise insertion point. -- **Sources Consulted**: `core_events.c:174-266` (`core_build_schedobj`, `core_build_primary/compound/alter_schedobj`, `core_build_job`), `schedobj.c:17-211` (`make_sched_obj`, `kill_sched_job`, sibling live-path implementation), `snapshot_load.c:58-371` (full `xce_sched_core_load_snapshot` body), `sched/types.h:30-116` (`xce_sched_obj_t`, `xce_sched_job_t` struct shapes). -- **Findings**: `core_build_schedobj(obj, j, resreq)` needs only `j->submit->user/queue/project_name` (raw wire strings — `snapshot_load.c`'s loop variable `wj` is the exact same `xce_job_t*` type) and `resreq` (`job->resreq`, already parsed by the time the loop reaches line 332). `glimits`/`hlimits` start empty; `configver`/`alloc`/`reason` start at zero/NULL in the live path too (confirmed `obj->alloc`'s only writers, `scheduler.c:194,228,257`, are pure per-round allocator scratch state, never persisted job state). -- **Implications**: The fix is a local, static helper in `snapshot_load.c` replicating `core_build_schedobj()`'s exact 3-part body, called right after the existing resreq-parsing block (after `snapshot_load.c:332`) and before `sched_add_job()` (line 334) — guarded on `job->resreq != NULL`. - -### Deep-copy vs. move semantics for `job->allocs` -- **Context**: `sim_reconcile_unaccounted_jobs()` already reconstructs `job->allocs`/`job->accounts` from a replay decision using MOVE_PTR ownership transfer, not a deep copy. Needed to confirm whether the same pattern applies here. -- **Sources Consulted**: `sim_loop.c:574-716` (`sim_reconcile_unaccounted_jobs`, full body incl. its ownership-hazard comment), `snapshot_load.c:58-60` (`xce_sched_core_load_snapshot` signature — `const sched_snapshot_data_t* in`), caller count from CodeGraph (`sim.c`, 4 call sites). -- **Findings**: `sim_reconcile_unaccounted_jobs()`'s source (`entry->body.dispatch->alloc`) is a single-use, about-to-be-freed replay decision — MOVE is safe there specifically because nothing else will ever reference it again. `xce_sched_core_load_snapshot()`'s source is a `const`-qualified snapshot struct with multiple call sites in `sim.c`, i.e., plausibly loaded into more than one core or inspected again after load. -- **Implications**: Confirms the brief's original approach — a genuine deep copy (composed from `make_alloc`/`make_host_alloc`/`make_job_alloc`), never an alias/move, paired with `free_JobAlloc_ptr` for cleanup. - -### Field fidelity of the deep copy: `xce_alloc_t.from`/`.duration` -- **Context**: `make_alloc()` (the constructor the deep-copy composes from) sets `->from = time(NULL)` internally rather than accepting the source's original value. Needed to determine whether this matters. -- **Sources Consulted**: `grep -rn "\.from\b\|->from\b"` / `"\.duration\b\|->duration\b"` across `src/` for `xce_alloc_t` usage. -- **Findings**: Both fields are write-only in live code (`xcealloc.c`'s own constructor) — the only reads are capnp wire round-trip test assertions (`test_e2e_capnp.c:794,896-897`), never live scheduling/accounting logic. -- **Implications**: Functionally inert today, but a real field-fidelity gap if left unaddressed — the deep-copy explicitly re-sets `->from`/`->duration` to the source's original values after calling `make_alloc()`, rather than accepting `make_alloc()`'s own `time(NULL)` stamp. Cheap, and keeps the copy a genuine field-complete mirror rather than a "doesn't matter yet" shortcut. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| Deep-copy via existing `make_*` constructors (selected) | Compose `dup_job_alloc()` from `make_alloc`/`make_host_alloc`/`make_job_alloc`, explicitly re-stamping `from`/`duration` | Reuses proven, tested constructors; correct independence guarantee; no capnp code added to snapshot files | One new function + explicit field re-stamp | Matches bug-1678's controller-side precedent (pure save/load glue, no schema change) | -| Alias/share the source snapshot's `xce_job_alloc_t*` directly | Just assign `job->allocs = wj->allocs` | Zero new code | Double-free the moment either core's normal job-lifecycle path frees it, or a `const`-violation if `wj` is truly shared across cores | Rejected — this is the exact hazard `scheduler-core`'s original scope decision flagged | -| Reuse `sim_reconcile_unaccounted_jobs()`'s MOVE_PTR pattern | Move the wire struct's pointer into the job, null the source | Less code than a full deep copy | Requires `wj` to be non-const and single-use; `in` is `const` and reusable | Rejected — wrong ownership model for this call site (see Research Log) | -| Call `core_build_schedobj()` directly from `snapshot_load.c` | Reuse the live construction function instead of duplicating it | No duplicated logic | `core_build_schedobj()` is `static` to `core_events.c`; calling it would violate the existing, deliberate no-shared-helper boundary between `core_events.c` and `snapshot_load.c` | Rejected — local replication chosen instead, consistent with existing convention | - -## Design Decisions - -### Decision: Local, static `schedobj` builder in `snapshot_load.c` rather than exporting `core_build_schedobj()` -- **Context**: `job->schedobj` needs identical construction logic in two files that must not share helpers. -- **Alternatives Considered**: - 1. Export `core_build_schedobj()` from `core_events.c` and call it from `snapshot_load.c`. - 2. Duplicate the construction logic locally in `snapshot_load.c` as a new static helper. -- **Selected Approach**: Option 2 — a new static helper in `snapshot_load.c`, field-for-field identical to `core_build_schedobj()`. -- **Rationale**: `scheduler-core`'s existing, documented convention (cited in `snapshot.c`'s own header) deliberately avoids `snapshot_load.c` depending on `core_events.c`'s static, unexported builders — the same reasoning that already applies to job/queue/worker reconstruction in this file. -- **Trade-offs**: Small duplication (roughly 25 lines) vs. preserving an existing, intentional architectural boundary. Duplication accepted. -- **Follow-up**: If `core_build_schedobj()`'s field set ever changes, `snapshot_load.c`'s mirror must be updated in lockstep (recorded as a Revalidation Trigger in `design.md`). - -### Decision: Genuine deep copy for `job->allocs`, with explicit `from`/`duration` re-stamping -- **Context**: See Research Log entries above. -- **Alternatives Considered**: alias/share; MOVE_PTR ownership transfer (both rejected, see Architecture Pattern Evaluation table). -- **Selected Approach**: New `dup_job_alloc(const xce_job_alloc_t*)` in `xcealloc.c`, composed from `make_alloc`/`make_host_alloc`/`make_job_alloc`, explicitly copying `from`/`duration` after each `make_alloc()` call. -- **Rationale**: Only option that satisfies the independence requirement (Req 1.4) without a capnp-layer round-trip inside the snapshot files (an existing architectural boundary). -- **Trade-offs**: None significant — this is the approach the brief already converged on; this session's discovery only refined the `from`/`duration` fidelity detail. -- **Follow-up**: none. - -## Risks & Mitigations -- **Risk**: `snapshot_load.c`'s local schedobj-builder drifts out of sync with `core_build_schedobj()` if the latter gains a new field later. — **Mitigation**: recorded as a Revalidation Trigger in `design.md`; `/xce-audit-invariants scheduler` dimension 4 is the standing check that would catch this class of drift. -- **Risk**: A job whose resreq failed to parse (pre-existing, unrelated `snapshot_load.c` gap) could crash schedobj construction if not guarded. — **Mitigation**: schedobj construction is explicitly gated on `job->resreq != NULL`, matching `core_build_schedobj()`'s own implicit precondition; the pre-existing resreq-failure behavior itself is unchanged (out of scope). -- **Risk**: `kill_sched_job()` not freeing `resreq`/`schedobj`/`allocs`/`accounts` is a pre-existing leak this feature's newly-populated fields will also flow through. — **Mitigation**: none needed for this spec — Requirement 2.4 asks for structural parity with live jobs, which this preserves (leak included); fixing job cleanup is an unrelated, broader job-lifecycle concern, explicitly out of boundary. -- **Risk** (found during `/kiro-validate-design` review): a PENDING job whose `resreq` fails to parse on load (pre-existing gap, plausible via a resmgr mismatch between the saving and loading cores) still reaches `sched_add_job()`/the picker with `job->schedobj == NULL`, since `snapshot_build_schedobj()`'s guard skips construction whenever `resreq` is NULL — if later selected by the picker, it hits the same bug-1569-class crash. Not a new regression (every restored job hit this unconditionally before this feature), but Requirement 2.2 is closed only for the general case, not this narrow sub-case. — **Mitigation**: documented explicitly in `design.md` (Components and Interfaces → Load-Side Job Reconstruction, and Out of Boundary) as a known, unclosed limitation rather than silently implied as covered; the actual fix belongs to the pre-existing resreq-parse-failure path itself, a distinct, unrelated defect. -- **Risk** (found during `/kiro-validate-design` review): `dup_job_alloc()`'s originally-drafted postcondition ("no leak on the failure path") was stricter than this file's established OOM-handling convention (`make_host_account()`/`make_job_account()` don't unwind partial state either). — **Mitigation**: relaxed the postcondition in `design.md` to explicitly allow a leak on the rare OOM path, matching the file's convention; confirmed this does not violate Req 1.4 (independence and leak-freedom are separate properties). - -## References -- `.wolf/buglog.json` bug-1569 — the exact prior instance of this bug class (`job->schedobj` unconditionally dereferenced by the allocator with no NULL check), already fixed once for the JOBS_PUSH path. -- `.wolf/buglog.json` bug-1678 — controller-side precedent (WAL_JOB_DISPATCH never persisting `job->allocs`), same shape, sibling persistence mechanism, already fixed with pure save/load glue and no schema change. -- `.kiro/steering/persistence.md` — "Snapshot Save/Load Field-Completeness (Scheduler)" section, the standing invariant this spec is the confirmed instance of. -- `src/schema/controller.capnp:326,341,438,443,448` — `Job.allocs`/`SchedJob.allocs`/`JobAlloc`/`HostAlloc`/`ResAlloc` schema, confirming zero schema changes needed. diff --git a/.kiro/specs/scheduler-snapshot-job-allocation/spec.json b/.kiro/specs/scheduler-snapshot-job-allocation/spec.json deleted file mode 100644 index ca8f5539..00000000 --- a/.kiro/specs/scheduler-snapshot-job-allocation/spec.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "feature_name": "scheduler-snapshot-job-allocation", - "created_at": "2026-08-27T09:07:31Z", - "updated_at": "2026-08-30T08:10:00Z", - "language": "en", - "phase": "implementation-complete", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} - diff --git a/.kiro/specs/scheduler-snapshot-job-allocation/tasks.md b/.kiro/specs/scheduler-snapshot-job-allocation/tasks.md deleted file mode 100644 index 3ca84b04..00000000 --- a/.kiro/specs/scheduler-snapshot-job-allocation/tasks.md +++ /dev/null @@ -1,95 +0,0 @@ -# Implementation Plan - -- [ ] 1. Core: independent snapshot-reconstruction primitives -- [x] 1.1 (P) Build a deep-copy primitive for a job's resource allocation - - Write failing unit tests first (TDD red): single-host/single-resource copy, multi-host copy, multi-resource-per-host copy, shared-resource-pool copy, an independence proof (mutate or free the source after copying and confirm the copy is unaffected), and a fidelity check that the copy's allocation start-time/duration match the source exactly rather than being reset to copy-time. - - Implement the primitive by composing the existing allocation-construction helpers, explicitly carrying over the source's original start-time/duration rather than accepting a fresh default. - - On an allocation failure partway through, return no result rather than a partial/inconsistent one, matching this area's existing convention (no forced full-rollback machinery). - - Observable completion: the new unit test suite goes from failing to passing, and a copy produced from a multi-host, multi-resource, shared-pool source is byte-for-byte equivalent in content to the source while sharing no pointers with it. - - _Requirements: 1.1, 1.4_ - - _Boundary: xcealloc (job allocation deep-copy)_ - -- [x] 1.2 (P) Construct a valid scheduling representation for every job restored from a snapshot - - Write a failing test first (TDD red) asserting that a job reconstructed from a snapshot has a non-NULL, correctly-shaped internal scheduling representation for a primary, a compound, and an alternative resource request. - - Implement construction using only information already available by the time a restored job's resource request has been parsed (its queue, user, project, and resource request) — no new snapshot data is captured or required. - - Skip construction (leaving today's behavior unchanged) when a restored job's resource request itself failed to parse — do not attempt to fix that separate, pre-existing failure path as part of this task. - - Observable completion: the new test goes from failing to passing, and a restored job's scheduling representation is structurally identical to what the same job would get if it had just been freshly submitted with the same queue/user/project/resource request. - - _Requirements: 2.1, 2.2, 2.3, 2.4_ - - _Boundary: snapshot load (scheduling-representation construction)_ - -- [ ] 2. Core: wire the allocation restoration into the save and load paths -- [x] 2.1 Populate a job's resource allocation on the save side - - Write a failing test first (TDD red) asserting that saving a running job with a resource allocation produces a saved representation carrying that allocation. - - Populate the already-existing allocation slot on the saved job representation using the deep-copy primitive from 1.1 whenever the live job has an allocation; leave it absent otherwise, matching how every other optional value here is already handled. - - Observable completion: the new test goes from failing to passing, and saving a job with a multi-host allocation produces a saved job whose allocation content matches the live job's. - - _Requirements: 1.1_ - - _Depends: 1.1_ - - _Boundary: snapshot save (allocation population)_ - - _Note: not marked parallel with 2.2 — both tasks' tests land in the same test file (test_snapshot_save_load.c), a shared-file conflict even though the production code they touch is disjoint._ - -- [x] 2.2 Restore a job's resource allocation and accounting on the load side - - Write a failing test first (TDD red) asserting that loading a saved job that carries a resource allocation reconstructs both the job's allocation and its per-host/per-resource accounting. - - Restore the allocation via the deep-copy primitive from 1.1 (never sharing the saved representation's own copy), then rebuild the job's accounting from the restored allocation using the existing accounting constructor. - - Position this restoration immediately after the scheduling-representation construction from 1.2 within the same job-reconstruction step, before the job becomes visible to the rest of the scheduler. - - Treat a job with no saved allocation data as the normal case (no error, no warning) — leave its allocation and accounting unset, identical to today's behavior. - - Observable completion: the new test goes from failing to passing, and loading a saved job with a multi-host, multi-resource allocation produces a restored job whose worker and shared resource usage accounting immediately reflects that allocation, with no further activity required. - - _Requirements: 1.1, 1.2, 1.3, 1.4, 3.5_ - - _Depends: 1.1, 1.2_ - - _Boundary: snapshot load (allocation and accounting restoration)_ - -- [ ] 3. Integration: prove the save/load round trip end to end -- [x] 3.1 Verify a full cross-instance save/load round trip preserves allocation, accounting, and scheduling representation together - - Save a scheduler instance containing a mix of running jobs (spanning multiple hosts, multiple resources, and a shared-resource-pool allocation) and pending jobs, then load that saved state into a second, independent scheduler instance. - - Assert every running job's allocation and accounting match the source instance, every job (running or pending) has a valid scheduling representation, and that ending either instance's job list afterward does not corrupt or crash the other. - - Observable completion: the round-trip test passes, demonstrating the restored instance's resource-in-use totals exactly match what the source instance reported before saving. - - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.3, 2.4_ - - _Depends: 1.2, 2.1, 2.2_ - -- [x] 3.2 Verify a snapshot with no allocation data loads cleanly - - Extend the round-trip coverage with a saved state containing no allocation data for its jobs (the natural case for a job that was never dispatched, or a save taken before this feature existed). - - Assert the load succeeds with no error, and every such job's resource contribution reads as zero/unavailable, identical to today's behavior. - - Observable completion: the backward-compatibility test passes without any special-casing visible to the caller. - - _Requirements: 3.5_ - - _Depends: 2.2_ - -- [ ] 4. Validation: close the reported symptom, prove no regressions, and keep documentation current -- [x] 4.1 (P) Prove a restored pending job with a valid resource request can be evaluated for allocation without crashing - - Load a saved state containing a pending job whose resource request parses successfully, then run that job through the scheduler's normal allocation-evaluation path. - - Assert no crash and no undefined behavior — this is the direct regression test for the scheduling-representation gap this feature closes. This does not cover a pending job whose resource request itself fails to parse on load — that pre-existing, separate failure path is explicitly out of this feature's boundary (see design.md's Known Limitation) and is not exercised by this test. - - Observable completion: the test passes cleanly under the project's memory-safety build configuration. - - _Requirements: 2.2, 2.4_ - - _Depends: 1.2_ - - _Boundary: snapshot load (scheduling-representation construction), test_snapshot_save_load.c_ - -- [x] 4.2 (P) Reproduce and close the originally-reported resource-accounting symptom end to end - - Extend the existing simulation-mode scenario that already reproduces the reported symptom (a saved state with many running jobs, replayed forward) to assert that every running job's resource contribution is correctly reflected immediately after loading, not only the jobs whose dispatch also happens to appear later in the replay. - - Observable completion: the extended end-to-end scenario passes, showing 100% of the saved running jobs' resource contribution accounted for immediately on load. - - _Requirements: 1.1, 1.2, 1.3_ - - _Depends: 3.1_ - - _Boundary: test_sim_e2e_original_motivation.c_ - -- [x] 4.3 (P) Confirm no regression to existing reconciliation and the rest of the snapshot format - - Run the existing replay-based reconciliation test coverage and the full existing test suite unchanged; confirm they all still pass exactly as before this feature. - - Confirm no other previously-supported snapshot field's save/load behavior changed. - - Observable completion: a full test run shows zero new failures and zero behavior change outside the allocation/scheduling-representation fields this feature touches. - - _Requirements: 3.1, 3.4_ - - _Depends: 2.1, 2.2_ - - _Boundary: existing test suite (read-only run, no source changes)_ - -- [x] 4.4 (P) Keep the snapshot format's own scope-decision documentation current - - Correct the existing source-level note that a job's resource allocation "is not captured" to reflect that it now is, and add a note documenting the job's latest resource-usage report as a known, still-uncaptured gap requiring a future format change. - - Observable completion: the source-level documentation and the feature's actual behavior no longer contradict each other. - - _Requirements: 3.2, 3.3_ - - _Depends: 2.2_ - - _Boundary: snapshot.c (scope-decision header comment only)_ - -- [x] 4.5 Run the project's scheduler design-invariant audit as final validation - - Run the scheduler component's field-completeness invariant audit dimension (or manually re-check it) against the finished implementation. - - Confirm the audit finds no undocumented gap introduced or left behind by this feature. - - Observable completion: the audit dimension reports clean, with this feature's fields correctly classified as fixed or documented. - - _Requirements: 3.2, 3.3, 3.4_ - - _Depends: 4.1, 4.2, 4.3, 4.4_ - -## Implementation Notes -- Task 4.4 follow-up (non-blocking, flag for 4.5): `snapshot.c`'s "internal_state contents" comment section (~lines 217-220, distinct from the "Scope decisions" section this task corrected) still cites the now-stale premise "this task also does not restore job->allocs/accounts (see 'Scope decisions' below)" as part of the rationale for why `sharedInUse`/`resInUse` aren't serialized. The exclusion conclusion itself still holds (session-transient state, re-summed every session by `sched_account_jobs()`), but the cited premise contradicts the corrected "Scope decisions" bullet below it. Out of task 4.4's boundary (scoped strictly to the "Scope decisions" header comment); confirmed by two independent readings (implementer + reviewer). Should be corrected — drop/rephrase the stale clause, keep the still-valid rationale — before task 4.5's field-completeness audit signs off clean. -- Task 4.2 side-finding (not fixed here, out of boundary): `core_upsert_queue()`'s existing-queue branch (`core_events.c:335-336`) calls `FREEUP(q->resreq)` (plain `free()`) on a `resreq` that was set via the rtti-backed `makeString()` (mismatched allocator/free, same bug class as already-fixed bug-1503/bug-1600). Reproducible via any second `QUEUE_UPDATE` for a queue whose `resreq` was ever populated via `makeString()` — not specific to snapshot restoration; the near-identical pattern already exists in the legacy `sched_update_existing_queue()` (`schedobj.c:573-575`). Confirmed real by two independent readings (implementer + reviewer). Needs its own buglog.json entry and likely a separate follow-up task — tracked here so it isn't lost. diff --git a/.kiro/specs/worker-join-refactor/design.md b/.kiro/specs/worker-join-refactor/design.md deleted file mode 100644 index 9d5c684c..00000000 --- a/.kiro/specs/worker-join-refactor/design.md +++ /dev/null @@ -1,450 +0,0 @@ -# Design Document: worker-join-refactor - -## Overview - -This refactor restructures the worker join path in the XCE controller to make the check → create-or-update → status/resource-update sequence explicit and readable at the `handle_worker_join` call site. - -The current implementation delegates the create-or-update decision to `add_worker`, which in turn delegates it to `makeWorker` in `wobj.c`. This hides the branching and introduces a latent deadlock: `handle_worker_join` holds the `workermgr` write lock while calling `add_worker → makeWorker`, which attempts to re-acquire the same lock. Since `my_rwlock_t` is `pthread_rwlock_t` (non-reentrant), this deadlocks. The refactor eliminates the outer lock, makes existence-checking visible at the join handler, and introduces two narrowly-scoped helpers for the create and rejoin paths. - -### Goals - -- Make the check → create-or-update → status/resource-update control flow visible in `handle_worker_join` -- Eliminate the double workermgr write lock acquisition (latent deadlock fix) -- Preserve all observable behavior: hooks, `on_worker_join`, admin-bit preservation - -### Non-Goals - -- Changes to `makeWorker` or any other function in `src/lib/wobj.c` -- Changes to other join handlers (`handle_client_join`, `handle_scheduler_join`, etc.) -- Job dispatch, WAL, or persist subsystem behavior changes - ---- - -## Boundary Commitments - -### This Spec Owns - -- `handle_worker_join` in `src/controller/join.c` — rewritten to expose explicit branching -- `add_worker` in `src/controller/worker.c` — replaced by `init_worker`, `rejoin_worker`, and `apply_join_state` -- `src/controller/include/worker.h` — updated exported declarations - -### Out of Boundary - -- `makeWorker` / `makeWorker_` / `override_worker` in `src/lib/wobj.c` — must not be modified -- Other join handler functions in `join.c` -- Persist/WAL hook internals - -### Allowed Dependencies - -- `key_find` from `kvmgr` (via RTTI) for existence check -- `makeWorker` from `wobj.c` for first-time worker creation -- `find_matched_hostconfig`, `add_hostattrs_resvals`, `set_host_slots`, `mark_host_res` from `worker.c` (already present) -- `xce_run_hooks` for BEFORE/AFTER event hooks -- `on_worker_join` for post-join config push - -### Revalidation Triggers - -- Changes to `makeWorker` behavior (create vs update path) require re-checking `init_worker` -- Changes to `XCE_WORKER_ADMIN_MASK` definition require re-checking the admin-bit preservation in `rejoin_worker` -- Adding new hconfig fields requires updating `apply_join_state` - ---- - -## Architecture - -### Existing Architecture Analysis - -**Current flow in `handle_worker_join` (join.c)**: - -``` -lock(workermgr, WRITE) - run_hooks(BEFORE_EVENT) - add_worker() ← delegates create-or-update decision - makeWorker() - lock(workermgr, WRITE) ← DEADLOCK: already held by caller - key_find + create-or-override - unlock(workermgr) - per-worker lock - clear UNAVAIL, set ep, apply hconfig - per-worker unlock - vget(workermgr, idx, &w) - run_hooks(AFTER_EVENT) - xce_reply(...) -unlock(workermgr) -on_worker_join(...) -``` - -**Problems**: -1. Double write lock on `workermgr` deadlocks (pthread_rwlock_t is non-reentrant) -2. Create-or-update branch hidden inside `makeWorker_` — invisible at call site -3. `xce_reply` called while holding the write lock (unnecessary) - -### Architecture Pattern & Boundary Map - -```mermaid -graph TB - HWJ[handle_worker_join] - FHC[find_matched_hostconfig] - HooksBefore[run_hooks BEFORE] - KeyFind[key_find workermgr READ lock] - InitW[init_worker] - RejoinW[rejoin_worker] - AJS[apply_join_state] - MakeW[makeWorker wobj.c] - HooksAfter[run_hooks AFTER] - OWJ[on_worker_join] - - HWJ --> FHC - HWJ --> HooksBefore - HWJ --> KeyFind - KeyFind -->|ENOENT| InitW - KeyFind -->|found| RejoinW - InitW --> MakeW - InitW --> AJS - RejoinW --> AJS - HWJ --> HooksAfter - HWJ --> OWJ -``` - -**Refactored flow**: -- `handle_worker_join` owns the explicit `key_find` branch decision (Req 1) -- `init_worker` owns first-time registration (Req 2); delegates creation locking to `makeWorker` -- `rejoin_worker` owns in-place attribute/endpoint update (Req 3); uses per-worker write lock -- `apply_join_state` is a static shared helper for status and resource update (Req 4) -- No outer `workermgr` write lock in `handle_worker_join` — eliminates deadlock - -**Locking model**: - -| Scope | Lock | Holder | Purpose | -|-------|------|--------|---------| -| Existence check | workermgr READ | `handle_worker_join` (brief) | Safe key_find | -| Worker creation | workermgr WRITE | `makeWorker` (internal) | key_add atomicity | -| Worker update | per-worker WRITE | `rejoin_worker` / `apply_join_state` | Field mutation | - -### Technology Stack - -| Layer | Choice | Role | Notes | -|-------|--------|------|-------| -| Language | C99 | Implementation | Existing codebase standard | -| Locking | pthread_rwlock_t (my_rwlock_t) | Concurrency | Existing primitive; non-reentrant | -| Object management | RTTI (key_find, key_add, makeInstance) | Worker lifecycle | wobj.c / kvmgr.c | -| Hooks | xce_run_hooks | Persistence integration | Unchanged hook fire points | - ---- - -## File Structure Plan - -### Modified Files - -- `src/controller/join.c` — `handle_worker_join`: remove outer workermgr write lock; add explicit `key_find` check; call `init_worker` or `rejoin_worker` based on result -- `src/controller/worker.c` — remove `add_worker`; add `apply_join_state` (static), `init_worker`, `rejoin_worker` -- `src/controller/include/worker.h` — replace `add_worker` declaration with `init_worker` and `rejoin_worker` - ---- - -## System Flows - -```mermaid -sequenceDiagram - participant JC as handle_worker_join - participant WC as worker.c helpers - participant WO as makeWorker (wobj.c) - participant HS as hooksys - - JC->>JC: find_matched_hostconfig (fail fast, no lock) - JC->>HS: run_hooks BEFORE_EVENT WORKER_JOIN - JC->>JC: lock(workermgr READ) - JC->>JC: key_find(workermgr, name, &w) - JC->>JC: unlock(workermgr READ) - - alt host not found (ENOENT) - JC->>WC: init_worker(c, ep, name, attrs, hconfig) - WC->>WO: makeWorker(workermgr, &tmp_host) - Note over WO: lock WRITE, create entry, unlock WRITE - WC->>WC: apply_join_state under per-worker WRITE lock - WC-->>JC: idx - else host found - JC->>WC: rejoin_worker(c, w, ep, attrs, hconfig) - WC->>WC: per-worker WRITE lock - WC->>WC: update hostattrs + ep (preserve admin bits) - WC->>WC: apply_join_state (status, resources) - WC->>WC: unlock per-worker - WC-->>JC: idx - else key_find error (not ENOENT) - JC-->>JC: reject join, return error - end - - JC->>JC: set priv->u.idx = idx - JC->>JC: xce_reply (outside any lock) - JC->>HS: run_hooks AFTER_EVENT WORKER_JOIN - JC->>WC: on_worker_join(c, ep) -``` - -**Key decisions**: -- `find_matched_hostconfig` fails fast before any lock (Req 4.4 — reject if NULL) -- Hooks fire outside the workermgr lock, bracketing the create-or-update step -- `xce_reply` and `on_worker_join` execute after all locks are released - ---- - -## Requirements Traceability - -| Requirement | Summary | Component | Interface | Flow | -|-------------|---------|-----------|-----------|------| -| 1.1 | key_find before create-or-update | `handle_worker_join` | — | Sequence: key_find step | -| 1.2 | Branch on key_find result | `handle_worker_join` | — | alt/else branches | -| 1.3 | Reject on non-ENOENT error | `handle_worker_join` | — | else-error branch | -| 2.1 | Create via makeWorker | `init_worker` | `init_worker` signature | create branch | -| 2.2 | Log failure, set reply status, no on_worker_join | `handle_worker_join` | — | error exit path | -| 2.3 | Assign priv->u.idx | `handle_worker_join` | — | after successful join | -| 3.1 | Update hostattrs on rejoin | `rejoin_worker` | `rejoin_worker` signature | update branch | -| 3.2 | Update ep on rejoin | `rejoin_worker` | — | update branch | -| 3.3 | Preserve admin bits | `rejoin_worker` | — | ADMIN_MASK pattern | -| 4.1 | Clear XCE_WORKER_UNAVAIL | `apply_join_state` | — | shared path | -| 4.2 | Apply hconfig overrides | `apply_join_state` | — | shared path | -| 4.3 | Update host-attr resource values | `apply_join_state` | — | shared path | -| 4.4 | Reject if find_matched_hostconfig NULL | `handle_worker_join` | — | fail-fast before lock | -| 5.1 | BEFORE/AFTER hook order preserved | `handle_worker_join` | — | hook call sites | -| 5.2 | on_worker_join still called | `handle_worker_join` | — | post-join step | -| 5.3 | Critical section as short as current | Locking model | — | per-lock scoping | -| 5.4 | Existing tests pass | All | — | behavioral preservation | - ---- - -## Components and Interfaces - -### Summary - -| Component | Layer | Intent | Req Coverage | Key Dependencies | -|-----------|-------|--------|--------------|-----------------| -| `handle_worker_join` | join.c | Explicit branch controller | 1.1–1.3, 2.2–2.3, 4.4, 5.1–5.3 | `key_find` (P0), `init_worker` (P0), `rejoin_worker` (P0) | -| `init_worker` | worker.c | First-time registration | 2.1–2.3 | `makeWorker` (P0), `apply_join_state` (P0) | -| `rejoin_worker` | worker.c | Reconnect update | 3.1–3.3 | `apply_join_state` (P0), per-worker lock (P0) | -| `apply_join_state` | worker.c (static) | Status + resource update | 4.1–4.3 | `add_hostattrs_resvals` (P0), `set_host_slots` (P0) | - ---- - -### join.c - -#### `handle_worker_join` (rewritten) - -| Field | Detail | -|-------|--------| -| Intent | Explicit check → create-or-update → status/resource sequence for worker join | -| Requirements | 1.1, 1.2, 1.3, 2.2, 2.3, 4.4, 5.1, 5.2, 5.3 | - -**Responsibilities & Constraints** -- Calls `find_matched_hostconfig` first; rejects join with appropriate error code if NULL (Req 4.4) -- Fires BEFORE_EVENT hook before the key_find step -- Acquires a brief workermgr READ lock for `key_find`; releases before any write path -- Branches explicitly on key_find result: `init_worker` or `rejoin_worker` -- Fires AFTER_EVENT hook after the create/update completes, outside any lock -- Calls `xce_reply` and `on_worker_join` outside any lock -- Does NOT hold the workermgr write lock at any point (prevents double-lock deadlock) - -**Dependencies** -- Inbound: `handle_join_` dispatches to this function (P0) -- Outbound: `find_matched_hostconfig` — hostconfig lookup (P0) -- Outbound: `xce_run_hooks` — BEFORE/AFTER_EVENT (P0) -- Outbound: `key_find(c->workermgr, ...)` — existence check (P0) -- Outbound: `init_worker` — first-time join path (P0) -- Outbound: `rejoin_worker` — rejoin path (P0) -- Outbound: `on_worker_join` — config push post-join (P0) - -**Contracts**: Service [x] / State [x] - -##### Service Interface (C) - -```c -/* Unchanged external signature */ -static int handle_worker_join(cluster_t* c, endpoint_t* ep, - xce_request_t* request); -``` - -- Preconditions: `request->body.joinmsg` non-NULL; `ep->data` non-NULL -- Postconditions: on success, `priv->u.idx` set; `on_worker_join` invoked; BEFORE/AFTER hooks fired -- Error return: negative errno; reply status set to error code; `on_worker_join` not called - -**Implementation Notes** -- Integration: The outer `lock(c->workermgr, RTTI_LOCK_WRITE)` present in the current code is **removed**. The `vget` lookup for the AFTER_EVENT hook (currently inside the lock) moves to after the create/update call. -- Validation: All early-exit paths (hostconfig NULL, key_find non-ENOENT error, init/rejoin failure) must set `reply.status` and call `xce_reply` before returning. -- Risks: `priv->u.idx` must be set before `on_worker_join` and after the lock is released; the existing order is preserved. - ---- - -### worker.c - -#### `apply_join_state` (static) - -| Field | Detail | -|-------|--------| -| Intent | Shared status and resource update applied to both new and rejoining workers | -| Requirements | 4.1, 4.2, 4.3 | - -**Responsibilities & Constraints** -- Called under the per-worker write lock (caller must hold it) -- Clears `XCE_WORKER_UNAVAIL` from `w->host.status` (Req 4.1) -- Sets `w->ep = ep` -- Applies hconfig hosttype/model overrides (dereference + makeString pattern) -- Calls `add_hostattrs_resvals` (Req 4.3) -- Computes mxj slots and calls `set_host_slots` (Req 4.2) -- Replaces `dispatch_window_config` and `thresholds` from hconfig -- Loops over `hconfig->nresources` calling `mark_host_res` (bool resources) - -**Contracts**: Service [x] - -##### Service Interface (C) - -```c -/* static — internal to worker.c */ -static int apply_join_state(cluster_t* c, worker_t* w, endpoint_t* ep, - xce_hostconfig_t* hconfig); -``` - -- Preconditions: caller holds per-worker WRITE lock; `w`, `hconfig` non-NULL -- Postconditions: worker status, ep, and all resource values updated -- Returns: 0 on success; negative errno on resource update failure - ---- - -#### `init_worker` - -| Field | Detail | -|-------|--------| -| Intent | Register a new worker entry for a first-time joining host | -| Requirements | 2.1, 2.2, 2.3 | - -**Responsibilities & Constraints** -- Builds a temporary `xce_worker_t` with name and attrs -- Calls `makeWorker(c->workermgr, &tmp)` — delegating write-lock acquisition and allocation to wobj.c -- On NULL return: logs failure, returns `-errno` (caller sets reply status) -- On success: acquires per-worker write lock, calls `apply_join_state`, unlocks -- Returns the worker index (`w->idx`) - -**Dependencies** -- Outbound: `makeWorker` (wobj.c) — allocation + key_add (P0) -- Outbound: `apply_join_state` — status/resource setup (P0) - -**Contracts**: Service [x] - -##### Service Interface (C) - -```c -/* exported — declared in worker.h */ -int init_worker(cluster_t* c, endpoint_t* ep, const char* host, - hostattrs_t* attrs, xce_hostconfig_t* hconfig); -``` - -- Preconditions: caller does NOT hold workermgr write lock; `host` non-NULL -- Postconditions: new worker registered in workermgr; status and resources initialized -- Returns: worker index (≥ 0) on success; negative errno on failure -- Error cases: `makeWorker` allocation failure → return `-errno` without calling `on_worker_join` - ---- - -#### `rejoin_worker` - -| Field | Detail | -|-------|--------| -| Intent | Update an existing worker entry when the same host reconnects | -| Requirements | 3.1, 3.2, 3.3 | - -**Responsibilities & Constraints** -- Accepts a `worker_t*` found by the caller's `key_find` -- Acquires per-worker write lock -- Saves admin status bits: `admin_status = w->host.status & XCE_WORKER_ADMIN_MASK` -- Frees existing `hosttype` and `hostmodel` strings; copies new ones from `attrs` -- Copies numeric attrs fields (`cpufactor`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks`) -- Restores `w->host.status` admin bits (Req 3.3) -- Calls `apply_join_state` (clears UNAVAIL, sets ep, applies hconfig overrides, updates resources) -- Releases per-worker write lock -- Returns `w->idx` - -**Dependencies** -- Outbound: `apply_join_state` — status/resource setup (P0) -- Outbound: `dereference` / `makeString` — string memory management (P0) - -**Contracts**: Service [x] - -##### Service Interface (C) - -```c -/* exported — declared in worker.h */ -int rejoin_worker(cluster_t* c, worker_t* w, endpoint_t* ep, - hostattrs_t* attrs, xce_hostconfig_t* hconfig); -``` - -- Preconditions: caller holds a reference on `w`; does NOT hold per-worker write lock -- Postconditions: `w->host.attrs` updated; `w->ep` set to `ep`; admin bits preserved; resources updated -- Returns: `w->idx` (≥ 0); negative errno on `apply_join_state` failure -- Invariant: `w->host.status & XCE_WORKER_ADMIN_MASK` unchanged from pre-call value - -```c -#define XCE_WORKER_ADMIN_MASK \ - (XCE_WORKER_LOCKEDU | XCE_WORKER_CLOSED | XCE_WORKER_HIDDEN \ - | XCE_WORKER_CLOSED_BY_ADMIN) -``` - ---- - -### worker.h - -#### Exported Declaration Changes - -Remove: -```c -extern int add_worker(void*, endpoint_t*, const char*, hostattrs_t*); -``` - -Add: -```c -extern int init_worker(cluster_t*, endpoint_t*, const char*, hostattrs_t*, - xce_hostconfig_t*); -extern int rejoin_worker(cluster_t*, worker_t*, endpoint_t*, hostattrs_t*, - xce_hostconfig_t*); -``` - ---- - -## Error Handling - -### Error Strategy - -Fail fast, explicit error propagation, no silent fallbacks. - -### Error Categories - -| Error | Source | Handler | Reply Status | -|-------|--------|---------|-------------| -| `find_matched_hostconfig` returns NULL | No matching hostconfig | Reject join; set `reply.status = -errno` | `-errno` | -| `key_find` returns error other than -ENOENT | workermgr internal error | Reject join (Req 1.3) | key_find return code | -| `init_worker` returns error | `makeWorker` allocation failure | Log, reject join (Req 2.2) | error code | -| `rejoin_worker` returns error | `apply_join_state` failure | Log, reject join | error code | -| `apply_join_state` partial failure | Resource update failure | Log with PERROR; continue (matches existing behavior) | 0 | - -### Monitoring - -- Existing `ERROR` / `TRACE` macros used throughout — no changes to log verbosity -- `PERROR("update_resval")` in `update_resvals` preserved for resource update failures - ---- - -## Testing Strategy - -### Unit Tests (src/controller/testsuite/) - -1. **First-time join creates worker**: call `init_worker` with a new hostname; verify `key_find` succeeds afterwards, `XCE_WORKER_UNAVAIL` is clear, and resource values are set. -2. **Rejoin updates attrs and ep**: call `init_worker`, then call `rejoin_worker` with different attrs; verify updated attrs, updated ep, and original admin bits preserved. -3. **Admin bits preserved on rejoin**: set `XCE_WORKER_CLOSED` before rejoin; verify flag still set after `rejoin_worker`. -4. **NULL hostconfig rejects join**: configure cluster with no matching host config; verify `handle_worker_join` path returns error without modifying workermgr. -5. **Non-ENOENT key_find error rejected**: simulate `key_find` returning `-EIO`; verify join rejected and `on_worker_join` not called. - -### Integration Tests - -1. **BEFORE/AFTER hook order**: verify hooks fire in the correct order relative to workermgr state changes using existing `test_e2e_hooks.c` patterns. -2. **on_worker_join called after successful join**: verify `worker_push_config` executes on both first-time and rejoin paths. -3. **Existing testsuite passes without modification**: `make test` in `src/frontend/` and testsuite build must remain green. - -### Behavioral Preservation - -The existing unit tests in `src/controller/testsuite/` exercise `add_worker` indirectly through cluster setup helpers. Test fixtures that call `add_worker` directly (if any) must be updated to call `init_worker` with the added `hconfig` parameter. All observable behaviors — hook ordering, resource initialization, admin-bit preservation, `on_worker_join` invocation — are explicitly covered by the traceability table and unit tests above. diff --git a/.kiro/specs/worker-join-refactor/requirements.md b/.kiro/specs/worker-join-refactor/requirements.md deleted file mode 100644 index 4878d982..00000000 --- a/.kiro/specs/worker-join-refactor/requirements.md +++ /dev/null @@ -1,71 +0,0 @@ -# Requirements Document - -## Project Description (Input) -Simplify the code of worker join in the controller by first checking whether the host is in `workermgr`; if it does not exist, create the host first; otherwise find the host in `workermgr` and update the host attrs and endpoint. Then update host status and resource information. - -## Introduction - -The controller's `handle_worker_join` function currently delegates the entire create-or-update decision to `add_worker`, which in turn delegates to `makeWorker` in `wobj.c`. The internal branching is invisible at the call site and the locking structure is confusing (the outer join handler holds the `workermgr` write lock while `makeWorker` re-acquires the same lock). This refactor restructures the worker join path so the check → create-or-update → status/resource-update sequence is explicit and readable. - -## Boundary Context - -- **In scope**: `handle_worker_join` in `src/controller/join.c` and `add_worker` in `src/controller/worker.c`. -- **Out of scope**: `makeWorker` in `src/lib/wobj.c`, other join handlers (`handle_client_join`, `handle_scheduler_join`, etc.), job dispatch, and the persist/WAL subsystem behavior. -- **Adjacent expectations**: Persist hooks (`XCE_CONTROLLER_HOOK_WORKER`) must still fire at the same event points (BEFORE_EVENT / AFTER_EVENT for HOOK_DETAIL_WORKER_JOIN). The `on_worker_join` callback that pushes configuration to the worker must still be called after a successful join. - -## Requirements - -### Requirement 1: Explicit Host Existence Check - -**Objective:** As a controller maintainer, I want the worker join code to make an explicit lookup before deciding whether to create or update, so that the control flow is readable without tracing into `makeWorker`. - -#### Acceptance Criteria -1. When a worker join request is received, the Controller shall look up the joining host by name in `workermgr` using `key_find` before taking any create-or-update action. -2. The Controller shall branch on the result of that lookup: one branch for a host that does not yet exist, one branch for a host that already exists. -3. If `key_find` returns an error other than `-ENOENT`, the Controller shall reject the join and return that error code. - ---- - -### Requirement 2: Host Creation for First-Time Workers - -**Objective:** As a controller maintainer, I want a new host entry to be created when a worker joins for the first time, so that the cluster knows about this node. - -#### Acceptance Criteria -1. When the host is not found in `workermgr`, the Controller shall allocate and register a new worker entry (via the existing `makeWorker` or equivalent creation path) using the joining worker's name and host attributes. -2. If creation fails, the Controller shall log the failure, set the reply status to the error code, and return without calling `on_worker_join`. -3. The Controller shall assign the new entry's index to `priv->u.idx` so subsequent disconnect handling can find the worker. - ---- - -### Requirement 3: Attribute and Endpoint Update for Rejoining Workers - -**Objective:** As a controller maintainer, I want an existing worker entry to be updated in-place when the same host rejoins, so that stale attributes and the old disconnected endpoint are replaced. - -#### Acceptance Criteria -1. When the host already exists in `workermgr`, the Controller shall update the existing worker's host attributes (`hostattrs`) with the values supplied in the join message. -2. When the host already exists in `workermgr`, the Controller shall update the existing worker's endpoint (`ep`) to the new connection's endpoint. -3. The Controller shall preserve any admin-set status bits (`XCE_WORKER_LOCKEDU`, `XCE_WORKER_CLOSED`, `XCE_WORKER_HIDDEN`, `XCE_WORKER_CLOSED_BY_ADMIN`) when updating an existing entry. - ---- - -### Requirement 4: Status and Resource Information Update - -**Objective:** As a controller maintainer, I want status and resource data to be updated after every successful join (both new and rejoining), so that the scheduler sees accurate host availability and capacity. - -#### Acceptance Criteria -1. After a successful create or update, the Controller shall clear the `XCE_WORKER_UNAVAIL` flag on the worker's status. -2. After a successful create or update, the Controller shall apply host-config overrides (hosttype, hostmodel, mxj slots, dispatch window, thresholds, and boolean resources) from the matched `xce_hostconfig_t`. -3. After a successful create or update, the Controller shall update all host-attribute-mapped resource values (`type`, `model`, `cpuf`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks`, `slots`). -4. If `find_matched_hostconfig` returns `NULL`, the Controller shall reject the join with an appropriate error code and not modify `workermgr`. - ---- - -### Requirement 5: Behavioral Preservation - -**Objective:** As a developer, I want the refactored code to preserve all existing observable behavior so that no regressions are introduced. - -#### Acceptance Criteria -1. The Controller shall fire `XCE_CONTROLLER_HOOK_WORKER / HOOK_BEFORE_EVENT / HOOK_DETAIL_WORKER_JOIN` before modifying the worker entry, and `HOOK_AFTER_EVENT / HOOK_DETAIL_WORKER_JOIN` after modification, matching the existing hook invocation order. -2. The Controller shall call `on_worker_join` after a successful join (both new and rejoin) to push configuration to the worker. -3. While the `workermgr` write lock is held during the join operation, the Controller shall keep the critical section as short as the current implementation. -4. The existing unit tests in `src/controller/testsuite/` shall pass without modification. diff --git a/.kiro/specs/worker-join-refactor/research.md b/.kiro/specs/worker-join-refactor/research.md deleted file mode 100644 index 8d5d0465..00000000 --- a/.kiro/specs/worker-join-refactor/research.md +++ /dev/null @@ -1,346 +0,0 @@ -# Research Log: worker-join-refactor - -## Summary - -**Discovery scope**: Light discovery (extension of existing C subsystem). - -**Key findings**: -1. **Latent deadlock in current code**: `handle_worker_join` acquires `lock(c->workermgr, WRITE)` then calls `add_worker → makeWorker`, which also calls `lock(workermgr, WRITE)`. `my_rwlock_t` is `pthread_rwlock_t` (non-reentrant), so this double-acquisition deadlocks. The outer lock was added for persistence-hook work on this branch and is the root cause of the "confusing locking structure." -2. **`makeWorker` in wobj.c is out of scope but must still be called**: It manages the create-or-update decision internally (`makeWorker_`). For the create path, the refactored design must NOT hold the `workermgr` write lock when calling `makeWorker`. -3. **`override_worker` already preserves admin bits**: The `XCE_WORKER_ADMIN_MASK` (`LOCKEDU | CLOSED | HIDDEN | CLOSED_BY_ADMIN`) is preserved by `override_worker` in wobj.c. The rejoin update path must replicate this guarantee when updating attrs in-place, bypassing `makeWorker`. - ---- - -## Research Log - -### Topic 1: Lock Reentrancy - -**Question**: Is `my_rwlock_t` reentrant? Is the existing double-lock intentional? - -**Investigation**: `src/include/mylock.h` defines `my_rwlock_t` as `pthread_rwlock_t`. POSIX `pthread_rwlock_wrlock` on an already-held write lock by the same thread is undefined behavior (most implementations deadlock). The outer write lock in `handle_worker_join` was introduced on branch `persistent_ha` for the AFTER_EVENT persist hook (the `vget` lookup comment confirms this). This is a newly introduced bug. - -**Implication**: The refactored design must eliminate the outer `workermgr` write lock in `handle_worker_join` before any path reaches `makeWorker`. - ---- - -### Topic 2: Existence Check and Locking Strategy - -**Question**: How should the explicit `key_find` check be locked? - -**Investigation**: `worker_ctrl` uses `lock(workermgr, READ)` → `key_find` → `reference(w)` → `unlock`. Other paths use read locks for existence checks and upgrade to per-object write locks for mutation. The workermgr write lock is only needed for structural changes (insert/delete of entries), not for updating existing worker fields. - -**Decision**: Use a brief `READ` lock for the `key_find` check. For the create path, delegate write locking to `makeWorker`. For the rejoin path, use a per-worker write lock. - -**TOCTOU note**: A brief window exists between the read-lock check and `makeWorker`'s write lock. This is benign: `makeWorker_` calls `key_find` internally under the write lock and, if the worker was created by another thread between our check and our call, will call `override_worker` (update) instead of `makeInstance` (create). The behavior remains correct. - ---- - -### Topic 3: Rejoin Update — Replicating `override_worker` Semantics - -**Question**: For the rejoin path (explicit update), how should attrs be updated in-place without calling `override_worker` (which is static in wobj.c)? - -**Investigation**: `override_worker` in wobj.c: -1. Saves admin bits (`status & XCE_WORKER_ADMIN_MASK`) -2. Calls `kill_worker` (frees name, attrs, dispatch_window, thresholds, resvals) -3. Calls `copy_worker` (re-allocates from new src) -4. Restores admin bits and idx - -For the rejoin path in worker.c, we can replicate this semantic without calling `override_worker` directly. The update needs to: free old `hostattrs` fields (hosttype/model strings), copy new attrs, update `ep`, and preserve admin bits on `host.status`. The existing `dereference/makeString` pattern is already used in `add_worker` for hosttype/model override, so the pattern is established. - -**Decision**: Implement `rejoin_worker` in worker.c that replicates the admin-bit-preserving update logic inline, avoiding the static `override_worker` dependency. - ---- - -### Topic 4: Resource and Status Update — Shared Path - -**Question**: Can the status/resource update logic in `add_worker` (req 4) be shared between create and rejoin paths? - -**Investigation**: The current `add_worker` applies the following after `makeWorker` returns, under a per-worker write lock: -- `w->host.status &= ~XCE_WORKER_UNAVAIL` -- `w->ep = ep` -- hconfig hosttype/model override -- `add_hostattrs_resvals` (resource vals for type, model, cpuf, etc.) -- `set_host_slots` (mxj or maxcpus) -- `dispatch_window_config` copy -- `thresholds` copy -- bool resource marking loop - -This block is identical for both create and rejoin. Extracting it into `static int apply_join_state(cluster_t*, worker_t*, endpoint_t*, xce_hostconfig_t*)` eliminates duplication. - ---- - -### Topic 5: Hook Firing and Lock Relationship - -**Question**: Must BEFORE_EVENT and AFTER_EVENT hooks fire while the workermgr write lock is held? - -**Investigation**: The current code fires both hooks under the outer workermgr write lock. However, `mark_worker_unavail` and `mark_worker_back` fire hooks while holding only the per-worker write lock (not the workermgr lock). The persist hook signature takes `(c, w)` — it reads from `w` directly, not through the workermgr index. There is no architectural requirement for the workermgr write lock to be held during hook dispatch. - -**Decision**: Hooks fire in `handle_worker_join` outside any workermgr lock. BEFORE_EVENT fires before `key_find`. AFTER_EVENT fires after the create/update completes. This matches the behavioral contract from req 5.1. - ---- - -## Architecture Decision: Removing `add_worker`, Adding `init_worker` + `rejoin_worker` - -The current `add_worker` is a monolithic function that internally calls `makeWorker` (which hides the create-or-update branch). The refactored design replaces it with: - -- `static int apply_join_state(cluster_t*, worker_t*, endpoint_t*, xce_hostconfig_t*)` — shared status/resource update logic (per-worker write lock) -- `int init_worker(cluster_t*, endpoint_t*, const char*, hostattrs_t*, xce_hostconfig_t*)` — first-time join: calls `makeWorker` + `apply_join_state` -- `int rejoin_worker(cluster_t*, worker_t*, endpoint_t*, hostattrs_t*, xce_hostconfig_t*)` — rejoin: updates attrs/ep in-place + `apply_join_state` - -`add_worker` declaration in `worker.h` is replaced with `init_worker` and `rejoin_worker`. - ---- - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|------|-----------|-----------| -| TOCTOU between key_find and makeWorker | Very low (same host reconnecting twice simultaneously) | makeWorker_ handles gracefully via override | -| Admin bits not preserved in rejoin path | Medium if not explicit | Explicit mask applied in rejoin_worker | -| Hook ordering regression | Low | Hooks remain in handle_worker_join, same relative order | -| Existing test breakage | Low | Tests exercise add_worker indirectly; signature change requires test fixture update | - ---- - -# Gap Analysis (2026-05-20) - -## Analysis Summary - -- **Scope**: Narrow brownfield refactor — one C file (`worker.c`) with three missing function bodies. All surrounding scaffolding is already in place. -- **Key challenge**: None blocking; the design and test contract are locked. Only implementation is missing. -- **Recommendation**: Option A (extend existing `worker.c`) is the only appropriate approach. -- **Effort**: S (1–3 days). All design decisions and lock semantics resolved; code to write is a mechanical extraction from the existing `add_worker` body. -- **Risk**: Low. Existing `add_worker` contains the exact logic to extract. Tests in `test_worker_join.c` specify every assertion. - -## Current Implementation State - -| Component | File | Status | -|-----------|------|--------| -| `handle_worker_join` (rewritten) | `join.c` | ✅ Done — explicit `key_find` branch, calls `init_worker`/`rejoin_worker` | -| `worker.h` declarations | `include/worker.h` | ✅ Done — `init_worker`, `rejoin_worker` exported; `add_worker` removed | -| TDD tests | `testsuite/test_worker_join.c` | ✅ Done — 7 tests for `apply_join_state`, `init_worker`, `rejoin_worker` | -| Test registered | `testsuite/controller_test.c` | ✅ Done — `test_worker_join` registered | -| `apply_join_state` (static helper) | `worker.c` | ❌ Missing — body not yet extracted | -| `init_worker` | `worker.c` | ❌ Missing — stub only; function body not yet written | -| `rejoin_worker` | `worker.c` | ❌ Missing — stub only; function body not yet written | -| `add_worker` removal | `worker.c` | ❌ Pending — old function body still present (unused by join.c) | - -## Requirement-to-Asset Map - -| Requirement | Asset Needed | Status | Gap Tag | -|-------------|-------------|--------|---------| -| 1.1 Explicit key_find lookup | `handle_worker_join` explicit key_find | ✅ Present in join.c | — | -| 1.2 Branch on key_find result | `init_worker`/`rejoin_worker` dispatch in join.c | ✅ Present in join.c | — | -| 1.3 Reject non-ENOENT errors | Error branch in join.c | ✅ Present in join.c | — | -| 2.1 Create via makeWorker | `init_worker` calling `makeWorker` | ❌ Missing implementation | Missing | -| 2.2 Log failure, no on_worker_join | Error path in join.c | ✅ Present in join.c | — | -| 2.3 Assign priv->u.idx | join.c post-join assignment | ✅ Present in join.c | — | -| 3.1 Update hostattrs on rejoin | `rejoin_worker` attrs copy | ❌ Missing implementation | Missing | -| 3.2 Update ep on rejoin | `rejoin_worker` ep assignment | ❌ Missing implementation | Missing | -| 3.3 Preserve admin bits | `XCE_WORKER_ADMIN_MASK` in `rejoin_worker` | ❌ Missing implementation | Missing | -| 4.1 Clear UNAVAIL | `apply_join_state` status clear | ❌ Missing implementation | Missing | -| 4.2 Apply hconfig overrides | `apply_join_state` hosttype/model/slots | ❌ Missing implementation | Missing | -| 4.3 Update host-attr resources | `apply_join_state` calling `add_hostattrs_resvals` | ❌ Missing implementation | Missing | -| 4.4 Reject if hostconfig NULL | join.c fail-fast before lock | ✅ Present in join.c | — | -| 5.1 BEFORE/AFTER hook order | join.c hook call sites | ✅ Present in join.c | — | -| 5.2 on_worker_join still called | join.c post-join call | ✅ Present in join.c | — | -| 5.3 Critical section brevity | Locking model in join.c | ✅ Present in join.c | — | -| 5.4 Existing tests pass | Test suite build | ❌ Pending — add_worker still present, init/rejoin stubs fail tests | Constraint | - -## Implementation Approach Options - -### Option A: Extend Existing worker.c (Recommended) - -**Rationale**: All design decisions are locked. The logic to implement already exists verbatim in `add_worker`. The task is extraction and reorganization: - -1. Add `apply_join_state` as a static helper — extract the `~XCE_WORKER_UNAVAIL` + ep + hconfig override + `add_hostattrs_resvals` + `set_host_slots` + `dispatch_window_config` + `thresholds` + bool resources loop from the bottom half of `add_worker`. -2. Implement `init_worker` — build a `xce_worker_t` from name/attrs, call `makeWorker`, then call `apply_join_state` under per-worker write lock. -3. Implement `rejoin_worker` — save admin bits, update attrs/ep in-place, call `apply_join_state`, restore admin bits. -4. Remove `add_worker` (now dead code). - -**Files changed**: `src/controller/worker.c` only. - -Trade-offs: -- ✅ Minimal surface — one file, pure extraction -- ✅ Tests already written and specify exact postconditions -- ✅ Lock semantics fully resolved in design.md -- ❌ None — this is the obvious approach - -### Option B: New file (not appropriate) - -These are not new responsibilities — they replace `add_worker` in the same file. Creating a new file adds navigation overhead with no benefit. - -### Option C: Hybrid (not applicable) - -No external integration or phased rollout needed. Pure in-file refactor. - -## Implementation Complexity & Risk - -- **Effort**: S (1–3 days) — logic is a mechanical extraction from `add_worker`; all design decisions resolved -- **Risk**: Low — no new patterns, no new dependencies, no new locking decisions; tests specify exact assertions - -## Recommendations for Design Phase - -Design phase is **already approved**. Proceed directly to implementation: - -1. Implement `apply_join_state` first (tasks 2 / task 1.1 green) -2. Implement `init_worker` (task 3.1 / task 1.2 green) -3. Implement `rejoin_worker` (task 3.2 / task 1.3 green) -4. Remove `add_worker` (task 4.1 cleanup) -5. Run `make test` in controller testsuite (task 5 validation) - -**No further research needed.** All unknowns from the original research log are resolved in `design.md`. - ---- - -# Cross-Spec Gap Audit (2026-06-08) - -> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. - -## Analysis Summary - -- **worker-join-refactor** is **~100% complete**. All 5 task groups, all subtasks (1.1–5) are `[x]` in tasks.md. -- **Prior gaps RESOLVED**: `init_worker` (worker.c:339), `rejoin_worker` (worker.c:366), and `apply_join_state` (static helper) are all implemented. `add_worker` declaration removed from `worker.h`. The double-lock deadlock (`handle_worker_join` held `workermgr` WRITE lock while `makeWorker` re-acquired it) is eliminated. -- **No remaining gaps**: `handle_worker_join` now has explicit `key_find` → branch → `init_worker`/`rejoin_worker` → `apply_join_state` flow visible at the call site. `BEFORE_EVENT`/`AFTER_EVENT` hooks fire at the same event points. `on_worker_join` called after successful join. Admin status bits (`LOCKEDU | CLOSED | HIDDEN | CLOSED_BY_ADMIN`) preserved in `rejoin_worker`. -- **Build verification**: Run `/xce-build controller` to confirm `test_worker_join.c` (7 test cases) pass. -- **Recommendation**: The spec can be marked `implemented`. - ---- - -# Gap Analysis — Validation Pass (2026-06-09) - -## Analysis Summary - -- **Status**: Implementation is **complete and committed** (`93ed282 feat(worker-join-refactor): implement apply_join_state, init_worker, rejoin_worker`). Working tree is clean for all spec-owned files. All tasks 1.1–5 are `[x]` in tasks.md. -- **Code verification**: Every requirement was re-checked against the actual source on branch `persistent_ha`. `handle_worker_join` (src/controller/join.c:33–119) now exposes the explicit check → create-or-update → status/resource-update flow; `apply_join_state` / `init_worker` / `rejoin_worker` are implemented in src/controller/worker.c:279–397; `add_worker` is fully removed (no remaining references anywhere in src/). -- **Only open item**: Req 5.4 (full testsuite green) could not be re-confirmed in this pass — no build artifact exists in the worktree. One `/xce-build controller` run is the remaining verification step. - -## Current State Investigation - -| Asset | Location | Finding | -|-------|----------|---------| -| `handle_worker_join` | `src/controller/join.c:33` | Rewritten. Fail-fast `find_matched_hostconfig` (no lock) → BEFORE_EVENT hook → brief workermgr READ lock + `key_find` + `reference(w)` → branch (`-ENOENT`→`init_worker`, found→`rejoin_worker`, other error→reject) → `vget` for AFTER_EVENT hook → `xce_reply` outside any lock → `priv->u.idx` set → `on_worker_join`. Outer workermgr WRITE lock is gone — the latent double-lock deadlock is eliminated. | -| `apply_join_state` | `src/controller/worker.c:283–337` | Static (exported under `-DXCE_TESTING`, which meson.build:148 sets for the test binary). Clears `XCE_WORKER_UNAVAIL`, sets `w->ep`, applies hosttype/model `"!"`-guarded overrides, `add_hostattrs_resvals`, mxj/maxcpus `set_host_slots`, dispatch-window + thresholds replacement, bool-resource loop. Precondition (per-worker WRITE lock held by caller) documented in comment. | -| `init_worker` | `src/controller/worker.c:339–364` | Builds temp `xce_worker_t`, calls `makeWorker` (which owns its own workermgr WRITE lock), `-errno` on NULL with ERROR log, then per-worker WRITE lock + `apply_join_state`, returns `w->idx`. | -| `rejoin_worker` | `src/controller/worker.c:366–397` | Per-worker WRITE lock; saves `admin_status = status & XCE_WORKER_ADMIN_MASK` (LOCKEDU\|CLOSED\|HIDDEN\|CLOSED_BY_ADMIN, defined worker.c:275); dereference/makeString replacement of hosttype/hostmodel; copies cpufactor/maxcpus/maxmem/maxswap/maxtmp/ndisks; `apply_join_state`; restores admin bits; returns `w->idx`. | -| `worker.h` | `src/controller/include/worker.h:11–14` | `init_worker` and `rejoin_worker` exported with `xce_hostconfig_t*` param; `add_worker` declaration removed (only unrelated `add_worker_to_xlist` remains in cluster.h). | -| Tests | `src/controller/testsuite/test_worker_join.c` | 7 `RUN_TEST` cases (3 × apply_join_state, 2 × init_worker, 2 × rejoin_worker), registered in `controller_test.c:154` and built via `meson.build:103`. | -| Out-of-scope guard | `src/lib/wobj.c` | `makeWorker`/`override_worker` untouched by the refactor commit (wobj.c admin-mask logic at lines 85–105 predates it, commit febb4a6). Boundary respected. | - -## Requirement-to-Asset Map - -| Req | Summary | Asset (file:line) | Status | -|-----|---------|-------------------|--------| -| 1.1 | key_find before create-or-update | join.c:75–79 (READ lock + key_find + reference) | Implemented | -| 1.2 | Branch on lookup result | join.c:81–97 (error / NULL / found branches) | Implemented | -| 1.3 | Reject non-ENOENT key_find error | join.c:81–84 (`idx = rc`, reply.status set, returned) | Implemented | -| 2.1 | Create via makeWorker | worker.c:349 (`makeWorker(c->workermgr, &tmp)`) | Implemented | -| 2.2 | Log failure, set reply status, skip on_worker_join | worker.c:350–353 + join.c:87–89, 107–112 (`if (idx < 0) return idx` before on_worker_join) | Implemented | -| 2.3 | Assign priv->u.idx | join.c:115 (before `on_worker_join`) | Implemented | -| 3.1 | Update hostattrs on rejoin | worker.c:376–386 (string + numeric attr copy) | Implemented | -| 3.2 | Update ep on rejoin | worker.c:294 (`w->ep = ep` inside apply_join_state, called from rejoin) | Implemented | -| 3.3 | Preserve admin bits | worker.c:374, 390 (save mask / restore after apply_join_state) | Implemented | -| 4.1 | Clear XCE_WORKER_UNAVAIL | worker.c:293 | Implemented | -| 4.2 | Apply hconfig overrides (type/model/mxj/window/thresholds/bool res) | worker.c:296–336 | Implemented | -| 4.3 | Host-attr resource values | worker.c:308 (`add_hostattrs_resvals`) | Implemented | -| 4.4 | Reject when hostconfig NULL, workermgr untouched | join.c:60–68 (fail-fast before BEFORE_EVENT hook and before any lock) | Implemented | -| 5.1 | BEFORE/AFTER hook order at same event points | join.c:71–72 (BEFORE, pre-key_find) and join.c:103–104 (AFTER, post-update, w via vget) | Implemented | -| 5.2 | on_worker_join after successful join (both paths) | join.c:116 (single post-branch call site) | Implemented | -| 5.3 | Critical section no longer than before | Locking narrowed vs. pre-refactor: brief READ for lookup, makeWorker-internal WRITE for create, per-worker WRITE for mutation; xce_reply outside all locks | Implemented | -| 5.4 | Existing testsuite passes unmodified | test_worker_join.c registered; no build dir in worktree to confirm a green run in this pass | Unknown — verify with `/xce-build controller` | - -**Constraints noted (no action required)**: -- `apply_join_state` visibility is toggled by `XCE_TESTING` (worker.c:282–288) — tests link against the non-static symbol. This is the established testsuite pattern; keep the meson `-DXCE_TESTING` arg in sync. -- TOCTOU window between READ-lock `key_find` and `makeWorker` remains by design; `makeWorker_` degrades to `override_worker` (admin-bit-preserving since febb4a6), so behavior stays correct. - -**Minor observation (LOW, optional)**: in the join.c error branch (rc < 0 && rc != -ENOENT), if `key_find` ever returned an error while also setting `w`, the `reference(w)` taken at join.c:77 would not be released. Current `key_find` semantics make this unreachable (`w` stays NULL on error), so this is a defensive-coding nit only. - -## Implementation Approach Options - -### Option A: Close out the spec as-is (Recommended) -Verify req 5.4 with one `/xce-build controller` run; if green, mark the spec `implemented` in spec.json and archive. -- ✅ Zero code change; everything else is verified at source level -- ✅ Matches the prior audit conclusion (2026-06-08) — now re-confirmed line-by-line -- ❌ None - -### Option B: Add the defensive dereference in the key_find error branch before closing -One-line hardening (`if (w) { dereference(w); w = NULL; }` in the error branch) plus rebuild. -- ✅ Removes the theoretical reference leak -- ❌ Touches a committed, behavior-frozen handler for an unreachable path; out of the spec's acceptance criteria - -### Option C: Fold verification into the next controller-wide build -Skip a dedicated build; rely on the next `/xce-build` from any active spec (e.g., HA work on this branch) to exercise `test_worker_join`. -- ✅ No extra build cycle -- ❌ Spec closure blocked on unrelated work; a regression would be attributed late - -## Effort & Risk - -- **Effort: S** — remaining work is one build/test run (Option A) or that plus a one-line nit (Option B); no design or implementation gaps remain. -- **Risk: Low** — pure verification of already-committed code following established patterns; all 16 functional criteria confirmed at source level; only the test-run evidence is missing. - -## Recommendations - -1. Run `/xce-build controller` to confirm the 7 `test_worker_join` cases and the rest of the controller testsuite pass (closes Req 5.4). -2. On green, update `spec.json` phase to `implemented` (or run `/kiro-verify-completion worker-join-refactor`). -3. Optionally apply the Option B one-line defensive dereference if the file is touched again for other reasons; do not reopen the spec for it. - -## Research Needed - -- None for this spec. Single carry-forward verification item: green `/xce-build controller` run on current HEAD (worker.c/join.c unmodified since 93ed282, but HA-branch changes to ha.c/raft.c in the working tree share the controller test binary — confirm no link/test interference). - ---- - -## Gap Re-Audit (2026-06-18) - -**Mode:** POST-IMPLEMENTATION GAP AUDIT (spec.json phase=implemented; tasks 12/12 [x]). -**Method:** EARS requirement → on-disk code verification via codegraph + targeted Read. Cites file:line at current HEAD. -**Scope reminder (roadmap line 26):** this spec covers **attribute/endpoint update on rejoin only**. The job-state resync (running/finished jobs re-reported on rejoin) is the SEPARATE, not-yet-started spec **worker-rejoin-resync** (planned, no effort). That is explicitly NOT a gap of THIS spec — see Boundary note below. - -### Requirement-to-Asset Map - -| Req | Statement (abbrev) | Asset (file:line) | Status | -|-----|--------------------|-------------------|--------| -| 1.1 | `key_find` before create-or-update | `handle_worker_join` brief READ lock + `key_find` — join.c:91-95 | **CLOSED** | -| 1.2 | Branch on lookup result (new vs exists) | `if (w==NULL) init_worker … else rejoin_worker` — join.c:101-113 | **CLOSED** | -| 1.3 | Reject on non-ENOENT `key_find` error | `if (rc<0 && rc!=-ENOENT){ … idx=rc; }` — join.c:97-100 | **CLOSED** | -| 2.1 | Create new entry via `makeWorker` | `init_worker` builds tmp `xce_worker_t`, calls `makeWorker(c->workermgr,&tmp)` — worker.c:339-354 | **CLOSED** | -| 2.2 | On create fail: log, set status, no `on_worker_join` | NULL→`ERROR`+`return -errno`; caller sets `reply.status=idx`, `if(idx<0) return idx` before `on_worker_join` — worker.c:351-353, join.c:104-105/138-147 | **CLOSED** | -| 2.3 | Assign new idx to `priv->u.idx` | `priv->u.idx = idx;` before `on_worker_join` — join.c:146 | **CLOSED** | -| 3.1 | Update `hostattrs` on rejoin | `rejoin_worker` frees+`makeString` hosttype/hostmodel, copies numeric attrs — worker.c:376-386 | **CLOSED** | -| 3.2 | Update `ep` on rejoin | `apply_join_state` sets `w->ep = ep` (worker.c:295), invoked by rejoin path — worker.c:388 | **CLOSED** | -| 3.3 | Preserve admin status bits | `admin_status = w->host.status & XCE_WORKER_ADMIN_MASK` (worker.c:374) restored `w->host.status |= admin_status` (worker.c:390); mask = LOCKEDU\|CLOSED\|HIDDEN\|CLOSED_BY_ADMIN — worker.c:275-277 | **CLOSED** | -| 4.1 | Clear `XCE_WORKER_UNAVAIL` | `w->host.status &= ~XCE_WORKER_UNAVAIL;` — worker.c:294 | **CLOSED** | -| 4.2 | Apply hconfig overrides (type/model/mxj slots/dispatch window/thresholds/bool res) | `apply_join_state` worker.c:297-334: hosttype/model "!"-guarded override, `set_host_slots(mxj, fallback maxcpus)`, dispatch_window dup, thresholds dup, `mark_host_res` loop | **CLOSED** | -| 4.3 | Update host-attr resource values (type,model,cpuf,maxcpus,maxmem,maxswap,maxtmp,ndisks,slots) | `add_hostattrs_resvals` (worker.c:223-250, 8 attrs) + `set_host_slots` (slots) — called worker.c:307,312 | **CLOSED** | -| 4.4 | Reject if `find_matched_hostconfig`==NULL, no workermgr mutation | fail-fast before any lock: `hconfig==NULL → reply.status=-errno/-EINVAL; return` — join.c:77-84 | **CLOSED** | -| 5.1 | BEFORE before mutation, AFTER after | `HOOK_BEFORE_EVENT` at join.c:87-88 (before key_find); `HOOK_AFTER_EVENT` at join.c:120-121 (after create/update, outside lock) | **CLOSED** | -| 5.2 | `on_worker_join` after successful join (both paths) | `on_worker_join(c, ep)` — join.c:147 (reached for both init and rejoin when idx>=0) | **CLOSED** | -| 5.3 | Critical section as short as current | No outer workermgr WRITE lock; only brief READ lock for key_find (join.c:91-95); mutation under per-worker WRITE lock in init/rejoin (worker.c:356-358, 372-392) | **CLOSED** | -| 5.4 | Existing tests pass; `add_worker` removed | Controller `add_worker` (old join path) deleted from worker.c & worker.h:11-14 (only `init_worker`/`rejoin_worker` declared); new `test_worker_join.c` registered in controller_test.c:37,166. NOTE: residual unrelated `add_worker` symbols exist — `add_worker_to_xlist` (misc.c:11) and worker-daemon `add_worker` (xcecd_worker_utils.c:12); both are distinct functions, not the refactored one | **CLOSED (verify build)** | - -**Score: 16/16 acceptance criteria CLOSED in code.** All 12 tasks have on-disk evidence. Design's planned architecture (explicit branch, no outer write lock, static `apply_join_state`, `init_worker`/`rejoin_worker`) matches the implementation exactly. - -### Implementation deltas vs design (not gaps — improvements/additions) - -1. **`validate_host` gate added** (join.c:65-72) — rejects unknown hosts with `-EPERM` before hostconfig lookup. Not in the original design; a correct hardening, no requirement conflict. -2. **`persist_commit_barrier` added** (join.c:129-136) — the success reply is gated on the join WAL entry being majority-committed (ha-commit-barrier-rollout Req 1.1-1.4). This is the cross-spec commit-durability integration; barrier failure becomes the error reply and skips `on_worker_join`, preserving Req 2.2/5.2 semantics. Landed after this spec's design was written; consistent with it. -3. **`apply_join_state` linkage** is `static` in production, non-`static` under `#ifdef XCE_TESTING` (worker.c:283-289) — enables the unit test to call it directly while keeping production encapsulation. Matches design intent. -4. **`reference(w)`/`dereference(w)` discipline** around the found worker (join.c:93-94, 109) — refcount safety across the lock release between key_find and rejoin. Correct; not specified but required given the read-lock-then-release pattern. - -### Residual / Deferred Gaps - -**No open requirement gaps for this spec.** All criteria are met in code. Residual items are verification-only or out-of-boundary: - -- **R1 (verification-only, carried from prior audit): green build/test on current HEAD.** worker.c/join.c are unmodified by HA-branch work, but join.c now depends on `persist_commit_barrier` and the controller test binary is shared with HA-branch changes (ha.c/raft.c) in the working tree. Action: run `/xce-build controller` and confirm `test_worker_join` (7 cases) plus `test_commit_barrier_reply` pass and no link interference. Effort: S (~1 build). Risk: Low. -- **R2 (latent, MEDIUM — was flagged in prior audit body): `apply_join_state` resource-update failures are swallowed.** `update_resvals` returns 0 unconditionally (worker.c:133-146; logs via `PERROR` but discards rc), and `mark_host_res` rc in the `apply_join_state` loop (worker.c:333-334) is ignored. So Req 4.3 "update all resource values" can partially fail silently and the join still reports success. This MATCHES pre-refactor behavior (Error Handling table, design.md:423 — "log with PERROR; continue"), so it is **not a regression and not a spec gap** — it is a pre-existing design-accepted behavior. Logged here only as a known limitation. -- **R3 (out-of-boundary — DO NOT count as this spec's gap): worker job-state resync on rejoin.** `rejoin_worker` updates attributes/endpoint/resources only. It does NOT recover running/finished job facts that were lost during the disconnect/failover window. `xce_join_req` carries only `hostattrs_t` (no job state). This is owned by the separate **worker-rejoin-resync** spec (roadmap line 26: "planned long ago, no effort yet"; dependencies: ha-consistency-invariants commit barrier + idempotency guards). Roadmap explicitly records this spec as "Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only)." **Boundary is clean and intentional.** - -### Options for the one actionable open item (R1) - -| Option | Description | Effort | Risk | -|--------|-------------|--------|------| -| **A (recommended)** | Run `/xce-build controller` on current HEAD; confirm `test_worker_join` + `test_commit_barrier_reply` green and no link interference with HA-branch ha.c/raft.c changes. Record result; close R1. | S (~1 build cycle) | Low | -| **B** | Defer until next controller build for any reason; treat R1 as a standing CI gate rather than a spec action. | None now | Low (build drift risk if join.c's `persist_commit_barrier` dep changes upstream) | -| **C** | Additionally harden R2: make `update_resvals`/`mark_host_res` propagate rc so a join with failed resource setup fails loud (per fail-loud preference). Re-scopes Req 4.3 error semantics — would need a spec amendment since current behavior is design-accepted. | M (touches worker.c shared helpers + tests; affects `restore_worker`/`mark_worker_unavail` callers) | Medium (behavior change to a shared path; out of original scope) | - -**Recommendation: Option A.** The spec is functionally complete (16/16 criteria CLOSED); only a build/test confirmation on the integrated HEAD remains. R2 is a pre-existing accepted behavior — pursue Option C only if a separate fail-loud hardening spec is opened. R3 belongs to worker-rejoin-resync. diff --git a/.kiro/specs/worker-join-refactor/spec.json b/.kiro/specs/worker-join-refactor/spec.json deleted file mode 100644 index 39a7cd64..00000000 --- a/.kiro/specs/worker-join-refactor/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "worker-join-refactor", - "created_at": "2026-05-15T00:00:00Z", - "updated_at": "2026-06-09T17:45:00Z", - "language": "en", - "phase": "implemented", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/worker-join-refactor/tasks.md b/.kiro/specs/worker-join-refactor/tasks.md deleted file mode 100644 index d9eeb284..00000000 --- a/.kiro/specs/worker-join-refactor/tasks.md +++ /dev/null @@ -1,86 +0,0 @@ -# Implementation Plan - -## Task Format - - - Detail items - - Observable completion condition - - _Requirements: X.X, Y.Y_ - ---- - -- [x] 1. Foundation: write failing tests for all new worker join behaviors (TDD Red) - -- [x] 1.1 Write failing tests for the shared status and resource update step - - Add a new test file (e.g., `test_worker_join.c`) to `src/controller/testsuite/` and register it in `controller_test.c` - - Tests assert: after the helper runs, `XCE_WORKER_UNAVAIL` is cleared on the worker's status, hosttype/model match the hconfig override values, slot count equals the hconfig mxj value, and host-attribute resources (type, model, cpuf, maxcpus, maxmem, maxswap, maxtmp, ndisks) hold the expected values - - Tests compile but fail because the helper does not yet exist - - _Requirements: 4.1, 4.2, 4.3_ - -- [x] 1.2 Write failing tests for the first-time worker creation path - - Tests assert: calling the creation function with a previously-unknown hostname produces a non-negative index; `key_find` on `workermgr` with that name succeeds after the call - - Tests assert: when the creation function returns an error, the error code is negative and no `on_worker_join` side effect occurs - - Tests compile but fail because the function does not yet exist - - _Requirements: 2.1, 2.2, 2.3_ - -- [x] 1.3 Write failing tests for the rejoin update path with admin-bit preservation - - Tests assert: after the update function runs, the worker's `host.attrs` reflects the new values and `w->ep` matches the new endpoint - - Tests assert: admin status bits (`XCE_WORKER_LOCKEDU`, `XCE_WORKER_CLOSED`, `XCE_WORKER_HIDDEN`, `XCE_WORKER_CLOSED_BY_ADMIN`) present before the call are still set after it - - Tests compile but fail because the function does not yet exist - - _Requirements: 3.1, 3.2, 3.3_ - ---- - -- [x] 2. Core: extract shared status and resource update into a standalone helper (TDD Green — 1.1) - - Move the status-clear + hconfig-override + resource-update block out of `add_worker` in `worker.c` into a new `static` helper function - - The helper covers: clearing `XCE_WORKER_UNAVAIL`, assigning `w->ep`, overriding hosttype/model strings from hconfig (with `dereference`/`makeString`), calling `add_hostattrs_resvals`, computing mxj slots via `set_host_slots`, replacing `dispatch_window_config` and `thresholds` from hconfig, and looping over `hconfig->nresources` with `mark_host_res` - - Must be called under the caller-held per-worker write lock; document this precondition with a comment - - Tests from 1.1 turn green; `add_worker` still compiles by delegating to the new helper - - _Requirements: 4.1, 4.2, 4.3_ - ---- - -- [x] 3. Core: implement the first-time creation and rejoin update helpers - -- [x] 3.1 Implement the first-time worker creation function (TDD Green — 1.2) - - Build a temporary `xce_worker_t` from the incoming hostname and attrs, then call `makeWorker` to register it in `workermgr` (makeWorker manages its own write lock) - - On a `NULL` return from `makeWorker`, log the failure and return `-errno` to the caller without touching `on_worker_join` - - On success, acquire the per-worker write lock, call the shared helper from Task 2, release the lock, and return `w->idx` - - Tests from 1.2 turn green - - _Requirements: 2.1, 2.2, 2.3_ - -- [x] 3.2 Implement the rejoin worker update function (TDD Green — 1.3) - - Acquire the per-worker write lock at the start; save `admin_status = w->host.status & XCE_WORKER_ADMIN_MASK` before any modification - - Free and replace the existing `hosttype` and `hostmodel` strings; overwrite numeric attrs fields (`cpufactor`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks`) from the incoming attrs argument - - Call the shared helper from Task 2 (status/resource update, ep assignment), then restore admin bits via `w->host.status |= admin_status`; release the lock - - Return `w->idx`; tests from 1.3 turn green and admin bits are unchanged after a full round-trip - - _Requirements: 3.1, 3.2, 3.3_ - ---- - -- [x] 4. Integration: wire the explicit existence check and new helpers into the join handler - -- [x] 4.1 Update exported worker function declarations - - Remove the `add_worker` declaration from `src/controller/include/worker.h` - - Add declarations for the first-time creation function and the rejoin update function with their full parameter lists (including the `xce_hostconfig_t*` parameter) - - Verify that the only direct caller of `add_worker` was `handle_worker_join`; if any other callers exist, update them to use the appropriate new function - - `worker.h` compiles cleanly with no remaining reference to `add_worker` - - _Requirements: 5.4_ - -- [x] 4.2 Rewrite the worker join handler with explicit existence check and branching - - Remove the outer `lock(c->workermgr, RTTI_LOCK_WRITE)` / `unlock` pair that currently wraps the `add_worker` call (eliminates the double write-lock deadlock) - - Move `find_matched_hostconfig` to the top of the handler, before any lock; reject the join with the appropriate error code and set `reply.status` if it returns `NULL` - - Acquire a brief workermgr read lock, call `key_find`, release the read lock immediately - - On `-ENOENT` call the first-time creation function; on a found entry call the rejoin update function; on any other error from `key_find` reject the join and return that error code - - Fire `BEFORE_EVENT / HOOK_DETAIL_WORKER_JOIN` before the `key_find` step; fire `AFTER_EVENT` after the create-or-update completes; call `xce_reply` and `on_worker_join` outside any lock - - Set `priv->u.idx` before calling `on_worker_join`; if create or update returned an error, set `reply.status`, call `xce_reply`, and return without calling `on_worker_join` - - The join handler compiles and the sequence check → create-or-update → status/resource-update is visible at the call site with no hidden branching inside `makeWorker` - - _Requirements: 1.1, 1.2, 1.3, 2.2, 2.3, 4.4, 5.1, 5.2, 5.3_ - ---- - -- [x] 5. Validation: run full test suite and confirm behavioral preservation - - Build the controller test binary and run all suites in `src/controller/testsuite/` - - Confirm `test_e2e_hooks` records `BEFORE_EVENT` before any workermgr mutation and `AFTER_EVENT` after it, matching the pre-refactor ordering - - If any test fixture in the suite constructed workers by calling `add_worker` directly, update the fixture to call the creation function with the added `xce_hostconfig_t*` argument - - `make test` exits 0; no previously-passing test is broken by the refactor - - _Requirements: 5.1, 5.2, 5.3, 5.4_ diff --git a/.kiro/specs/worker-rejoin-resync/brief.md b/.kiro/specs/worker-rejoin-resync/brief.md deleted file mode 100644 index ca3e5eff..00000000 --- a/.kiro/specs/worker-rejoin-resync/brief.md +++ /dev/null @@ -1,53 +0,0 @@ -# Brief: worker-rejoin-resync - -## Problem - -When a worker disconnects and reconnects (rejoin after network blip, or reconnect to a new primary after controller failover), any job start/finish/fail fact it sent during the disconnected window — and never got acknowledged — is permanently lost. Worker notify today is fire-and-forget with zero redelivery (bug-396). The controller has no way to ask "what happened to my jobs while you were gone," and the worker has no way to answer. - -Evidence: `xce_join_req` (`src/lib/request.c:1398-1429`) only attaches `hostattrs_t` when `role == HostRole_worker` (`joinmsg.data.attrs`) — no job-state payload exists on the join wire message at all. `handle_worker_join`/`handle_join_` (`src/controller/join.c`) process the join purely as a host-attribute registration; there is no code path that reconciles worker-reported job state against the controller's job table on (re)join. - -## Current State - -- `worker-join-refactor` (implemented) covers only attribute/endpoint update on rejoin — no job-state content. -- The controller-side pending-facts queue (`ha-consistency-invariants` Req 2.8) covers quorum-loss windows *while the worker stays connected* — it retries facts the controller received but couldn't commit. It does NOT cover facts that never arrived because the worker was disconnected or the controller itself failed over mid-flight. This spec is the designated backstop for that second case (`ha-consistency-invariants` Req 2.8 boundary context, elevated 2026-06-11). -- `raft-apply-decoupling` (implemented) elevated this spec to a **hard prerequisite** for the facts half of its own work (2026-06-25) — commit-then-apply idempotency guards this spec needs are now in place. - -## Desired Outcome - -On worker rejoin (disconnect/reconnect or failover reconnect), the worker reports its current running + recently-finished job state; the controller reconciles that report against its job table through the existing commit barrier, idempotently (no duplicate facts, no clobbering newer state the controller already has). Facts that would otherwise be silently lost in a disconnect/failover window are recovered. - -## Approach - -Extend the join/hello exchange (or add a post-join follow-up message) so a worker attaches a running+recent-job-state report, analogous to how `hostattrs_t` already rides the join message today. Controller-side, add a reconciliation path that runs this report through `mark_job_*`-style idempotent transitions (reusing the commit-then-apply guards `raft-apply-decoupling`/`ha-consistency-invariants` already built) rather than a new ad hoc mutation path. - -## Scope - -- **In**: Worker-side job-state rejoin reporting (running jobs + recently-finished/failed jobs not yet acked); controller-side idempotent reconciliation against the job table via the existing commit barrier; wire schema addition to the join exchange (or a new post-join message type); regression coverage for disconnect-during-job and failover-during-job scenarios. -- **Out**: apiserver/scheduler resync-on-reconnect (different data — cluster/queue/job *list* state, not job *execution facts* — tracked separately, see roadmap's deferred "lightweight rejoin re-attach" item, YAGNI until measured). Delta/incremental state transfer optimizations (session-token re-attach) — separate, deferred backlog item, not this spec's concern. Any change to the primary-side pending-facts queue itself (already implemented, this spec only complements it). - -## Boundary Candidates - -- Wire/schema: extending `xce_join_req`/join message (or new message type) to carry job-state report — worker + controller.capnp touchpoint. -- Worker-side: collecting "running + recently-finished-but-unacked" job state at rejoin time (worker.c / controller_connector.c). -- Controller-side: reconciliation logic reusing `mark_job_update`/`mark_job_fail`-style idempotent commit-then-apply transitions (job.c). - -## Out of Boundary - -- apiserver rejoin content (full state resend — separate, deferred item). -- scheduler rejoin content (separate, deferred item). -- Delta-based / session-token re-attach optimization (separate backlog item, explicit YAGNI until measured). - -## Upstream / Downstream - -- **Upstream (hard deps)**: `ha-consistency-invariants` (commit barrier + idempotency guards — implemented). `raft-apply-decoupling` (commit-then-apply seam this reuses — implemented, and the thing that elevated this spec to hard-prereq status for its own facts half). -- **Downstream**: `job-array-single-wal-record` and the broader Raft Apply-Ordering family reference this as a complements-not-replaces sibling to the pending-facts queue. The deferred "lightweight rejoin re-attach" backlog item lists this as one of its own dependencies (worker job-state content). - -## Existing Spec Touchpoints - -- **Extends**: `worker-join-refactor` (implemented — attribute/endpoint update only; this adds the job-state content that was explicitly out of that spec's scope). -- **Adjacent**: `ha-consistency-invariants` (Req 2.8 designates this as the official backstop — do not duplicate the pending-facts queue's mechanism, complement it). `component-failover-adoption` (in-progress sibling — makes daemons exercise libxce failover; does not itself cover job-state resync content). - -## Constraints - -- Must not introduce a second, competing idempotency mechanism — reconciliation goes through the same commit-then-apply guards the pending-facts queue and `raft-apply-decoupling` already use. -- Must not regress `worker-join-refactor`'s existing attribute/endpoint update behavior on rejoin. diff --git a/.kiro/specs/worker-rejoin-resync/spec.json b/.kiro/specs/worker-rejoin-resync/spec.json deleted file mode 100644 index 370e3939..00000000 --- a/.kiro/specs/worker-rejoin-resync/spec.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "feature_name": "worker-rejoin-resync", - "created_at": "2026-08-10T00:00:00Z", - "updated_at": "2026-08-10T00:00:00Z", - "language": "en", - "phase": "brief", - "description": "Worker re-reports running/finished job state on rejoin (disconnect or controller failover) so job-execution facts lost during the disconnected window are recovered via commit-then-apply idempotent reconciliation.", - "approvals": { - "requirements": { - "generated": false, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false, - "blocked_on": [] -} diff --git a/.kiro/specs/xce-test-cli-ux/requirements.md b/.kiro/specs/xce-test-cli-ux/requirements.md deleted file mode 100644 index 1c7138f3..00000000 --- a/.kiro/specs/xce-test-cli-ux/requirements.md +++ /dev/null @@ -1,92 +0,0 @@ -# Requirements Document - -## Introduction - -`xce_test` (`src/controller/testsuite/test.c`) is the REPL/scripting client engineers use — both interactively and via piped scripts — to drive controller, worker, scheduler, and HA soak/E2E testing. A `/cli-ux` review found that its interrupt handling, flag validation, and output-stream discipline do not give scripted callers a reliable pass/fail signal, and give interactive users no way to recover from a stuck command without losing their session. This feature makes `xce_test`'s command-line surface predictable and fail-loud: a mistyped flag, a stuck RPC, or a misrouted error message should never be silently absorbed or misreported. `xce_test`'s existing exit-code/auto-variable verdict machinery (`xce-test-e2e-primitives`, implemented) and its per-command `help <cmd>` texts are unaffected and are the foundation this feature builds on. - -## Boundary Context - -- **In scope**: process-level `-h`/`--help`/`--version` behavior; uniform rejection of unrecognized or malformed command-line flags across every `xce_test` command; consistent routing of error/diagnostic text to stderr and command-result data to stdout; Ctrl-C (SIGINT) behavior that cancels the in-flight operation or the current input line instead of silently doing nothing or exiting the session; a machine-readable output mode for the tabular listing commands (`workerlist`, `joblist`, `reslist`); command-not-found guidance that tells the user when a command exists in a different context; adding a long-form flag equivalent, consistently named per concept, for every command's existing short-only flags; command-name tab completion at the interactive prompt. -- **Out of scope**: the `xce_test` → `xce_harness`-style rename (tracked separately, to be done later); any change to controller/worker/scheduler wire protocol or RPC semantics; the auto-variable, exit-code-verdict, or shell-escape (`!`) mechanisms already shipped by `xce-test-e2e-primitives`; a full curses/TUI mode; color/theming; a `.xce_testrc` auto-executed startup-file feature (tracked as a separate, later task); changing what a successful command's result data actually contains; **renaming, removing, or changing the meaning of any existing short flag** (e.g. resolving `-m`'s four different meanings, or `-c`'s three, by reassigning a letter) — that is a breaking change to every script already using these flags and is deferred to a future, explicitly-scoped decision, not bundled into this feature; tab completion of flag *values* (jobids, hostnames, resource names) — only command-*name* completion is in scope; a per-command `-h`/`--help` flag handled by each `cmd_*` function individually — the existing in-session `help <cmd>` command already serves this need and is not being duplicated. -- **Adjacent expectations**: `xce-test-e2e-primitives` explicitly deferred structured/JSON output as out of its boundary — this feature now owns that gap for the listing commands only (`jobsub`/`joblist`'s existing `$jobid`/`$jobcount` auto-variable capture is unaffected and remains the scripted way to read those specific values). Cancelling a client-side wait on Ctrl-C does not need to cancel the request on the controller/worker/scheduler side — the daemon's own RPC handling and timeouts are unaffected. Existing scripts that parse the current human-readable table output must keep working unchanged when the new machine-readable mode is not requested. Requirement 4's process-level `-h`/`--help` and the existing in-session `help <cmd>` command remain the sole help surfaces — Requirement 7's long-form flags are additive documentation of an existing command's own flags, not a new help mechanism. - -## Requirements - -### Requirement 1: Responsive Interrupt Handling - -**Objective:** As an operator running `xce_test` interactively against a live or degraded HA cluster, I want Ctrl-C to cancel whatever is currently happening instead of silently doing nothing or exiting my whole session, so that I can recover control during a stuck command without losing my session state. - -#### Acceptance Criteria -1. When the operator presses Ctrl-C while a command is blocked awaiting a cluster RPC reply, the xce_test session shall abandon the wait immediately, without waiting for the RPC's own timeout to elapse, and return control to the prompt. -2. When the operator presses Ctrl-C at the command prompt with no command in flight, the xce_test session shall discard the currently-typed input line and redisplay the prompt. -3. If a command is interrupted by Ctrl-C, then the xce_test session shall report that command as failed, consistent with how any other failing command is reported. -4. The xce_test session shall exit only as a direct result of an explicit `quit` command, an EOF at the top-level prompt, or an abort-on-failure condition in scripted mode — never as a side effect of a single Ctrl-C keypress. -5. While a command is being interrupted by Ctrl-C, the xce_test session shall leave the active worker/scheduler/client context and its underlying connection intact. - -### Requirement 2: Fail-Loud Flag Validation - -**Objective:** As an author of HA soak/E2E test scripts driven through `xce_test`, I want every command to reject a flag it doesn't recognize or that is missing a required value, so that a typo in a script fails loudly instead of silently running with an unintended default. - -#### Acceptance Criteria -1. When any xce_test command is invoked with a flag it does not recognize, the xce_test session shall reject the command, report the invalid flag, and return a nonzero result, instead of running the command while ignoring that flag. -2. When any xce_test command is invoked with a flag that requires a value but none is given, the xce_test session shall reject the command and report which flag is missing its value, instead of proceeding with an empty or default value. -3. The xce_test session shall apply Acceptance Criteria 1 and 2 uniformly across every command that accepts flags — no command shall silently ignore an unrecognized or malformed flag. -4. When the xce_test session rejects a command for an invalid or malformed flag, the xce_test session shall include that command's usage summary in the error output. - -### Requirement 3: Consistent Error and Output Stream Routing - -**Objective:** As an author of scripts that pipe or redirect `xce_test`'s output, I want every error and diagnostic message on stderr and every command result on stdout, so that I can reliably separate machine-parseable results from human diagnostics no matter which command fails. - -#### Acceptance Criteria -1. The xce_test session shall write every error, warning, or diagnostic message to stderr. -2. The xce_test session shall write a command's successful result data to stdout only. -3. The xce_test session shall apply Acceptance Criteria 1 and 2 uniformly across every command and every error path, including flag-validation errors, RPC failures, and internal errors. -4. While xce_test's stdout and stderr are redirected to different destinations (a script or pipe), the xce_test session shall route a given message to the same stream it would use in an interactive TTY session. - -### Requirement 4: Process-Level Help and Version Discovery - -**Objective:** As a new or infrequent user of `xce_test`, I want `-h`/`--help` and `--version` to behave the way they do in other command-line tools, so that I can learn how to use `xce_test` without already knowing its internals. - -#### Acceptance Criteria -1. When xce_test is invoked with `-h` or `--help`, the xce_test process shall print usage help and exit without starting the interactive/scripted command session. -2. When xce_test is invoked with `--version`, the xce_test process shall print its version information and exit without starting the interactive/scripted command session. -3. The xce_test process's `-h`/`--help` output shall describe the in-session `help` command as the way to list and learn available commands. - -### Requirement 5: Machine-Readable Output for Listing Commands - -**Objective:** As an author of an automated soak/E2E script, I want the tabular listing commands to offer a structured, machine-parseable output mode, so that my script can reliably extract fields without depending on fixed-width column alignment. - -#### Acceptance Criteria -1. Where a machine-readable output mode is requested for `workerlist`, `joblist`, or `reslist`, the xce_test session shall emit each record in a structured, parseable format (for example, JSON) containing the same fields as that command's human-readable table. -2. When no machine-readable output mode is requested, the xce_test session shall continue to print the existing human-readable table for `workerlist`, `joblist`, and `reslist`, unchanged. -3. The xce_test session shall not truncate or misalign a field's value in machine-readable output regardless of the field value's length. -4. The xce_test session shall document, via `help <cmd>`, how to request the machine-readable output mode for each command that supports it. - -### Requirement 6: Context-Aware Command-Not-Found Guidance - -**Objective:** As a user new to `xce_test`'s top/worker/scheduler/client context tree, I want an unrecognized command to tell me if it exists in a different context, so that I can find where to run it without trial and error. - -#### Acceptance Criteria -1. When a typed command is not found in the active context, and a command of that exact name exists in a different context, the xce_test session shall report the command as not found in the current context and identify the context where it does exist. -2. When a typed command is not found in the active context and does not exist in any context, the xce_test session shall report it as not found without naming an unrelated context. -3. The xce_test session shall not change the active context or execute any command as a result of a not-found lookup. - -### Requirement 7: Long-Form, Cross-Command-Consistent Flags - -**Objective:** As a user or script author who has learned one `xce_test` command's flags, I want a long-form equivalent of every flag, named the same way for the same concept across every command that accepts it, so that muscle memory from one command doesn't mislead me in another and my scripts can be self-documenting. - -#### Acceptance Criteria -1. The xce_test session shall provide a long-form flag equivalent for every flag that is currently available only as a short (single-letter) flag. -2. The xce_test session shall use the same long-form flag name for the same real-world concept (for example: a target host, a command line to run, a resource name, a job identifier) in every command that accepts that concept, regardless of what short letter each command already uses for it. -3. When a long-form flag equivalent is added for an existing short flag, the xce_test session shall leave that short flag's accepted values, default, and behavior unchanged. -4. The xce_test process's help output for a command (`help <cmd>`) shall list both the short and long form of each of that command's flags. - -### Requirement 8: Command-Name Tab Completion - -**Objective:** As an interactive user, I want tab completion for command names at the prompt, so that I can discover and enter commands without memorizing the full command table or re-running `help`. - -#### Acceptance Criteria -1. While the session is interactive and the operator presses tab while typing a command name, the xce_test session shall offer completions drawn only from the command names valid in the currently active context. -2. When the partially-typed text unambiguously matches exactly one command name in the active context, pressing tab shall complete it in full. -3. When the partially-typed text matches more than one command name in the active context, pressing tab shall list the matching candidates without altering the input line. -4. The xce_test session shall not offer a command name as a completion candidate when that command is not valid in the currently active context. diff --git a/.kiro/specs/xce-test-cli-ux/spec.json b/.kiro/specs/xce-test-cli-ux/spec.json deleted file mode 100644 index f19d0cdd..00000000 --- a/.kiro/specs/xce-test-cli-ux/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "xce-test-cli-ux", - "created_at": "2026-08-26T08:38:15Z", - "updated_at": "2026-08-26T09:44:21Z", - "language": "en", - "phase": "requirements-generated", - "approvals": { - "requirements": { - "generated": true, - "approved": false - }, - "design": { - "generated": false, - "approved": false - }, - "tasks": { - "generated": false, - "approved": false - } - }, - "ready_for_implementation": false -} diff --git a/.kiro/specs/xce-test-e2e-primitives/design.md b/.kiro/specs/xce-test-e2e-primitives/design.md deleted file mode 100644 index c77aa5d2..00000000 --- a/.kiro/specs/xce-test-e2e-primitives/design.md +++ /dev/null @@ -1,154 +0,0 @@ -# Design Document — xce-test-e2e-primitives - -## Overview - -**Purpose:** Make the `xce_test` client usable as a script-driven E2E driver: a piped command sequence whose **process exit code is the pass/fail verdict**, able to drive node lifecycle via a shell-escape, wait for failover deterministically, and assert outcomes in-tool. - -**Users:** Test/CI authors and maintainers writing failover/durability/cold-restart E2E scenarios (e.g. the `test-commit-durability.sh` soak). - -**Impact:** Today `xce_test` is an interactive client: `execute_line()` parses a line into argv, dispatches to a `command_t.func` (returning `int`, `<0` = error), and the main loop logs failures but **continues and exits 0**. Success is only inferable by grepping prose, and `jobsub` can print a garbage JobId (bug-786). This feature adds: session failure tracking + a real exit code; a `!` shell-escape; a fixed auto-variable substitution pass (`$?`, `$jobid`, `$jobcount`); `wait-primary`, `expect-jobs`, `expect-job`, `echo` commands; `#`/blank-line tolerance; and a `jobsub` reply-status fix. Everything lives in the single test-client TU; the shell remains the orchestrator. - -### Goals -- The process exit code reflects whether any command failed (Req 1). -- A `!cmd` shell-escape lets one script drive node lifecycle; its status feeds `$?` (Req 2). -- A fixed, tool-maintained auto-variable set (`$?`, `$jobid`, `$jobcount`) — no user variables/expressions (Req 3). -- `wait-primary` blocks on the existing client failover until an op succeeds (Req 4). -- `expect-jobs`/`expect-job` assertions fail the session on mismatch (Req 5). -- `jobsub` reports success only on a clean ack with a valid JobId (Req 7). - -### Non-Goals -- Structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test`; user-assignable variables, expressions, conditionals, or loops (the shell owns these). - -## Boundary Commitments - -### This Spec Owns -- The `xce_test` command loop's **failure tracking + process exit code**, an **abort-vs-continue** mode flag, and `#`/blank-line handling. -- A **line pre-processing step** in the client that (a) recognizes a leading `!` as a shell-escape and (b) substitutes the fixed auto-variables before tokenizing. -- The fixed **auto-variable state** (`$?`, `$jobid`, `$jobcount`) and the points that set it. -- New client commands: `wait-primary`, `expect-jobs`, `expect-job`, `echo`; the `set continue-on-error` toggle. -- The **`cmd_jobsub` reply-status correctness** fix (bug-786). -- The conversion of `src/tools/test-commit-durability.sh` to use these primitives. - -### Out of Boundary -- Controller/replication behavior and opcodes (consumed unchanged). -- libxce client failover/master-rotation internals (consumed by `wait-primary`/`jobsub`/`joblist`). -- Node lifecycle (deploy/kill/restart) — performed by the shell, reachable via `!cmd`. -- Any general scripting language (variables/expressions/control flow). - -### Allowed Dependencies -- The existing `command_t`/`cmd_func_t` dispatch, `execute_line`, `parse_argv`, `find_command`, and the main read loop in `test.c`. -- The existing client context (`active_ctx`), `cmd_jobsub`/`cmd_joblist` paths, and the libxce request/failover APIs they already call. -- Standard C `system()` for the shell-escape; a monotonic clock helper for `wait-primary` timeout. - -### Revalidation Triggers -- Any change to the `cmd_func_t` contract (return-code semantics) → re-check all existing command handlers. -- A change to how `jobsub`/`joblist` report acks or list jobs → re-check `$jobid`/`$jobcount` capture and the assertions. - -## Architecture - -### Existing Architecture Analysis -- `execute_line(char* line)`: `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; returns the handler's `int` rc (`<0` error). -- Main loop: reads a line, `rc = execute_line(s)`; on `rc < 0` sets errno via `SET_ERROR` and **continues**; `main` returns 0. So failures never affect the exit code. -- Commands return `int` (`0` ok, `<0` errno-style). `cmd_jobsub`/`cmd_joblist` already drive the client + libxce failover. - -### Pattern -Extend-in-place. Introduce a thin **pre-processing + verdict** wrapper around the existing dispatch; the new commands are ordinary `command_t` entries. No new module, no new files. - -```mermaid -graph TD - Loop[main read loop] --> Pre[preprocess line] - Pre -->|leading '#'/blank| Skip[ignore] - Pre -->|leading '!'| Shell[run via system; set $?] - Pre -->|else| Subst[substitute $? / $jobid / $jobcount] - Subst --> Exec[execute_line -> find_command -> func] - Exec --> Track[update session-failed + $?] - Shell --> Track - Track -->|abort mode + failed| Stop[stop loop] - Track --> Loop - Loop --> Verdict[main returns 1 if any failure else 0] -``` - -## File Structure Plan - -### Modified Files -- `src/controller/testsuite/test.c` — all client changes: - - **Session/verdict state** (file-scope): `g_session_failed` (int), `g_continue_on_error` (bool), and auto-vars `g_last_status` (int), `g_last_jobid[XCE_MAXNAME_LEN]` (string), `g_last_jobcount` (int). - - **`preprocess_and_run(char* line)`** (new): handle blank/`#`; leading `!` → `system(rest)`, set `g_last_status` from `WEXITSTATUS`; else substitute auto-vars then call `execute_line`. Update `g_session_failed`/`g_last_status` from the result; honor abort-vs-continue. Called from the main loop in place of the direct `execute_line`. - - **`substitute_autovars(line) → expanded`** (new): replace whole-token `$?`, `$jobid`, `$jobcount` with current values; a reference with no current value yields a command failure (Req 3.6). - - **New command handlers**: `cmd_wait_primary`, `cmd_expect_jobs`, `cmd_expect_job`, `cmd_echo`, registered in the command table(s). - - **`cmd_set`**: add `continue-on-error` toggle. - - **`cmd_jobsub`**: gate the "ok" report on a successful reply status + valid JobId; on success set `g_last_jobid`; otherwise print a typed error and return `<0` (bug-786). - - **`cmd_joblist`**: set `g_last_jobcount`; expose a reusable internal count/contains helper for the assertions. - - **`main`**: return `g_session_failed ? 1 : 0`. -- `src/tools/test-commit-durability.sh` — convert to drive `xce_test` via piped scripts using `wait-primary`/`expect-jobs`/`$jobid` and the exit-code verdict instead of prose-grepping + sanitizer filtering. - -### New Files -- None. - -> Note: the process exit code is only a trustworthy verdict if a clean (leak-free) exit is possible. Reuse the existing test run convention (`ASAN_OPTIONS=detect_leaks=0`) for the soak; leak cleanup of the client is out of scope here and tracked separately. - -## Components and Interfaces - -| Component | Intent | Req Coverage | -|-----------|--------|--------------| -| Verdict & loop wrapper (`preprocess_and_run`, `g_session_failed`, exit code) | Turn command outcomes into a process verdict; abort/continue | 1.1–1.6, 6.1, 6.2 | -| Shell-escape (`!cmd` via `system`) | Drive node lifecycle / setup from a script | 2.1–2.4 | -| Auto-vars (`substitute_autovars`, `g_last_*`) | Reference last status / JobId / job count | 3.1–3.6 | -| `cmd_wait_primary` | Block until a client op succeeds (failover-aware) | 4.1–4.4 | -| `cmd_expect_jobs` / `cmd_expect_job` | Assert job count / membership; fail session on mismatch | 5.1–5.4 | -| `cmd_echo` | Progress/section messages | 6.3 | -| `cmd_jobsub` fix | Report success only on clean ack + valid JobId; set `$jobid` | 7.1–7.3, 3.2 | -| `test-commit-durability.sh` driver | Reference E2E using the primitives | 8.1–8.3 | - -### Verdict & loop wrapper -- **State (file-scope):** `g_session_failed`, `g_continue_on_error`, `g_last_status`, `g_last_jobid[]`, `g_last_jobcount`. -- **Contract:** every executed command updates `g_last_status` and, on failure, sets `g_session_failed`. In default mode the loop stops on first failure; under `continue-on-error` it keeps going. `main` returns non-zero iff `g_session_failed`. - -### Shell-escape -- A line whose first non-whitespace char is `!` runs `system(rest_of_line)`. `g_last_status = WEXITSTATUS(rc)` (or non-zero on `system` failure). A non-zero status counts as a command failure. - -### Auto-variables -- Whole-token substitution of `$?` (→ `g_last_status`), `$jobid` (→ `g_last_jobid`), `$jobcount` (→ `g_last_jobcount`) before tokenizing. Unset `$jobid`/`$jobcount` → command failure (no garbage). No other variable syntax is recognized. - -### cmd_wait_primary -- `wait-primary [timeout_ms]`: loop a cheap client op (e.g. the `joblist`/probe path through `active_ctx`, which already does master rotation) until success or the timeout (monotonic clock); return 0 on success, `<0` on timeout. Default timeout when arg omitted. - -### cmd_expect_jobs / cmd_expect_job -- `expect-jobs N`: obtain the current job count (shared helper with `cmd_joblist`); equal → 0, else print expected/actual and return `<0`. -- `expect-job <jobid>`: list jobs; present → 0, else print the missing id and return `<0`. - -### cmd_jobsub (bug-786 fix) -- Inspect the reply: only when the controller returns a success status with a well-formed JobId does it print `jobsubmit ok: JobId: <id>` and set `g_last_jobid`. Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id. - -## Error Handling -- **Command failure** → `g_last_status != 0`, `g_session_failed = 1`; abort (default) or continue. Non-zero process exit at end. -- **Shell-escape failure** → treated as a command failure via `$?`. -- **Unset auto-var reference** → command failure with a clear message (never substitutes garbage). -- **`wait-primary` timeout** and **assertion mismatch** → command failure with expected/actual reported. - -## Testing Strategy - -### Unit / behavioral (driven through `xce_test` itself; no controller needed for most) -- Exit code: a script with all-passing commands exits 0; a script with a failing `expect-jobs` exits non-zero (Req 1.1–1.3). -- Abort vs continue: a failing command mid-script stops execution by default; under `set continue-on-error` later commands still run and the final exit is non-zero (Req 1.4, 1.5). -- Shell-escape: `!true` → `$?`=0, success; `!false` → `$?`≠0, session fails (Req 2.1–2.4). -- Auto-vars: `$?` reflects the prior command; an unset `$jobid` reference fails rather than substituting garbage (Req 3.1, 3.6). -- Ergonomics: `#` comment and blank lines are ignored; `echo` succeeds (Req 6). -- `jobsub` reporting: a non-acked submit returns failure and does not set `$jobid`; never prints a malformed id (Req 7) — verifiable against a live single node or by simulating a non-ack. - -### Integration / E2E (live `/xce-deploy-ha` cluster) -- Reference driver: converted `test-commit-durability.sh` runs a scenario — `wait-primary`, `jobsub` (capture `$jobid`), `expect-jobs`, `!kill` the primary, `wait-primary`, `expect-job $jobid` — and gates on the `xce_test` exit code (Req 4, 5, 8). Adding a `!`-driven cold-restart-all + `expect-jobs` step exercises the durability path (would catch a bug-778-class regression). - -## Requirements Traceability - -| Requirement | Component | Test | -|-------------|-----------|------| -| 1.1–1.6 | Verdict & loop wrapper | exit-code + abort/continue tests | -| 2.1–2.4 | Shell-escape | `!true`/`!false` tests | -| 3.1–3.6 | Auto-vars | `$?`/`$jobid` substitution + unset-fails tests | -| 4.1–4.4 | cmd_wait_primary | live failover wait | -| 5.1–5.4 | cmd_expect_jobs / cmd_expect_job | assertion pass/fail tests | -| 6.1–6.3 | loop wrapper + cmd_echo | comment/blank/echo tests | -| 7.1–7.3 | cmd_jobsub fix | non-ack returns failure; no garbage id | -| 8.1–8.3 | test-commit-durability.sh | converted soak runs, exit-code-gated | -``` diff --git a/.kiro/specs/xce-test-e2e-primitives/requirements.md b/.kiro/specs/xce-test-e2e-primitives/requirements.md deleted file mode 100644 index 9a88b465..00000000 --- a/.kiro/specs/xce-test-e2e-primitives/requirements.md +++ /dev/null @@ -1,94 +0,0 @@ -# Requirements Document - -## Introduction - -This feature adds a small set of scripting primitives to the `xce_test` client so that a complete end-to-end scenario — submit work, kill the primary, wait for failover, restart/cold-restart nodes, and assert that committed work survived — can be driven from a single piped or scripted command sequence whose **process exit code is the pass/fail verdict**. Today `xce_test` is an interactive client whose success can only be inferred by grepping human-readable output (and which can emit a malformed JobId), so E2E scripts must filter prose, suppress sanitizer noise, and poll controller logs for cluster state. These primitives let the shell remain the orchestrator (node lifecycle, loops) while `xce_test` reliably reports outcomes. - -## Boundary Context - -- **In scope**: command-failure tracking and a process exit code that reflects it; a `!`-prefixed shell-escape command; a fixed set of tool-maintained auto-variables (`$?`, `$jobid`, optional `$jobcount`); a `wait-primary` blocking command; `expect-jobs` / `expect-job` assertion commands; script ergonomics (`#` comments, blank lines, `echo`); correct `jobsub` success/failure reporting (bug-786); converting `src/tools/test-commit-durability.sh` to use these primitives. -- **Out of scope**: structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test` (kill/restart/deploy stay in the shell, reachable via `!cmd`); user-assignable variables, expression evaluation, conditionals, or loops (the shell owns these); any change to controller/replication behavior. -- **Adjacent expectations**: `wait-primary` and `jobsub`/`joblist` rely on the existing libxce client-side master-rotation/failover already built into `xce_test`; this feature consumes that behavior and does not change it. - -## Requirements - -### Requirement 1: Exit code reflects command outcomes - -**Objective:** As an E2E script author, I want the `xce_test` process to exit non-zero when any command in the session failed, so that a shell can use the exit code alone as the pass/fail verdict. - -#### Acceptance Criteria -1. When a command in the session fails, the xce_test client shall record the session as failed. -2. When the session ends and at least one command failed, the xce_test client shall exit with a non-zero status. -3. When the session ends and every command succeeded, the xce_test client shall exit with status zero. -4. While default (abort-on-failure) mode is active, when a command fails, the xce_test client shall stop executing subsequent commands and exit non-zero. -5. Where `set continue-on-error` has been issued, when a command fails, the xce_test client shall continue executing subsequent commands and still exit non-zero at session end. -6. The xce_test client shall send all human-readable log and diagnostic output in a way that does not corrupt the determination of the exit-code verdict. - -### Requirement 2: Shell-escape command - -**Objective:** As an E2E script author, I want to run an arbitrary shell command from within an `xce_test` script, so that a single script can drive node lifecycle (kill, restart, deploy) and environment setup without a separate orchestrator. - -#### Acceptance Criteria -1. When a script line begins with `!`, the xce_test client shall execute the remainder of the line as a shell command. -2. When the shell command exits zero, the xce_test client shall treat the `!` command as successful. -3. If the shell command exits non-zero, then the xce_test client shall treat the `!` command as failed (subject to Requirement 1's exit-code and abort/continue behavior). -4. When a `!` command completes, the xce_test client shall set `$?` to the shell command's exit status. - -### Requirement 3: Tool-maintained auto-variables - -**Objective:** As an E2E script author, I want a fixed set of auto-variables that later commands can reference, so that I can assert on outcomes without parsing client output in the shell. - -#### Acceptance Criteria -1. When any command completes, the xce_test client shall set `$?` to that command's status (zero on success, non-zero on failure). -2. When a `jobsub` command receives a successful acknowledgment, the xce_test client shall set `$jobid` to the acknowledged JobId. -3. When a subsequent command references `$jobid` (e.g. `expect-job $jobid`), the xce_test client shall substitute the most recently acknowledged JobId. -4. Where the `$jobcount` auto-variable is provided, when a `joblist` command completes, the xce_test client shall set `$jobcount` to the number of jobs returned. -5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables. -6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value. - -### Requirement 4: Wait for an elected primary - -**Objective:** As an E2E script author, I want a command that blocks until the cluster has a serving primary, so that scripts can proceed deterministically after a kill or restart without fixed sleeps or log polling. - -#### Acceptance Criteria -1. When `wait-primary` is issued, the xce_test client shall repeatedly attempt a client operation against the configured masters until one succeeds. -2. When a client operation succeeds within the timeout, the xce_test client shall complete `wait-primary` successfully. -3. If no client operation succeeds within the timeout, then the xce_test client shall fail `wait-primary`. -4. Where a timeout argument is supplied (`wait-primary <timeout_ms>`), the xce_test client shall use it as the maximum wait; where it is omitted, the client shall use a default timeout. - -### Requirement 5: Outcome assertions - -**Objective:** As an E2E script author, I want assertion commands that fail the session on mismatch, so that durability/availability expectations are checked in-tool rather than by parsing output. - -#### Acceptance Criteria -1. When `expect-jobs N` is issued and the current job count equals `N`, the xce_test client shall treat the assertion as passed. -2. If `expect-jobs N` is issued and the current job count does not equal `N`, then the xce_test client shall fail the assertion (non-zero status) and report the expected and actual counts. -3. When `expect-job <jobid>` is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed. -4. If `expect-job <jobid>` is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId. - -### Requirement 6: Script ergonomics - -**Objective:** As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting. - -#### Acceptance Criteria -1. When a script line's first non-whitespace character is `#`, the xce_test client shall treat the line as a comment and ignore it. -2. When a script line is empty or whitespace-only, the xce_test client shall ignore it without error. -3. When `echo <message>` is issued, the xce_test client shall emit the message to output and treat the command as successful. - -### Requirement 7: Correct job-submission reporting - -**Objective:** As an E2E script author, I want `jobsub` to report success only when a job was genuinely accepted, so that scripts never chase a job the cluster never committed (bug-786). - -#### Acceptance Criteria -1. When `jobsub` receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set `$jobid` to that JobId. -2. If `jobsub` does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status. -3. The xce_test client shall never report `jobsub` success with a malformed or uninitialized JobId. - -### Requirement 8: Reference E2E driver - -**Objective:** As a maintainer, I want the existing durability soak to use these primitives, so that there is a working reference for script-driven E2E and the primitives are exercised. - -#### Acceptance Criteria -1. The `test-commit-durability.sh` soak shall be converted to drive `xce_test` using the exit-code verdict and the new primitives instead of grepping prose and filtering sanitizer output. -2. When the converted soak runs against a healthy cluster, it shall report pass/fail using `xce_test` exit codes and the assertion commands. -3. The converted soak shall preserve its existing coverage (rolling primary kill + restart, zero acknowledged-job loss across failover). diff --git a/.kiro/specs/xce-test-e2e-primitives/research.md b/.kiro/specs/xce-test-e2e-primitives/research.md deleted file mode 100644 index b2bab377..00000000 --- a/.kiro/specs/xce-test-e2e-primitives/research.md +++ /dev/null @@ -1,21 +0,0 @@ -# Research / Discovery — xce-test-e2e-primitives - -## Discovery type -LIGHT (extension to an existing single-file test client). No external research needed. - -## Key findings (from src/controller/testsuite/test.c) -- Dispatch: `execute_line(line)` → `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; handlers return `int` (`<0` = errno-style error). Command table is `command_t {name, func, doc}` arrays per context. -- Main read loop reads a line, `rc = execute_line(s)`, and on `rc<0` only `SET_ERROR(-rc)` + continues; `main` returns 0 → **failures never affect the exit code** (the core gap for scriptable E2E). -- `cmd_jobsub`/`cmd_joblist` already drive the client via `active_ctx` and libxce client-side master-rotation/failover — `wait-primary` and the assertions can reuse these paths. -- bug-786: `cmd_jobsub` can print `jobsubmit ok: JobId: 63-15--1-...` (uninitialized id) on a non-clean ack; controller then logs `handle_jobsubmit fail to lock job <garbage>`. Root: success is reported without verifying the reply status/jobid. - -## Design decisions -- **Extend-in-place, single TU.** Add a `preprocess_and_run` wrapper (blank/`#`/`!`/auto-var-substitution) + file-scope verdict state; new commands as ordinary table entries; `main` returns `g_session_failed ? 1 : 0`. No new files, no new opcode, no JSON. -- **Fixed auto-vars only** (`$?`, `$jobid`, `$jobcount`) — substitution, not a variable system; unset reference fails (no garbage). -- **Shell-escape via `system()`**, status → `$?`; node lifecycle stays in the shell. -- **`wait-primary`** loops an existing failover-aware client op until success/timeout (monotonic clock). -- Exit-code trustworthiness depends on a clean exit; reuse `ASAN_OPTIONS=detect_leaks=0` for the soak (client leak cleanup is separate/out of scope). - -## Risks -- `system()` shell-escape = arbitrary execution — acceptable for a dev/test client only. -- Auto-var substitution must be whole-token and conservative to avoid surprising expansions. diff --git a/.kiro/specs/xce-test-e2e-primitives/spec.json b/.kiro/specs/xce-test-e2e-primitives/spec.json deleted file mode 100644 index 3929fdb5..00000000 --- a/.kiro/specs/xce-test-e2e-primitives/spec.json +++ /dev/null @@ -1,27 +0,0 @@ -{ - "feature_name": "xce-test-e2e-primitives", - "created_at": "2026-06-19T15:11:01Z", - "updated_at": "2026-06-19T00:00:00Z", - "language": "en", - "phase": "implemented", - "implementation": { - "completed": true, - "tasks_done": "1.1-7.2 (all)", - "notes": "All primitives + soak conversion landed and reviewed; verified live on a 3-node cluster. bug-786 fully closed (gate strengthened to jobid == submitted id). The soak's bonus cold-restart-all step surfaced upstream bug-795 (total-outage committed-job loss) — owned by ha-commit-durability Phase 6, out of this spec's boundary. Reqs 8.2/8.3 met (exit-code verdict + rolling-kill zero-loss preserved)." - }, - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} \ No newline at end of file diff --git a/.kiro/specs/xce-test-e2e-primitives/tasks.md b/.kiro/specs/xce-test-e2e-primitives/tasks.md deleted file mode 100644 index 6234c681..00000000 --- a/.kiro/specs/xce-test-e2e-primitives/tasks.md +++ /dev/null @@ -1,100 +0,0 @@ -# Implementation Plan - -> TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor). -> Almost all work is in the single test client TU `src/controller/testsuite/test.c`, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through `xce_test` itself (piped scripts), gating on its exit code. - -- [x] 1. Foundation: verdict, loop wrapper, ergonomics -- [x] 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling - - Add file-scope state (`g_session_failed`, `g_continue_on_error`, `g_last_status`) and a `preprocess_and_run(line)` wrapper used by the main read loop: ignore blank and `#`-comment lines; run normal lines via `execute_line`; update `g_last_status`/`g_session_failed` from the result; in default mode stop on first failure, under `continue-on-error` keep going. `main` returns `g_session_failed ? 1 : 0`. Extend `cmd_set` with a `continue-on-error` toggle. - - Observable: a piped script with a bogus/failing command makes `xce_test` exit non-zero; an all-valid script exits 0; with `set continue-on-error` a failing line does not stop later lines but the final exit is still non-zero; `#`/blank lines are ignored. - - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 6.1, 6.2_ - - _Boundary: VerdictLoopWrapper_ -- [x] 1.2 `echo` command - - Add `cmd_echo` (table-registered) that prints its argument message and returns success. - - Observable: `echo hello` emits `hello` and the command succeeds (exit unaffected). - - _Requirements: 6.3_ - - _Boundary: VerdictLoopWrapper_ - - _Depends: 1.1_ - -- [x] 2. Shell-escape -- [x] 2.1 `!<cmd>` runs a shell command and feeds `$?` - - In `preprocess_and_run`, when the first non-whitespace char is `!`, run the remainder via `system()`; set `g_last_status = WEXITSTATUS(rc)` (non-zero on `system` failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code). - - Observable: `!true` succeeds and sets `$?`=0; `!false` fails the session and sets `$?`≠0; a script can `!kill ...` / `!bash start.sh ...` to drive node lifecycle. - - _Requirements: 2.1, 2.2, 2.3, 2.4_ - - _Boundary: ShellEscape_ - - _Depends: 1.1_ - -- [x] 3. Auto-variable substitution -- [x] 3.1 Substitute the fixed auto-vars `$?`, `$jobid`, `$jobcount` - - Add `substitute_autovars(line)` (run before tokenizing in `preprocess_and_run`): replace whole-token `$?`→`g_last_status`, `$jobid`→`g_last_jobid`, `$jobcount`→`g_last_jobcount`. A reference to an unset `$jobid`/`$jobcount` fails the command (no garbage substitution). No other variable syntax is recognized. - - Observable: after a command, `echo $?` prints the prior status; referencing `$jobid` before any successful `jobsub` fails that command rather than substituting an empty/garbage value. - - _Requirements: 3.1, 3.3, 3.5, 3.6_ - - _Boundary: AutoVars_ - - _Depends: 1.1_ - -- [x] 4. Correct job-submission reporting (bug-786) + `$jobid` capture -- [x] 4.1 `cmd_jobsub` reports success only on a clean ack with a valid JobId - - Gate the `jobsubmit ok: JobId: <id>` report on a successful reply status AND a well-formed JobId; on success set `g_last_jobid` (feeds `$jobid`, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id (`63-15--1-...`). - - Observable: a submit that does not get a clean ack returns failure and does NOT set `$jobid`; `jobsub` never prints success with a malformed id; a successful submit sets `$jobid` to the acked JobId. - - _Requirements: 7.1, 7.2, 7.3, 3.2_ - - _Boundary: JobsubFix_ - - _Depends: 1.1, 3.1_ - -- [x] 5. Job-count capture and assertions -- [x] 5.1 `cmd_joblist` sets `$jobcount`; extract a shared count/contains helper - - Refactor `cmd_joblist` to expose an internal helper that returns the current job count and tests membership of a JobId; set `g_last_jobcount` after a `joblist`. - - Observable: after `joblist`, `echo $jobcount` prints the number of jobs returned; the helper is reusable by the assertions. - - _Requirements: 3.4_ - - _Boundary: JoblistCount_ - - _Depends: 1.1, 3.1_ -- [x] 5.2 `expect-jobs N` assertion - - Add `cmd_expect_jobs`: pass when the current job count equals `N`; otherwise print expected vs actual and return `<0`. - - Observable: `expect-jobs <correct>` succeeds; `expect-jobs <wrong>` fails the session and prints expected/actual. - - _Requirements: 5.1, 5.2_ - - _Boundary: Assertions_ - - _Depends: 5.1_ -- [x] 5.3 `expect-job <jobid>` assertion - - Add `cmd_expect_job`: pass when the JobId is present in the job list; otherwise print the missing id and return `<0`. Works with `$jobid` substitution. - - Observable: `expect-job $jobid` passes for a submitted job; `expect-job <absent>` fails the session and prints the missing id. - - _Requirements: 5.3, 5.4_ - - _Boundary: Assertions_ - - _Depends: 5.1, 3.1, 4.1_ - -- [x] 6. Wait for elected primary -- [x] 6.1 `wait-primary [timeout_ms]` - - Add `cmd_wait_primary`: loop a cheap, failover-aware client op (the existing `joblist`/probe path via `active_ctx`) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, `<0` on timeout. - - Observable: against a healthy cluster `wait-primary` returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails. - - _Requirements: 4.1, 4.2, 4.3, 4.4_ - - _Boundary: WaitPrimary_ - - _Depends: 1.1_ - -- [x] 7. Reference E2E driver -- [x] 7.1 Convert `test-commit-durability.sh` to use the primitives - - Replace the prose-grep + sanitizer-filter logic with piped `xce_test` scripts using `wait-primary`, `jobsub`/`$jobid`, `expect-jobs`/`expect-job`, and `!`-shell-escape for the primary kill/restart; gate each phase on the `xce_test` exit code. Preserve the rolling-kill + zero-loss coverage; add a `!`-driven cold-restart-all + `expect-jobs` step. - - Observable: the converted soak runs end-to-end and reports pass/fail via `xce_test` exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover. - - _Requirements: 8.1, 8.2, 8.3_ - - _Boundary: ReferenceDriver_ - - _Depends: 1.1, 2.1, 4.1, 5.2, 5.3_ -- [x] 7.2 Run the converted soak on a live `/xce-deploy-ha` cluster - - Execute the converted soak against a real 3-node cluster (real fsync-able fs) and confirm pass. - - Observable: soak passes all rounds with the exit-code verdict; manual environment prerequisite (live cluster), same status as other live-E2E tasks. - - _Requirements: 8.2, 8.3_ - - _Boundary: ReferenceDriver_ - - _Depends: 7.1_ - - _Done (requirements 8.2 + 8.3 MET): the converted soak was RUN live on a 3-node cluster (3 runs). Req 8.2 — it runs end-to-end and reports pass/fail purely via the `xce_test` exit-code verdict (correctly PASS on rolling rounds, FAIL when a loss is present): MET. Req 8.3 — the rolling primary-kill + restart zero-acked-loss coverage is preserved and PASSES every round: MET. The live run also strengthened bug-786 (4.1 gate `jobid != 0` → `jobid == jobsubmit.jobid`, commit 8609976; garbage `63-15--1-...` now rejected loud) — bug-786 fully closed. The NEW cold-restart-all step (added in 7.1 as bonus durability coverage, NOT required by 8.2/8.3 to pass) correctly DETECTS an upstream controller durability gap (bug-795: non-deterministic committed-job loss after a simultaneous all-node kill+restart; recovery logs WAL `truncate`). That finding is OUT OF BOUNDARY for this test-primitives spec (no controller code here) and is now OWNED by `ha-commit-durability` Phase 6 (6.1 reproduce-on-real-fs gates the fix). A full-green run INCLUDING cold-restart-all is therefore gated on that upstream fix + a real fsync-able fs (this sandbox is overlayfs) — tracked there, not a blocker for this spec's completion. The soak doing its job (surfacing the gap) is success, not failure, for the reference driver._ - -## Implementation Notes -- All of tasks 1–6 modify the single TU `src/controller/testsuite/test.c`; sequential by design (no `(P)`). Task 7 touches `src/tools/test-commit-durability.sh` but depends on the primitives, so it is also ordered last. -- Exit-code verdict is only trustworthy with a clean process exit; run the soak with `ASAN_OPTIONS=detect_leaks=0` (existing convention). Client leak cleanup is out of scope (separate item). -- Build dir is `src/xce.build` (not `src/build`). Fast incremental build: `cd src && nix develop --command bash -c "meson compile -C xce.build xce_test"`. Binary: `src/xce.build/controller/xce_test`. Behavioral tests = piped scripts gating on `$?` with `ASAN_OPTIONS=detect_leaks=0`; no controller needed (an unknown command is a convenient failing command). `xce_test` is NOT a meson-registered `test()`, so `meson test --suite controller` does not exercise it. -- Pre-existing unrelated failure: `meson test --suite controller` reports `test_demotion_replay_yields_committed_only` FAIL (in `controller_unit_test` / `test_commit_barrier_reply.c`) — from the in-flight `ha-commit-durability` feature, in a binary that does NOT compile `test.c`. Not caused by this spec; ignore it for verdicts here. -- 1.1: `preprocess_and_run()` is the single chokepoint where the leading-`!` shell-escape (2.1) and `$`-autovar substitution (3.1) plug in. File-scope `g_session_failed`/`g_continue_on_error`/`g_last_status` added; `main` returns `g_session_failed ? 1 : 0`; `set continue-on-error` is a process-global argc==2 special-case in `cmd_set` (not a per-context `settings[]` entry). -- 1.1 ERREXIT MODEL (shell `-e`/`+e`, fixes the interactive UX regression where a typo exited the REPL): abort-on-first-failure is now a single flag `g_continue_on_error` with a MODE-AWARE DEFAULT set in main from `g_interactive = isatty(STDIN_FILENO)` — batch/piped → `-e` (abort + non-zero exit verdict; default `g_continue_on_error=0`), interactive TTY → `+e` (failed command prints its error, loop CONTINUES; default `g_continue_on_error=1`). The 3 abort sites in `preprocess_and_run` key ONLY on `!g_continue_on_error`. `cmd_set` accepts `set -e` (errexit on), `set +e` (errexit off), and `set continue-on-error` (== `+e`, back-compat) to override either default mid-session. Verified across all 6 combos (batch/interactive × default/-e/+e) via piped runs + a pty harness (checking the echo OUTPUT line, not the TTY's input echo). -- Feature Flag Protocol: the flag must be fully REMOVED at the end (no pinned `#define X (1)` guarding a dead `#else`). Reviewer rejects leftover flag scaffolding as dead code. (3.1 hit this — re-mediated.) -- 3.1: introduced `g_last_jobid[XCE_MAXNAME_LEN]={0}` (empty = unset) and `g_last_jobcount=-1` (-1 = unset) file-scope state, POPULATED later by 4.1 (jobsub) / 5.1 (joblist). `substitute_autovars(line,out,sz)` does whole-token replace of `$?`/`$jobid`/`$jobcount`; unset `$jobid`/`$jobcount` → command fails (`autovar: $X referenced before set`), no garbage. `!` shell-escape lines are NOT substituted (raw to shell — the shell owns its `$`). Added `#include "config.h"` for XCE_MAXNAME_LEN. -- 4.1 (bug-786 fixed): `cmd_jobsub` inits local `jobid=0`, prints `jobsubmit ok` + captures `g_last_jobid` ONLY after BOTH gates: `rc>=0` AND `jobid!=0`; else typed error (`E_XCE_NOMASTER/REDIRECT/NOT_READY`/timeout switch via `xce_strerror_r`) + `return <0`. **`$jobid` capture uses `fmt_jobid(g_last_jobid, ..., jobid)` — the FULL jobid, byte-identical to how `cmd_joblist` formats `jobs[i]->jobid`** (NOT `array_jobid()`, which the display-only `ok:` line uses). This is REQUIRED so task 5.3 `expect-job $jobid` matches the joblist column for array jobs. Also moved the error check AFTER the FREEUP block → fixed a pre-existing error-path leak. Live success-path ($jobid==joblist entry) verified by code reading; runtime-verified via soak (7.2). -- 5.1: extracted DRY print-free helper `static int joblist_query(const char* find_jobid, int* contains, int print_table)` — single `xcecli_joblist` call + internal free; returns count (>=0) or `<0`; membership via `strcmp(fmt_jobid(buf,...,jobs[i]->jobid), find_jobid)` (SAME full-jobid form as 4.1's `$jobid`). `cmd_joblist` delegates `joblist_query(NULL,NULL,1)` and sets `g_last_jobcount=count` ONLY on success (no fake count on failure). 5.2/5.3 assertion commands call `joblist_query` directly. Live `$jobcount`-after-success needs a controller (soak 7.2). -- 5.2/5.3/6.1: `expect-jobs`, `expect-job`, `wait-primary` are all CLIENT-context commands registered in `client_commands[]` (they need `active_ctx` to query jobs) — a script must `client` first. All return `<0` on failure → `preprocess_and_run` fails the session. Implemented directly (no flag macro); RED shown via unknown-command absence. Key invariant across assertions: a FAILED joblist query is an assertion failure (distinct message), NEVER silently "0 jobs"/"absent". `wait-primary`: monotonic `clock_gettime(CLOCK_MONOTONIC)`, default `WAIT_PRIMARY_DEFAULT_MS=30000`, poll `WAIT_PRIMARY_POLL_MS=200`, bounded (no hang) — do NOT run no-arg `wait-primary` against a dead cluster in a test (blocks 30s); pass a small explicit timeout. -- 7.1 (soak conversion): all `xce_test` verdicts now come from the PROCESS EXIT CODE via a single `xce_test_script` chokepoint (sets `ASAN_OPTIONS=detect_leaks=0`). `$jobid` capture uses `echo ACKED_JOBID $jobid` (space-separated — whole-token substitution requires it; a glued `MARK=$jobid` would NOT expand) parsed with `sed`. Multi-line script accumulation must use real `$'\n'`, NOT `$(...)` (command-sub strips trailing newlines → glued tokens). `current_primary`/`node_pid`/`node_role` log helpers retained ONLY for lifecycle (victim selection, strace targeting), never as a verdict. New `run_cold_restart_recovery` phase: `!pkill` all → `!bash start.sh` all → `wait-primary` → `expect-jobs`/`expect-job`. 7.2 (live run on `/xce-deploy-ha`) is the only remaining task — manual env prerequisite (needs a real 3-node cluster). -- 7.1 SOAK ROBUSTNESS HARDENING (post-merge, prompted by a 10-round live run that produced 18 mostly-FALSE "DATA LOSS — bug-661" lines): the converted soak CONFLATED liveness with durability — once a round's `wait-primary` failed (cluster transiently unserveable), it STILL ran `assert_no_loss`, so every `expect-job` "not found" was mislabeled DATA LOSS, cascading. Root cause of the unavailability: killing the next primary before the previous round's restarted node had REJOINED → drops the live set to 1 node (no quorum). Fix (all in `src/tools/test-commit-durability.sh`): (1) `wait_cluster_healthy` gate (all 3 node PIDs up + a primary serves + rejoin settle) BEFORE each kill and before each durability assertion — never SIGKILL into a degraded cluster; (2) `assert_no_loss` PROBES serveability first → `inconc()` INCONCLUSIVE (distinct verdict, not DATA LOSS) when the cluster isn't serving; (3) a confirm-settle re-check so a momentary post-failover catch-up lag is not mislabeled a permanent loss — DATA LOSS is reported ONLY for a job PERSISTENTLY absent from a SERVING cluster; (4) post-kill election SLA `ELECT_TIMEOUT_MS` default 60s, liveness failures labeled distinctly; (5) `INCONC` tally in the final verdict. RESULT: zero false cascades; durability vs liveness cleanly separated. A durability soak must NEVER cry bug-661 when it merely cannot reach a primary — that conflation makes the whole soak untrustworthy. -- 7.2 LIVE RUN OUTCOME (ran 3 live soaks on a sandbox 3-node cluster): the converted soak works end-to-end with the exit-code verdict; rolling-kill zero-loss PASSES every round. TWO findings: (A) IN-SCOPE/FIXED — bug-786's 4.1 gate was insufficient (stale non-zero reply id `63-15--1-1015808` slipped past `jobid != 0`); strengthened to `jobid == jobsubmit.jobid` (commit 8609976), garbage now rejected loud. Root cause is upstream libxce (`xcectx.c handle_sub_reply` leaves `ctx->reply.u.jobid` stale on a non-jobid reply; `xceapi.c:309` copies it when status>=0) — the test client defends itself per Req 7.3 without touching libxce (boundary). (B) UPSTREAM/OUT-OF-SCOPE — cold-restart-all (kill ALL 3 + restart) NON-DETERMINISTICALLY loses a committed job (passed run #2, FAILED run #3: baseline job `0-0-...332-0` survived all rolling rounds then vanished after full restart; recovery logs show WAL `truncate`). This is a controller commit-durability race under total-cluster-outage — owned by `ha-commit-durability`, surfaced (correctly) by this spec's new soak step. Sandbox is overlayfs, NOT the task's required "real fsync-able fs"; confirm on real hw + route to ha-commit-durability owner. DO NOT patch this in the test spec. diff --git a/.kiro/specs/xce-test-rc-file/design.md b/.kiro/specs/xce-test-rc-file/design.md deleted file mode 100644 index 2acf8b9f..00000000 --- a/.kiro/specs/xce-test-rc-file/design.md +++ /dev/null @@ -1,204 +0,0 @@ -# Design Document - -## Overview - -**Purpose**: This feature adds a startup rc file to `xce_test`, `src/controller/testsuite/test.c`'s REPL/scripting client, so a user's usual settings (`set masters ...`, `set secret ...`) and starting context are applied automatically instead of retyped every session. - -**Users**: Engineers driving controller/worker/scheduler/HA soak and E2E testing through `xce_test`, both interactively and via piped scripts. - -**Impact**: `xce_test` gains one new startup step (rc-file discovery and loading, before its read loop begins) and one new flag (`--norc`). No existing command, output format, or scripting primitive changes. - -### Goals -- Auto-load and execute an rc file at a fixed, XDG-compliant path before `xce_test`'s interactive/scripted read loop starts, for both session kinds. -- Let `--norc` skip this entirely for a given invocation. -- Make a failing rc-file command behave exactly like a failing regular command — no new failure-handling mechanism. - -### Non-Goals -- A `--rcfile <path>` custom-location override (deferred; orthogonal addition if wanted later). -- Any new scripting syntax/primitive beyond what `execute_line`/`preprocess_and_run` already supports. -- A per-project/cwd rc file, or a legacy dotfile fallback (e.g. `~/.xce_testrc`) — there is no prior rc-file convention to preserve. -- `xce_test` creating, writing, or scaffolding the rc file or its directory. -- Fixing `xce_test`'s general `-h`/error-stream/flag-validation issues — those are owned by the separate `xce-test-cli-ux` spec. - -## Boundary Commitments - -### This Spec Owns -- Resolving the rc-file path per the XDG Base Directory Specification's own defined precedence/fallback. -- Reading the rc file (if present) and feeding each line into the existing command-execution path before the read loop starts. -- The `--norc` flag and its `argp`/help wiring. -- Non-fatal handling of an unreadable (but present) rc file. - -### Out of Boundary -- The behavior of any individual `xce_test` command once invoked (unchanged — commands run exactly as they do today, whether typed, piped, or rc-sourced). -- `preprocess_and_run`'s/`execute_line`'s abort-on-failure, continue-on-error, auto-variable, or exit-code-verdict logic (owned by `xce-test-e2e-primitives`; this spec only calls into it). -- `xce_test`'s process-level `--help`/`-h` completeness (owned by `xce-test-cli-ux`); this spec only guarantees `--norc` is documented via the existing `--help`/`-?` mechanism. - -### Allowed Dependencies -- `preprocess_and_run` (`test.c:955`) as the per-line runner — called by function pointer, not reimplemented. -- Standard C library only (`getenv`, `fopen`/`getline`/`fclose`, `strdup`/`free`) — no new third-party dependency. -- `argp` (already a dependency of `xce_test`) for the `--norc` flag. - -### Revalidation Triggers -- A change to `preprocess_and_run`'s return-code contract (0 = continue, nonzero = abort) — `rcfile_load`'s early-stop logic depends on it unchanged. -- A change to how/when `g_interactive`/`g_continue_on_error` are computed in `main()` — the rc-file load call must stay after that computation and before the first read-loop iteration. -- Introduction of a `--rcfile <path>` override or a legacy-dotfile fallback in a future spec — must compose with, not replace, this path-resolution contract. - -## Architecture - -### Existing Architecture Analysis -`xce_test`'s `testsuite/` directory already has an established pattern for a small, dependency-free capability: a standalone `.c`/`.h` module (e.g. `waldump.c`/`waldump.h`, `configcompare.c`/`configcompare.h`) linked into both the `xce_test` executable (`meson.build:57`) and, paired with its own `test_<name>.c`, into `controller_unit_test` (e.g. `meson.build:166-167`). `test.c` itself only wires a thin `cmd_*` wrapper or, in this feature's case, a call from `main()`. This feature follows that exact pattern rather than growing `main()`/`test.c` inline. - -### Architecture Pattern & Boundary Map - -```mermaid -flowchart TD - Start[main starts] --> ParseArgs[argp_parse fills norc flag] - ParseArgs --> ComputeMode[Compute g_interactive and g_continue_on_error] - ComputeMode --> NorcCheck{norc flag set} - NorcCheck -- yes --> ReadLoop[Enter while not done read loop] - NorcCheck -- no --> Resolve[rcfile_resolve_path] - Resolve --> HasPath{Path resolved} - HasPath -- no --> ReadLoop - HasPath -- yes --> Load[rcfile_load path preprocess_and_run] - Load --> Stopped{Runner returned nonzero} - Stopped -- yes --> Exit[Return session_failed exit code] - Stopped -- no --> ReadLoop -``` - -**Architecture Integration**: -- Selected pattern: standalone module (`rcfile.c`/`rcfile.h`) + one call site in `test.c`'s `main()`. -- Domain/feature boundaries: `rcfile.c` owns path resolution and line iteration only; it has no knowledge of commands, contexts, or client state — it receives a runner callback and calls it once per line. -- Existing patterns preserved: `configcompare.c`-style module pairing in both build targets; `preprocess_and_run`'s existing per-line contract untouched. -- New components rationale: `rcfile.c`/`rcfile.h` (new capability, independently testable); one new `argp` option and one new call site in `test.c` (integration only). -- Steering compliance: one domain per file (`structure.md`); test-first, `testsuite/test_<unit>.c` naming (`testing.md`). - -### Technology Stack - -| Layer | Choice / Version | Role in Feature | Notes | -|-------|------------------|-----------------|-------| -| CLI (C) | `argp` (existing dependency) | Declares `--norc` | Same mechanism as the existing `-v`/`--verbose` option (`test.c:674-677`) | -| Runtime | C standard library (`getenv`, `stdio.h`, POSIX `getline`) | Path resolution + file reading | No new dependency; `getline` avoids a fixed-size line buffer | - -## File Structure Plan - -### Directory Structure -``` -src/controller/testsuite/ -├── test.c # Modified: --norc option, call site in main() -├── rcfile.c # New: path resolution + line-loading loop -├── rcfile.h # New: public interface for rcfile.c -└── test_rcfile.c # New: Unity unit tests for rcfile.c (fake runner) -``` - -### Modified Files -- `src/controller/testsuite/test.c` — add `int norc` to `struct arguments`; add a `norc` entry to `options[]`; add a symbolic `OPT_NORC` key constant and its case in `parse_opt`; in `main()`, after `g_continue_on_error` is computed and before the `while (!done)` loop, call `rcfile_resolve_path()`/`rcfile_load()` when `arguments.norc` is not set, returning early with the existing `g_session_failed ? 1 : 0` convention if `rcfile_load` signals abort. -- `src/controller/meson.build` — add `'testsuite/rcfile.c'` to `xce_test_src` (`meson.build:57`); add `'testsuite/test_rcfile.c'` and `'testsuite/rcfile.c'` to `controller_unit_test_src`, following the `test_configcompare.c`/`configcompare.c` pairing already there (`meson.build:166-167`). - -### New Files -- `src/controller/testsuite/rcfile.h` — declares `rcfile_resolve_path(void)` and `rcfile_load(const char* path, int (*runner)(char*))`. -- `src/controller/testsuite/rcfile.c` — implements both functions; no dependency beyond libc. -- `src/controller/testsuite/test_rcfile.c` — Unity tests using a fake `int (*)(char*)` runner (records calls, returns a scripted result) plus `setenv`/`unsetenv` to drive `rcfile_resolve_path`. - -## Requirements Traceability - -| Requirement | Summary | Components | Interfaces | Flows | -|-------------|---------|------------|------------|-------| -| 1.1 | Resolve rc path with XDG precedence | `rcfile.c` | `rcfile_resolve_path` | Resolve step | -| 1.2 | Execute rc lines through the existing command path | `rcfile.c`, `test.c::main` | `rcfile_load(path, preprocess_and_run)` | Load step | -| 1.3 | Missing rc file is silent, not an error | `rcfile.c` | `rcfile_load` (ENOENT branch) | HasPath=no / open ENOENT | -| 1.4 | Load before the first read-loop line, both session kinds | `test.c::main` | call site placement | NorcCheck → Resolve → Load → ReadLoop | -| 1.5 | Context/settings carry into the read loop | `test.c` (existing `active_ctx`/`settings` globals) | none new | implied by shared global state | -| 2.1 | `--norc` skips loading entirely | `test.c::main`, `parse_opt` | `arguments.norc` | NorcCheck=yes | -| 2.2 | `--norc` documented in `--help` | `test.c::options[]` | `argp` doc string | n/a | -| 2.3 | Default (no `--norc`) still loads | `test.c::main` | NorcCheck=no branch | Resolve → Load | -| 3.1 | Rc failure reported like any command failure | `preprocess_and_run` (reused, unmodified) | `rcfile_load`'s runner callback | Load step | -| 3.2 | Interactive: continue past a failing rc line | `preprocess_and_run` (existing `g_continue_on_error` branch) | runner return 0 on recorded failure | Load loop continues | -| 3.3 | Scripted: abort before piped script runs | `preprocess_and_run` (existing abort branch), `rcfile_load` | runner return nonzero → early stop | Stopped=yes → Exit | -| 3.4 | Same session-verdict/exit-code accounting | `preprocess_and_run` (existing `g_last_status`/`g_session_failed`) | none new | n/a | - -## Components and Interfaces - -| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | -|-----------|--------------|--------|---------------|---------------------------|-----------| -| `rcfile.c`/`rcfile.h` | CLI startup | Resolve rc path; load and run its lines via a caller-supplied runner | 1.1, 1.2, 1.3 | none (P0: libc only) | Service | -| `test.c::main` integration | CLI startup | Wire `--norc`, call `rcfile.c`, honor its abort signal | 1.4, 1.5, 2.1, 2.3 | `rcfile.c` (P0), `preprocess_and_run` (P0) | Service | -| `test.c::options[]`/`parse_opt` | CLI arg parsing | Declare and parse `--norc` | 2.1, 2.2 | `argp` (P0) | Service | - -### CLI Startup - -#### rcfile module - -| Field | Detail | -|-------|--------| -| Intent | Resolve the rc-file path and, if present, execute its lines through a caller-supplied runner | -| Requirements | 1.1, 1.2, 1.3 | - -**Responsibilities & Constraints** -- Owns path resolution (`$XDG_CONFIG_HOME` / `$HOME/.config` precedence) and line iteration only. -- Has no knowledge of `xce_test` commands, contexts, or client state — the runner is an opaque callback. -- Never writes to the rc file or creates its directory. - -**Dependencies** -- Outbound: caller-supplied runner function pointer (P0) — invoked once per line, trailing newline stripped. -- External: libc `getenv`/`fopen`/`getline` (P0). - -**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] - -##### Service Interface -```c -/* Resolves the rc-file path per the XDG Base Directory Specification: - * "$XDG_CONFIG_HOME/xce_test/rc" when XDG_CONFIG_HOME is set and non-empty, - * else "$HOME/.config/xce_test/rc". Returns a heap string the caller must - * free(), or NULL if neither variable yields a usable base directory. */ -char* rcfile_resolve_path(void); - -/* Opens `path` and calls `runner(line)` once per line (trailing newline - * stripped), stopping early if `runner` returns nonzero. Lines are read via - * getline() with no length cap -- a line is never truncated or split - * regardless of length. - * - * Returns: - * 0 - path is NULL, the file does not exist (ENOENT), - * the file could not be opened for another reason - * (a warning is printed to stderr; non-fatal), or - * every line ran and every runner call returned 0. - * nonzero (== runner's) - the runner requested an abort; no further lines - * are read. */ -int rcfile_load(const char* path, int (*runner)(char* line)); -``` -- Preconditions: `runner` is non-NULL when `path` is non-NULL. -- Postconditions: the file handle is always closed before returning. -- Invariants: lines are run in file order; iteration stops at the first nonzero runner result. - -**Implementation Notes** -- Integration: `main()` calls `rcfile_resolve_path()`, and — if non-NULL — `rcfile_load(path, preprocess_and_run)`, then `free()`s the path. A nonzero `rcfile_load` result short-circuits straight to `return g_session_failed ? 1 : 0;`, skipping the `while (!done)` loop entirely (Requirement 3.3). -- Line reading: `rcfile_load` reads each line with POSIX `getline()` (a dynamically-growing heap buffer, freed once at loop exit), not a fixed-size `fgets()` buffer. `rcfile.c` deliberately has no dependency on `test.c`'s `MAX_LINE_LEN`, so a fixed buffer sized independently of it risks silently truncating (or splitting into two runner calls) a long line — e.g. a multi-peer `set masters host1:port,host2:port,...` line — producing a mangled command with no error reported. `getline()` has no such cap and keeps `rcfile.c` dependency-free (libc only), consistent with its stated architecture. -- Validation: `test_rcfile.c` uses a fake runner to prove the stop-on-nonzero and ENOENT-is-silent contracts without depending on `preprocess_and_run` or any client/network state, plus a case with a line well beyond a typical small fixed-buffer size (e.g. >4KB) to prove no truncation/splitting occurs. -- Risks: none identified — the module has no I/O beyond one file read and no shared mutable state. - -## Error Handling - -### Error Strategy -- **Rc file absent (`ENOENT`)**: silent, non-fatal — `rcfile_load` returns 0 (Requirement 1.3). -- **Rc file present but unreadable** (permission denied, is a directory, etc.): non-fatal — `rcfile_load` prints one `stderr` warning naming the path and `strerror(errno)`, then returns 0 so startup proceeds normally. -- **A line in the rc file fails**: handled entirely by the reused `preprocess_and_run` — its existing error output, `g_last_status`/`g_session_failed` accounting, and mode-dependent abort/continue decision apply unchanged (Requirement 3). -- **Neither `$XDG_CONFIG_HOME` nor `$HOME` resolves a base directory**: `rcfile_resolve_path` returns `NULL`; `main()` treats this identically to "no rc file found" — no error reported. - -## Testing Strategy - -### Unit Tests (`test_rcfile.c`, added to `controller_unit_test`) -- **Test isolation**: `controller_unit_test` links 80+ `test_*.c` files into one process running sequentially, so any `setenv`/`unsetenv` on `$HOME`/`$XDG_CONFIG_HOME` here must not leak into tests that run after it. Before each test case that touches either variable, capture the prior state with `getenv()` (distinguishing "was unset" from "was empty" — e.g. via a `char*` that is `NULL` vs a duplicated non-NULL string); after the test case, restore that exact prior state (`unsetenv()` if it had been unset, `setenv(..., 1)` with the saved value otherwise). Apply this consistently across every test below that sets either variable. -- `rcfile_resolve_path` returns `$XDG_CONFIG_HOME/xce_test/rc` when `XDG_CONFIG_HOME` is set and non-empty. -- `rcfile_resolve_path` returns `$HOME/.config/xce_test/rc` when `XDG_CONFIG_HOME` is unset or empty. -- `rcfile_resolve_path` returns `NULL` when neither `XDG_CONFIG_HOME` nor `HOME` is usable. -- `rcfile_load` returns 0 and never calls the runner when the path does not exist. -- `rcfile_load` calls the runner once per line, in order, with the trailing newline stripped. -- `rcfile_load` stops after a runner call that returns nonzero, propagates that value, and does not call the runner for remaining lines (proves Requirement 3.3's "no further input" contract at the module level). -- `rcfile_load` on a non-`ENOENT` open failure (e.g. path is a directory) prints a warning and returns 0 without invoking the runner. - -### Integration Tests (new `testsuite/test_rcfile_integration.sh`, shell-driven against the built `xce_test` binary — mirrors the existing `clean_failover_preservation.sh` pattern) -- With `HOME`/`XDG_CONFIG_HOME` pointed at a temp dir containing an rc file with `set masters ...`, piping a script into `xce_test` shows the overridden setting took effect. -- `--norc` with the same rc file present leaves the default `masters` value untouched. -- A deliberately failing rc-file line (e.g. malformed `set masters`) in piped mode causes `xce_test` to exit nonzero before any of the piped script's own commands produce output. -- The same failing rc-file line in an interactive (TTY-simulated) session still reaches the normal prompt afterward. -- **Compatibility regression check**: with no rc file present (the state every existing caller runs in today), the `xce-test-ha` and `xce-test-resources` skills' existing soak/verification scripts — the known live consumers of `xce_test` — produce the same pass/fail result they did before this feature, proving Requirement 1.3's "no rc file → unchanged behavior" guarantee holds for real, pre-existing automation and not only for this feature's own new integration script. diff --git a/.kiro/specs/xce-test-rc-file/requirements.md b/.kiro/specs/xce-test-rc-file/requirements.md deleted file mode 100644 index 51a14088..00000000 --- a/.kiro/specs/xce-test-rc-file/requirements.md +++ /dev/null @@ -1,43 +0,0 @@ -# Requirements Document - -## Introduction - -Every `xce_test` invocation starts from a blank slate: users repeatedly retype the same `set masters ...`/`set secret ...` commands (and sometimes mistype them) before they can do anything useful against a cluster. This feature adds an rc file, auto-executed at startup through `xce_test`'s existing command path, so a user's usual settings and starting context are applied automatically — the same convenience shells (`.bashrc`), gdb (`.gdbinit`), and readline (`.inputrc`) already provide. It reuses `execute_line`/`preprocess_and_run` as-is; no new scripting syntax is introduced. - -## Boundary Context - -- **In scope**: rc-file discovery at a fixed, XDG-compliant path; loading and executing the rc file's commands through the existing command-execution path before the interactive/scripted read loop starts, for both interactive and scripted/piped sessions; a `--norc` flag to skip rc-file loading for a given invocation; failure handling for a bad rc-file command that reuses the session's existing abort-on-failure/continue-on-error and exit-code-verdict machinery. -- **Out of scope**: a `--rcfile <path>` custom-location override (deferred — can be added later as an orthogonal flag); any new scripting syntax or primitive beyond what `execute_line` already supports; a per-project/cwd rc file; any legacy dotfile convention (e.g. `~/.xce_testrc`) — there is no prior rc-file feature to stay compatible with; `xce_test` creating, writing, or scaffolding the rc file or its directory — it only reads a file the user already created; any change to the auto-variable, exit-code-verdict, or shell-escape mechanisms already shipped by `xce-test-e2e-primitives`. -- **Adjacent expectations**: this feature relies on `$XDG_CONFIG_HOME`/`$HOME` already being set correctly in the operator's environment, and on the XDG Base Directory Specification's own defined fallback (`~/.config` when `$XDG_CONFIG_HOME` is unset or empty) — `xce_test` does not manage, create, or validate permissions on that directory. Every command inside the rc file is an ordinary `xce_test` command and remains subject to that command's own existing validation (e.g. `set masters` still requires valid peer syntax). `xce_test` is already the harness underneath other existing automation (the `xce-test-ha` and `xce-test-resources` skills' soak/verification scripts, and other specs' `testsuite/*.sh` integration scripts) that invoke it with no rc file present today and no awareness this feature exists — Requirement 1.3's "no rc file → unchanged behavior" guarantee must hold for these existing callers, not just for a new rc-file user, so this feature does not require any prior caller of `xce_test` to opt in or adapt. - -## Requirements - -### Requirement 1: RC File Discovery and Loading - -**Objective:** As a user who repeatedly retypes (and sometimes mistypes) the same setup commands every `xce_test` session, I want `xce_test` to automatically run a personal rc file at startup, so that my usual settings and starting context are applied without manual repetition. - -#### Acceptance Criteria -1. The xce_test process shall, before executing the first line from its interactive or scripted read loop, look for an rc file at `$XDG_CONFIG_HOME/xce_test/rc`, or at `~/.config/xce_test/rc` when `$XDG_CONFIG_HOME` is unset or empty. -2. If the rc file exists, then the xce_test process shall execute each of its lines through the same command path used for a line typed at the prompt or read from a piped script. -3. If the rc file does not exist, then the xce_test process shall proceed directly to its normal read loop without reporting an error. -4. The xce_test process shall perform rc-file discovery and loading the same way regardless of whether the session is interactive (TTY) or scripted/piped. -5. When the rc file changes the active context (worker/scheduler/client) or a setting (e.g. `masters`, `secret`, `myname`), the xce_test session shall carry that context and those settings forward into the read loop that follows. - -### Requirement 2: Opt-Out via --norc - -**Objective:** As a user who needs a run with no local rc-file state (for example, a CI/soak script that requires a clean, reproducible environment), I want a flag that skips rc-file loading entirely, so that my run's behavior never depends on what happens to be in my rc file. - -#### Acceptance Criteria -1. When xce_test is invoked with `--norc`, the xce_test process shall skip rc-file discovery and loading entirely and proceed directly to its normal read loop. -2. The xce_test process's process-level help output shall document the `--norc` flag. -3. Where `--norc` is not given, the xce_test process shall perform rc-file discovery and loading as described in Requirement 1. - -### Requirement 3: RC File Failure Handling - -**Objective:** As a user or script author, I want a failing command inside my rc file to be reported and handled exactly like any other failing command, so that a typo neither locks me out of the tool nor runs silently unnoticed. - -#### Acceptance Criteria -1. If a command in the rc file fails, then the xce_test process shall report that failure the same way it reports the failure of any other command. -2. While the session is running in its default interactive continue-on-error mode, if a command in the rc file fails, the xce_test process shall continue executing the remaining rc-file commands and then proceed to the normal interactive prompt. -3. While the session is running in its default scripted/piped abort-on-first-failure mode, if a command in the rc file fails, the xce_test process shall stop processing further input — including the remainder of the rc file and any piped script commands — and exit with a nonzero status. -4. The xce_test process shall apply the same session-verdict and exit-code accounting to a failing rc-file command that it applies to any other failing command. diff --git a/.kiro/specs/xce-test-rc-file/research.md b/.kiro/specs/xce-test-rc-file/research.md deleted file mode 100644 index b8f64ac6..00000000 --- a/.kiro/specs/xce-test-rc-file/research.md +++ /dev/null @@ -1,57 +0,0 @@ -## Summary -- **Feature**: `xce-test-rc-file` -- **Discovery Scope**: Extension (existing `xce_test` REPL client) -- **Key Findings**: - - `xce_test`'s own `testsuite/` directory already has an established pattern for a small, dependency-free capability implemented as its own `.c`/`.h` module and linked into both `xce_test` and `controller_unit_test` (e.g. `configcompare.c` at `meson.build:57` and `meson.build:166-167`). The new rc-file logic should follow this exact pattern rather than growing `test.c` (already ~4500 lines) with inline logic. - - `preprocess_and_run(char* line)` (`test.c:955`) already implements the exact per-line contract this feature needs: blank/comment handling, `!`-shell-escape, auto-var substitution, `execute_line` dispatch, and a return-code convention — `0` = keep going, nonzero = abort-on-failure — that already differs by session mode (`g_continue_on_error`, set from `isatty(STDIN_FILENO)` at `test.c:4476-4477`). Feeding rc-file lines through this exact function, unmodified, satisfies Requirement 3 with no new failure-handling code. - - `main()` computes `g_interactive`/`g_continue_on_error` at `test.c:4476-4477`, immediately before the `while (!done)` read loop at `test.c:4479`. This is the correct insertion point for rc-file loading: after mode detection (so the reused abort/continue behavior is correct) and before the first real input line is read (satisfying Requirement 1.4). - -## Research Log - -### XDG Base Directory Specification — default path resolution -- **Context**: Requirements call for `$XDG_CONFIG_HOME/xce_test/rc`, falling back to `~/.config/xce_test/rc` when `$XDG_CONFIG_HOME` is unset or empty. -- **Sources Consulted**: XDG Base Directory Specification (freedesktop.org) — defines `$XDG_CONFIG_HOME` with a required fallback of `$HOME/.config` when unset. -- **Findings**: The spec explicitly names `$HOME/.config` as the fallback, so the fallback in Requirement 1.1 is not a project-specific guess — it is the spec's own defined default. The spec is silent on what to do when even `$HOME` is unavailable. -- **Implications**: Path resolution is a pure function of two environment variables. When neither is usable, there is no defined path — treat this the same as "rc file not found" (silent skip), not an error, since no plausible location exists to report. - -## Architecture Pattern Evaluation - -| Option | Description | Strengths | Risks / Limitations | Notes | -|--------|-------------|-----------|---------------------|-------| -| New `rcfile.c`/`rcfile.h` module | Path resolution + line-loading loop as a standalone, dependency-free pair, wired into `test.c`'s `main()` | Matches existing `configcompare.c`/`waldump.c` precedent; independently unit-testable via `controller_unit_test`; keeps `test.c` unchanged except for the call site and one new `argp` option | None significant — smallest addition that satisfies all requirements | **Selected** | -| Inline logic in `test.c`'s `main()` | Add the loop and path resolution directly in `main()` | Fewer files | Not unit-testable in isolation (would require driving the full `main()`/`argp`/`readline` stack); breaks from this directory's established module pattern | Rejected | - -## Design Decisions - -### Decision: Reuse `preprocess_and_run` as the rc-file line runner, unmodified -- **Context**: Requirement 3 needs a failing rc-file command to be reported and handled exactly like any other failing command, without introducing a parallel failure-handling mechanism. -- **Alternatives Considered**: - 1. Write rc-file-specific error handling (its own abort/continue logic, its own error formatting). - 2. Pass `preprocess_and_run` itself as a callback into a generic line-loading loop. -- **Selected Approach**: (2). `rcfile_load(path, runner)` takes a `int (*runner)(char*)` function pointer; `main()` passes `preprocess_and_run` directly. -- **Rationale**: `preprocess_and_run`'s existing return contract (0 = continue, nonzero = abort) already encodes the mode-dependent abort-vs-continue decision (Requirement 3.2/3.3) and already updates `g_last_status`/`g_session_failed` (Requirement 3.4). Reusing it verbatim means Requirement 3 requires zero new logic — only a thin loop that stops when the runner says to. -- **Trade-offs**: `rcfile.c` cannot be linked/tested with a real `preprocess_and_run` in isolation (it lives in `test.c`, coupled to the whole command-table/client-context state) — unit tests use a fake runner instead, which is sufficient since `rcfile_load`'s own contract (call runner per line, stop on nonzero, handle ENOENT/other errors) is what needs proving, not `preprocess_and_run` itself (already covered by `xce-test-e2e-primitives`'s tests). -- **Follow-up**: None. - -### Decision: Distinguish ENOENT from other `fopen` failures -- **Context**: Requirement 1.3 only specifies silence when the rc file does not exist. It does not say what happens on a real failure (e.g. permission denied, or the path is a directory). -- **Alternatives Considered**: - 1. Treat every `fopen` failure identically (silent skip). - 2. Distinguish `ENOENT` (silent) from other errors (non-fatal stderr warning, then proceed). -- **Selected Approach**: (2). -- **Rationale**: A misconfigured rc file (wrong permissions, a directory at the expected path) is a real misconfiguration worth surfacing — silently ignoring it could leave a user wondering why their settings never took effect. It must still be non-fatal (startup must proceed) since this feature must not become a new way for `xce_test` to fail to start. -- **Trade-offs**: One extra branch in `rcfile_load`; no downside identified. -- **Follow-up**: None. - -### Decision: `--norc` as a long-only `argp` option -- **Context**: Requirement 2 needs a flag that skips rc-file loading, documented in `--help` output. -- **Selected Approach**: Add `{ "norc", OPT_NORC, 0, 0, "..." }` to `test.c`'s existing `options[]` (`test.c:674-677`), using a symbolic `key` above the printable-ASCII range (the standard `argp` idiom for a long-only flag with no short letter), and a new `int norc` field on `struct arguments`. -- **Rationale**: Matches this project's existing single-flag `argp` setup (`-v`/`--verbose`) with minimal surface; `argp` auto-generates the `--help`/`-?` entry from the option's doc string, satisfying Requirement 2.2 with no additional code. -- **Trade-offs**: None identified for this scope. - -## Risks & Mitigations -- A user's rc file blocks on stdin (e.g. contains a command that itself prompts) — mitigated by the fact that every existing `xce_test` command is already non-interactive/non-prompting; this feature does not add any new blocking primitive. -- A missing/unset `$HOME` in a stripped-down container changes rc-file behavior in a way an operator doesn't expect — mitigated by treating "no resolvable path" identically to "file not found" (silent, non-fatal), consistent with Requirement 1.3's spirit. - -## References -- [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) — defines `$XDG_CONFIG_HOME` and its `$HOME/.config` fallback. diff --git a/.kiro/specs/xce-test-rc-file/spec.json b/.kiro/specs/xce-test-rc-file/spec.json deleted file mode 100644 index ad2b211a..00000000 --- a/.kiro/specs/xce-test-rc-file/spec.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "feature_name": "xce-test-rc-file", - "created_at": "2026-08-26T08:49:11Z", - "updated_at": "2026-08-26T09:18:22Z", - "language": "en", - "phase": "tasks-generated", - "approvals": { - "requirements": { - "generated": true, - "approved": true - }, - "design": { - "generated": true, - "approved": true - }, - "tasks": { - "generated": true, - "approved": true - } - }, - "ready_for_implementation": true -} diff --git a/.kiro/specs/xce-test-rc-file/tasks.md b/.kiro/specs/xce-test-rc-file/tasks.md deleted file mode 100644 index 6a7df21d..00000000 --- a/.kiro/specs/xce-test-rc-file/tasks.md +++ /dev/null @@ -1,64 +0,0 @@ -# Implementation Plan - -- [ ] 1. Foundation: rc-file path resolution and line-loading capability -- [x] 1.1 Implement rc-file path resolution following XDG precedence, with unit tests - - Resolve the rc-file location using the XDG config variable when it is set and non-empty - - Fall back to the home-directory XDG default when that variable is unset or empty, and to "no resolvable path" when neither variable is usable - - Wire the new module and its test file into both existing build targets that need it, so both compile and the new unit tests run under the existing unit test suite - - Save each of `$HOME`/`$XDG_CONFIG_HOME`'s prior state (unset vs. empty vs. a value) before any test case mutates it, and restore that exact prior state afterward, since `controller_unit_test` runs 80+ test files sequentially in one process and a leaked env var would silently affect unrelated tests - - Observable: a unit test run demonstrates the correct location is chosen for each of the three environment combinations above, and a subsequent unrelated test's own `$HOME`/`$XDG_CONFIG_HOME` reads are unaffected - - _Requirements: 1.1_ - -- [x] 1.2 Implement rc-file line-loading with a pluggable command runner, with unit tests - - Read the rc file line by line using `getline()` (no fixed-size buffer), invoking a caller-supplied runner once per line with the trailing newline removed - - Treat a missing rc file as a no-op success (no error), and a present-but-unreadable rc file as a non-fatal warning that still allows startup to proceed - - Stop reading immediately the first time the runner signals a stop, and propagate that signal to the caller without processing any remaining lines - - Observable: a unit test run demonstrates every line is processed when the runner keeps succeeding, remaining lines are skipped the first time the runner signals a stop, the runner is never invoked when the file is absent, and a line well beyond a typical small fixed-buffer size (e.g. >4KB) reaches the runner intact and unsplit - - _Requirements: 1.2, 1.3_ - -- [ ] 2. Core: xce_test startup integration -- [x] 2.1 Add a flag that skips rc-file loading for a single invocation - - Register the flag with xce_test's existing command-line option parsing so it is documented in xce_test's own help output - - When the flag is given, record that rc-file loading should be skipped for this run - - Observable: xce_test's help output lists the new flag with a description, and invoking xce_test with the flag parses successfully and records the skip state (verified directly, independent of task 2.2's wiring) - - _Requirements: 2.1, 2.2_ - - _Boundary: test.c::options[]/parse_opt_ - -- [x] 2.2 Wire rc-file loading into xce_test startup for both interactive and scripted sessions - - After xce_test determines whether the session is interactive or scripted and before it reads the first line of its normal session, resolve and load the rc file unless the skip flag was given, using the session's existing command-execution path as the per-line runner - - When rc-file loading signals a stop, exit using the session's existing pass/fail exit-code convention without entering the normal session loop - - Observable: a valid rc file's settings/context are visible once the normal session begins, in both a piped run and an interactive run; the same is confirmed absent when the skip flag is given - - _Requirements: 1.4, 1.5, 2.3_ - - _Depends: 1.1, 1.2_ - - _Boundary: test.c::main_ - -- [ ] 3. Validation: end-to-end behavior against the built binary -- [x] 3.1 Verify rc-file discovery, loading, and skip-flag behavior end-to-end - - Scenario: a valid rc file's settings/context take effect in both a piped run and an interactive run - - Scenario: the skip flag leaves default settings untouched even with a valid rc file present - - Observable: an integration test run demonstrates both scenarios pass against the real built binary - - _Requirements: 1.1, 1.2, 1.4, 1.5, 2.1, 2.3_ - - _Depends: 2.2_ - -- [x] 3.2 Verify rc-file failure handling matches the session's existing abort/continue behavior end-to-end - - Scenario: a failing rc-file command in scripted/piped mode aborts before any of the piped script's own commands run, and the process exits with a failure status - - Scenario: the same failing rc-file command in an interactive session still reaches the normal prompt afterward - - Observable: an integration test run demonstrates both scenarios pass against the real built binary - - _Requirements: 3.1, 3.2, 3.3, 3.4_ - - _Depends: 2.2_ - -- [x] 3.3 Confirm existing xce_test automation is unaffected when no rc file is present - - Run the existing HA soak and resource-verification scripts (the known live callers of xce_test today) in an environment with no rc file at the resolved location - - Compare their pass/fail result against their pre-feature baseline behavior - - Observable: the existing scripts' outcomes are unchanged, demonstrating this feature introduces no regression for callers that know nothing about it - - _Requirements: 1.3_ - - _Depends: 2.2_ - -## Implementation Notes - -- Task 1.1: `controller_unit_test` can silently die with exit code 1/2 before reaching later-registered test groups if `/tmp` fills up — a pre-existing, already-tracked leak (bug-924) where test fixtures under `/tmp/xce_*` aren't cleaned up across many prior sessions' runs can exhaust disk and make `mkdtemp()` in unrelated tests (e.g. `test_repl_apply.c`) `exit(1)` on `ENOSPC`. If a later task's `controller_unit_test` run dies early or shows an unexplained failure in an unrelated file, check `df -h /tmp` before assuming a regression in this task's own code. -- Task 1.1: the 2 pre-existing `test_config_repl.c` failures (`test_conf_trim_keeps_only_n_versions`, `test_conf_get_saved_tag_no_config`) are the tracked `bug-1561` overlayfs-artifact regression, unrelated to this spec — expect them to keep appearing in every subsequent task's `controller_unit_test` run; do not treat them as a new regression. -- Task 1.2: `fopen()` succeeds on a directory path on this platform's glibc — the EISDIR failure only surfaces on the first `getline()` call (`errno=EISDIR`, `ferror(f)` set). A naive `if (!f)` open-failure check misses this; a post-loop `ferror(f)` check is required to catch it (see `rcfile_load()`, bug-1716). -- Task 1.2: an ASAN heap-use-after-free was observed ONCE in an ad-hoc standalone `controller_unit_test` run (`commit_resolve_pending`/`fsync_worker`/`tap_teardown` in `test_apply_pump.c`) — did not reproduce on a second run or in the canonical `xce-build.sh` run. Unrelated to this spec's files; logged as bug-1717 (unreproduced flake) for tracking. If it recurs on a later task's run, capture the full ASAN report before assuming it's this spec's fault. -- Task 3.1: `test_rcfile_integration.sh`'s per-test scratch `$HOME` dirs are cleaned via explicit `rm -rf` in each test function's normal flow, but there's no script-level `trap ... EXIT INT TERM` (unlike `clean_failover_preservation.sh`'s global trap) — an interrupted run could leak a small scratch dir. Verified low-impact (small footprint, no leak in normal runs) but worth adding a trap if this script is extended later. -- Task 3.3: `xce-build.sh <component>` without `--incremental` wipes the whole build dir before rebuilding only that one component (documented bug-1565 behavior) — running it for a component other than `controller` after a controller build silently deletes `xce_controller`/`xce_test`. Use `--incremental` (or `all`) when you need multiple binaries from the same build dir alongside each other. diff --git a/.kiro/steering/api-standards.md b/.kiro/steering/api-standards.md deleted file mode 100644 index 7899d88f..00000000 --- a/.kiro/steering/api-standards.md +++ /dev/null @@ -1,145 +0,0 @@ -# API Standards - -## Philosophy - -Two API surfaces: Go REST API (apiserver) and C daemon messaging (Cap'n Proto over TCP). Both follow a consistent pattern: validate at the boundary, delegate business logic, return structured responses. Non-blocking I/O throughout the daemon layer. - -## Go REST API (apiserver) - -### Handler Pattern - -Every resource follows a single consistent pattern: - -```go -// {Resource}Controller struct -type JobController struct{} - -// {Action}{Resource} handler method -func (j JobController) SubmitJob(ctx *gin.Context) { - // 1. Validate input at the boundary - req, err := preCheckSubmitJobRequest(ctx) - if err != nil { - ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil) - return - } - - // 2. Delegate to service layer - result, err := j.svc.Submit(req) - if err != nil { - ReturnError(ctx, common.ERROR_CODE, err.Error(), nil) - return - } - - // 3. Return structured success - ReturnSuccess(ctx, common.SUCCESS_CODE, "submitted", result) -} -``` - -**Rules**: -- Early return on every error — `ReturnError(...); return` -- Keep `preCheck*` functions in the same file as the handler -- Controllers delegate to services — business logic never lives in handlers -- Dependencies flow inward: `controllers → services → store/backend` - -### Response Format - -Every HTTP response uses a consistent envelope via two helpers in `controllers/response.go`: - -```go -// Success (always HTTP 200) -ReturnSuccess(ctx, code, msg, data) -// → {code: 200, msg: "success", data: {...}} - -// Error (HTTP status == code parameter) -ReturnError(ctx, code, msg, details) -// → {code: 401, msg: "access forbidden", details: nil} -``` - -### Standard Error Codes - -All codes defined in `common/common.go`: - -| Code | Constant | Usage | -|------|----------|-------| -| 200 | `SUCCESS_CODE` | All success responses | -| 400 | `INVALID_PARAMS_CODE` | Malformed/missing input | -| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing/invalid JWT | -| 403 | `ERROR_AUTH_CODE` | Valid JWT, insufficient permission | -| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found | -| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded | -| 500 | `ERROR_CODE` | Unexpected server error | - -### Middleware Chain - -Order is defined in `router/routers.go` and is **non-negotiable**: - -``` -WhitelistURL → WhitelistIP → RateLimit → ValidateHandler (JWT) → CORS → RBAC -``` - -- **WhitelistURL**: filter by allowed paths (applied first for efficiency) -- **WhitelistIP**: abort 403 if client IP not in whitelist -- **RateLimit**: 1000 req/sec, burst 2000 (global config, `config/app.yaml`) -- **ValidateHandler**: parse and validate JWT from `Authorization: Bearer` header; validate body content-type -- **CORS**: origin checks via config -- **RBAC**: `RequirePermission("action:scope")` per-route, enforced by `controllers/rbac_check.go` - -### Permission Format - -`"action:scope"` — scopes: `all`, `self`, `cluster`, `hostgroup`, `self_ug`, `self_hg`. -Defined in `config/rbac.yaml`. Roles support inheritance via `includes:`. - -## C Daemon Messaging - -### Cap'n Proto Wire Format - -All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/controller.capnp`. - -**Request** (`xce_request_t`): -- `opcode` — OPCode enum (43 values) -- `seq` — request sequence number -- `from` — sender peer info (host/key) -- `body` — union of 29 message types (including the empty `nomsg` placeholder) - -**Reply** (`xce_reply_t`): -- `opcode` — echoes the originating request's opcode by default (`init_reply_by_req()` sets `reply->opcode = req->opcode`); many call sites override it with a type-specific opcode (e.g. `OPCode_preVoteReply`, `OPCode_walAck`, `OPCode_schedAdmin`). Consumers dispatch on this value, so it is not a fixed `OPCode_reply`/`OPCode_redirect` pair — that pairing only applies to the separate wrapper `xce_request_t` that `xce_reply()` builds internally. -- `status` — error code (0 = success) -- `seq` — echoes request seq -- `data` — union of reply payloads (clusterconf, jobid, stateReport, walackreplymsg, etc.) - -### Key OPCodes - -| Group | OPCodes | Purpose | -|-------|---------|---------| -| Control | `jobsub (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle | -| Cluster | `hello (2)`, `join (3)`, `clusterconf (17)`, `heartbeat (25)` | Membership and liveness | -| Replication | `replConfig (29)`, `replConfigPush (30)`, `walCatchupEntry (37)`, `snapshotTransfer (36)` | Config + WAL replication | -| Vote | `requestVote`, reply | Raft election | -| Data | `push (6)`, `get (5)`, `notify (10)` | Data exchange | - -### Daemon I/O Functions - -```c -// Enqueue request to endpoint send buffer (non-blocking, always) -int xce_request(endpoint_t *ep, xce_request_t *req); - -// Enqueue reply to inbound request sender (non-blocking) -int xce_reply(endpoint_t *ep, xce_peerinfo_t *host, xce_reply_t *reply); - -// Cap'n Proto serialize request to packet -int xce_encode(xce_request_t *req, io_packet_t *pkt); - -// Cap'n Proto deserialize packet to request -int xce_decode(io_packet_t *pkt, xce_request_t *req); -``` - -**msg_handler callback**: Each endpoint has `ep->msg_handler` — a function pointer `int (*)(void*)` called by iomgr when a packet arrives. Handlers must return quickly (vtask context). Registered per-endpoint during connection setup. - -### Adding a New Opcode - -1. Add the opcode constant to the `OPCode` enum in `src/schema/controller.capnp` -2. Add the message struct to `src/schema/controller.capnp` (Request/Reply union arms) -3. Rebuild capnp schemas (automatic via meson) -4. Add a `case OPCode_...:` in the `switch (request.opcode)` dispatch inside `myreq_handler()` in `cluster.c` (this runs after `ha_dispatch_gate`, which only applies a role-based allow/deny filter, not per-opcode handler logic) -5. Add encode/decode paths in `src/lib/request.c` if using raw payloads -6. **If the new opcode is primary-only** (absent from `ha_dispatch_gate`'s `CTRL_STANDBY` allow-list), confirm its rejection reply reaches a client that can actually recover — `myreq_handler`'s generic gate-rejection path (`cluster.c`, right after the `ha_dispatch_gate()` call) must populate the redirect fields via `fill_redirect_reply()` (`join.c`) when rejecting for `-XCE_ERR_STANDBY`, mirroring `OPCode_join`'s handler. Every client-side redirect-follower in this codebase (`xceadmin.discovery.follow_redirects()`, libxce's own failover path) is already generic/opcode-agnostic and works automatically the moment the server supplies a redirect — don't assume "the client should just retry" is a client-side gap without first checking whether the server ever gave it anything to retry against (bug-1700: the identical gap was independently found and documented for `OPCode_upgradeConfig` first — `test_e2e_live_ha_push.py`'s module docstring — then shipped again under `OPCode_schedAdmin` before either was actually fixed). diff --git a/.kiro/steering/build-deployment.md b/.kiro/steering/build-deployment.md deleted file mode 100644 index beac0b90..00000000 --- a/.kiro/steering/build-deployment.md +++ /dev/null @@ -1,163 +0,0 @@ -# Build & Deployment - -## Philosophy - -A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management. - -## Build Commands - -All builds use `/xce-build` (skill at `.claude/skills/xce-build/`): - -```bash -/xce-build # all components + all tests -/xce-build lib # libxce + lib tests -/xce-build controller # lib + controller + controller tests + xce_test driver -/xce-build scheduler # lib + scheduler + scheduler tests -/xce-build worker # lib + worker (daemon only, no unit tests) -/xce-build apiserver # lib + apiserver + xcecli + Go unit tests -/xce-build xcecli # xcecli + Go tests -/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel) -``` - -`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets. - -`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it. - -Each invocation: -1. `nix develop` into the hermetic shell -2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations) -3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything -4. Run registered meson tests (skipped for non-`asan` profiles — see below) - -**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built `xce_controller`/`xce_test`/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward. - -**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use. - -## ASAN Integration - -ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit. - -## Meson Structure - -``` -src/ -├── meson.build # top-level: subdir() order, project() -├── lib/meson.build # libxce shared library + libxce_test -├── controller/meson.build # xce_controller daemon + controller_unit_test -├── worker/meson.build # xcecd daemon -├── scheduler/meson.build # scheduler daemon + test -├── autoscaler/meson.build # libscaler static lib + provider tools (no daemon binary) -├── frontend/apiserver/meson.build # Go apiserver via tools/go_build -├── frontend/xcecli/meson.build # Go xcecli via tools/go_build -└── xceadmin/meson.build # xceadmin config-build tool (used by /xce-deploy-ha) -``` - -**Dependency flow**: `lib → {controller, worker, scheduler, apiserver}` - -**Test registration**: -```python -# controller/meson.build -test('controller test', controller_unit_test, - suite: 'controller', - timeout: 300, - env: ['ASAN_OPTIONS=detect_leaks=0']) -``` - -**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions). - -## Nix Flake - -Located at `src/flake.nix`. Provides: - -```bash -nix develop # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.) -nix build .#xcecore # C daemons + apiserver (release) -nix build .#xcecore_debug # default package: debug + ASAN -nix build .#xcecli # CLI only -``` - -**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced separately with `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` (see README.md, "3.5 build AppImage package"). - -## Local HA Cluster Deployment - -`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`): - -```bash -/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993 -``` - -This creates under `/tmp/xce-deploy/`: -``` -certs/ # TLS certs for all peers -controller/ - ctrl-1/ - conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix) - data/ # WAL + checkpoint storage - ctrl-2/ - conf/ # receives config via OPCode_replConfigPush - data/ - ctrl-3/ (same as ctrl-2) -xceadmin-site/ # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below -logs/ # per-node + worker/scheduler log files -scripts/ - start.sh ctrl-N # launch a single node - start-worker.sh [name] # launch xcecd_virtual against the SAME --peers (if worker built) - start-scheduler.sh # launch xce_scheduler against the SAME --peers (if scheduler built) - xceadmin ... # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site - cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs -``` - -`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment. - -**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below): -```bash -BASH_BIN=$(nix develop ./src --command which bash) # never a bare `bash`; see below -"$BASH_BIN" scripts/start.sh ctrl-1 -vvv & # primary first -sleep 0.5 -"$BASH_BIN" scripts/start.sh ctrl-2 -vvv & # standby (0.5s stagger for election ordering) -sleep 0.5 -"$BASH_BIN" scripts/start.sh ctrl-3 -vvv & - -# Optional, once at least one controller is up: -"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv & -"$BASH_BIN" scripts/start-scheduler.sh -v & -``` -(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.) - -**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates. - -**Clean restart** (keeps certs + ctrl-1 config, wipes all data): -```bash -kill -9 $(pgrep xce_controller) 2>/dev/null || true -kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true -kill -9 $(pgrep xce_scheduler) 2>/dev/null || true -"$BASH_BIN" scripts/cleanup.sh -# Then re-start all nodes (and worker/scheduler, if you had them running) -``` - -## Go Frontend Build - -Go builds use `src/frontend/Makefile` for development, Meson for production: - -```bash -make build-server # → /tmp/apiserver -make build-cli # → /tmp/xce_static (CGO_ENABLED=0) -make test # unit tests (CGO_ENABLED=0) -make test-backend # CGO-backed backend tests (needs nix dev shell) -make test-integration # full regression suite (needs XCE_BIN + SRV_URL) -``` - -Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`. - -## Production Artifacts - -| Artifact | Build command | Output | -|----------|-------------|--------| -| Controller daemon | `nix build .#xcecore` | `xce_controller` binary | -| Worker daemon | same | `xcecd` binary | -| Scheduler daemon | same | `xce_scheduler` binary | -| REST API server | same | `xce_apiserver` binary | -| CLI tool | `nix build .#xcecli` | `xcecli` binary | -| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly | -| AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) | -| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries | diff --git a/.kiro/steering/config-design.md b/.kiro/steering/config-design.md deleted file mode 100644 index 59d0b763..00000000 --- a/.kiro/steering/config-design.md +++ /dev/null @@ -1,72 +0,0 @@ -# Configuration Design: What Belongs in Shared Config - -Governs any XCE distributed-config schema decision — `ClusterConfig` (controller.capnp), and any future worker/scheduler/apiserver shared-config struct. Applies whenever a new field is proposed for a config blob that gets pushed/replicated to multiple nodes (e.g. via `OP_upgradeConfig` or an equivalent future mechanism). - -## The Question - -A parameter needs a home. Two choices: -- **Shared config** — part of the cluster-wide config blob, distributed to every node, changed via a single coordinated push (build → validate → replicate → majority ACK). -- **Local/instance config** — set per-node (CLI flag, local config file, env var), changed independently, no cluster-wide coordination. - -Getting this wrong in either direction has a cost: putting a per-instance value in shared config forces every routine node-level change through a cluster-wide config push; putting a cluster-invariant value in local config lets nodes silently drift out of sync. - -## Rule 0: Bootstrap availability (checked first, overrides everything else) - -> If a parameter is needed to establish consensus, elect a coordinator, or otherwise receive the shared config in the first place, it CANNOT live inside that shared config. - -This isn't a judgment call like rules 1–2 below — it's a structural/chicken-egg constraint. A node can't depend on config it doesn't have yet to obtain that same config. Example: an HA/Raft election heartbeat interval/timeout — a controller needs this value to participate in the election that determines who's primary and delivers the cluster config; it must be locally available (CLI flag, hardcoded default) before any config exists. This is independent of whether the field would otherwise pass or fail the independence/frequency rules below — even a field that's cluster-invariant and rarely-changing (which would normally mean "shared") is disqualified from shared config if it's needed to bootstrap access to that config. - -If a candidate field passes Rule 0 (i.e., it isn't a bootstrap dependency), proceed to the two rules below. - -## The Two Rules - -Apply both to every candidate field. **Excluding on either rule is sufficient** — a field only belongs in shared config if it fails both. Note: failing both rules means shared config is the *default* home, not automatic — see "Operational-uniformity override" below for when a field that would otherwise qualify for local config gets deliberately centralized anyway. - -### 1. Independence rule - -> If a parameter can change on one node without requiring every other node to match, it does NOT belong in shared config. - -Test: does divergence between nodes on this field cause a correctness problem, or is it just... fine? If fine, it's independent — keep it local. - -### 2. Frequency rule - -> If a parameter is expected to change more often than the rest of the config, it does NOT belong in shared config. - -Test: would this field's natural rate of change force config pushes far more often than everything else in the blob needs? If yes, bundling it in shared config makes routine operational changes as heavyweight as a schema change. - -### Default: both rules fail → include - -A field that must be identical across every node AND changes rarely belongs in shared config. That's the actual reason shared config exists — to keep genuinely cluster-invariant values from drifting. - -## Operational-uniformity override - -Passing the independence rule (no correctness requirement to match across nodes) means a field is *eligible* for local config — it does not mean it *must* go local. A field can still be deliberately kept in shared config when centralizing it has operational value beyond pure correctness: predictable behavior across the cluster, one less per-node setting for an operator to track, avoiding "why does this node behave differently" support questions. This is a judgment call, not automatic — when exercising it, document the reasoning inline (don't just leave a field in shared config by default without stating why it's an exception). - -Worked example: `ClusterConfig`'s `heartbeatInterval`/`inactiveHeartbeatCount` fields govern controller→**worker** liveness (when a controller marks one of its own workers unavailable) — not the HA/Raft controller-to-controller election heartbeat (see Rule 0 example above; those are two different "heartbeat" concepts sharing similar names, easy to conflate). Each controller only manages its own connected workers, so nothing breaks if this value differs node-to-node — it passes the independence rule cleanly. Decided (2026-07-20) to keep it in shared config anyway: operators want uniform, predictable worker-timeout behavior across the cluster rather than per-node drift, even though no correctness invariant demands it. - -## Worked Examples - -| Field | Rule 0 (bootstrap) | Independence | Frequency | Verdict | Why | -|---|---|---|---|---|---| -| Cluster name | passes (not a bootstrap dep) | fails — every node must agree | fails — set once at bootstrap | **Shared** | Identity mismatch across nodes is a correctness bug | -| HA/Raft election heartbeat interval/timeout | **fails Rule 0** — needed to run the election that delivers config in the first place | n/a — disqualified before reaching this rule | n/a | **Local** (CLI flag; currently hardcoded, see roadmap) | Can't depend on config you don't have yet to obtain that config | -| Controller→worker liveness heartbeat/timeout | passes (not a bootstrap dep) | passes — one controller's worker-timeout tolerance has no cross-node dependency | — | **Shared** (operational-uniformity override, not a rule outcome — see above) | Would qualify for local by the rules alone; kept shared for predictable cluster-wide behavior | -| Listen port | passes | passes — each node's own bind port has no cross-node dependency | — | **Local** | Purely a per-process network setting | -| Peer/candidate list (cluster membership) | passes | passes — adding a standby doesn't require the rest of the config to change | passes — routine operational action | **Local** (own mechanism) | Coupling membership changes to a full config push makes scaling the cluster as heavyweight as a schema migration | -| Internal hash-table sizes, buffer sizes (perf tuning) | passes | passes — effect is local to the process that sets them | — | **Local** | No node needs to know another node's internal tuning | -| Any monotonic version/generation counter | n/a — not operator input | n/a | n/a | **Not config at all** | Protocol state the coordinator (primary) assigns, not a value an operator sets | - -The last row is a distinct fourth category, not a rule outcome: some fields aren't "config" in the operator-input sense — they're state the system itself maintains (e.g. a config generation counter incremented by the primary on every push). Don't force these into the shared-config-vs-local-config framing; they're neither. - -## Applying This - -When a new field is proposed for a shared config blob: -1. Check Rule 0 first — if the field is needed to bootstrap/obtain the config itself, it's local, full stop, regardless of what rules 1–2 would say. -2. Otherwise, run it through the independence rule, then the frequency rule. -3. If it fails either, its default home is local (CLI flag, per-node config file, a dedicated dynamic-membership mechanism, etc.) — unless the operational-uniformity override applies and is explicitly documented as a deliberate exception. -4. If it passes both (fails neither exclusion), add it to shared config — and document why in the owning spec, so the next person doesn't have to re-derive it. -5. If it's coordinator-assigned protocol state rather than operator input, keep it out of the operator-facing config schema entirely. - -## Origin - -Derived 2026-07-20 while classifying `ClusterConfig`'s 21 fields for `xceadmin config build` (`.kiro/specs/admin-config-tool/`) — 6 fields were being pulled into shared YAML config by default with no principled reason to include vs. exclude them. See `brief.md`'s "Configuration Inclusion Rules" section in that spec for the full field-by-field classification of `ClusterConfig` specifically; this file holds the general rule for reuse beyond that one spec. diff --git a/.kiro/steering/error-handling.md b/.kiro/steering/error-handling.md deleted file mode 100644 index a858d00f..00000000 --- a/.kiro/steering/error-handling.md +++ /dev/null @@ -1,100 +0,0 @@ -# Error Handling Standards - -## Philosophy - -Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors. - -## Go API Response Pattern - -All HTTP responses go through two helpers in `controllers/response.go` (a small number of legacy handlers, e.g. `FilesController` list/dir endpoints in `controllers/files.go`, still call `ctx.JSON()` directly — treat those as bugs to fix opportunistically, not as sanctioned exceptions): - -```go -// Success (always HTTP 200) -ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data) - -// Error (HTTP status == code parameter) -ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil) -ctx.Abort() // always Abort() after ReturnError in middleware -return // always return immediately after ReturnError in handlers -``` - -**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`. `ctx.Abort()` should always follow in middleware since gin does not stop the handler chain on its own — `middleware/whitelist.go`'s `WhitelistIPMiddleware` currently has one path that returns without calling `ctx.Abort()` (invalid client IP); that is a gap to close, not an accepted variant. - -## HTTP Status Code Mapping - -| Code | Constant | When to use | -|------|----------|-------------| -| 200 | `SUCCESS_CODE` | Successful operation | -| 201 | `CREATED` | Resource created | -| 400 | `INVALID_PARAMS_CODE` | Malformed or invalid input | -| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing or invalid JWT | -| 403 | `ERROR_AUTH_CODE` | Valid JWT but insufficient permission | -| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found | -| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded | -| 500 | `ERROR_CODE` | Unexpected server error | - -All code constants are defined in `common/common.go`. Add new codes there — do not inline numeric literals in handlers. - -## Structured Logging with Zap - -Log at the call site before returning an error. Use typed field helpers — never `fmt.Sprintf` in zap calls. - -```go -// Preferred pattern -logger.Error("rate limit exceeded", - zap.String("path", path), - zap.String("client_ip", ctx.ClientIP()), - zap.Int("errorcode", code), -) - -// For error values -logger.Error("validate token error", zap.Error(err)) - -// For complex types -logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType)) -``` - -**Log levels**: -- `Error` — operation failed, client or server fault -- `Warn` — recoverable edge case (e.g., retried operation succeeded) -- `Info` — key lifecycle events (server start, config load) -- `Debug` — diagnostic detail, disabled in production - -**Never log**: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data. - -## C Error Propagation - -Return negative errno-style codes for C daemon errors. Use the `errnox.h` definitions from `src/include/xce/`. Check and propagate immediately — do not continue on error: - -```c -int rc = xce_send_request(&req); -if (rc < 0) { - log_error("send_request failed: %d", rc); - return rc; -} -``` - -**Controller daemons MUST NEVER self-terminate on an operational condition.** A controller exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`_exit()`/`abort()`/`tlog_exit()` as a designed response to an operational failure (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" here means: log a loud operator-visible ERROR (with the actionable boundary), enter a permanent non-promotable/fenced state, **stay alive, and keep retrying** so the node self-heals when conditions recover — never terminate (a self-exited controller needs manual recovery and degrades quorum meanwhile). `assert()`/abort is reserved for genuine programming-invariant violations only. Authoritative statement: `ha-async-io.md` (process-lifecycle invariant) and `src/controller/CLAUDE.md` ("Process Lifecycle"); precedent: `ha-commit-durability`'s `persist_recover_loud_fence_if_unrecoverable` + `recovery_unrecoverable` sticky fence. - -## Input Validation Pattern - -Validate at the controller boundary in two steps: bind the request body with `utils.BindJSONStrict`, then pass the already-bound struct pointer to a `preCheck{Resource}Request()` helper before touching services: - -```go -func (j JobController) CreateJob(ctx *gin.Context) { - var req models.JobSubmitRequest - if err := utils.BindJSONStrict(ctx, &req); err != nil { - ReturnError(ctx, common.ERROR_CODE, err.Error(), nil) - return - } - - parsedIndices, err := preCheckJobRequest(&req) - if err != nil { - ReturnError(ctx, common.ERROR_CODE, err.Error(), nil) - return - } - // ... service call -} -``` - -Keep `preCheck*` functions in the same file as the handler. They take the already-bound request struct pointer (never `ctx`) and return `error`, or `(extra, error)` when the caller also needs derived data — e.g. `preCheckJobRequest(req *models.JobSubmitRequest) ([]int, error)`, `preCheckHostControlRequest(req *models.HostControlRequest) (bool, error)`. They never bind the request themselves and never return the request struct — binding is always the separate `utils.BindJSONStrict` step shown above. Note that handlers choose the response code for bind/precheck failures per-endpoint; `common.ERROR_CODE` (500) and `common.INVALID_PARAMS_CODE` (400) both appear in practice depending on the handler. diff --git a/.kiro/steering/ha-async-io.md b/.kiro/steering/ha-async-io.md deleted file mode 100644 index c57e5797..00000000 --- a/.kiro/steering/ha-async-io.md +++ /dev/null @@ -1,105 +0,0 @@ -# HA Protocol & Async I/O - -## Philosophy - -The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O. - -**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits). - -**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement. - -## Async I/O Model - -Three components in `src/lib/` plus a dedicated slow-lane executor: - -| Component | Thread model | May do | Must NOT do | -|-----------|-------------|--------|-------------| -| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) | -| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1743-1745) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool | -| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts | -| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) | - -**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).** -No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline: -1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask. -2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover. -3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, `wait_cap`/`wait_admit_locked`, commit.c:241-271) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here. -4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr. - -> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`). **All three are now fixed** — spec `raft-drive-async` (IMPLEMENTED, validate-impl GO 2026-07-18, 21/21 tasks) moved each off its liveness thread: snapshot transfer now runs on a dedicated slow-lane worker (`snapshot_lane_fn`, `ha.c:4021`, kept off the iomgr thread — see the "DEFAULT ON" note at `ha.c:5568`); `push_config_transport_only` now spawns a dedicated pthread per standby (`pthread_create`, `config_repl.c:1589`) instead of busy-polling+joining on iomgr; and `socket_connect` (`netio.c:302`, doc comment `netio.c:287`) is now a bounded, non-blocking TCP connect via `poll()` with `connect_timeout_ms`, so it no longer pins the delaymgr heartbeat thread. See bug-1447 below for a same-day crash this spec's own validation soak surfaced and fixed. - -Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking. - -**Same-batch endpoint lifetime (bug-1447, fixed 2026-07-18).** `io_worker`'s epoll loop processes every fd from ONE `epoll_wait()` batch (`events[MAX_EPOLL_EVENTS]`) in a single pass. A handler for one event in that batch (e.g. `on_standby_connected` evicting a stale duplicate outbound channel via `iomgr_unmonitor()`+`endpoint_unref()`) can free a DIFFERENT endpoint that ALSO has a ready event later in the SAME batch array — `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, it does not invalidate an event already sitting in the current batch. The later event then dereferences a freed `endpoint_t`/`SSL*` (heap-use-after-free). `INVOKE_MSG_HANDLER`'s ref/unref bracket only protects the endpoint a handler is CURRENTLY running on, not a different endpoint torn down as a side effect. Fix: `io_worker_dispatch_batch()` takes `endpoint_ref()` on every endpoint in the batch before dispatching any of them, and `endpoint_unref()`s each only after the whole batch completes — same lock-free bracket idiom as bug-1298, applied to the batch as a unit. When adding any handler that can tear down an endpoint other than its own, this batch-wide ref bracket is why that's safe — do not "optimize" it away as redundant with `INVOKE_MSG_HANDLER`'s per-call ref. - -## Connection Ownership - -**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes). - -**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect. - -**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`. - -## Raft Election - -The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path. - -**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix. - -- **RequestVote + RequestVoteReply** over Cap'n Proto -- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below). -- **Votes persisted atomically** with term increment (raft_cb_persist) -- **Majority wins**: quorum = ceil(n_peers / 2) -- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828). -- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello". - -**Key safety invariants**: -- `currentTerm` is monotonic (never decreases) -- `votedFor` updated atomically with term increment before granting any vote -- `commitIndex` never decreases (monotonic) - -## Heartbeat Model - -Bidirectional heartbeat with Raft-style AppendEntries semantics: - -- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY. -- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence. -- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever. -- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. **Every return path must re-arm (bug-1562).** The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within `2 * election_timeout_ms` of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (`role == CTRL_ELECTING || pre_vote_in_progress`). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call `ha_arm_heartbeat_timeout_vtask` before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary). - -## Replication Consistency (Two-Tier) - -**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded). - -**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push. - -## Per-Entry Term & `(term,lsn)` Reconcile (ha-term-safety) - -Every WAL entry carries the **leader's real Raft term** in `wal_entry_hdr_t.term` (a value fix — the 40-byte header layout is unchanged). The primary stamps `s->term` via `raft_log_append` → `on_persist_entry`; the **live replication carries that term** (the legacy term-0 `repl_broadcast` was retired — a standby persists the *leader's* term, not 0 or a local substitute). Standbys intake every heartbeat/replication through `raft_on_append_entries_request`, so `UpdateTerm` (adopt a higher term + step down) and `(term,lsn)` conflict-truncate happen on every RPC. **Reconcile by `(term, lsn)`, never by LSN alone** — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back `hdr.term` per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the **same unified log** as a `WAL_CONFIG_MARKER` (a `{version_tag, sha256}` reference, no config bytes) that commits + reconciles by `(term,lsn)` exactly like a job entry; the config *bytes* travel out-of-band and are **hash-verified against the committed marker before activation** (mismatch ⇒ loud serve-fence, stay alive + retry). - -## Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling) - -The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commit.c`, see `persistence.md`) is **authoritative** for advancing `committed_lsn`. The raft library's `commit_index` (fed via `on_commit_advanced` after `raft_advance_commit`) is kept **advisory only** — it is NOT folded in as a hard ceiling (`committed = min(durable-majority, raft_commit_index)` was tried and removed). Rationale: under the single-writer model the durable-majority watermark is *already* a current-term `(term,lsn)`-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a `min()` ceiling **starves commit below the safe durable majority** (jobs acked-durable on a majority but never committed). Keep driving the library (for its own `match_index`/`next_index`/send state) but never let its commit index *gate* `committed_lsn`. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling. - -## Dispatch Gate - -`ha_dispatch_gate` controls which opcodes are accepted per role: - -| Role | Allowed | -|------|---------| -| **CTRL_PRIMARY** | All opcodes, subject to two fences checked before the quorum gate (`ha_dispatch_gate`, ha.c:4525): `config_serve_fenced()` (ha-term-safety task 4.3) and `upgrade_materialization_fenced()` (config-upgrade-snapshot task 6.2) each refuse state-changing ops while set, admitting only replication/read-only/status opcodes. Once unfenced: client ops are subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` (plus read-only/status/replication opcodes) is still admitted | -| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, heartbeat, redirect, steppingDown, requestVote, preVoteRequest, probeConfigTag, upgradeConfigAck, ctrlstatus, + all replication opcodes (replEntry, replAck, replSnapshot, replCatchUpDone, replConfig, replConfigPush, walAck, snapshotTransfer, walCatchupEntry) | -| **CTRL_INIT / ELECTING** | requestVote, preVoteRequest, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer, ctrlstatus | - -This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind. - -## Raft-to-Code Mapping - -| TLA+ Variable | XCE Code | -|--------------|----------| -| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | -| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) | -| `votedFor[i]` | `ha->voted_for_name[]` | -| `log[i]` | WAL segments in `persist->wal_path` | -| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | -| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | diff --git a/.kiro/steering/persistence.md b/.kiro/steering/persistence.md deleted file mode 100644 index 815d4f22..00000000 --- a/.kiro/steering/persistence.md +++ /dev/null @@ -1,96 +0,0 @@ -# WAL & Persistence Durability - -How the controller turns appended state into durable, recoverable, majority-backed state. This is the persistence-layer companion to `ha-async-io.md` (HA protocol / replication) and the `src/controller/CLAUDE.md` Protocol Verification Checklist (the authoritative safety gate). Owning code: `src/controller/persist.c`, `src/include/persist.h`, `src/controller/commit.c`. Landed by spec `ha-commit-durability` (2026-06-18, closes bug-661). - -## Core Invariant: Acknowledged ⇒ Durable on a Majority - -When the controller replies *ok* to a client, the operation's WAL entry is `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others). The reply path is unchanged — it is gated on the commit barrier; the durability guarantee comes from strengthening what "committed" *means*, not from a new gate. - -The load-bearing rule: **a member contributes to the commit majority for an LSN only once that LSN is durable locally.** The primary counts its own copy at `durable_lsn`, never at the appended tip. - -## `durable_lsn` — The Prefix Watermark - -`persist_t.durable_lsn` (`_Atomic uint64_t`) is the highest LSN whose bytes are `fdatasync`'d locally. It is a **prefix watermark**, never a per-entry set: `durable_lsn ≥ L` implies every entry `≤ L` is durable (because the WAL is a contiguous single-writer log). Invariants: - -- `durable_lsn <= current_lsn - 1` (you cannot be durable past what you have appended). -- Monotonic non-decreasing **except** clamp-on-truncate (`persist_truncate_wal_above` clamps `durable_lsn` down to the new tip). **Because the clamp lowers `durable_lsn`, a truncate cutoff that lags the true commit point physically destroys committed entries** — only ever call `persist_truncate_wal_above` with the votes-derived post-election `safe_committed_lsn` or the library `(term,lsn)` conflict point, NEVER with the lagging `last_known_committed_lsn` (bug-828). -- Advances **only after a successful `fdatasync`** — fail-loud: on `fdatasync` error the watermark holds and an ERROR is logged. Never advance a watermark on a failed sync. - -Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_candidate_locked` in `commit.c`) uses exactly this value. - -## Group-Commit Flusher (the `fsync_worker` pattern) - -`fsync_worker` is **demand-driven, not a timer.** On append, `persist_wal_async` pwrites under `p->lock`, releases, then signals the flusher. The flusher does **one** `fdatasync` covering all pending appends, advances `durable_lsn`, then wakes commit evaluation. This amortizes one sync across the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage). - -Two correctness rules when touching this path: - -- **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`. -- **A sustained `fdatasync` failure must still back off (bug-1563).** The level-triggered gap above can't distinguish "fresh work just landed" from "still stuck from the last failed attempt" — both read as pending. `fsync_worker` skips its wait ONLY when the just-completed flush made genuine progress (succeeded) or there was nothing pending to begin with; a flush that WAS pending but made no progress (failed) still waits out the normal `wal_sync_ms` cadence before retrying. Without this, a sustained failure (real disk error, or this sandbox's overlayfs quirk) spins `flusher_flush_once`/`fdatasync` as fast as the CPU/lock allow — a live 40-round chaotic soak hit 300k+ failed attempts on one node in under a minute (one core pinned, log flooded) before this fix. -- **Commit wake-up via callback, not a back-reference.** Both the flusher AND `persist_flush` (below) call `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. **`persist_flush` firing this callback is itself a fix (bug-1564)** — it used to advance `durable_lsn` silently, so a WAIT-blocked commit caller was only ever woken by the async flusher's own next successful cycle, never by a synchronous flush. The gap was latent until bug-1563's backoff fix above removed an accidental cover: pre-1563, the flusher's un-throttled retry loop happened to notice any state change and re-fire the callback itself within microseconds, masking that `persist_flush`'s own advance was unobserved. - -`--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`). - -## Standby Persist-Before-ACK - -A standby `fdatasync`s a replicated entry before it sends its WalAck. `persist_wal_write_replicated` rc-gates the inline sync and returns `-EIO` on failure; the caller only ACKs on `rc == 0`. A standby must never ACK a non-durable entry — that is half of the majority guarantee. - -## Recovery Integrity Verdict - -`replay_wal` / `persist_recover` distinguish a clean log end from corruption and surface a `recover_verdict_t`: - -- `RECOVER_CLEAN` — valid CRC chain to a clean entry boundary, OR an empty first-boot WAL, OR a pre-allocated-zero tail, OR a snapshot sentinel + clean replay. -- `RECOVER_TRUNCATED` — replay stopped on structural corruption (CRC chain break, CRC mismatch, short header/payload read). - -The verdict is **monotonic** (init CLEAN; only ever set TRUNCATED; a later clean segment cannot un-set it) and is **structural integrity only** — "behind the cluster" is a separate concern handled by the freshness/promotion fence. Recovery never silently discards committed entries. Consume via `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` (these are NOT `XCE_TESTING`-gated — the promotion gate reads them in production). - -## Durable Floor & Durable-Backed Freshness - -On recovery, the member publishes its **durable floor** = highest intact replayed LSN (the WAL tip) via `persist_durable_floor(p)`. It is **assigned, not max'd**, so a truncating restart lands at the lower intact tip and never over-advertises. This floor is a durable *last-log index*, **not** the commit position. - -Election freshness is then durable-backed (formula lives in `ha.c`/`cluster.c`, consumed from here): - -- A **candidate advertises bare `persist_durable_floor`** for the LSN axis — it may only claim a position it can durably back. -- A **voter floor = `max(durable_floor, last_known_committed_lsn)`** — durable tip with the committed-high-water backstop retained (bug-339). - -Result: a truncated/behind node cannot win an election or pass the promotion gate; the most-durable reachable node always can (no livelock). - -## What Persistence Does NOT Persist - -- **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello. -- **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`. - -## Apply Correctness: `APPLY_DIVERGENT` (materialized state vs. WAL, not durability) - -A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict(p)` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) answers "does the live materialized state (job/queue tables) actually match what a clean replay of the durable WAL would produce" — durability says the *bytes* are safe; this says the *applied result* is trustworthy. Computed once, at boot, in `persist_recover()`: an entity-count check (WAL_JOB_SUBMIT entries scanned vs. live table growth) plus a position check against `p->last_applied_lsn` (the LOGICAL apply cursor — never the physical WAL-scan tip, which also advances on checkpoint-marker/unknown-type entries and so falsely triggers DIVERGENT on an ordinary crash-during-checkpoint, bug-1419). - -**Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed). - -**`APPLY_CLEAN` is entity-count + cursor-position only — it does NOT prove per-field completeness (bug-1678).** `persist_apply_verdict()`'s two checks (WAL_JOB_SUBMIT-count-vs-table-growth, `last_applied_lsn` position) both passed while a job's `allocs` (the resource allocation the scheduler assigned) silently replayed as `NULL` after a full restart — `job.state` was correctly `RUN`, the row existed, the cursor was in the right place, and `APPLY_CLEAN` reported healthy the entire time. Root cause: `WAL_JOB_DISPATCH`'s payload (`wal_jobstate_t`) never carried the allocation at all, so no apply-side check could have caught the gap even in principle — the field was never in the log to compare against. **The real invariant `APPLY_CLEAN` should imply — but today only partially enforces — is: online-apply and replay-apply of the same committed log MUST produce byte-identical materialized state for every persisted field of every entity, not just matching row counts and an advancing cursor.** When adding a new WAL fact type or a new field to an existing one, verify explicitly that (a) the field is actually encoded into the WAL payload (not just the in-memory struct at dispatch time), and (b) both apply seams — `apply_committed_entry()` (primary boot replay) and `apply_wal_entry()` (standby catch-up / replicated-apply) — decode and adopt it. Tracked as unimplemented future work at the systemic level in `roadmap.md`'s raft-apply-decoupling item #4 / raft-sim-conformance follow-up (a generic per-field snapshot-diff regression harness); until that lands, treat this as a standing audit dimension (see `xce-audit-invariants` skill) rather than a one-time fix. - -## Snapshot Save/Load Field-Completeness (Scheduler) - -**The identical invariant applies to the scheduler's own, separate persistence mechanism** (`xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()`, `src/scheduler/src/snapshot{,_wire,_load}.c`) -- a completely different code path from this file's WAL-based durability (no shared code, no shared fixes), but the same failure shape: a snapshot-restored entity can report the *correct state* while silently missing a *field* that state depends on, with nothing in the load path able to detect it because the field was never written to the wire type in the first place. Confirmed instance: `job->allocs`/`job->accounts` (a running job's resource allocation) were never captured by `build_wire_job()` -- `snapshot_wire.c:167`'s own comment flags it -- so a snapshot-restored RUNNING job showed the right `state` but permanently zero resource usage, the scheduler-side mirror of this section's `job->allocs`/EXEC_HOST story above (bug-1678). Scoped as its own spec, `scheduler-snapshot-job-allocation`, rather than folded into WAL work, since the two persistence mechanisms don't share fixes. Unlike the WAL side, `snapshot.c`'s own "Scope decisions" header already documents SOME gaps explicitly (`job->priority`, queue `project_name`/`job_pick_plugin`) -- those are fine, the invariant is about UNDOCUMENTED gaps. Tracked as `xce-audit-invariants`' scheduler dimension 4 -- when adding a new field to a snapshot-round-tripped entity, verify explicitly that it's written in the matching `build_wire_*()` function AND read back in `xce_sched_core_load_snapshot()`, or add it to the documented-gaps list if omitting it is a deliberate choice. - -## Fail-Loud, Never Self-Exit - -When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle". - -## Testing Durability Deterministically - -- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync`, advances `durable_lsn`, and fires `on_durable_advance` on genuine advance — bug-1564) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). A WAIT-blocked commit caller now wakes from either path — don't assume only the async flusher can release one. -- **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`. -- **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures). -- Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`. - -## Quick Reference - -| Concern | Symbol / Site | -|---------|---------------| -| Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` | -| Durable floor (WAL tip) | `persist_durable_floor(p)` | -| Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` | -| Group-commit wake (async + sync) | `persist_set_durable_cb` → `on_durable_advance(ctx)`, fired by both `fsync_worker` and `persist_flush` on genuine advance | -| Synchronous flush (tests) | `persist_flush(p)` — also fires `on_durable_advance` (bug-1564) | -| Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` | -| Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` | -| Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` | diff --git a/.kiro/steering/product.md b/.kiro/steering/product.md deleted file mode 100644 index e24d420f..00000000 --- a/.kiro/steering/product.md +++ /dev/null @@ -1,70 +0,0 @@ -# Product Overview - -XCE is an HPC (High-Performance Computing) cluster management system providing job scheduling, queue management, and cluster resource control. It targets organizations running batch workloads on on-premises or air-gapped compute clusters who need deterministic scheduling, multi-tenant access control, and zero external database dependencies. - -## Core Capabilities - -1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API. -2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering. -3. **Cluster and host management** — node status monitoring (OK, UNAVAIL, LOCKEDU, LOCKEDW, BUSY, CLOSED, CLOSED_BY_ADMIN), cluster topology tracking, and host control operations. -4. **Multi-tenant RBAC + pluggable auth** — two-part `action:operation` permission IDs (e.g., `job:view`) with a separately-keyed domain scope per role (e.g., `type: self`, `type: self_ug`, `type: all`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting. -5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with Raft-style quorum election (RequestVote, term-based voting, freshness gates) for standby failover and live replication over TCP (Cap'n Proto serialization). - -## Target Use Cases - -- **Local HPC batch computing** — long-running scientific/engineering workloads requiring deterministic scheduling, queue priority, and resource isolation. -- **Air-gapped / secure environments** — standalone deployments with file-based auth + SSL, no external cloud dependencies. -- **Multi-team shared clusters** — multiple teams sharing a cluster through RBAC with JWT token authentication. -- **No-shared-storage HPC** — controller uses local filesystem (ext4/xfs) for storage; job files transferred via API upload/download rather than network shares. - -## Value Proposition - -XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable. - -**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the `src/lib/raft/` shared module (`raft.c`/`raft.h`, consumed by the controller through `ha_init_raft`), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database. - -## Active Development Context - -The project is in a **persistence and HA transformation** (branch `persistent_ha`). Phases: - -| Phase | Goal | Status | -|-------|------|--------| -| P0 | Restart survival (WAL + checkpoint) | Complete | -| P1 | Standby election and takeover (Raft-style) | Complete | -| P2 | Live replication primary → standby (config + WAL) | Complete | -| P2.5 | Worker/scheduler auto-reconnect on failover | In progress | -| P3–P4 | Hardening, metrics, docs | In progress | - -## Spec-Driven Development - -Features are designed and implemented via the kiro spec system (`.kiro/specs/`). -Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation. - -**Status as of 2026-07-18**: the ha-consistency-invariants umbrella (ha-commit-engine, -ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) is -COMPLETE (closed 2026-07-13). The Raft TLA+ Conformance family (raft-library, -raft-log-replication, raft-commit-safety, raft-sim-conformance) and its -XCE-integration umbrella ha-term-safety are implemented + validate-impl GO. -job-array-single-wal-record shipped complete (2026-07-17, including a -user-directed residual-gap closure pass). A backlog sweep (2026-07-18) ran -`/kiro-validate-impl` on 9 specs whose tasks were 100% done but had never -been formally closed — raft-drive-async, ha-wal-replication, -ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle, -ha-entity-snapshots, ha-commit-durability, libxce-failover, -raft-sim-conformance — all now GO. That sweep surfaced one real bug -(bug-1447: same-batch endpoint use-after-free in `io_worker`'s epoll -dispatch loop, see `ha-async-io.md`), root-caused and fixed the same day. - -Remaining backlog (requirements-only, design-pending, or partially implemented): -admin-config-tool (spec finalized), config-upgrade-snapshot (tasks generated, -unblocked, implementation nearly complete). scheduler-framework was decomposed -into scheduler-core (implementation-complete), scheduler-allocation, and -scheduler-preemption (both active). See `roadmap.md` for dependency order -and status. - -Key design decisions already locked: -- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller) -- Raft + CP adopted (2026-06-10, supersedes earlier "No Raft"): Raft is the consensus model for both election and log commitment — majority-ACK commit barrier on all state-changing writes, reject (never degrade) when quorum is unavailable. Single-writer log retained as XCE's simplification: the controller is the sole log writer, avoiding concurrent-proposal complexity while inheriting Raft's safety rules (see `.kiro/specs/ha-consistency-invariants/brief.md`) -- No BoltDB (using dual zarray COW for application-level checkpointing) -- No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt) -- No WAL logging for job deletion (deferred to checkpoint time to simplify replay) diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md deleted file mode 100644 index dc91152b..00000000 --- a/.kiro/steering/roadmap.md +++ /dev/null @@ -1,122 +0,0 @@ -# Roadmap - -## Overview - -XCE is in the persistence + HA transformation (branch `persistent_ha`). Governing model: Raft consensus + CP (see `product.md` key decisions). Active spec work is tracked per-spec in `.kiro/specs/`; this file is the **single consolidated future-work document** — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former `src/docs/FUTURE_WORK.md` (2026-05-12, config-replication-design review), which now redirects here. - -## In Flight - -- ~~**scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`.~~ **DECOMPOSED 2026-07-27** into three child specs (`.kiro/specs/scheduler-framework/spec.json` `child_specs`): **scheduler-core** — core/driver separation, pluggable job picker interface, and simulation mode; **IMPLEMENTATION-COMPLETE**, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups: (a) tlog logging-integration built but never activated, (b) `fcfs.c`/`builtin_fcfs.c` doc+traceability gap — design.md text was corrected same-day, but tasks 2.1-2.6 in `tasks.md` still carry no `_Verified:` lines, **still open**), 45/45 sub-tasks done as of that validation. **Updated 2026-08-19**: task 12.5 (REPL live-operator-testing fixes, Req 5.22 amendment) was added after that validation from live-operator testing gaps (bug-1593/1594/1595/1596) and is now also done, independently re-verified with fresh evidence (`/kiro-verify-completion`) — see `tasks.md`'s own `_Verified:` line on that task; this closes follow-up (a) too (the tlog gap is now activated — see `design.md`'s `SchedulerCore` component and `research.md`). A separate retrospective design review (2026-08-19, unrelated to the original 2026-08-07 follow-ups) additionally found and fixed an `OPCode_schedAdmin` blocking-wait admission gap and extended `core.c`'s isolation-boundary test coverage to its `core_events.c`/`core_session.c`/`core_clock.c` siblings — see `research.md`'s "SUPERSEDES" notes. **scheduler-allocation** — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. **scheduler-preemption** — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. The parent `scheduler-framework` spec is retained only for discovery-context provenance and is no longer in flight itself. Next: `/kiro-spec-design scheduler-allocation` and `/kiro-spec-design scheduler-preemption`. -- ~~**ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.~~ **COMPLETE 2026-07-13** — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight. -- ~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it~~ **task 5.4 CLOSED 2026-07-18** via `/kiro-validate-impl`'s live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight. -- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:598-599`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this. - -## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21) - -**Goal:** evolve the election-only `src/lib/raft/raft.{c,h}` into a STRICT implementation of `raft/raft.tla` (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (`raft_sim.c`) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: **bug-851** — the log protocol is currently hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c` and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates". - -- [x] raft-log-replication — *(specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.)* Extend the library to OWN the replicated log. TLA+ actions: `ClientRequest`, `AppendEntries`, `HandleAppendEntriesRequest`, `UpdateTerm`-on-AppendEntries. Per-entry term in the library log model (closes **G1**); `AppendEntries` `logOk` = `prevLogTerm` match (**G2**); conflict branch deletes the divergent `(term,lsn)` suffix (**G3**); ANY higher-term RPC incl. AppendEntries/heartbeat advances `currentTerm` and steps down (**G5**). New pure callbacks: `on_persist_entry`, `on_get_entry_term(lsn)`, `on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx)`, `on_truncate_from(lsn)`; controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5. -- [x] raft-commit-safety — *(specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.)* `AdvanceCommitIndex` with the Figure-8 current-term rule (commit `Max(agreeIndexes)` only if `log[Max].term == currentTerm`) (**G4**); `RequestVote`/PreVote freshness becomes `(lastLogTerm, lastLogIndex)` instead of LSN-only (**G6**). New callback `on_commit_advanced(lsn)` feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6. -- [x] raft-sim-conformance — *(specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.)* Extend `raft_sim.c` into a conformance/verification harness driving the library through `raft.tla`-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of `raft.tla`. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. **IMPLEMENTED, validate-impl GO 2026-07-18** (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — `raft_tla` — is named for). - -> **ha-term-safety** (`.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. **IMPLEMENTED, validate-impl GO 2026-07-17** (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix). - -> **raft-adapter-conformance** (`.kiro/specs/raft-adapter-conformance/`) — NEW 2026-07-29, sibling audit to the library-side conformance work above: a systematic, action-by-action pass over the controller's Raft **adapter** (the wire-level send/receive/reply code in `ha.c`/`repl.c`/`config_repl.c`) against `raft.tla`'s actions — distinct from `raft-sim-conformance`'s in-process library harness. Motivated by a live-soak finding that the controller's `WalAckReply` constructor hardcoded success (no failing reply ever reached the primary) plus a standby catch-up streaming-range self-deadlock, both already fixed (bug-1479) before this spec was seeded. This pass found and fixed two more instances of the same defect class: configuration-push rejects previously sent no reply at all (bug-1480), and a vote/PreVote endpoint failure arriving *after* connect routed through a legacy tally path instead of the production one (extending bug-1423's connect-time fix to that remaining timing window). Also records, as a single explicit decision, which `AppendEntries` mechanism is production-authoritative (`repl_broadcast` + `ha_stream_wal_to_standby`; the per-peer index-driven `raft_send_append_entries` stays dormant by documented design, not oversight). Requirements/design/tasks generated and approved, all 15 leaf tasks (1–7.3) checked; `src/controller/CLAUDE.md`'s Protocol Verification Checklist marks the corresponding invariants "post `raft-adapter-conformance`, landed 2026-07-29" — `spec.json` phase (`tasks-generated`) has not been flipped to reflect this, same administrative-staleness pattern seen elsewhere in this doc (cf. the former libxce-failover entry above). Dependencies: raft-log-replication, raft-commit-safety, ha-term-safety, raft-sim-conformance (adjacent/informational — this spec fixes the adapter layer those validate underneath). - -## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`); IMPLEMENTED — see `raft-apply-decoupling` below - -**Goal (historical — achieved):** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. At the time this family was opened, `persist_job_hook_fun` (an `AFTER_EVENT` hook) performed the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutated in-memory state BEFORE that commit — the structural root of the bug-795/828/851/1060 divergence family. A failed/timed-out commit did not cleanly revert the optimistic mutation. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decomposed via Path E (2 new specs + touchpoints), delivered as `raft-apply-decoupling` (**IMPLEMENTED 2026-07-07**, see below). Current code reflects the fix: `persist_job_hook_fun` (`persist.c` ~6509-6553) now explicitly returns early for `WAL_MODE_WAIT` entries, with a comment noting job-submit durability is invoked directly by `handle_jobsubmit` — `persist_submit_job_payloads_and_commit()`, called strictly before any job object is constructed — never from this hook; `create_job()` is now defined at `job.c:48`. - -### Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`) - -The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_handler` runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (`ha-async-io.md`, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § Keystone + "Two distinct prerequisites" + P2 offload mechanism): - -- **P1 — commit-barrier async** (the *commit* drive). Convert `repl_wait_majority_ack()` from blocking `pthread_cond_timedwait` (commit.c:359) to event-driven `resolve(lsn,outcome)` on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. **Owned within `raft-apply-decoupling` as its keystone** (sequenced before its Phase A). -- **P2 — drive hardening** (the *transport/timer* drive). Now its own spec **`raft-drive-async`** (brief: `.kiro/specs/raft-drive-async/brief.md`) — promoted because it threatens liveness independently. See the Specs list below. - -### Specs (dependency order — Raft Drive & Apply Async family) - -- [x] **raft-drive-async** — **IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in `io_worker`'s epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see `ha-async-io.md`; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. *(was: START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.** -- [x] raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single `apply_committed_entry()` seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). **Contains P1 (commit-barrier async) as its keystone.** Commands (WAL_MODE_WAIT) apply-after-commit; **facts (WAL_MODE_RETAIN) routed through commit-then-apply too** — D1 RESOLVED (block-but-durable under quorum loss); formalized as `ha-consistency-invariants` criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: `.kiro/specs/raft-apply-decoupling/brief.md` + `facts-in-raft.md`. **IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO** — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on `persist_t`, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned `repl_submit_majority_commit()` was never built; apply pump wired via `commit_resolve_pending()` not `ha.c`'s `on_commit_advanced`; fact hooks collapse to generic `HOOK_DETAIL_JOB_UPDATE` not per-type); task 5.3 touched `ha.c` without a formal `_Boundary:_` exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous `jobsub` path (resembles but not confirmed as bug-1416's class), and `trim_wal_segments`'s retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap). -- [ ] worker-rejoin-resync — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/worker-rejoin-resync/brief.md`). **Elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Scoped narrowly to **worker** job-execution-fact resync only — apiserver/scheduler resync-on-reconnect is a different data shape (cluster/queue/job *list* state, not job *execution facts*) and stays out of scope (see the deferred "lightweight rejoin re-attach" backlog item below, still YAGNI). Dependencies: ha-consistency-invariants (commit barrier + idempotency guards) — both implemented, unblocked. Next: `/kiro-spec-requirements worker-rejoin-resync`. -- [ ] job-array-single-wal-record — **PROMOTED from backlog 2026-06-24.** Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: `.kiro/specs/job-array-single-wal-record/brief.md`. - -### Existing Spec Updates (Raft Apply-Ordering phase) - -- [ ] raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with `/kiro-spec-requirements raft-sim-conformance` once raft-apply-decoupling requirements land. -- [ ] ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors. - -### Direct Implementation Candidates (Raft Apply-Ordering phase) - -- [ ] xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec. - -## Specs (dependency order) - -> `[x]` = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions). - -- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes) -- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied) -- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks) -- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks) -- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.) - -## Future Specs / Backlog (dependency order) - -- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.** -- [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands. -- [x] ~~worker-rejoin-resync~~ — **PROMOTED to spec 2026-08-10, see In Flight (Raft Apply-Ordering family) above** for current status and brief. No longer tracked here. -- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) was blocked on `entity-bearing-snapshots`; that blocker is now cleared (see Future Specs/Backlog above — `ha-entity-snapshots` implemented, `config-upgrade-snapshot` `blocked_on: []`). Next: `/kiro-spec-requirements admin-config-tool`. -- [ ] **component-rejoin-resync** — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/component-rejoin-resync/brief.md`) from the "lightweight rejoin re-attach" item below. **apiserver + scheduler** reconnect/rejoin resync efficiency — today each does a full re-join and receives full state again (whole job list, queue/resource/config state), not a delta; for a large cluster this is sizable but not yet a measured problem. Sibling of `worker-rejoin-resync` — disjoint data (list/config state vs. job-execution facts), NOT merged into that spec. Not started: first step is investigation (is reconnect cost actually a measured problem at realistic cluster sizes?) before committing to any delta/session-token mechanism — original YAGNI discipline still applies, this spec is the promoted vehicle for that investigation, not a rubber-stamped design. Original decision (libxce-failover design review) retained below for provenance. Dependencies: libxce-failover (rejoin hook — implemented), component-failover-adoption (daemon rejoin content — in progress, sequence alongside/after). Next: `/kiro-spec-requirements component-rejoin-resync` when ready to begin (not immediate). -- [x] ~~lightweight rejoin re-attach (failover optimization)~~ — **recorded 2026-06-16; deferred (YAGNI)**; apiserver/scheduler half **PROMOTED to `component-rejoin-resync` 2026-08-10** (see above); worker half already promoted to `worker-rejoin-resync`. No longer tracked as a bare backlog item — provenance only. Original text: on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. -- [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready. -- [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships. -- [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item. -- [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization). -- [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants). -- [x] entity-bearing snapshots ("Task 11" format) — **PROMOTED to spec `ha-entity-snapshots` (created 2026-07-01); IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks 1.1-8.1; 962 controller tests green including all 9 `test_persist_snapshot_entities.c` cases; live `--force-snapshot` soak: committed_divergence=0, committed_job_loss=0, confirmed real "HA: snapshot" install log line). `write_snapshot()` (`persist.c` ~2304-2372) now collects and serializes job/queue/host entity frames (`collect_job_frames`/`collect_queue_frames`/`collect_host_frames`) ahead of the header — no longer a bare 16-byte LSN sentinel. Original backlog note retained for provenance: real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs was already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream: `config-upgrade-snapshot`'s former HARD dependency on this is now cleared** (`blocked_on: []`) — see that item above. **Also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0)** is now live: the standby-side snapshot-install/apply path is wired (`ha.c` ~3112-3147 wire-receive path task 6.1; `persist.c` ~4277-4285 local boot recovery task 8.1), with a decision path that requests a snapshot transfer when a standby's cursor falls below the leader's oldest retained WAL LSN (`ha.c:4413`). The bug-1147 WAL-trim cursor-bound guard (see Hardening below) is implemented in production, no longer just a stopgap. - -## Hardening / Direct-Implementation Items - -Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work. - -- [x] ~~eliminate src/frontend/Makefile, consolidate Go builds onto meson~~ — **PROMOTED to spec 2026-08-27** (`/kiro-discovery`; brief at `.kiro/specs/eliminate-frontend-makefile/brief.md`). Investigation (recorded 2026-08-21, from `dirty-content-fingerprint` spec's task 4.3) confirmed the risk is real and independently corroborated: `build-server` is separately confirmed BROKEN today (bug-1597, bare `go build` with no CGO flags, `fatal error: client.h: No such file or directory`; `src/docs/deployment.md:243-247` already tells developers to use `/xce-build apiserver` instead). Blast radius is narrower than originally framed: `/xce-build` already builds both binaries via meson exclusively (`build-cli`/`build-server` are dead weight from that path today); only 2 live call sites (`xce-build.sh:240,287`) depend on `make test`, with an already-proven direct-`go test` replacement 3 lines away in the same file. One genuine gap found: no meson `test()` exists for the CGO backend suite (`test-backend`) — that must be added before the Makefile can be safely removed, or backend CGO coverage is lost, not just inconvenienced. Next: `/kiro-spec-init eliminate-frontend-makefile`. -- [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later. -- [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060. -- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_ -- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`. -- [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.** -- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.** -- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_ -- [x] ~~OP_upgradeConfig standby redirect verification~~ — **RESOLVED 2026-08-19, then the underlying gap itself CLOSED 2026-08-26 (bug-1700/1701).** As of 2026-08-19, a standby did NOT return the primary's address for `OPCode_upgradeConfig` — `ha_dispatch_gate` rejected it before the handler's own dead defensive check ever ran, replying flat `status=-XCE_ERR_STANDBY, data_which=nodata` with no redirect fields; `fill_redirect_reply()` (`join.c:27`) was never called for this opcode, a documented, deliberate feature gap (`admin-config-tool` design.md's "Revalidation Triggers"). That gap is now closed: `cluster.c`'s generic dispatch-gate rejection path calls `fill_redirect_reply()` on every `-XCE_ERR_STANDBY` rejection (bug-1700/1701, gate-level fix, not opcode-specific), and a standby pushing `config push` now redirects to the primary and succeeds. Coverage: client-side redirect-following remains unit-tested against mocks (`test_discovery.py`, `test_config_push_mock.py::test_push_config_follows_one_redirect_to_primary`); the real server-side behavior is live-verified against a real 3-node cluster (`src/xceadmin/tests/test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds`, renamed from `..._returns_standby_rejection_not_redirect` to match the now-correct outcome). -- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1. -- [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump <snap_dir|snap_file>`** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare <dirA> <dirB>`** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_ -- [x] ~~xce_test shell command support~~ — **IMPLEMENTED** (task 2.1, spec `xce-test-e2e-primitives`, closed 2026-06-19; stale entry corrected 2026-08-19). `preprocess_and_run()` (`src/controller/testsuite/test.c:950`) detects a leading `!` and runs the remainder via `system()`; the shell command's exit status becomes `g_last_status` (the `$?` auto-variable) and a non-zero status sets `g_session_failed`, aborting the script the same way any other failing command does — `system()`'s child inherits the parent's stdout/stderr fds, so output lands in whatever the invoking `xce_test` session's own output is already redirected to. Works in every REPL context (no `leave_ctx()`/disconnect), including mid-session inside `worker`/`client` contexts — used this session to fix bug-1579 (resource-value-verification's nested controller-check pattern). No parser/harness-layer change remains to do. -- [x] scheduler framework enhancement — **PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27** into `scheduler-core` (**IMPLEMENTATION-COMPLETE**), `scheduler-allocation`, and `scheduler-preemption` — see **In Flight** above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here. -- [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass. -- [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`. -- [ ] **resource-distribution-mapping** — **created 2026-07-23, never added here (found 2026-08-19, itself a small doc gap).** Physical resources have no "configured total vs. currently-consumed" concept anywhere in this codebase, live or dead — `resources.yaml`'s `resourcemap` field (which expresses exactly this: a value shared across N hosts) is deliberately dropped by `xceadmin config build` (`build.py:41-43`), and the matching `controller.capnp` fields are commented out (`controller.capnp:208-209`). `mark_host_res()` (`worker.c`) actively rejects any resource flagged `XCE_RESF_SHARED` today, so "shared pool" and "per-host value" are mutually exclusive by construction in the live system right now. A working total/current pooling mechanism DOES exist in the scheduler (`resource.c`'s `sched_check_shared_satisfy_()`), but it exclusively serves job-limit virtual resources (queue `MAX_RUNNING`-style), never physical shared resources — architecturally reusable, never connected. This spec (research.md done, 3 scoped options, requirements.md never generated — sitting at the decision point) is the existing owner of closing this gap; surfaced again 2026-08-19 when scheduler-core's REPL `info resource` command couldn't show a total for a GLOBAL/SHARED resource because there genuinely isn't one to show. Next: pick a scope option in research.md, then `/kiro-spec-requirements resource-distribution-mapping`. -- [ ] **scheduler-sim-diagnostics** — **NEW 2026-08-10 (`/kiro-discovery`).** Open-ended, continuously-extended spec for scheduler simulation-mode debug/troubleshoot capability — added to over time as new investigation needs surface from discussion or from other in-flight scheduler specs (`scheduler-allocation`, `scheduler-preemption`). Extends `scheduler-core`'s REPL (`info job/queue/worker/resource/summary`, breakpoints, snapshots). Two seed capabilities identified so far: (1) **limit inspection** — `xce_limit_t`/`xce_vres_t` config + live usage currently live only in the `limit` plugin's private kvmgr tables, invisible to the REPL; needs a new `info limit <name|all>` command + accessor plumbing. (2) **decision-reasoning trace** — `xce_pendreason_t` is captured on pend decisions but never printed by the REPL, and dispatch decisions have no reasoning trail at all (no record of which limit/resource checks passed) — needs new instrumentation in the allocation path (`resource.c`'s `sched_try_alloc_on_host_`, `plugin.c`'s `limit_prepare_allocation`) plus a REPL command to surface it. Explicitly simulation-mode only (no live/production controller-connected diagnostics) and read-only (no state mutation) per user decision 2026-08-10. Brief: `.kiro/specs/scheduler-sim-diagnostics/brief.md`. Not started — requirements to be added per-capability, not fully speced upfront. Next: `/kiro-spec-init scheduler-sim-diagnostics` when ready to begin. -- [ ] **xcecd_virtual concurrent-job support (MXJ > 1)** — **NEW 2026-08-19, recorded during bug-1589 investigation** (`.wolf/buglog.json`). Live testing found that when a job array (multiple elements) is dispatched to a single `xcecd_virtual` worker in quick succession, only the first element's start ever gets confirmed back to the controller (`on_job_start`) — the rest are dispatched but never confirmed. Root-caused (or being root-caused — see bug-1589 for the confirmed mechanism once the in-flight investigation lands) to `xcecd_virtual`'s job-simulation path: unlike the real `xcecd` binary, which launches an actual dbus-managed worker process per job (tracked by real pid, real IPC status reports), `xcecd_virtual` (`src/worker/src/xcecd.c`'s `xcecd_process_command()`, `VIRTUAL_CLUSTER` branch) never launches a real process at all — it assigns a bare incrementing fake pid (`static int virtual_pid`, `xcecd.c:27`) and simulates job execution via a sleep-timer mechanism (`xcecd_worker_utils.h`'s `VIRTUAL_CLUSTER`-only `xcecd_worker_t` fields `start_time`/`sleep_time`/`killed_signal`, `check_worker_sleep_time_elapsed()`). The immediate bug-1589 fix (once landed) addresses the specific reported symptom; this item is the broader follow-up: **properly support N concurrent simulated jobs per `xcecd_virtual` worker**, since MXJ (max jobs per host) > 1 is a legitimate, common configuration to want to exercise in virtual-cluster-mode testing/CI (this project's own dev workflow relies on `xcecd_virtual` for HA soak tests, resource-value verification, and now scheduler-core's live validation — several `.wolf/buglog.json` entries this session alone used it), not just a single-job-at-a-time edge case. Scope sketch (to refine once bug-1589's root cause is fully confirmed): audit every VIRTUAL_CLUSTER-only code path in `src/worker/` (`xcecd.c`, `main.c`, `xcecd_worker_utils.h`/`.c`, `process_worker_manager.c`) for single-job assumptions (a lone poll/early-return that only processes one pending simulated job per tick, a shared mutable scratch struct reused across concurrent jobs, etc.) and generalize each to iterate the full set of in-flight simulated jobs, matching the pattern the real-process path (`process_manager.c`'s `process_info_t manager->processes[]`, a proper per-pid array) already establishes correctly. Dependencies: bug-1589 (must land first — its root-cause trace is the concrete map of what needs generalizing). Promote via `/kiro-discovery` or `/kiro-spec-init` once bug-1589 is closed and the exact mechanism is confirmed. - -## Superseded / Absorbed Items - -- ~~Quorum-ACK gate before config restart~~ _(FUTURE_WORK.md §2)_ — **largely superseded by ha-consistency-invariants Requirement 5**: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes). -- ~~Lost-and-found keyword validation~~ _(FUTURE_WORK.md §8)_ — folded into the admin-config-tool spec (brief: `.kiro/specs/admin-config-tool/brief.md`). -- ~~admin-config-tool backlog item~~ _(FUTURE_WORK.md §1 + §8, 2026-05-12)_ — **absorbed into `admin-config-tool` spec 2026-07-06** (`/kiro-discovery`). The one-paragraph backlog sketch is superseded by the full brief with architecture decisions, decomposition, and boundary contracts at `.kiro/specs/admin-config-tool/brief.md`. - -## Notes - -- Backlog items here are seeds for `/kiro-discovery` / `/kiro-spec-init`; promote an item by creating its spec under `.kiro/specs/`. -- Provenance: items marked _(FUTURE_WORK.md §N)_ originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10. -- **2026-07-06 admin-config-tool discovery** (`/kiro-discovery`, Path E mixed decomposition): - - | # | Workstream | Type | Language | Dependencies | - |---|-----------|------|----------|--------------| - | 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only | - | 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) | - | 3 | **perf-tuning-extraction** | DIRECT work | doc | None | - - **Key architecture decisions:** Python (future open-source, shared capnp infra), pure Python wire (no CGO/libxce), direct TCP+TLS to controller (not apiserver), single `controller.capnp` dependency, `xceadmin` separate from `xcecli` (user=REST, admin=TLS), auto primary detection via peer list + redirect. Full details: `.kiro/specs/admin-config-tool/brief.md`. - - **2026-07-06 scheduler-framework discovery** (`/kiro-discovery`, Path C single-scope): - - **Key architecture decisions:** Iterator-driven allocation (lazy plan iterator + host iterator composition), configurable job picker (plugin name from config, not hardcoded), combinatorial parallel job allocation (1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 sorted by preference), session-level unsatisfiable resreq fingerprint cache. Scope narrowed from original backlog seed: commit barrier wiring deferred, new job selection policies deferred (framework enables them), GPU/affinity allocation deferred (pluggable via allocator interface). Alternatives documented in brief.md (Approach A: incremental/rejected, Approach C: full framework/deferred). Full details: `.kiro/specs/scheduler-framework/brief.md`. diff --git a/.kiro/steering/scheduler-driver-equivalence.md b/.kiro/steering/scheduler-driver-equivalence.md deleted file mode 100644 index 6d54d000..00000000 --- a/.kiro/steering/scheduler-driver-equivalence.md +++ /dev/null @@ -1,36 +0,0 @@ -# Scheduler Driver Equivalence - -## Philosophy - -`scheduler-core` split the scheduling engine into a mode-agnostic **core** state machine (`core.c`/`core_events.c`/`core_session.c`, `xce_sched_core_t`) driven by two interchangeable **drivers**: `driver.c` (controller-connected, capnp-driven, live) and `sim.c`/`sim_loop.c` (standalone, `.replay`-file-driven). The entire value of simulation/replay mode rests on one premise: replaying the exact recording of a normal-mode run reproduces what actually happened, so an operator can debug and tune scheduling against recorded production data and trust the result. `src/scheduler/CLAUDE.md` states this as "one core, two consumers" — this file makes it an enforced constraint, not a description. - -## DESIGN INVARIANT: State equivalence, not just decision equivalence - -**Replaying the exact recording of a normal-mode run MUST reproduce identical scheduling decisions AND identical resource-accounting state** (`w->resInUse`, `core->sharedInUse`, `job->accounts`) **at every point in the replay** — not just a matching final decision sequence. - -Decision-only equivalence is an insufficient test of this invariant. A fixture with no resource contention (every job fits regardless of what `resInUse` says) can produce identical decisions between normal and sim mode while the underlying accounting state has silently diverged — because nothing in that fixture ever depends on `resInUse` being correct. The divergence only becomes decision-visible under contention, which is exactly the scenario most replay-based debugging exists to investigate. **Treat accounting-state parity as load-bearing, not incidental to decision parity.** - -This is a stricter, broader sibling of the REPL-vs-batch-sim invariant (`--interactive` and batch `--sim` both drive `sched_sim_driver_run()` gated only by `sim->interactive`, proven decision-equivalent by construction and enforced by `test_repl_equivalence.c`). That invariant is solid — both paths share the exact same driver code. Normal-mode-vs-sim-mode equivalence is weaker: they are *separate* driver implementations that must each independently uphold the same per-round contract around the shared core, and nothing mechanically forces them to stay in sync when one changes. - -## Why a driver can silently violate this - -The core deliberately does not own accounting. `xce_core_begin_session()`'s resval reset zeroes `resInUse` every session; re-summing it from currently-running jobs (`sched_account_jobs()`) is documented as "real, load-bearing production behavior... `xce_scheduler_t*`-coupled and therefore outside this pure-core module's reach" (`core.h`) — i.e. each driver must call it itself, in the gap between `xce_core_begin_session()` and `xce_core_signal_go()`. `driver.c`'s `xce_sched_driver_run_normal_mode_round()` does this correctly. Nothing forces a second driver implementation to remember to. - -Similarly, a `ReplaySession` carries a `decisions` list — the recorded decisions from the original live run — decoded every round via `decode_ReplaySession_ptr()`. There is no structural guarantee a driver applies that data (reconstructing `job->allocs`/`job->accounts` for a job already resolved in the recording) rather than only using it for optional comparison. - -## Known violations (evidence trail) - -- **bug-1613** (confirmed): `sched_account_jobs()` is called by `driver.c` but never by `sim_loop.c`, which still uses the older monolithic `xce_core_go()` with no interleaving point for it. Compounding this, `sched_account_jobs()`/`sched_charge_account()`/`sched_update_account()`/`sched_update_res_accounts()` are signatured on `xce_scheduler_t*` (the normal-mode-only driver struct) even though their bodies only ever touch `sched->core` — `xce_sched_sim_driver_t` deliberately has no `xce_scheduler_t*` field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. `sched_resolve_job_picker()` went through this exact `xce_scheduler_t*` → `xce_sched_core_t*` relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent. -- **bug-1611** (suspected same root cause): `sim_loop.c` only reads `decoded_session->decisions` when `sim->verify` is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct state for a job the live picker in this replay isn't independently re-deciding. This plausibly explains both bug-1611 (no `DISPATCH` decision ever visible for such jobs) and the accounting gap in bug-1613 (their usage is never counted, not even transiently) as one underlying gap, not two. -- Both are tracked in `.wolf/buglog.json`; a dedicated fix spec is scoped separately (see `.kiro/specs/` — do not fix inline as a drive-by patch, per the testing discipline below). - -## Testing discipline - -Any change touching resource accounting (`resource.c`), core event application (`core_events.c`), or driver session-boundary handling (`driver.c`, `sim_loop.c`, `core_session.c`) must be validated against **a resource-contention fixture** — multiple jobs competing for capacity insufficient for all of them — not only a fixture where every job fits. A no-contention fixture cannot distinguish "accounting is correct" from "accounting is wrong but nothing currently reads it." Existing coverage in this space (a normal-mode-recording-vs-simulation-replay decision-identity test) evidently does not exercise contention, or bug-1613 would have failed it. - -When adding or modifying driver-specific per-round logic in either `driver.c` or `sim_loop.c`, check the other driver for the equivalent step before assuming none is needed — the two are not required to look identical, but any core-derived state (`resInUse`, `sharedInUse`, `job->accounts`, decision counts) they produce from equivalent input MUST match. - -## See Also - -- `src/scheduler/CLAUDE.md` — architecture overview; "Drivers" section points here for the equivalence contract. -- `.kiro/specs/scheduler-core/` — original driver-split design and requirements. diff --git a/.kiro/steering/security.md b/.kiro/steering/security.md deleted file mode 100644 index 55d2125e..00000000 --- a/.kiro/steering/security.md +++ /dev/null @@ -1,71 +0,0 @@ -# Security Standards - -## Philosophy - -Defense in depth, least privilege, fail closed. Authentication verifies *who*; authorization verifies *what they can do*. Centralize both — never duplicate permission checks across handlers. - -## RBAC Enforcement - -All protected routes use `RequirePermission` middleware attached at registration time: - -```go -// In router/routers.go -hostsGroup.GET("", RequirePermission("host:view"), handler) -hostsGroup.POST("/control", RequirePermission("host:control"), handler) -``` - -`RequirePermission` in `controllers/rbac_check.go` resolves the user from the JWT, looks up their effective permissions (including inherited role permissions), and aborts with 403 if the required permission is absent. - -**cluster_admin** bypasses all permission checks via early return — do not add new bypass paths outside this single mechanism. - -**Known gap (not yet scoped): runtime RBAC mutations are not persisted.** `store.Roles`/`store.UserRoles` (`store/stores.go`) are loaded from `config/rbac.yaml` at startup via `LoadRBACConfig()`, but are also mutated at runtime with no write-back to disk or any DB — `RoleController.AddPermissionsForRole` (`controllers/roles.go`), and `store.AddRoleForUser`/`DeleteRoleForUser` (called from `AuthController.Logon`'s auto-role-assignment and the user-role-assignment endpoints in `controllers/users.go`). On apiserver restart, any role/permission granted or user/role assignment made via API since the last startup silently reverts to whatever `rbac.yaml` says, with no error or audit trail of the loss. This is the same save/restore field-completeness failure shape documented in `persistence.md` (state that "recovers correctly" while silently dropping data it depends on), just in the apiserver's in-memory RBAC store rather than a WAL/snapshot. Flagged during the `scheduler-snapshot-job-allocation` cross-component persistence survey (2026-08-27); intentionally deferred rather than fixed inline — needs its own scoped design (write-through to `rbac.yaml`? a small embedded store? explicit "runtime-only, does not survive restart" documented as accepted scope?) rather than a quick patch. - -## Permission String Format - -`"action:scope"` — e.g., `"queue:submit"`, `"host:view"`, `"queue:control"`. - -Scopes: `all` (unrestricted), `self` (own resources), `cluster`, `hostgroup`, `self_ug`, `self_hg`. - -Permissions are defined in `config/rbac.yaml`. Roles support inheritance via `includes:` — resolution is recursive. Add new permissions to the YAML first, then wire them onto routes. - -## JWT Authentication - -Token flow: -1. `POST /auth/logon` → validates credentials via configured auth backend → issues JWT with `{username, IP, expiry}` claims -2. Every subsequent request: `Authorization: Bearer <token>` header -3. `middleware/validatehandler.go` strips the prefix, parses with golang-jwt, **validates the IP claim against the request IP** - -IP-bound tokens prevent token theft across network boundaries. Do not remove the IP validation. - -Token secret is loaded from `config/auth.yaml` at startup. It must never appear in source code or logs. - -## Auth Backends - -Backend selection is controlled by `config/auth.yaml`'s `provider:` field, matched literally against a `switch` in `backend/auth/init.go` — an unrecognized value (e.g. `configfile` without the underscore) fails apiserver startup with "unsupported auth provider". Valid identifiers: `config_file`, `pam`, `ldap`, `oauth2`. Only `config_file` (dev mode, users defined in the YAML) and `pam` (Linux + CGO only) are functional today — `ldap` and `oauth2` are unconditional stubs whose `Authenticate()` always returns "not yet implemented". All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware. - -## Input Validation - -Validate at the API boundary inside `preCheck{Resource}Request()` helpers — before any service or store access. Return 400 with a clear message on validation failure. Do not pass unvalidated input to downstream layers. - -## Secrets - -- JWT secret key: `config/auth.yaml` — should be file-system protected (e.g. chmod 0600); this is not currently enforced by the code or deploy tooling, so treat it as an operator responsibility, not a guarantee -- LDAP bind credentials: `config/auth.yaml` -- **Never log** tokens, passwords, secret keys, or full request bodies containing credentials -- Validate all required config at startup; fail fast with a clear error if missing - -## Rate Limiting and IP Whitelist - -Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/whitelist.go`, which implements both `WhitelistURLMiddleware()` and `WhitelistIPMiddleware()`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds. - -## Transport - -All daemon-to-daemon communication is over TCP. The WAL and replication wire format uses Cap'n Proto — schema files in `src/schema/`. Do not add unvalidated deserialization paths for external input. - -XCE uses **server-side TLS only** — encryption without peer certificate verification. There is no mTLS and no CA-pinning anywhere in the system; peer verification is deferred as optional P3 hardening (see bug-1453, which deleted all CA-cert/client-cert plumbing from both the C tier and the Go tier as dead code). Do not reintroduce CA-cert loading, client certificates, or cert-pinning without a design doc — this posture is intentional, not an oversight. - -**Clients talking to the API server** (xcecli, or any HTTP client of `apiserver`) must support **both `http://` and `https://`** — the scheme is a per-deployment/user choice (`APIServerInfo.URL`), not something the client hardcodes. When the scheme is `https://`, set `InsecureSkipVerify: true` on the TLS client config (see `xcecli/common/request/request.go`'s `NewAPIClient`) rather than attempting cert validation — the API server's cert is typically self-signed/auto-generated and the client has no CA to validate it against. - -**Clients talking to an XCE controller directly** (worker, scheduler, and any other libxce-based client) need **only server-side certs — no client-side certs are required or supported**. The live connection path is `xcecd_join_cluster()` (`src/worker/src/controller_connector.c`) calling `xcecli_start`/`xcecli_join` (`src/lib/client.c`), which drives `endpoint_connect()` in `src/lib/netio.c` over an `SSL_CTX` built by `create_ssl_context(NULL, NULL)` (`src/lib/xcectx.c`) — note that `cluster.c`'s `do_connect()`/`xce_connect_cluster()`/`xce_reconnect_init`/`xce_reconnect_step` are an unused stub path (explicitly documented as "to be wired in a later task") with no callers outside `cluster.c` and its own testsuite; don't point at them as the live mechanism. `xce_cluster_config_t` has no `ca_cert_path`/client-cert fields; do not add any. TLS verification mode is `SSL_VERIFY_NONE` throughout — this matches the API-server posture above, just at the C/libxce layer instead of Go/HTTP. - -**Known follow-up (not yet scoped):** a HAProxy reverse proxy in front of `apiserver` is planned to terminate HTTPS, likely with a CA-trusted cert (Let's Encrypt candidate, unconfirmed), and whether `apiserver`'s own `RunTLS`/`httpsConfig.Enabled` listener stays in the picture is also undecided. Once both are settled, revisit xcecli's blanket `InsecureSkipVerify: true`: if the client-facing endpoint ends up CA-trusted, default to real verification (Go's default `tls.Config{}` validates against the system trust store with no extra plumbing) and make skip-verify an explicit opt-in for self-signed deployments only — matching the automated security review's suggested fix. Don't implement this speculatively; the right shape depends on which of those two decisions lands where. diff --git a/.kiro/steering/structure.md b/.kiro/steering/structure.md deleted file mode 100644 index 535534b5..00000000 --- a/.kiro/steering/structure.md +++ /dev/null @@ -1,106 +0,0 @@ -# Project Structure - -## Organization Philosophy - -**Layered monorepo, component-per-directory.** Each major subsystem lives under `src/<component>/` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/`. - -``` -src/ -├── lib/ C — shared library (libxce): data structures, IPC, utilities -│ ├── raft/ C — reusable Raft consensus module (raft.c/raft.h) -│ └── testsuite/ C — libxce unit tests (test_*.c) -├── controller/ C — cluster controller daemon -├── worker/ C — per-node job execution daemon (xcecd) -├── scheduler/ C — job scheduling engine (plugin architecture) -├── autoscaler/ C — autoscaling engine (provider plugins) -├── frontend/ Go — apiserver (Gin REST) + xcecli (Cobra CLI) -├── xceadmin/ Python — admin CLI tool -├── include/ C — public shared headers consumed across all daemons -│ └── xce/ C — subset of public headers under the xce/ prefix -├── schema/ Cap'n Proto schema files (c.capnp, controller.capnp) -└── docs/ Project-wide documentation (markdown) -``` - -## C Daemon Layout Pattern - -`worker/` and `scheduler/` follow: - -``` -src/<component>/ - meson.build — build definition - CLAUDE.md — component-specific AI guide - include/ — component-internal headers - src/ — .c implementation files (flat, one domain per file) - testsuite/ — test programs (test_*.c) -``` - -**Exceptions**: -- `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`), and does have its own `testsuite/`. -- `worker/` has no `testsuite/` — no `test_*.c` files exist anywhere in the component. -- `autoscaler/` does not follow this pattern at all: it has no CLAUDE.md, and its daemon code lives two levels deeper in `src/autoscaler/scaler/` (flat `.c` files plus its own `include/` and `testsuite/`), alongside `src/autoscaler/providers/`. - -Plugin-based components add their own subdirectory: -- `scheduler/` — plugins in `src/plugins/` -- `autoscaler/` — providers in `src/autoscaler/providers/` - -## Go Frontend Layout Pattern - -**apiserver** (`src/frontend/apiserver/`): - -| Package | Purpose | -|---------|---------| -| `cmd/` | Entry point, init sequence | -| `router/` | Route registration + middleware chain | -| `controllers/` | HTTP handlers — one file per resource | -| `services/` | Business logic | -| `store/` | Thread-safe in-memory KV stores (generics) | -| `models/` | Request/response DTOs, domain structs | -| `middleware/` | CORS, rate limiting, JWT, IP whitelist | -| `backend/` | CGO bridge to C library | -| `backend/auth/` | Pluggable auth providers | -| `utils/` | Zap logger wrapper, config parsing, crypto | -| `common/` | Global error codes and response helpers | -| `config/` | YAML configs + JSON schemas | - -## Naming Conventions - -### C - -- **Files**: `snake_case.c` / `snake_case.h` — one domain per file -- **Public headers**: `src/include/<name>.h` (e.g. `persist.h`, `netio.h`, `resreq.h`) — the libxce API boundary; a small subset (`client_failover.h`, `cluster.h`, `cluster_def.h`, `errnox.h`, `types.h`) lives under `src/include/xce/` and is included as `xce/<name>.h` -- **Component headers**: `src/<component>/include/<name>.h` -- **Test files**: `test_<unit>.c` (e.g., `test_ha_cluster.c`, `test_persist_wal.c`) -- **Functions**: `snake_case` with component prefix — `ha_*`, `cluster_*`, `xcecd_*`, `xce_*` -- **Types**: `snake_case_t` suffix (e.g., `cluster_t`, `ha_state_t`) -- **Struct tags**: `struct <name>_s` (e.g., `struct cluster_s`) -- **Enums/Macros**: `UPPER_SNAKE_CASE` with component prefix (`CTRL_PRIMARY`, `HA_DEFAULT_*`) - -### Go - -- **Files**: one file per resource — `jobs.go`, `queues.go`, `hosts.go` -- **Tests**: `*_test.go` co-located; integration tests as `integration_test.go`, `<resource>_control_test.go` -- **Controller structs**: `{Resource}Controller` (e.g., `JobController`, `ClusterController`, `QueueController`); one exception — `hosts.go` uses `HostHandler` -- **Handler methods**: `{Action}{Resource}` (e.g., `CreateJob`, `GetJobsInfo`, `ControlJobByID`) -- **Permission strings**: `"action:scope"` (e.g., `"job:view:self"`) -- **JSON tags**: `snake_case` - -## Test Organization - -**C tests** (`testsuite/` inside each component): -- Standalone `.c` programs, one per unit: `test_<unit>.c` -- `test.c` provides the test runner harness -- Tiered naming: `test_p1_integration.c`, `test_e2e_wal.c`, `test_p3_hardening.c` -- Test-only symbols guarded by `#ifdef XCE_TESTING` - -**Go tests** (co-located `_test.go`): -- Table-driven unit tests, `t.TempDir()` for fixtures -- Global state reset helpers (e.g., `resetRBACState()`) for test isolation -- `make test` = unit only; `make test-all` = unit + integration - -## Code Organization Principles - -- **One domain per file** in C — `ha.c` owns HA logic, `job.c` owns job logic. No megafiles. -- **Single Go module per binary** — apiserver and xcecli share no Go code at the module level. -- **Controllers delegate to services** — HTTP handlers validate input and delegate; business logic lives in `services/`. -- **Dependencies flow inward**: `controllers → services → store/backend`, never the reverse. -- **Cap'n Proto for wire formats** — any new inter-daemon or persistence protocol uses `.capnp` schemas in `src/schema/`. diff --git a/.kiro/steering/tech.md b/.kiro/steering/tech.md deleted file mode 100644 index b66a35aa..00000000 --- a/.kiro/steering/tech.md +++ /dev/null @@ -1,102 +0,0 @@ -# Technology Stack - -## Architecture - -``` -xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer - controller / scheduler / worker / autoscaler - ↑↑↑ all linked against libxce (src/lib/) -``` - -Request path inside apiserver: `Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store` - -All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level. - -## Core Technologies - -- **Go 1.23.4** — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no `go.work`. -- **C (gnu17)** — controller, worker, scheduler, autoscaler, libxce. No C++. -- **Gin v1.10.0** — HTTP router, middleware, binding -- **Cobra v1.9.1** — CLI framework for xcecli -- **Zap v1.27.0** — structured logging throughout (both Go modules) -- **Viper v1.19.0 / v1.20.1** — YAML config management -- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication -- **iomgr / vtaskmgr / delaymgr** — libxce async I/O framework: single-threaded epoll loop (iomgr), thread-pool packet dispatch (vtaskmgr), timer thread for periodic/blocking tasks (delaymgr) - -## Build Systems - -| System | Scope | -|--------|-------| -| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper | -| **Make** | Go development loop (build/test) from `src/frontend/` | -| **Nix flakes** | Hermetic dev environment, AppImage packaging | - -```bash -# Full build (C + Go, debug + ASAN) -/xce-build - -# Go development loop (from src/frontend/) -make build-server # → /tmp/apiserver -make build-cli # → /tmp/xce_static (CGO_ENABLED=0) -make test # unit tests -make test-integration # regression suite (needs XCE_BIN + SRV_URL) - -# Nix package builds (from src/) -nix build .#xcecore # C daemons + apiserver -nix build .#xcecore_debug # debug + ASAN -nix build .#xcecli # CLI only -``` - -## Development Standards - -### Controller Pattern (Go) - -```go -// Struct: {Resource}Controller (typically an empty struct{}) -type JobController struct{} - -// Handler: {Action}{Resource} -func (j JobController) CreateJob(ctx *gin.Context) { - // Validate input - var req models.JobSubmitRequest - if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return } - parsedIndices, err := preCheckJobRequest(&req) - if err != nil { ReturnError(ctx, code, err.Error(), nil); return } - - // Business logic — package-level services function, not a controller field - err = services.SubmitJob(&newJob, arraySize) - if err != nil { ReturnError(ctx, code, err.Error(), nil); return } - - ReturnSuccess(ctx, code, "submitted", result) -} -``` - -Early return on every error. Log with zap fields before returning. - -### Testing Requirements - -- **TDD mandatory**: Write failing test first, then implementation. -- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures. -- **C tests**: Add test case in `testsuite/` before modifying implementation. -- `make test` must pass without a live server. -- `make test-integration` requires `XCE_BIN` + `SRV_URL` env vars. - -### Code Formatting - -- Go: `gofmt` / `goimports` (post-edit hooks configured) -- C: `clang-format` using project config at `xce/.clang-format` - -### Auth and RBAC - -- Auth backend set in `config/auth.yaml` (`provider: config_file | pam | ldap | oauth2`). Only `config_file` and `pam` (Linux + CGO) are functional; `ldap` and `oauth2` are documented as future/stub providers whose `Authenticate()` always returns a "not yet implemented" error. -- Permissions are a plain action string (e.g. `"job:view"`, `"queue:submit"`) paired with a separate structured `domain` list (e.g. `{type: self}`, `{type: all}`) in `rbac.yaml` — not a combined `"action:scope"` string. `controllers.RequirePermission()` checks the action string only; domain/scope is evaluated separately. -- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware. - -## Key Technical Decisions - -1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both. -2. **In-memory API state** — no database in Go tier; `store.InitStores()` allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (`backend.RegisterToBackend()` + `backend.StartNotificationListener()` in `cmd/main.go`). Only RBAC (`config/rbac.yaml`) and auth (`config/auth.yaml`) are actually loaded from YAML at startup. -3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell. -4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin. -5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output. -6. **Single-writer HA with full Raft log replication** — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, `AppendEntries` request/reply, per-peer `next_index[]`/`match_index[]` progress vectors, `(term,index)` log matching/truncation, commit-index advance) implemented in the `src/lib/raft/` shared module (`raft.c`/`raft.h`) and validated by the `raft-log-replication`, `raft-commit-safety`, `raft-ha-core-safety`, and `ha-term-safety` specs. diff --git a/.kiro/steering/testing.md b/.kiro/steering/testing.md deleted file mode 100644 index 0a2fed27..00000000 --- a/.kiro/steering/testing.md +++ /dev/null @@ -1,95 +0,0 @@ -# Testing Standards - -## Philosophy - -- **TDD mandatory**: Write a failing test first, then the minimal implementation. The cycle is Red → Green → Refactor. -- Test behavior, not implementation details. -- Tests must run without a live server or external services. -- A change is not complete until `make test` (and `make test-integration` where applicable) passes. - -## Go Tests (apiserver / xcecli) - -**Co-located** alongside source: `jobs.go` → `jobs_test.go`. Integration tests use descriptive names: `integration_test.go`, `host_control_test.go`. - -**Table-driven** for validation/filter logic: - -```go -tests := []struct { - name string - input SomeRequest - wantErr bool -}{ - {"valid request", validReq, false}, - {"missing field", missingReq, true}, -} -for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - got, err := preCheckRequest(tt.input) - if (err != nil) != tt.wantErr { t.Errorf(...) } - }) -} -``` - -**State isolation** — reset global store state between tests: - -```go -func resetRBACState() { /* clear in-memory stores */ } - -func TestPermissions(t *testing.T) { - t.Cleanup(resetRBACState) - // ... -} -``` - -**Fixtures**: use `t.TempDir()` for file-system state; never rely on absolute paths or external services. - -**Build targets**: -- `make test` — unit tests only (`CGO_ENABLED=0`) -- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir) -- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars) - -## C Tests (controller / scheduler / lib) - -**Location**: `testsuite/` directory inside each C component. - -**Note**: `src/worker` has no `testsuite/` directory and no meson `test()` registrations today — it currently carries no C test coverage of its own. - -**Structure**: one `.c` test program per unit: - -``` -src/controller/testsuite/ - test.c ← test runner harness - test_ha_cluster.c ← tests for ha.c - test_persist_wal.c ← WAL-specific tests - test_e2e_checkpoint.c ← end-to-end checkpoint flow - test_p1_integration.c ← P1 HA integration tier -``` - -File naming convention: `test_<unit>.c`. E2E tests: `test_e2e_<topic>.c`. Integration tiers: `test_p1_*.c`, `test_p2_*.c`, `test_p3_hardening.c`. - -**Test-only symbols** are guarded to prevent production bloat: - -```c -#ifdef XCE_TESTING -extern void reset_ha_state(ha_state_t *s); -#endif -``` - -**Add a test case in `testsuite/` before modifying any C implementation.** - -## Verifying a Test Run Actually Passed - -**Meson's aggregate `Ok:N Fail:0` is not sufficient evidence.** Meson can report a suite `Fail:0` while Unity sub-groups inside it fail — meson only tracks the process exit code of each *meson test unit*, not each Unity `RUN_TEST`. Always cross-check the **standalone test binary's own stdout** (`controller_unit_test`, `xcelib_test`, etc., run directly) for its authoritative `:FAIL:` count / `N Tests M Failures` summary before treating a run as green. This was the root cause of a masked-failure bug in the Unity harness itself (nested `UNITY_BEGIN()`/`UNITY_END()` across 83 `controller/testsuite/*.c` files swallowed the real exit code — fixed 2026-07-17) and is now the standard verification step for every `/kiro-validate-impl` pass. - -## Coverage Targets - -- Go: 80% minimum on business logic (controllers, services, store) -- C: every new function in a daemon must have at least one test case in `testsuite/` -- Integration: all public REST endpoints must have at least one integration test case in `src/frontend/tests/regression/` - -## What Not to Mock - -- Mock **external dependencies only** (network calls, file I/O that crosses component boundaries) -- Do **not** mock the system under test -- In-memory stores (`store/`) are real — use them directly in unit tests -- CGO bridge (`backend/`) is covered by `make test-backend` — do not stub it in unit tests diff --git a/CHANGES.md b/CHANGES.md deleted file mode 100644 index 0062dff5..00000000 --- a/CHANGES.md +++ /dev/null @@ -1,374 +0,0 @@ -# XCE Change Log - -Auto-generated by /xce-changes — component-grouped, newest week first. -`.wolf/` bookkeeping commits are excluded. - -## Aug 24, 2026 - Aug 29, 2026 -<!-- window: 2026-08-24..2026-08-29 --> - -### controller -- `67009815` fix(bug-1731): fire HOOK_DETAIL_JOB_FAIL on WAL_JOB_FAIL apply (2026-08-29) -- `ebe691c3` fix(bug-1731): fire HOOK_DETAIL_JOB_START/JOB_EXIT on WAL apply (2026-08-29) -- `47532f0a` fix(ha): first-boot promotion evaluated eligibility before its own WAL recovery ran (bug-1696) (2026-08-27) -- `f886c809` fix(controller): standby dispatch-gate rejections now redirect to a known primary (bug-1700, bug-1701) (2026-08-27) -- `ac1e312e` fix(controller): job.start_time never set -- dispatch/WAL_JOB_START RUN-state collision (bug-1697) (2026-08-27) -- `7880ab42` chore(xce-test-rc-file): pass --norc from existing xce_test automation (2026-08-27) -- `19bc8192` test(xce-test-rc-file): task 3.2 end-to-end rc-file failure handling (2026-08-26) -- `d258aa3d` test(xce-test-rc-file): task 3.1 end-to-end rc-file discovery/loading/--norc (2026-08-26) -- `e863d699` feat(xce-test-rc-file): task 2.2 wire rc-file loading into xce_test startup (2026-08-26) -- `68fcd657` feat(xce-test-rc-file): task 2.1 add --norc flag to xce_test (2026-08-26) -- `77aba21f` feat(xce-test-rc-file): task 1.2 rc-file line-loading via pluggable runner (2026-08-26) -- `9de4db95` feat(xce-test-rc-file): task 1.1 rc-file path resolution with XDG precedence (2026-08-26) -- `ac46bfa6` fix(controller): restored-but-stale raft term permanently blocked re-election after restart (bug-1695) (2026-08-25) -- `b4eb9ec8` fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694) (2026-08-25) -- `3eba7c16` fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693) (2026-08-25) -- `d0e5f85a` fix(controller): WAL_JOB_PREASON loses job.pend_reasons on replay (bug-1692) (2026-08-25) -- `97760761` fix(controller): WAL_JOB_CKPT loses job.ckptinfo on replay (bug-1691) (2026-08-25) -- `f7b732a3` fix(controller): WAL_JOB_RUSAGE loses job.run_usage on replay (bug-1690) (2026-08-25) -- `c2690cbf` fix(controller): WAL_JOB_FINISH loses exit_info/finish_time on replay -- and exit_info was never set at all in production (bug-1689) (2026-08-25) -- `955efecd` fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687) (2026-08-25) -- `e5e6cd18` fix(controller): raft_cb_persist must deny vote and roll back state on persist failure (bug-1684) (2026-08-25) -- `8d6f610e` test(controller): cover live dispatch's apiserver notification (bug-1679) (2026-08-25) -- `da7813bf` fix(controller): notify apiserver synchronously on live dispatch (bug-1679) (2026-08-25) -- `f8851353` test(controller): cover WAL_JOB_DISPATCH allocation persistence (bug-1678) (2026-08-25) -- `db142d41` fix(controller): persist dispatch-time resource allocation into WAL_JOB_DISPATCH (bug-1678, gitea #43) (2026-08-25) - -### worker -- `347e996e` feat(worker): distinguish signal-killed job exits from real crashes in logs (2026-08-29) - -### scheduler -- `14f95b06` fix(scheduler): re-derive worker/shared resource used-totals on snapshot load (2026-08-29) -- `ebe691c3` fix(bug-1731): fire HOOK_DETAIL_JOB_START/JOB_EXIT on WAL apply (2026-08-29) -- `b1323f8d` fix(bug-1730): free job accounts/allocs on JOB_UPDATE terminal transition (2026-08-29) -- `5948e079` docs(scheduler-snapshot-job-allocation): close final invariant-audit follow-up, feature complete (2026-08-28) -- `93d791c6` docs(scheduler-snapshot-job-allocation): correct snapshot.c scope-decision documentation (2026-08-28) -- `2654e03f` test(scheduler-snapshot-job-allocation): close originally-reported symptom end to end (2026-08-28) -- `de6ad514` test(scheduler-snapshot-job-allocation): regression test for allocator crash (2026-08-27) -- `f0fbebde` test(scheduler-snapshot-job-allocation): backward-compat for allocation-less snapshots (2026-08-27) -- `7481cedb` test(scheduler-snapshot-job-allocation): full cross-instance round-trip proof (2026-08-27) -- `182cb555` feat(scheduler-snapshot-job-allocation): restore allocs/accounts on load (2026-08-27) -- `6fa2d970` feat(scheduler-snapshot-job-allocation): populate allocs on snapshot save (2026-08-27) -- `22353849` feat(scheduler-snapshot-job-allocation): construct schedobj for restored jobs (2026-08-27) -- `0ebf3a81` fix(scheduler): repeat WORKERS_PUSH with a NULL resval clobbers good values (bug-1688) (2026-08-27) -- `ab077916` test(scheduler-sim-eager-reconciliation): task 3.2 sim-verify regression (2026-08-27) -- `c971f7dc` test(scheduler-sim-eager-reconciliation): task 3.1 resource-contention proof (2026-08-27) -- `c67382e6` test(scheduler-sim-eager-reconciliation): task 1.2 capture-safety unit coverage (2026-08-27) -- `0893fd50` test(scheduler-sim-eager-reconciliation): task 2.1 resInUse deferral proof (2026-08-27) -- `e2bb98c8` feat(scheduler-sim-eager-reconciliation): task 1.1 state-independent dispatch capture (2026-08-27) -- `bc3f76b4` fix(scheduler): invalidate stale builtin_fcfs picker cursors across calls (bug-1686) (2026-08-25) -- `b040dfc0` fix(scheduler): remove double-free in limit plugin config update (bug-1685) (2026-08-25) -- `6400f96d` fix(scheduler): mirror plugin_begin_session/plugin_end_session into sim-mode driver (bug-1681) (2026-08-25) - -### frontend -- `ebe691c3` fix(bug-1731): fire HOOK_DETAIL_JOB_START/JOB_EXIT on WAL apply (2026-08-29) -- `9a230f08` fix(apiserver): FilterRows must not route non-bitmask operators through bitmask matching (bug-1680) (2026-08-25) -- `1a61b04a` fix(apiserver): thread from_host through the CGO conversion layer (bug-1676, gitea #44) (2026-08-24) -- `fe8818b1` fix(apiserver): pass client-asserted FromHost through unchanged (bug-1676, gitea #44) (2026-08-24) -- `4fad7884` fix(xcecli): assert own hostname as FROM_HOST at submission time (bug-1676, gitea #44) (2026-08-24) -- `fe87dafc` fix(apiserver): fall back to default HostConfig for unmatched workers (bug-1673, gitea #40) (2026-08-24) -- `88c7c05d` fix(apiserver): jobs list -l/-o validates against the wrong struct (bug-1674, gitea #41) (2026-08-24) -- `a58c2a19` fix(xcecli): friendly message for connection-refused instead of raw OS error (bug-1675, gitea #42) (2026-08-24) - -### xceadmin -- `6193fa7e` test(admin-config-tool): update standby-redirect E2E for bug-1700/1701 (2026-08-29) -- `cfd7db8c` fix(xceadmin): retry transient scheduler-admin failures after a redirect (bug-1713, bug-1715) (2026-08-27) -- `3891bd90` test(xceadmin): add env-var override for live E2E binary discovery (2026-08-27) - -### lib -- `42e1eeaa` fix(bug-1733): fix heap-use-after-free race in vtask_enq() (2026-08-29) -- `097e33ae` feat(scheduler-snapshot-job-allocation): add dup_job_alloc deep-copy primitive (2026-08-27) -- `00207503` fix(lib): heap buffer overflow in make_job_account() for multi-host jobs (bug-1722) (2026-08-27) -- `363709b0` fix(lib): thread from_host through the wire format and the job-submit interning object (bug-1676, gitea #44) (2026-08-24) - -### include -- `097e33ae` feat(scheduler-snapshot-job-allocation): add dup_job_alloc deep-copy primitive (2026-08-27) -- `ac1e312e` fix(controller): job.start_time never set -- dispatch/WAL_JOB_START RUN-state collision (bug-1697) (2026-08-27) -- `b4eb9ec8` fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694) (2026-08-25) -- `d0e5f85a` fix(controller): WAL_JOB_PREASON loses job.pend_reasons on replay (bug-1692) (2026-08-25) -- `97760761` fix(controller): WAL_JOB_CKPT loses job.ckptinfo on replay (bug-1691) (2026-08-25) -- `f7b732a3` fix(controller): WAL_JOB_RUSAGE loses job.run_usage on replay (bug-1690) (2026-08-25) -- `c2690cbf` fix(controller): WAL_JOB_FINISH loses exit_info/finish_time on replay -- and exit_info was never set at all in production (bug-1689) (2026-08-25) -- `955efecd` fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687) (2026-08-25) -- `da7813bf` fix(controller): notify apiserver synchronously on live dispatch (bug-1679) (2026-08-25) -- `db142d41` fix(controller): persist dispatch-time resource allocation into WAL_JOB_DISPATCH (bug-1678, gitea #43) (2026-08-25) -- `363709b0` fix(lib): thread from_host through the wire format and the job-submit interning object (bug-1676, gitea #44) (2026-08-24) - -### schema -- `f7b732a3` fix(controller): WAL_JOB_RUSAGE loses job.run_usage on replay (bug-1690) (2026-08-25) -- `363709b0` fix(lib): thread from_host through the wire format and the job-submit interning object (bug-1676, gitea #44) (2026-08-24) - -### specs -- `6193fa7e` test(admin-config-tool): update standby-redirect E2E for bug-1700/1701 (2026-08-29) -- `42fdeb66` docs: save scheduler-config-version-tag spec (requirements/design/tasks) (2026-08-29) -- `b64682dc` docs: save scheduler-snapshot-job-allocation and xce-test-cli-ux spec docs, sync OpenWolf state (2026-08-28) -- `5948e079` docs(scheduler-snapshot-job-allocation): close final invariant-audit follow-up, feature complete (2026-08-28) -- `93d791c6` docs(scheduler-snapshot-job-allocation): correct snapshot.c scope-decision documentation (2026-08-28) -- `9988e714` test(scheduler-snapshot-job-allocation): confirm no regression to reconciliation or snapshot format (2026-08-28) -- `2654e03f` test(scheduler-snapshot-job-allocation): close originally-reported symptom end to end (2026-08-28) -- `de6ad514` test(scheduler-snapshot-job-allocation): regression test for allocator crash (2026-08-27) -- `f0fbebde` test(scheduler-snapshot-job-allocation): backward-compat for allocation-less snapshots (2026-08-27) -- `7481cedb` test(scheduler-snapshot-job-allocation): full cross-instance round-trip proof (2026-08-27) -- `182cb555` feat(scheduler-snapshot-job-allocation): restore allocs/accounts on load (2026-08-27) -- `6fa2d970` feat(scheduler-snapshot-job-allocation): populate allocs on snapshot save (2026-08-27) -- `22353849` feat(scheduler-snapshot-job-allocation): construct schedobj for restored jobs (2026-08-27) -- `097e33ae` feat(scheduler-snapshot-job-allocation): add dup_job_alloc deep-copy primitive (2026-08-27) -- `d32f06e5` docs: formalize save/restore field-completeness as a standing invariant (2026-08-27) -- `5e68fcc0` docs(spec): promote eliminate-frontend-makefile backlog item to a brief (2026-08-27) -- `13246a89` docs: OpenWolf pointer cleanup + redirect-hint checklist item (bug-1700 lesson) (2026-08-27) -- `446437bf` test(scheduler-sim-eager-reconciliation): task 3.3 full regression confirmation (2026-08-27) -- `ab077916` test(scheduler-sim-eager-reconciliation): task 3.2 sim-verify regression (2026-08-27) -- `c971f7dc` test(scheduler-sim-eager-reconciliation): task 3.1 resource-contention proof (2026-08-27) -- `c67382e6` test(scheduler-sim-eager-reconciliation): task 1.2 capture-safety unit coverage (2026-08-27) -- `0893fd50` test(scheduler-sim-eager-reconciliation): task 2.1 resInUse deferral proof (2026-08-27) -- `e2bb98c8` feat(scheduler-sim-eager-reconciliation): task 1.1 state-independent dispatch capture (2026-08-27) -- `8fc86ef4` test(xce-test-rc-file): task 3.3 confirm existing automation unaffected (2026-08-26) -- `19bc8192` test(xce-test-rc-file): task 3.2 end-to-end rc-file failure handling (2026-08-26) -- `d258aa3d` test(xce-test-rc-file): task 3.1 end-to-end rc-file discovery/loading/--norc (2026-08-26) -- `e863d699` feat(xce-test-rc-file): task 2.2 wire rc-file loading into xce_test startup (2026-08-26) -- `68fcd657` feat(xce-test-rc-file): task 2.1 add --norc flag to xce_test (2026-08-26) -- `77aba21f` feat(xce-test-rc-file): task 1.2 rc-file line-loading via pluggable runner (2026-08-26) -- `9de4db95` feat(xce-test-rc-file): task 1.1 rc-file path resolution with XDG precedence (2026-08-26) -- `71a814f3` docs(steering): add field-complete apply-parity invariant; extend xce-audit-invariants (2026-08-25) -- `5c82ad0b` chore(specs): reconcile stale phase/checkbox metadata across 4 specs (2026-08-24) -- `0c736c79` chore(specs): add scheduler-sim-eager-reconciliation spec, weekly report, cross-spec sequencing notes (2026-08-24) - -### build -- `83cb660b` fix(bug-1732): build xcecd_virtual in xce-build.sh's worker target (2026-08-29) -- `faedc6fc` fix(deploy-ha): guard against double-launching a live HA deploy (2026-08-27) - -### misc -- `f0a3729f` ignore daemon.log token-ledger (2026-08-28) -- `1b473736` docs: general design-time cross-check against project invariant dimensions (2026-08-27) -- `d32f06e5` docs: formalize save/restore field-completeness as a standing invariant (2026-08-27) -- `13246a89` docs: OpenWolf pointer cleanup + redirect-hint checklist item (bug-1700 lesson) (2026-08-27) -- `7880ab42` chore(xce-test-rc-file): pass --norc from existing xce_test automation (2026-08-27) -- `5441d1b8` docs(steering): add Decision Log fidelity as controller audit dimension 8 (2026-08-25) -- `71a814f3` docs(steering): add field-complete apply-parity invariant; extend xce-audit-invariants (2026-08-25) -- `643741fa` fix(gitea-sync): unwrap buglog.json's {version,bugs} wrapper before iterating (bug-1677) (2026-08-24) -- `8fd232c1` chore: gitignore generated capnp-c codegen output (2026-08-24) -- `f2b5f9c9` chore: untrack .wolf/dashboard-token, gitignore it going forward (2026-08-24) -- `0c736c79` chore(specs): add scheduler-sim-eager-reconciliation spec, weekly report, cross-spec sequencing notes (2026-08-24) - -## Aug 17, 2026 - Aug 23, 2026 -<!-- window: 2026-08-17..2026-08-23 --> - -### controller -- `a9cfee5f` test(scheduler-dispatch-commit): fix last stale pinned assertion, full regression clean (task 5.4, final) (2026-08-23) -- `bea95699` test(scheduler-dispatch-commit): terminal dispatch failure e2e coverage (task 5.3) (2026-08-23) -- `16d90319` test(scheduler-dispatch-commit): retryable and bounded-retry dispatch failure e2e coverage (task 5.2) (2026-08-23) -- `7b3dfed5` feat(scheduler-dispatch-commit): handle_jobdispatch no longer duplicates dispatch-outcome reporting (task 4.1) (2026-08-23) -- `05c45ef3` feat(scheduler-dispatch-commit): controller resolves retryable dispatch failures with bounded escalation (task 3.3) (2026-08-23) -- `8a4266ad` feat(scheduler-dispatch-commit): controller resolves terminal dispatch failures to EXIT (task 3.2) (2026-08-23) -- `d9aedc4b` feat(scheduler-dispatch-commit): controller commits alloc+RUN transition together on successful dispatch (task 3.1) (2026-08-23) -- `19bbac6a` feat(scheduler-dispatch-commit): add dispatch-fail retry counter to job_t (task 1.3) (2026-08-23) -- `8eea7490` feat(scheduler-dispatch-commit): add durable WAL_JOB_DISPATCH fact type (task 1.2) (2026-08-23) -- `5724d1e9` feat(scheduler-dispatch-commit): add controller dispatch-failure classification (task 1.1) (2026-08-23) -- `1c906e95` fix(worker): job ids print human-readable format via fmt_jobid() (bug-1588) (2026-08-22) -- `fd62e26b` feat(dirty-content-fingerprint): controller/scheduler show fingerprint in --version (2026-08-21) -- `88649a04` feat(build-time-versioning): controller/scheduler --version and startup log (2026-08-20) -- `5147396d` fix(controller): job start_time/finish_time/run_time never set by apply-pump (2026-08-20) -- `e52c68f4` fix(controller): admission-gate OPCode_schedAdmin's blocking wait (Req 5 AC 10) (2026-08-19) -- `515f4ca0` fix(controller): job.c TRACE lines print human-readable jobid (bug-1588) (2026-08-19) -- `b406dd76` fix(controller): sched_worker_change() use-after-free on worker disconnect (bug-1578) (2026-08-18) -- `f5390c14` feat(resource-value-verification): per-name accept/reject verification in resupdate (task 2.2) (2026-08-18) -- `058335d2` feat(resource-value-verification): wire resolver into resupdate, add -t flag (task 2.1) (2026-08-17) -- `98b8c34b` feat(resource-value-verification): extract resupdate type-resolution logic (task 1.1) (2026-08-17) -- `aabddf60` fix(scheduler,controller): worker resource values always display as 0.00 (bug-1568) (2026-08-17) -- `986a3304` fix(controller): persist_flush now fires durable-advance callback (bug-1564) (2026-08-17) -- `846713ab` fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563) (2026-08-17) - -### worker -- `1c906e95` fix(worker): job ids print human-readable format via fmt_jobid() (bug-1588) (2026-08-22) -- `d3d9522f` feat(dirty-content-fingerprint): worker shows fingerprint in --version (2026-08-21) -- `bc5c669b` feat(build-time-versioning): worker --version and startup log (2026-08-20) -- `70fe9e46` fix(worker): dbus_workers_update() starves workers after first DONE/EXITED (bug-1589) (2026-08-19) -- `6e5c3a89` fix(worker): bug-1569 periodic resval reports tagged with wrong host (2026-08-17) - -### scheduler -- `01c83066` test(scheduler-dispatch-commit): bug-1624 multi-round regression coverage (task 5.1) (2026-08-23) -- `281a026d` feat(scheduler-dispatch-commit): scheduler processes dispatch-fail notify, returns job to PEND (task 2.2) (2026-08-23) -- `8607348c` feat(scheduler-dispatch-commit): scheduler transitions job to RUN at dispatch commit (task 2.1) (2026-08-23) -- `3c4091d8` test(scheduler-sim-decision-parity): live-verify against a real 24-round recording (task 4.3, final) (2026-08-22) -- `88bed529` test(scheduler-sim-decision-parity): regression suite for existing guarantees (task 4.2) (2026-08-22) -- `b3190732` test(scheduler-sim-decision-parity): prove capacity-aware parity under contention (task 4.1) (2026-08-22) -- `12644094` feat(scheduler-sim-decision-parity): wire reconciliation into the round boundary (task 3) (2026-08-22) -- `0c237f60` feat(scheduler-sim-decision-parity): report reconciled decisions through the existing channel (task 2.4) (2026-08-22) -- `eccba841` feat(scheduler-sim-decision-parity): reconstruct job allocation with safe ownership (task 2.3) (2026-08-22) -- `81358b52` feat(scheduler-sim-decision-parity): add reconciliation trigger and lookup (task 2.2) (2026-08-22) -- `134380b8` fix(scheduler-sim-decision-parity): give sim mode driver.c's round-boundary split (task 2.1) (2026-08-22) -- `f11f0df3` test(scheduler-sim-decision-parity): add resource-contention replay fixture (task 1.2) (2026-08-22) -- `3106c82e` feat(scheduler-sim-decision-parity): relocate resource accounting to core-level (task 1.1) (2026-08-22) -- `25e35bac` docs(scheduler): document driver-equivalence invariant, scope decision-parity fix spec (2026-08-22) -- `fd62e26b` feat(dirty-content-fingerprint): controller/scheduler show fingerprint in --version (2026-08-21) -- `88649a04` feat(build-time-versioning): controller/scheduler --version and startup log (2026-08-20) -- `8abab049` feat(scheduler): show event time/session in sim echo, total sessions on replay load (2026-08-20) -- `94733616` fix(scheduler): gate resource value/used display on XCE_RESF_CONSUMABLE (2026-08-20) -- `185feca2` fix(scheduler): "info resource all" printed nothing when no global/shared resources exist (2026-08-20) -- `8cb304d2` fix(scheduler): sim_print_decision() bypassed tlog, producing inconsistent PEND/DISPATCH/PREEMPT output (2026-08-20) -- `1bc65d2c` docs(resource-value-verification): close design-review doc/comment gaps (2026-08-19) -- `a13068dc` fix(scheduler): extend core-isolation nm-audit to core_events.c/core_session.c/core_clock.c (Req 5 AC 1/2) (2026-08-19) -- `b450a680` fix(scheduler): activate virtual-clock logging for sim/REPL sessions (Req 5 AC 3) (2026-08-19) -- `c9acd502` fix(scheduler): clock/resval/jobid-format fixes plus REPL live-operator gaps (task 12.5) (2026-08-19) -- `e68444f1` feat(resource-value-verification): add scheduler resval trace log (task 4.1) (2026-08-18) -- `7041d988` fix(scheduler): bug-1569 three stacked crashes blocking ALL job dispatch (2026-08-17) -- `6e5c3a89` fix(worker): bug-1569 periodic resval reports tagged with wrong host (2026-08-17) -- `aabddf60` fix(scheduler,controller): worker resource values always display as 0.00 (bug-1568) (2026-08-17) -- `0c8e417b` fix(scheduler): sim mode picker never learns about the initial snapshot's own jobs/queues (bug-1567) (2026-08-17) - -### frontend -- `d65d18ad` feat(dirty-content-fingerprint): xcecli shows fingerprint in --version (2026-08-21) -- `a4c1b5ad` feat(dirty-content-fingerprint): apiserver shows fingerprint in -version (2026-08-21) -- `aa3391a9` feat(dirty-content-fingerprint): Makefile computes dev-loop fingerprint (2026-08-21) -- `502dd023` feat(dirty-content-fingerprint): apiserver/xcecli meson pass fingerprint to go_build (2026-08-21) -- `fb09afe6` feat(build-time-versioning): xcecli --version via Cobra rootCmd.Version (2026-08-20) -- `17e837b1` feat(build-time-versioning): apiserver -version flag and startup log (2026-08-20) -- `45c9b026` feat(build-time-versioning): Makefile ldflags for make build-server/build-cli (2026-08-20) -- `071f3895` feat(build-time-versioning): thread commit_id/build_time into go_build env (2026-08-20) -- `1bc65d2c` docs(resource-value-verification): close design-review doc/comment gaps (2026-08-19) -- `8b4fbffa` feat(resource-value-verification): add apiserver resval trace log (task 4.2) (2026-08-18) - -### xceadmin -- `c79ccf0f` feat(dirty-content-fingerprint): xceadmin shows fingerprint in --version (2026-08-21) -- `2bf62a64` feat(dirty-content-fingerprint): xceadmin build-info module gains fingerprint (2026-08-21) -- `7f5c87ce` feat(build-time-versioning): xceadmin committed fallback stub and CLI version wiring (2026-08-20) -- `e1cd9d25` feat(build-time-versioning): xceadmin generates build-info module (2026-08-20) -- `0f1df499` feat(deploy,xceadmin): standalone xceadmin packaging + deploy-ha usability (bug-1566) (2026-08-17) - -### lib -- `7ebcba6d` feat(dirty-content-fingerprint): add --version-only composition with dirty_fingerprint (2026-08-21) -- `48a86cda` feat(build-time-versioning): version macro/helper functions with unknown fallback (2026-08-20) -- `ddee884c` fix(libxce): bug-1600 -- makeGroup() freed rtti makeString() pointer via raw free() (2026-08-20) - -### include -- `19bbac6a` feat(scheduler-dispatch-commit): add dispatch-fail retry counter to job_t (task 1.3) (2026-08-23) -- `8eea7490` feat(scheduler-dispatch-commit): add durable WAL_JOB_DISPATCH fact type (task 1.2) (2026-08-23) -- `7ebcba6d` feat(dirty-content-fingerprint): add --version-only composition with dirty_fingerprint (2026-08-21) -- `48a86cda` feat(build-time-versioning): version macro/helper functions with unknown fallback (2026-08-20) -- `846713ab` fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563) (2026-08-17) - -### specs -- `f784e2bd` chore(scheduler-dispatch-commit): mark spec implementation-complete (2026-08-23) -- `a9cfee5f` test(scheduler-dispatch-commit): fix last stale pinned assertion, full regression clean (task 5.4, final) (2026-08-23) -- `bea95699` test(scheduler-dispatch-commit): terminal dispatch failure e2e coverage (task 5.3) (2026-08-23) -- `16d90319` test(scheduler-dispatch-commit): retryable and bounded-retry dispatch failure e2e coverage (task 5.2) (2026-08-23) -- `01c83066` test(scheduler-dispatch-commit): bug-1624 multi-round regression coverage (task 5.1) (2026-08-23) -- `7b3dfed5` feat(scheduler-dispatch-commit): handle_jobdispatch no longer duplicates dispatch-outcome reporting (task 4.1) (2026-08-23) -- `05c45ef3` feat(scheduler-dispatch-commit): controller resolves retryable dispatch failures with bounded escalation (task 3.3) (2026-08-23) -- `8a4266ad` feat(scheduler-dispatch-commit): controller resolves terminal dispatch failures to EXIT (task 3.2) (2026-08-23) -- `d9aedc4b` feat(scheduler-dispatch-commit): controller commits alloc+RUN transition together on successful dispatch (task 3.1) (2026-08-23) -- `281a026d` feat(scheduler-dispatch-commit): scheduler processes dispatch-fail notify, returns job to PEND (task 2.2) (2026-08-23) -- `8607348c` feat(scheduler-dispatch-commit): scheduler transitions job to RUN at dispatch commit (task 2.1) (2026-08-23) -- `19bbac6a` feat(scheduler-dispatch-commit): add dispatch-fail retry counter to job_t (task 1.3) (2026-08-23) -- `8eea7490` feat(scheduler-dispatch-commit): add durable WAL_JOB_DISPATCH fact type (task 1.2) (2026-08-23) -- `5724d1e9` feat(scheduler-dispatch-commit): add controller dispatch-failure classification (task 1.1) (2026-08-23) -- `ca01509f` chore(scheduler-sim-decision-parity): mark feature complete (2026-08-22) -- `3c4091d8` test(scheduler-sim-decision-parity): live-verify against a real 24-round recording (task 4.3, final) (2026-08-22) -- `88bed529` test(scheduler-sim-decision-parity): regression suite for existing guarantees (task 4.2) (2026-08-22) -- `b3190732` test(scheduler-sim-decision-parity): prove capacity-aware parity under contention (task 4.1) (2026-08-22) -- `12644094` feat(scheduler-sim-decision-parity): wire reconciliation into the round boundary (task 3) (2026-08-22) -- `0c237f60` feat(scheduler-sim-decision-parity): report reconciled decisions through the existing channel (task 2.4) (2026-08-22) -- `eccba841` feat(scheduler-sim-decision-parity): reconstruct job allocation with safe ownership (task 2.3) (2026-08-22) -- `81358b52` feat(scheduler-sim-decision-parity): add reconciliation trigger and lookup (task 2.2) (2026-08-22) -- `134380b8` fix(scheduler-sim-decision-parity): give sim mode driver.c's round-boundary split (task 2.1) (2026-08-22) -- `f11f0df3` test(scheduler-sim-decision-parity): add resource-contention replay fixture (task 1.2) (2026-08-22) -- `3106c82e` feat(scheduler-sim-decision-parity): relocate resource accounting to core-level (task 1.1) (2026-08-22) -- `0b2c856d` docs(scheduler-sim-decision-parity): finalize design and generate tasks (2026-08-22) -- `25e35bac` docs(scheduler): document driver-equivalence invariant, scope decision-parity fix spec (2026-08-22) -- `94fd1504` chore: sync harness/session state and OpenWolf bookkeeping (2026-08-22) -- `979cd24f` docs(scheduler-sim-repl-verbosity): add approved spec (requirements/design/tasks) (2026-08-22) -- `aaca3768` docs(dirty-content-fingerprint): mark task 2 container checkbox complete (2026-08-21) -- `f4d2c123` test(dirty-content-fingerprint): add Nix production path fingerprint check (2026-08-21) -- `a4785418` test(dirty-content-fingerprint): add startup-log-unchanged consistency check (2026-08-21) -- `a7d64c82` test(dirty-content-fingerprint): add dev-loop distinguishability regression (2026-08-21) -- `c79ccf0f` feat(dirty-content-fingerprint): xceadmin shows fingerprint in --version (2026-08-21) -- `2bf62a64` feat(dirty-content-fingerprint): xceadmin build-info module gains fingerprint (2026-08-21) -- `d65d18ad` feat(dirty-content-fingerprint): xcecli shows fingerprint in --version (2026-08-21) -- `a4c1b5ad` feat(dirty-content-fingerprint): apiserver shows fingerprint in -version (2026-08-21) -- `aa3391a9` feat(dirty-content-fingerprint): Makefile computes dev-loop fingerprint (2026-08-21) -- `502dd023` feat(dirty-content-fingerprint): apiserver/xcecli meson pass fingerprint to go_build (2026-08-21) -- `b9499493` feat(dirty-content-fingerprint): go_build injects fingerprint via ldflags (2026-08-21) -- `d3d9522f` feat(dirty-content-fingerprint): worker shows fingerprint in --version (2026-08-21) -- `fd62e26b` feat(dirty-content-fingerprint): controller/scheduler show fingerprint in --version (2026-08-21) -- `7ebcba6d` feat(dirty-content-fingerprint): add --version-only composition with dirty_fingerprint (2026-08-21) -- `fc5cba15` feat(dirty-content-fingerprint): Nix flake computes fingerprint via self.narHash (2026-08-21) -- `c9d40ba1` feat(dirty-content-fingerprint): xce-build.sh computes dev-loop fingerprint (2026-08-21) -- `a860141b` feat(dirty-content-fingerprint): add dirty_fingerprint meson option and C define (2026-08-21) -- `d3c583b3` docs(dirty-content-fingerprint): add spec (requirements, design, tasks) (2026-08-21) -- `084a5a95` test(build-time-versioning): add xceadmin end-to-end build/install/version cycle check (2026-08-20) -- `79941a9f` test(build-time-versioning): add version/startup-log consistency check (2026-08-20) -- `0eefc95f` test(build-time-versioning): add motivating-incident regression script (2026-08-20) -- `7f5c87ce` feat(build-time-versioning): xceadmin committed fallback stub and CLI version wiring (2026-08-20) -- `e1cd9d25` feat(build-time-versioning): xceadmin generates build-info module (2026-08-20) -- `fb09afe6` feat(build-time-versioning): xcecli --version via Cobra rootCmd.Version (2026-08-20) -- `17e837b1` feat(build-time-versioning): apiserver -version flag and startup log (2026-08-20) -- `45c9b026` feat(build-time-versioning): Makefile ldflags for make build-server/build-cli (2026-08-20) -- `071f3895` feat(build-time-versioning): thread commit_id/build_time into go_build env (2026-08-20) -- `18ae0cc3` feat(build-time-versioning): go_build injects commit/build-time via ldflags (2026-08-20) -- `bc5c669b` feat(build-time-versioning): worker --version and startup log (2026-08-20) -- `88649a04` feat(build-time-versioning): controller/scheduler --version and startup log (2026-08-20) -- `48a86cda` feat(build-time-versioning): version macro/helper functions with unknown fallback (2026-08-20) -- `d08fda76` feat(build-time-versioning): nix packages compute build time at build phase (2026-08-20) -- `bca63974` feat(build-time-versioning): xce-build.sh computes real commit id/build time (2026-08-20) -- `1ff9e66c` docs(build-time-versioning): add spec (requirements, design, research) (2026-08-20) -- `4b7cc786` feat(build-time-versioning): add build_time meson option and C-wide define (2026-08-20) -- `e61fa2b3` docs(roadmap): add resource-distribution-mapping (existed since 2026-07-23, never tracked here) (2026-08-20) -- `a16d32e3` docs(roadmap): close out four low-risk backlog cleanup items (2026-08-19) -- `0519d24d` docs(scheduler-core): reconcile stale doc references after today's fixes (2026-08-19) -- `aa42752d` docs(scheduler-core): mark task 12.5 complete with fresh verification evidence (2026-08-19) -- `1bc65d2c` docs(resource-value-verification): close design-review doc/comment gaps (2026-08-19) -- `e52c68f4` fix(controller): admission-gate OPCode_schedAdmin's blocking wait (Req 5 AC 10) (2026-08-19) -- `c9acd502` fix(scheduler): clock/resval/jobid-format fixes plus REPL live-operator gaps (task 12.5) (2026-08-19) -- `70fe9e46` fix(worker): dbus_workers_update() starves workers after first DONE/EXITED (bug-1589) (2026-08-19) -- `ef404131` docs(resource-value-verification): commit spec files, mark implementation complete (2026-08-18) -- `9be9e954` test(resource-value-verification): verdict-logic regression selftest (task 8.2) (2026-08-18) -- `19104d31` test(resource-value-verification): full live e2e validation run (task 8.1) (2026-08-18) -- `13646647` feat(resource-value-verification): verdict aggregation, reporting, cleanup (task 7.1) (2026-08-18) -- `8222c031` feat(resource-value-verification): apiserver-side verification check (task 6.4) (2026-08-18) -- `b673b7c9` fix(resource-value-verification): close bug-1579 for real via xce_test's ! shell-escape (2026-08-18) -- `72b6fdea` feat(resource-value-verification): scheduler-side verification check (task 6.3) (2026-08-18) -- `cb5bbfcc` feat(resource-value-verification): controller-side verification check (task 6.2) (2026-08-18) -- `75a0839d` feat(resource-value-verification): worker join and resource injection (task 6.1) (2026-08-18) -- `da41cce2` feat(resource-value-verification): wire deployment into skill via start-all.sh (task 5.1) (2026-08-18) -- `8b4fbffa` feat(resource-value-verification): add apiserver resval trace log (task 4.2) (2026-08-18) -- `e68444f1` feat(resource-value-verification): add scheduler resval trace log (task 4.1) (2026-08-18) -- `8f542a38` feat(resource-value-verification): extend xce-deploy-ha with apiserver step + start-all.sh (task 3.1) (2026-08-18) -- `f5390c14` feat(resource-value-verification): per-name accept/reject verification in resupdate (task 2.2) (2026-08-18) -- `058335d2` feat(resource-value-verification): wire resolver into resupdate, add -t flag (task 2.1) (2026-08-17) -- `642e2362` feat(resource-value-verification): scaffold xce-test-resources skill (task 1.2) (2026-08-17) -- `98b8c34b` feat(resource-value-verification): extract resupdate type-resolution logic (task 1.1) (2026-08-17) -- `9f539583` docs: sync skills and steering docs with bug-1562..1566 findings (2026-08-17) - -### build -- `f4d2c123` test(dirty-content-fingerprint): add Nix production path fingerprint check (2026-08-21) -- `a4785418` test(dirty-content-fingerprint): add startup-log-unchanged consistency check (2026-08-21) -- `a7d64c82` test(dirty-content-fingerprint): add dev-loop distinguishability regression (2026-08-21) -- `b9499493` feat(dirty-content-fingerprint): go_build injects fingerprint via ldflags (2026-08-21) -- `fc5cba15` feat(dirty-content-fingerprint): Nix flake computes fingerprint via self.narHash (2026-08-21) -- `c9d40ba1` feat(dirty-content-fingerprint): xce-build.sh computes dev-loop fingerprint (2026-08-21) -- `a860141b` feat(dirty-content-fingerprint): add dirty_fingerprint meson option and C define (2026-08-21) -- `084a5a95` test(build-time-versioning): add xceadmin end-to-end build/install/version cycle check (2026-08-20) -- `79941a9f` test(build-time-versioning): add version/startup-log consistency check (2026-08-20) -- `0eefc95f` test(build-time-versioning): add motivating-incident regression script (2026-08-20) -- `18ae0cc3` feat(build-time-versioning): go_build injects commit/build-time via ldflags (2026-08-20) -- `d08fda76` feat(build-time-versioning): nix packages compute build time at build phase (2026-08-20) -- `bca63974` feat(build-time-versioning): xce-build.sh computes real commit id/build time (2026-08-20) -- `4b7cc786` feat(build-time-versioning): add build_time meson option and C-wide define (2026-08-20) -- `83e61eec` feat(xce-deploy-ha): support real xcecd worker deployment alongside virtual (2026-08-19) -- `8f542a38` feat(resource-value-verification): extend xce-deploy-ha with apiserver step + start-all.sh (task 3.1) (2026-08-18) -- `0f1df499` feat(deploy,xceadmin): standalone xceadmin packaging + deploy-ha usability (bug-1566) (2026-08-17) -- `1c943e98` feat(build): add --incremental flag to xce-build.sh (bug-1565) (2026-08-17) - -### misc -- `94fd1504` chore: sync harness/session state and OpenWolf bookkeeping (2026-08-22) -- `83e61eec` feat(xce-deploy-ha): support real xcecd worker deployment alongside virtual (2026-08-19) -- `9be9e954` test(resource-value-verification): verdict-logic regression selftest (task 8.2) (2026-08-18) -- `13646647` feat(resource-value-verification): verdict aggregation, reporting, cleanup (task 7.1) (2026-08-18) -- `8222c031` feat(resource-value-verification): apiserver-side verification check (task 6.4) (2026-08-18) -- `b673b7c9` fix(resource-value-verification): close bug-1579 for real via xce_test's ! shell-escape (2026-08-18) -- `72b6fdea` feat(resource-value-verification): scheduler-side verification check (task 6.3) (2026-08-18) -- `cb5bbfcc` feat(resource-value-verification): controller-side verification check (task 6.2) (2026-08-18) -- `75a0839d` feat(resource-value-verification): worker join and resource injection (task 6.1) (2026-08-18) -- `da41cce2` feat(resource-value-verification): wire deployment into skill via start-all.sh (task 5.1) (2026-08-18) -- `8f542a38` feat(resource-value-verification): extend xce-deploy-ha with apiserver step + start-all.sh (task 3.1) (2026-08-18) -- `642e2362` feat(resource-value-verification): scaffold xce-test-resources skill (task 1.2) (2026-08-17) -- `9f539583` docs: sync skills and steering docs with bug-1562..1566 findings (2026-08-17) diff --git a/reports/aug03-aug09-2026.md b/reports/aug03-aug09-2026.md deleted file mode 100644 index 99229b2f..00000000 --- a/reports/aug03-aug09-2026.md +++ /dev/null @@ -1,86 +0,0 @@ -# XCE Project Status — Aug 03 – Aug 07, 2026 - -## TL;DR - -**scheduler-core: fully implemented.** All 13 task groups landed across the week — picker plugin contract, core extraction, normal-mode driver wiring, replay wire format, snapshot save/load, recording lifecycle, admin opcode dispatch, simulation mode, interactive REPL, and HA soak regression. One controller bug fixed (bug-1505: WalCatchupEntry SIGSEGV). Aug 08 (today) was documentation cleanup: deployment guide, CLAUDE.md sync, and a broad audit of 33 closed specs. The scheduler-framework feature branch is now implementation-complete. - -## Commits by day - -### Aug 03 (Sun) — Group 1 (picker contract) + Group 2 (builtin_fcfs) + Group 4 (core extraction) + Group 5 (driver wiring) - -10 commits, closing out Groups 1–5: - -- **Group 1 — Picker Plugin Contract** (tasks 1.1–1.3): defined `job_elect_fn` plugin contract, registered picker plugin category, added picker configuration fields -- **Group 2 — builtin_fcfs** (tasks 2.1–2.6): lifecycle, state-change hooks, session/post-allocation hooks, wired as registered default picker, scheduling-loop ordering guard regression tests, data-sharing invariant audit -- **Group 4 — Core Extraction** (tasks 4.1–4.6): extracted `xce_sched_core_t` from monolithic `scheduler_t`, pinned jbuf capacity + kvmgr determinism audit, event batch push interface (CONFIG_PUSH decomposition approved round 2), session-start signal + decision callback, clock virtualization + session timing, core contract test (full push-go-drain-end cycle) -- **Group 5 — Driver Wiring** (tasks 5.1–5.2): normal-mode event translation path, decision publishing + mode-selection wired into `main.c` - -### Aug 04 (Mon) — Group 6 (HA soak gate) + Group 7 (replay wire format) + Group 8 (snapshot) - -9 commits: - -- **Group 6** (task 6): Phase 2 HA soak regression gate closed -- **Group 7 — Replay Wire Format** (tasks 7.1–7.5): replay stream envelope schema, event payload types (JobStateChangeEvent), decision payload types + snapshot schema (SnapshotData, ReplayDecision), admin opcode schema, round-trip coverage gap (1087/1087 tests pass) -- **Group 8 — Snapshot** (tasks 8.1–8.2): snapshot save/load with cross-thread concurrency fix, `-EBUSY` guard against active session - -### Aug 05 (Tue) — Group 9 (recording lifecycle) + Group 10 (admin dispatch + xceadmin CLI) - -9 commits: - -- **Group 9 — Recording Lifecycle** (tasks 9.1–9.5): directory provisioning + path validation, record start/stop lifecycle (O_TRUNC→ftruncate hardlink fix, CWE-59 patched), periodic snapshot checkpointing, scheduler-side admin request handler, integration tests (26/26 pass) -- **Group 10 — Admin Dispatch + xceadmin CLI** (tasks 10.1–10.3): controller-side `OPCode_schedAdmin` dispatch (fmt-noise + leak fix), dispatch test suite (timeout/RPC-status/error paths), xceadmin recording client functions - -### Aug 06 (Wed) — Group 10 (cont.) + Group 11 (simulation mode) - -9 commits: - -- **Group 10 cont.** (tasks 10.4–10.6): xceadmin scheduler dump command, CLI wiring (record_start/stop/status), mock-server E2E tests -- **Group 11 — Simulation Mode** (tasks 11.1–11.3): simulation-mode startup, virtual clock advancement + event loop, simulation isolation enforcement - -### Aug 07 (Thu) — Group 11 (cont.) + Group 12 (REPL) + Group 13 (validation) - -12 commits: - -- **Group 11 cont.** (tasks 11.4–11.6): simulation decision verification, what-if branching + output recording, normal↔sim decision equivalence tests (5/5 pass, 33/33 regression) -- **Group 12 — REPL** (tasks 12.1–12.4): execution control (readline→libedit GPL→BSD), breakpoints + state inspection, snapshot capture + simulation parameter commands, step/continue equivalence to batch (37/37 tests pass) -- **fix(controller):** stop SIGSEGV in WalCatchupEntry capnp encode (**bug-1505**) -- **Group 13:** full cross-phase HA soak regression (15 rounds sequential pass) -- **Validation:** spec marked implementation-complete; feature-level validation findings recorded - -### Aug 08 (Fri, today) — Documentation Cleanup - -5 commits (documentation only, not implementation): - -- `docs:` add scheduler recording/simulation/REPL to deployment guide -- `docs(scheduler-core):` sync CLAUDE.md for core/driver/picker/sim/REPL architecture -- `docs(scheduler-core):` commit accumulated brief/requirements/roadmap edits -- `docs(scheduler-core):` backfill `_Verified` notes for tasks 2.1–2.6 -- `docs(scheduler-core):` commit design.md/research.md, sync design.md with as-built code - -Plus a broad documentation audit (not yet committed): 33 closed `.kiro/specs` design.md files audited for drift against production source, with confirmed corrections across 8+ HA/Raft spec design docs. A separate stream of work (S1275–S1278) covered 28 free-standing docs → 126 audit findings, and TLS encryption claims in deployment.md were verified against `libxce` source (server-side TLS confirmed, `do_connect()` is an uncalled stub). - -## Bugs opened/fixed this window - -| Bug | Status | Summary | -|---|---|---| -| bug-1505 | **Fixed** (2026-08-07, `1fd467d0`) | `WalCatchupEntry` capnp encode SIGSEGV — `Data` fields in capnp schema caused null-deref; fixed by migrating to `List(UInt8)` (same class as bug-1504 in SnapshotData) | - -## Spec status changes - -- **scheduler-core** — **implementation-complete** (2026-08-07). All 13 task groups (1.1–13) delivered; spec.json phase → `implemented`. 55 commits, 112 files changed, ~42K insertions. - - Architecture: picker plugin system (job_elect_fn contract), builtin_fcfs, extracted xce_sched_core_t, normal-mode driver, capnp replay wire format, snapshot persistence, recording lifecycle, xceadmin CLI, simulation mode (normal↔sim equivalence), interactive REPL (BSD libedit). - -## Stats - -- **55 commits** in 5 days -- **112 files changed**, 42,005 insertions, 803 deletions -- **1 bug fixed** (bug-1505) -- **13 task groups** completed (Groups 1–13) -- **1 spec** moved to implemented (scheduler-core) -- **1,087+ roundtrip tests**, **26 integration tests**, **37 REPL tests**, **15-round HA soak** all passing - -## Open threads heading into next session - -1. scheduler-core is implementation-complete — next step is either PR review + merge to master, or starting scheduler-allocation/scheduler-preemption design. -2. Documentation audit (33 specs) is in progress — design.md corrections applied to ~8 HA/Raft specs; remaining ~24 specs need verification. -3. Broad documentation audit (S1278) still running — 28 docs with 126 findings, Tier 3 workflow targeting 32 closed spec design.md files. diff --git a/reports/aug08-aug15-2026.md b/reports/aug08-aug15-2026.md deleted file mode 100644 index 14155afb..00000000 --- a/reports/aug08-aug15-2026.md +++ /dev/null @@ -1,111 +0,0 @@ -# XCE Project Status — Aug 08 – Aug 15, 2026 - -## TL;DR - -**component-failover-adoption: implementation-complete and GO-validated — but not yet committed.** The week opened with scheduler-core documentation cleanup (carried over from last week), then a worker CLI/config cleanup pass, then the bulk of the week (Aug 11–13) was the 21-task component-failover-adoption spec: unifying worker/scheduler/apiserver's dual connect paths into one candidate-list pipeline, adding rejoin-on-failover for all three components, and an apiserver backend-health circuit breaker. Live E2E validation during that work found two real production bugs (bug-1548/bug-1549: apiserver never rejoined a new primary during an idle/heartbeat-only failover, so job pushes silently stopped reaching it). `/kiro-validate-impl` caught this as an in-scope gap and returned **NO-GO** on Aug 15 at 13:10 UTC; a same-day Phase 6 remediation (a monotonic `reconnect_gen` counter) fixed it, live-reverified, and the decision flipped to **GO** at 14:20 UTC. That fix, plus a new `xceadmin` cluster-config dump feature and a batch of build-tooling fixes, are sitting validated-but-uncommitted in the working tree as of end of week — 57 modified/untracked files, no commits since Aug 13. - -## Commits by day - -### Aug 08 (Sat) — scheduler-core documentation cleanup (carried over from prior week) - -5 commits, docs-only: -- `docs:` add scheduler recording/simulation/REPL to deployment guide -- `docs(scheduler-core):` sync CLAUDE.md for core/driver/picker/sim/REPL architecture -- `docs(scheduler-core):` commit accumulated brief/requirements/roadmap edits -- `docs(scheduler-core):` backfill `_Verified` notes for tasks 2.1–2.6 -- `docs(scheduler-core):` commit design.md/research.md, sync design.md with as-built code - -### Aug 09 (Sun) — no commits - -### Aug 10 (Mon) — worker CLI/config cleanup + probeConfigTag fix - -11 commits: -- **Worker cleanup:** removed the config-file mechanism and deprecated CLI flags entirely (`5e4baa27`, `f0f46f87`, `b3124767`), fixed `xcecd -h` exiting before root/config checks (`9ffb1d51`, **bug-1509** class), default logdir to `/tmp` when `-l` omitted (`e959ee61`), added `-v` verbose flag (`0df1cf5f`) -- **fix(xceadmin)/feat(controller):** `OPCode_probeConfigTag` handler with Request-envelope reply, and `probe_config` updated to match (`90d7dd82`, `b00e97ff` — **bug-1506, bug-1507**) -- `feat(scheduler):` working default record-dir, improved REPL output, job ID formatting (`c705cada`) - -### Aug 11 (Tue) — component-failover-adoption: foundation + worker + scheduler connect unification - -11 commits — Task groups 1–3 begin: -- **Group 1 (Foundation):** reconciled `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` (`fe8b4b86`), added the candidate-list→client-config conversion helper (`210faa2b`) -- **Group 2 (Worker):** unified worker's dual connect paths into the single pipeline (`ce9f734a`), fixed startup ordering + added rejoin retry-on-failure (`331fa024`), validated thread isolation and fixed rejoin bugs found by live E2E (`619b119e`) -- **Group 3 (Scheduler):** unified scheduler's dual connect paths (`a9f283dc`), added decision-publish retry-until-success-or-shutdown (`7c260e0c`), validated backpressure + fixed candidate-name diagnostics (`b7df8d60`), added decision-redelivery queue coverage (`d383d043`) - -### Aug 12 (Wed) — component-failover-adoption: apiserver connect unification + rejoin + circuit breaker - -7 commits — Task group 4: -- Routed apiserver's candidate parsing through the real C parser (`8b1e13a5`) -- Unified apiserver's dual connect paths — found and fixed a genuine use-after-free on `ctx->conf->masters[i]` during failover rotation (`e6359039`, **bug-1532**) -- Added apiserver's rejoin logic with correctly-placed cgo registration (`2a205291`) -- Wired rejoin to trigger from the idle-period heartbeat path (`ecf5eddc`) -- Added the backend-health circuit breaker (`c8325e75`) -- Closed an apiserver concurrency coverage gap (`79273a7f`) -- Group 5 begins: full-cluster E2E reconnect-after-failover test passes (`6cb8f22b`) — **this run first surfaced bug-1548/bug-1549** (apiserver silently fails to rejoin on an idle/heartbeat-only failover) - -### Aug 13 (Thu) — component-failover-adoption closes; resource/config bugfixes; buglog integrity hardening - -15 commits (6 feature/fix, 9 wolf-buglog checkpoints): -- Task 5.2 doubly-confirms bug-1548 in an isolated heartbeat-only scenario (`04d41d29`); task 5.3 validates the circuit breaker live (`51589b68`); task 5.4 closes the spec on a full regression pass (`d170623b`) — bug-1548/1549 were (incorrectly, see below) treated as out-of-scope follow-up items at this point -- `chore(wolf):` buglog.json integrity guard, dedupe rule, auto-checkpoint hook (`ba4e5fd2`) -- `fix(controller,xceadmin):` zero-slot scheduling block, misleading resval logs, byte-unit display (`e3b3ffaa` — **bug-1555, bug-1556**) -- `refactor(controller,xceadmin):` dropped generic `Resource.unit`, hardcoded bytes for maxmem (`5bf123cc`) - -### Aug 14–15 (Fri–Sat) — validation reopened, Phase 6 remediation, tooling fixes — all uncommitted - -No commits landed, but substantial validated work sits in the working tree: -- `/kiro-validate-impl` reopened component-failover-adoption: **NO-GO** (Aug 15, 13:10 UTC) — bug-1548 lives inside task 4.4's own `ApiserverBackendBridge` boundary against an already-existing acceptance criterion, so it doesn't qualify as legitimately out-of-scope -- **Phase 6 remediation** (tasks 6.1/6.2): added a monotonic `reconnect_gen` counter to `endpoint_t` (`src/include/netio.h`, `src/lib/xcectx.c`), bumped on every successful transport reconnect; `heartbeat.go`'s `runHeartbeat()` replaced its failure-to-success transition detector with a per-tick generation comparison. New regression tests in `heartbeat_test.go`/`heartbeat_rejoin_test.go`. Live E2E re-run of the exact task-5.2 scenario passed: rejoin fired 22s after a primary kill (12s after election) with zero write traffic in the preceding 5.5 minutes; a post-recovery job submit correctly appeared via `GET /xce/v1/jobs` -- Decision flipped to **GO** (14:20 UTC) -- **bug-1558** fixed: `make test-backend` now self-derives `CGO_CFLAGS`/`CGO_LDFLAGS` via `pkg-config` -- **bug-1559** fixed: `xce-build.sh`'s scheduler case now runs `meson test --suite scheduler` (all 41 registered targets, not just one binary by path) -- **New, separate uncommitted feature:** `xceadmin` cluster-config dump/pretty-printer (`build.py`: `dump_cluster_config` + formatters, ~236 new lines) and a `parse_size_or_number` helper in `mapping.py` for size-string threshold parsing, with new tests (`test_cli.py`, `test_mapping.py`, untracked `test_config_dump.py`) - -## Bugs opened/fixed this window - -Named/significant bugs (55 total opened this window, `bug-1506`–`bug-1560`; most of the unlisted ones are terse review-catches — "significant refactor of X", "incorrect value in code" — logged during component-failover-adoption's two-round review process): - -| Bug | Status | Summary | -|---|---|---| -| bug-1506, bug-1507 | Fixed (`90d7dd82`, `b00e97ff`) | `probeConfigTag` missing Request-envelope handling; SIGABRT in REPL breakpoint inspection | -| bug-1508 | **Open** | `handle_fetchfile()` — possible job-ownership/path-traversal gap found during doc audit; flagged, needs a security-reviewer pass | -| bug-1509 | Fixed | `extralib` flake dependency removed; ported trivial linked-list code local to autoscaler | -| bug-1514, bug-1515 | Fixed | design.md/requirements.md defects in component-failover-adoption's scheduler-backpressure description, found by `/kiro-validate-design` and a deep re-review | -| bug-1519 | **Open** | AddressSanitizer "joining already joined thread" abort under `unshare -r`; worked around, not root-caused | -| bug-1532 | Fixed (`e6359039`) | apiserver `tryConnect()` use-after-free on `ctx->conf->masters[i]` during failover rotation | -| bug-1542 | Fixed | Illegal `import "C"` in a `_test.go` file; exposed the needed constant via a plain-Go package var instead | -| **bug-1548** | Fixed (uncommitted, Phase 6) | apiserver never rejoined a new primary on an idle/heartbeat-only failover — root cause: `XCECLI_FF_CALL`'s silent inline reconnect was invisible to the old failure→success transition detector | -| **bug-1549** | Fixed (uncommitted, Phase 6) | Downstream symptom of bug-1548 — job pushes never reached apiserver's local store after failover | -| bug-1553 | **Open** | `controller_unit_test` segfaults intermittently (~1 in 3 runs); crash site not yet isolated | -| bug-1554 | **Open, deferred by user** | `maxswap`/`maxtmp` report hardcoded stub values instead of real hardware detection | -| bug-1555, bug-1556 | Fixed (`e3b3ffaa`) | Misleading `update_resval` error logs; zero-slot scheduling block on the documented `hosts.yaml` auto-detect default | -| bug-1557 | **Open** | `xce_test worker join` defaults `maxmem=64` (bytes, not MB) | -| bug-1558, bug-1559 | Fixed (uncommitted) | `make test-backend` CGO flags now self-derived via `pkg-config`; `xce-build.sh` scheduler suite now runs all 41 test targets | -| bug-1560 | **Open, low priority** | `test_recordstart_symlink_escape_replies_error` fails under sandbox root — confirmed pre-existing, environment-specific, not a regression | - -## Spec status changes - -- **component-failover-adoption** — tasks 1.1–5.4 (21 tasks) implemented and initially marked complete (Aug 13). Reopened same week by its own `/kiro-validate-impl` run: **NO-GO** (Aug 15, 13:10 UTC) → Phase 6 remediation (2 tasks) → **GO** (Aug 15, 14:20 UTC). **Implementation and validation are done; the fix itself is not yet committed to git.** -- **scheduler-core** — carried into this window only for Aug 08's documentation cleanup; no further status change. -- Three new specs created this week, all still at brief/pre-requirements stage, not yet started: **component-rejoin-resync** and **worker-rejoin-resync** (both born from this week's bug-1548/1549 investigation — full-state resend cost on rejoin), and **scheduler-sim-diagnostics** (limit/pend-reason visibility in the REPL; not yet even `spec.json`-initialized). - -## Stats - -- **49 commits** in the window (39 feature/fix/docs, 10 wolf-infra/checkpoint) -- **~124 files changed** across the week's commits (109,317 insertions / 18,170 deletions, dominated by component-failover-adoption) -- **55 bugs opened** (bug-1506–bug-1560), **48 fixed**, **7 still open** (bug-1508, 1519, 1553, 1554, 1557, 1560, plus bug-1548/1549 fixed-but-uncommitted) -- **1 spec closed and reopened same week**, remediated, now GO -- **3 new specs** opened at brief stage -- **0 commits on Aug 14–15** despite validated GO work landing — 57 files (modified + untracked) sitting uncommitted at week's end - -## Open threads / to-do heading into next session - -1. ~~**Commit the Phase 6 failover fix**~~ — **Done (2026-08-15, `d479b70`).** Verified via a full `lib asan` rebuild (all libxce tests incl. the new `test_netio_reconnect_gen` pass) and `make test-backend` (apiserver backend + auth suites pass) before committing. Scoped to `netio.h`, `xcectx.c`, `lib/meson.build`, `lib/testsuite/test.c`, `lib/testsuite/test_netio_reconnect_gen.c`, `heartbeat.go`, `clib.go`, `heartbeat_test.go`, `heartbeat_rejoin_test.go`, `spec.json`, `tasks.md`. Note: `design.md`/`requirements.md`/`research.md` were deliberately left out — they've been stale since the spec's very first commit (`4ad31d77`, never updated across all 21+2 tasks), so syncing them is a separate, broader doc-catch-up task, not part of the Phase 6 diff. -2. ~~**Commit or explicitly shelve the xceadmin config-dump feature**~~ — **Done (2026-08-15, `3afe3d5`).** Verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install + pytest run (256 passed, 3 skipped) before committing. Scoped to `build.py`, `cli.py`, `errors.py`, `mapping.py`, `hosts.schema.json`, `queues.schema.json` (the `SizeOrNumber` schema type is tightly coupled to `mapping.py`'s new `parse_size_or_number`, so it had to travel with it), plus the three test files. **New discovery while scoping this:** `resources.yaml`'s diff also contains unrelated changes — new resource catalog entries (`r15m`, `ut`, `pg`, `ls`, `it`, `swp`, `io`) that the worker (`resource_sync.c`) already registers but the shipped default `resources.yaml` never declared, plus `mem`/`tmp` byte-unit description fixes matching `node_information.c`'s uncommitted change (worker-reported `mem`/`tmp` now actually returns bytes, not MB) and `xcecd.c`'s `maxswap`/`maxtmp` stub-value unit fix (bug-1554's stub, now byte-scaled). These three files (`resources.yaml`, `worker/src/node_information.c`, `worker/src/xcecd.c`) were deliberately left out of both commits so far — added as new item 10 below. -3. **`.wolf/dashboard-token`** is untracked, holds a live-looking token, and is not covered by `.gitignore` — add a gitignore rule before it lands in history via a broad `git add`. -4. **Triage the remaining uncommitted files** not obviously tied to items 1, 2, or 10 (`deploy-ha.sh`, `xce-build.sh`, `controller/testsuite/test.c` — the `xce_test` detailed-help-text feature, unrelated to bug-1557's one-line default fix it happens to share a file with). -5. **bug-1553** (intermittent controller test segfault) is the most concerning open item — a real memory-safety bug in the shared test binary, unisolated. Needs a bisect/gdb session. -6. **bug-1508** (possible `handle_fetchfile()` ownership/path-traversal gap) — flagged during a doc audit, never given a security-reviewer pass. -7. **bug-1557** — one-line fix (`maxmem` default should be `64 * 1024 * 1024`, not `64`), still open. -8. **bug-1554** — swap/tmp-disk detection deferred by explicit user request; needs a design discussion when picked back up. -9. Decide which of the three new brief-stage specs (component-rejoin-resync, worker-rejoin-resync, scheduler-sim-diagnostics) to move into `/kiro-spec-requirements` next. -10. **New (found while scoping item 2):** commit the worker-side byte-unit completion + resource-catalog sync (`resources.yaml`, `src/worker/src/node_information.c`, `src/worker/src/xcecd.c`) — finishes what `5bf123cc`/`e3b3ffaa` started (mem/tmp now genuinely report bytes, not just documented as bytes) and adds the 7 missing built-in resource entries (`r15m`, `ut`, `pg`, `ls`, `it`, `swp`, `io`) the worker already reports but the catalog never declared. Worth its own scoped commit with a build+test pass first, same as items 1 and 2. diff --git a/reports/aug16-aug22-2026.md b/reports/aug16-aug22-2026.md deleted file mode 100644 index cab84b61..00000000 --- a/reports/aug16-aug22-2026.md +++ /dev/null @@ -1,103 +0,0 @@ -# XCE Project Status — Aug 16 – Aug 22, 2026 - -## TL;DR - -The week opened by clearing the entire uncommitted backlog the Aug 08–15 report ended on, then ran four specs start-to-finish. **resource-value-verification** (Tue), **build-time-versioning** (Thu), **dirty-content-fingerprint** (Fri), and **scheduler-sim-decision-parity** (Sat) each went spec → implementation → tests inside a single day. Monday was a firefight: seven stacked bugs, one of which (**bug-1569**) was blocking *all* job dispatch. Every one of the prior week's 7 open carry-over bugs is now closed, and 64 new bugs (bug-1561–1624) were opened and fixed within the window. The one thing that got worse: `.wolf/buglog.json` picked up **39 duplicate IDs** from a restarted counter on Aug 15–17. - -## Commits by day - -### Aug 16 (Sun) — clearing the prior week's uncommitted backlog - -- **`d479b707`** — component-failover-adoption Phase 6: closed apiserver's idle-period rejoin gap (bug-1548/bug-1549). This was open thread #1 from last week's report. -- **`3afe3d55`** — xceadmin `config dump` command + size-string thresholds (open thread #2). -- **`45400c3a`** — controller: closed 6 data-safety races, an apply-pump O(N²) regression, and a boot-time TOCTOU. -- **`8a6c339a`** — libxce bug-1425: `jobsubmit` now populates `*jobid` on idempotent-replay success. -- **`ce0ee78b`, `5a08dac5`, `570e464d`** — spec metadata reconciliation: promoted 3 new specs, closed 3 stalled specs whose follow-up work had already landed, closed **admin-config-tool** task 6.3 (now 27/27). -- **`69f44c2f`** — published the Aug 8–15 weekly report. - -### Aug 17 (Mon) — HA + dispatch firefight (bug-1562 → bug-1569) - -- **`846713ab`** — bug-1562 standby election freeze + bug-1563 flusher retry storm. -- **`986a3304`** — bug-1564: `persist_flush` now fires the durable-advance callback. -- **`7041d988`** — **bug-1569: three stacked crashes blocking ALL job dispatch.** The week's most severe fix. -- **`6e5c3a89`** — bug-1569 companion: worker periodic resval reports were tagged with the wrong host. -- **`aabddf60`** — bug-1568: worker resource values always displayed as 0.00. -- **`0c8e417b`** — bug-1567: sim-mode picker never learned about the initial snapshot's own jobs/queues. -- **`1c943e98`** — bug-1565: `--incremental` flag for `xce-build.sh`. -- **`0f1df499`** — bug-1566: standalone xceadmin packaging + deploy-ha usability. -- Started **resource-value-verification** (tasks 1.1, 1.2, 2.1). - -### Aug 18 (Tue) — resource-value-verification, spec to done in one day - -- Tasks 2.2 → 8.2 landed: resupdate per-name accept/reject verification, `xce-deploy-ha` apiserver step + `start-all.sh`, scheduler and apiserver resval trace logs, then the four-tier verification chain — worker join/injection (6.1), controller (6.2), scheduler (6.3), apiserver (6.4) — plus verdict aggregation (7.1), live E2E run (8.1), and a verdict-logic selftest (8.2). -- **`b406dd76`** — bug-1578 (CRITICAL, reviewer-discovered): `sched_worker_change()` use-after-free on worker disconnect, reproducible controller SIGSEGV. -- **`b673b7c9`** — bug-1579 closed for real via `xce_test`'s `!` shell-escape. - -### Aug 19 (Wed) — scheduler-core task 12.5 closeout - -- **`c9acd502`** — clock/resval/jobid-format fixes plus REPL live-operator gaps. -- **`a13068dc`** — extended the core-isolation `nm` audit to `core_events.c`/`core_session.c`/`core_clock.c` (Req 5 AC 1/2). -- **`b450a680`** — activated virtual-clock logging for sim/REPL sessions (Req 5 AC 3). -- **`e52c68f4`** — admission-gated `OPCode_schedAdmin`'s blocking wait (Req 5 AC 10). -- **`70fe9e46`** — bug-1589: `dbus_workers_update()` starved workers after the first DONE/EXITED. -- **`515f4ca0`** — bug-1588: `job.c` TRACE lines now print human-readable jobids. -- **`83e61eec`** — `xce-deploy-ha` gained real `xcecd` worker deployment alongside virtual. - -### Aug 20 (Thu) — build-time-versioning, spec to done in one day (21 tasks) - -- **`1ff9e66c`** spec → **`4b7cc786`** `build_time` meson option → full fan-out across every binary: controller/scheduler `--version` (`88649a04`), worker (`bc5c669b`), apiserver `-version` (`17e837b1`), xcecli via Cobra (`fb09afe6`), xceadmin build-info module + fallback stub (`e1cd9d25`, `7f5c87ce`). -- Build plumbing: `xce-build.sh` real commit id/build time (`bca63974`), nix build-phase timestamps (`d08fda76`), `go_build` ldflags injection (`18ae0cc3`, `071f3895`), Makefile ldflags (`45c9b026`). -- Three regression scripts: motivating-incident (`0eefc95f`), version/startup-log consistency (`79941a9f`), xceadmin end-to-end build/install/version cycle (`084a5a95`). -- Also **`ddee884c`** — bug-1600: `makeGroup()` freed an rtti `makeString()` pointer via raw `free()`, crashing the scheduler during controller failover. -- Also **`5147396d`** — controller job `start_time`/`finish_time`/`run_time` were never set by the apply-pump. - -### Aug 21 (Fri) — dirty-content-fingerprint, spec to done in one day (21 tasks) - -- **`d3c583b3`** spec → **`a860141b`** `dirty_fingerprint` meson option + C define → fingerprint surfaced in all six binaries (`fd62e26b`, `d3d9522f`, `a4c1b5ad`, `d65d18ad`, `2bf62a64`, `c79ccf0f`). -- Computation paths: nix via `self.narHash` (`fc5cba15`), `xce-build.sh` dev-loop (`c9d40ba1`), Makefile dev-loop (`aa3391a9`), `--version-only` composition (`7ebcba6d`). -- Three regression tests: dev-loop distinguishability (`a7d64c82`), startup-log-unchanged consistency (`a4785418`), Nix production-path fingerprint (`f4d2c123`). - -### Aug 22 (Sat) — scheduler-sim-decision-parity, spec to done in one day (13 tasks) - -- **`5f7e1e98`/`25e35bac`** documented the driver-equivalence invariant and scoped the fix; **`0b2c856d`** finalized design + tasks. -- Implementation: relocated resource accounting to core level (`3106c82e`), gave sim mode `driver.c`'s round-boundary split (`134380b8`), added reconciliation trigger/lookup (`81358b52`), reconstructed job allocation with safe ownership (`eccba841`), reported reconciled decisions through the existing channel (`0c237f60`), wired reconciliation into the round boundary (`12644094`). -- Validation: contention-parity proof (`b3190732`), regression suite for existing guarantees (`88bed529`), live verification against a real 24-round recording (`3c4091d8`), then **`ca01509f`** marked the feature complete. -- **`979cd24f`** — scheduler-sim-repl-verbosity spec approved (requirements/design/tasks). -- **`1c906e95`** — bug-1588 follow-up: worker job ids print via `fmt_jobid()`. - -## Bugs opened/fixed this window - -- **64 genuinely new bugs** (bug-1561 – bug-1624), **all 64 fixed** — zero left open at window end. -- **All 8 carry-over bugs** the prior report listed as open are now FIXED: bug-1508, 1519, 1548, 1549, 1553, 1554, 1557, 1560. That includes bug-1553, which last week's report flagged as "the most concerning open item" (intermittent controller test segfault). -- Highest-severity fixes: **bug-1569** (three stacked crashes blocking all job dispatch), **bug-1578** (CRITICAL controller UAF on worker disconnect), **bug-1600** (allocator-mismatch heap corruption during failover), **bug-1572** (scheduler double-free/dangling pointer on resvals delta). -- ⚠️ **`.wolf/buglog.json` integrity regressed.** The file grew 917 → 1004 entries, but **39 IDs are now duplicated**: entries appended on Aug 15 (8), Aug 16 (14), and Aug 17 (22) reused the already-taken `bug-875`–`bug-949` range. Example: `bug-908` is both a 2026-06-21 `raft_sim.c` guard-clause bug and a 2026-08-16 `config-upgrade-snapshot/spec.json` refactor note. This is exactly the counter-restart failure the OpenWolf bug-logging rule warns is "unrecoverable without a git-history merge." - -## Spec status changes - -- **resource-value-verification** — created and driven to `implementation-complete` on Aug 17–18. Task checkboxes read 15/23 despite the phase; the trailing 8 are unchecked. -- **build-time-versioning** — created and fully implemented Aug 20, **21/21 tasks**, but `spec.json` phase still reads `tasks-generated`. Metadata lag, not work lag. -- **dirty-content-fingerprint** — created and fully implemented Aug 21, **21/21 tasks**, same `tasks-generated` phase lag. -- **scheduler-sim-decision-parity** — created and completed Aug 22, **13/13 tasks**, phase `implementation-complete`. -- **scheduler-sim-repl-verbosity** — spec approved Aug 22, **0/15 tasks**, not started. -- **admin-config-tool** — closed Aug 16 at 27/27. -- **component-failover-adoption** — Phase 6 fix finally committed Aug 16, closing last week's #1 open thread; checkboxes read 22/27. -- Three stalled specs closed Aug 16 as already-superseded by landed work. - -## Stats - -- **192 commits** in the window — **103 feature/fix/docs**, 89 `.wolf` bookkeeping/checkpoint commits. -- **218 non-`.wolf` files changed**, **+28,657 / −2,268** lines (excluding `.wolf/`; including it the raw number is +288,983 / −22,630, dominated by `token-ledger.json` and `buglog.json` churn). -- Busiest areas: `src/scheduler` (55 files), `.kiro` (54), `src/controller` (25), `src/worker` (13), `src/frontend` (13). -- **4 specs implemented end-to-end**, 1 spec closed, 1 spec approved-not-started, 3 stalled specs retired. -- **19 sessions** logged in `.wolf/memory.md` across Aug 16–21. -- Commits landed on **all 7 days**; peak days Mon and Thu (35 each). - -## Open threads / to-do heading into next session - -1. **Repair the 39 duplicate buglog IDs.** The Aug 15–17 counter restart collided new entries with `bug-875`–`bug-949`. Needs a git-history-informed merge: pull the pre-collision file, identify which of each duplicate pair is the newcomer, and renumber the newcomers above the current max (1624+). The Stop hook validates duplicates, so this will keep flagging until fixed. -2. **Sync spec.json phase metadata** for build-time-versioning and dirty-content-fingerprint — both are 21/21 implemented but still read `tasks-generated`. -3. **Reconcile resource-value-verification's checkboxes** — phase says `implementation-complete`, tasks say 15/23. Either the last 8 tasks are done-but-unchecked or the phase is premature. -4. **scheduler-core is 49/59** and **component-failover-adoption is 22/27** — both sit at `implementation-complete` with unchecked tasks. Same reconcile-or-reopen question. -5. **Decide the next spec to start.** scheduler-sim-repl-verbosity is approved at 0/15. A new untracked spec, `scheduler-sim-eager-reconciliation`, appeared in the working tree and is not yet committed. -6. **Uncommitted at window end:** `src/scheduler/src/handler.c`, `src/scheduler/testsuite/test_handlers.c`, and untracked generated `src/lib/controller.capnp.{c,h}` — the generated pair should probably be gitignored rather than committed. -7. **`.wolf/dashboard-token`** — still untracked, still holding a live-looking token, still not covered by `.gitignore`. Carried over unresolved from last week's item #3. diff --git a/reports/aug23-aug29-2026.md b/reports/aug23-aug29-2026.md deleted file mode 100644 index 416f1200..00000000 --- a/reports/aug23-aug29-2026.md +++ /dev/null @@ -1,111 +0,0 @@ -# XCE Project Status — Aug 23 – Aug 29, 2026 - -> Note: the calendar-default window (Aug 17–22) is already covered by -> `aug16-aug22-2026.md`; this report covers the actual gap — the last -> complete day before this report was generated (Aug 30 is in progress). - -## TL;DR - -An unusually dense week: 97 real commits shipped two specs end-to-end -(`scheduler-dispatch-commit`, `xce-test-rc-file`), drove -`scheduler-sim-eager-reconciliation` and `scheduler-snapshot-job-allocation` -to substantial completion, and closed out a systematic five-bug sweep of -WAL-replay field-completeness gaps alongside several safety-critical HA -correctness fixes (vote-denial on persist failure, stale-term -re-election blocking, fail-loud disconnect handling). Last week's #1 -open thread — 39 duplicate buglog IDs — was fully repaired on Monday. -The week closed with a deep multi-session live-debugging investigation -that fixed three confirmed production bugs (a scheduler resource leak, -a controller heap-use-after-free that crashed 2 of 3 controllers under -load, and a set of permanently-dead job-notification hooks that could -leave a job's state stuck as RUN forever in the scheduler and apiserver) -and made major forensic progress on a fourth, still-open scheduler crash -(bug-1714) — proved it's a genuine cross-thread use-after-free, not the -overflow first suspected, but the exact stale-pointer source remains -unisolated. - -## Commits by day - -### Sun 08-23 — `scheduler-dispatch-commit` spec, start to finish in one day - -- **`5724d1e9`→`a9cfee5f`** (13 commits) — dispatch-failure classification, a durable `WAL_JOB_DISPATCH` fact type, scheduler transitions to RUN at commit time, controller-side retryable/terminal dispatch-failure resolution with bounded escalation, and full multi-round + terminal + retryable e2e coverage. Spec marked implementation-complete (`f784e2bd`). - -### Mon 08-24 — Gitea issue triage and buglog integrity repair - -- **`1c9cfc1f`, `e83d207a`** — reproduced and logged 4 Gitea issues (#40–42, #44) plus a `gitea-sync` bug found while doing it; corrected an earlier false-negative on #43 (bug-1678 — turned out reproducible). -- **`db142d41`→`88c7c05d`** — fixed the 4 reproduced issues: dispatch-time resource allocation not persisted into `WAL_JOB_DISPATCH` (bug-1678), live job-status pushes to the apiserver never firing (bug-1679), `jobs list -l/-o` validating against the wrong struct (bug-1674), unmatched workers not falling back to a default `HostConfig` (bug-1673) — plus `from_host` threaded end-to-end through the wire format, CGO layer, and job-submit interning object (bug-1676). -- **`755976fc`** — **repaired last week's #1 open thread**: renumbered 40 buglog entries that had collided with `bug-875`–`bug-949` after the Aug 13–17 counter restart. -- **`5c82ad0b`** — **repaired last week's #2 open thread**: synced `spec.json` phase metadata for `build-time-versioning`/`dirty-content-fingerprint` (both were 21/21 done but stuck on `tasks-generated`) and flipped 23 stale section-header checkboxes across 3 more specs whose numbered subtasks were already complete. - -### Tue 08-25 — peak day (46 commits): WAL-replay field-completeness sweep + HA correctness - -- **`955efecd`→`d0e5f85a`** — five-bug systematic sweep: `WAL_JOB_START` losing pidinfo/start_time (bug-1687), `WAL_JOB_FINISH` losing exit_info/finish_time — and exit_info was never set in production at all (bug-1689), `WAL_JOB_RUSAGE` losing run_usage (bug-1690), `WAL_JOB_CKPT` losing ckptinfo (bug-1691), `WAL_JOB_PREASON` losing pend_reasons (bug-1692) — all on restart replay. -- **`e5e6cd18`** — **`raft_cb_persist` must deny the vote and roll back state on a persist failure** (bug-1684): `ha_persist_term()`'s every open/write/fsync/rename failure was silently swallowed; the raft library always granted the vote regardless. Safety-critical. -- **`3eba7c16`** — fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693). -- **`b4eb9ec8`** — offload `checkpoint_task_fn` off the delaymgr thread (bug-1694, async-I/O blocking-violation). -- **`ac46bfa6`** — restored-but-stale raft term permanently blocked re-election after restart (bug-1695), found via the mandatory live HA chaotic soak for this session's controller audit campaign. -- **`b040dfc0`, `bc3f76b4`, `6400f96d`** — removed a double-free in the limit plugin's config update (bug-1685), invalidated stale `builtin_fcfs` picker cursors across calls (bug-1686), mirrored `plugin_begin_session`/`plugin_end_session` into the sim-mode driver (bug-1681). -- **`9a230f08`** — apiserver `FilterRows` routing non-bitmask operators through bitmask matching (bug-1680, found while verifying bug-1679). - -### Wed 08-26 — `xce-test-rc-file` spec - -- **`9de4db95`→`8fc86ef4`** (7 commits) — rc-file path resolution with XDG precedence, `--norc` flag, pluggable rc-line-loading runner wired into `xce_test` startup, and full e2e coverage for discovery/loading/`--norc`/failure-handling, confirmed not to disturb existing automation. - -### Thu 08-27 — `scheduler-sim-eager-reconciliation` spec + a fix batch + `scheduler-snapshot-job-allocation` kickoff - -- **`e2bb98c8`→`446437bf`** (6 commits) — state-independent dispatch capture, resInUse deferral proof, capture-safety unit coverage, resource-contention/sim-verify/full regression proofs. 6/9 tasks done. -- **`f886c809`** — standby dispatch-gate rejections now redirect to a known primary (bug-1700, bug-1701) — closes a gap `admin-config-tool`'s own design doc had flagged as a documented blocker. -- **`ac1e312e`, `0ebf3a81`, `47532f0a`, `cfd7db8c`** — `job.start_time` never set due to a dispatch/`WAL_JOB_START` RUN-state collision (bug-1697); repeated `WORKERS_PUSH` with a NULL resval clobbering good values (bug-1688); first-boot promotion evaluating eligibility before its own WAL recovery ran (bug-1696); xceadmin retrying transient scheduler-admin failures after a redirect (bug-1713, bug-1715). -- **`097e33ae`→`22353849`** — `scheduler-snapshot-job-allocation` begins: `dup_job_alloc` deep-copy primitive, schedobj construction for restored jobs. - -### Fri 08-28 — `scheduler-snapshot-job-allocation` completion + a heap-overflow fix - -- **`6fa2d970`→`5948e079`** — allocs populated on snapshot save, restored on load, full cross-instance round-trip proof, backward-compat for allocation-less snapshots, allocator-crash regression test, and the final invariant-audit follow-up closed. Spec docs mark it feature-complete. -- **`00207503`** — **heap buffer overflow in `make_job_account()` for multi-host jobs** (bug-1722): the function `calloc`'d room for exactly 1 host account pointer but its own loop wrote up to `nhosts-1` — any job spanning more than one host overflowed the allocation. - -### Sat 08-29 — live-debugging investigation: 4 bugs found, 3 fixed, 1 deeply diagnosed - -- **`b1323f8d`** — scheduler leaked `job->accounts`/`job->allocs` on a `JOB_UPDATE` terminal transition (bug-1730). -- **`42e1eeaa`** — **heap-use-after-free in `vtask_enq()`** (bug-1733): a task was published to the dispatch queue before its own `owner` field finished initializing; a concurrent worker could dequeue, run, and free it before the producer's own write executed. ASAN-confirmed, crashed 2 of 3 controllers under a live burst-array-job HA workload. -- **`ebe691c3`, `67009815`** — `apply_committed_entry()` only ever fired the generic `HOOK_DETAIL_JOB_UPDATE` for job-state changes; `HOOK_DETAIL_JOB_START`/`JOB_EXIT`/`JOB_FAIL` were permanently dead code, leaving the Go apiserver's already-wired `OnJobFinished` handler starved of data — a finished (or failed) job's state could stay stuck as RUN in the scheduler's own view forever (bug-1731). Live-verified end to end on both the scheduler and apiserver sides. -- **`83cb660b`** — `xce-build.sh`'s `worker` target never built `xcecd_virtual`, so deploy tooling silently skipped generating the worker-start script in sandboxes with no real systemd manager (bug-1732). -- **`347e996e`, `14f95b06`** — worker now distinguishes signal-killed job exits from genuine crashes in its logs; scheduler snapshot load re-derives worker/shared resource "used" totals instead of leaving them at zero until the first session. -- **`6193fa7e`** — `test_e2e_live_ha_push.py`'s standby-redirect test rewritten to assert the new success behavior (was asserting the old, now-fixed rejection behavior). -- **Not landed as a commit, but the week's deepest investigation**: `bug-1714` (a scheduler `SIGSEGV` in `sched_get_project`, first reported live days earlier) was reproduced twice more this session — once under ASAN (no shadow-memory violation fired), once via a scripted gdb watchpoint that caught the actual corrupting write: freed `job_submit_data_t` memory being recycled by malloc for an unrelated Cap'n Proto decode buffer on a different thread. The free itself was verified legitimate by static reading — the bug is an unidentified second/stale reader elsewhere. Two related side-bugs found during the chase (an orphaned duplicate job silently handed to the picker, and the dedup-race that made it reachable) were fixed and confirmed independent of bug-1714's own root cause, which remains open. - -## Bugs opened/fixed this window - -- **69 genuinely new bugs** (bug-1625 – bug-1733), **66 fixed** across the window's 75 touched entries, **9 still open** at window end. Zero ID collisions — the buglog counter stayed clean all week. -- **Both of last week's open threads are resolved**: the 39 duplicate IDs were repaired Monday (`755976fc`); `build-time-versioning`/`dirty-content-fingerprint`'s phase-metadata lag was fixed the same day (`5c82ad0b`). -- Highest-severity fixes: **bug-1722** (heap buffer overflow in `make_job_account()` for any multi-host job — memory corruption, not just a crash), **bug-1733** (controller UAF crashing 2/3 nodes under live HA load), **bug-1684** (raft would grant a vote even when its own persist call failed — HA safety-critical), **bug-1695** (a restored-but-stale raft term could permanently block re-election after restart). -- Still open at window end (carry-over risk into next week): - - **bug-1714** — scheduler UAF crash, mechanism proven, stale-reader source not yet isolated (highest priority — this is the week's one unresolved thread with active investigation momentum). - - **bug-1429** — `controller_unit_test` 300s TIMEOUT, deadlock-suspected, open since Jul 10, no owner. - - **bug-1560** — `test_recordstart_symlink_escape_replies_error`, a known sandbox filesystem-permissions flake, open since Aug 15. - - **bug-1630, bug-1632, bug-1723, bug-1726, bug-1727, bug-1728** — six newer, lower-priority findings (a pre-existing ASAN flake in the sim driver, a test-fixture memory leak, a flaky HA e2e test distinct from bug-1485, a REPL resource-display bug, and a false-ERROR log severity on worker join) — none blocking, none investigated further this week. - - **Note on bug-1630 specifically**: its own `fix` field is still empty in the buglog, but the underlying problem it describes (the 39 duplicate IDs) is verifiably resolved — `755976fc`'s renumbering left **zero duplicate IDs** in the current file (re-verified directly). The entry itself just never got marked fixed; worth a one-line buglog update, not further investigation. - -## Spec status changes - -- **`scheduler-dispatch-commit`** — went 0→implementation-complete in one day (Sun); `spec.json` still reads 13/18 tasks despite the phase — metadata lag, not work lag (same pattern as last week's build-time-versioning). -- **`xce-test-rc-file`** — 7/10 tasks, phase `tasks-generated`. -- **`scheduler-sim-eager-reconciliation`** — 6/9 tasks, phase `tasks-generated`. -- **`scheduler-snapshot-job-allocation`** — driven to feature-complete per its own closing commit message, but `spec.json` phase still reads `tasks-generated` at 11/15 tasks — another phase/reality mismatch to sync. -- **`scheduler-config-version-tag`** — new spec created (requirements/design/research/tasks all approved), 0/8 tasks — not started. -- **`admin-config-tool`** — no new work, but its own design doc's last documented blocker (the standby-redirect gap) was closed this week by `f886c809`. - -## Stats - -- **164 commits** in the window — **97 real** (feature/fix/docs/test), **67 `.wolf` bookkeeping/checkpoint commits**. -- **341 non-`.wolf` files changed**, **+26,205 / −1,002** lines (excluding `.wolf/`; including it the raw figure is +220,722 / −1,049,394, dominated by buglog renumbering and token-ledger churn). -- Busiest areas: `src/controller` (46 files), `.kiro` (42), `src/scheduler` (35), `.wolf` (25), `src/frontend` (16). -- **2 specs shipped end-to-end** (`scheduler-dispatch-commit`, `xce-test-rc-file`), **2 more substantially advanced** (`scheduler-sim-eager-reconciliation`, `scheduler-snapshot-job-allocation`), **1 new spec created**. -- **18 sessions** logged in `.wolf/memory.md`, single busiest day Aug 23 (383 logged actions). -- Commits landed on **all 7 days**; peak day **Tue 08-25** (46 commits — the WAL-replay + HA sweep). - -## Open threads / to-do heading into next session - -1. **Isolate bug-1714's stale-reader.** The UAF mechanism is proven (freed `job_submit_data_t` recycled across threads); the actual second reader that still holds a pointer past the job's legitimate free is not yet found. Two leads already ruled unlikely or weak by this week's own static reading — needs either deeper tracing of the pending-update queue lifecycle or another live session with pointer-tracking instrumentation. -2. **Sync `scheduler-dispatch-commit` and `scheduler-snapshot-job-allocation`'s `spec.json` phase metadata** — both report a phase behind their actual completion state, the same lag class fixed for two other specs this Monday. -3. **Five lower-priority open bugs** (1632, 1723, 1726, 1727, 1728) have no assigned follow-up yet — worth a triage pass to confirm none has quietly become higher-priority. (bug-1630 is housekeeping only — see the note above; just needs its buglog entry marked fixed.) -4. **bug-1429** (controller test-suite deadlock-suspected timeout) has been open since Jul 10 with no owner — the oldest unresolved item in the log. diff --git a/reports/jul06-jul11-2026.md b/reports/jul06-jul11-2026.md deleted file mode 100644 index 4dcbb449..00000000 --- a/reports/jul06-jul11-2026.md +++ /dev/null @@ -1,191 +0,0 @@ -# Jul 6 → Jul 11 Progress Report - -## Overview - -Five days of HA stability hardening, chaotic soak debugging, spec work, and controller invariant audit. **raft-apply-decoupling completed**. **Two controller invariant bugs fixed** (1433/1434) bringing HA soaks to clean PASS. **Three specs advanced** (scheduler-framework design, admin-config-tool finalized, ha-elect-ctx-lifecycle initiated). - ---- - -## Jul 6-7: raft-apply-decoupling — COMPLETE - -Final tasks closed out the raft-apply-decoupling spec: - -| Task | Detail | Commit | -|------|--------|--------| -| 7.2 | kill-mid-fact-stream soak, fix bug-1416 boot-time TOCTOU | `a5e29a6` | -| 7.3 | Stale apply-before-commit assertions updated | `16b890b` | -| 7.4 | O(n²) apply-pump WAL rescan → O(log n) throughput benchmarked | `40196be` | - -**Bugs fixed during close-out**: -- **bug-1416**: persist recover TOCTOU — boot-time applied_lsn > committed_lsn gap -- **bug-1419** (×3 layers): resubmit-dedup gap — layer 1 (fe4352f), layer 2 (0eb1054), layer 3 (f28ce57) -- **bug-1420**: xce_test CLI harness jobid collision within same wall-clock second - -Spec marked complete: `903b284`. - ---- - -## Jul 7-9: HA Election Liveness — Cascade of Bugs - -Six bugs discovered and fixed in rapid succession, all around election/term/vote correctness: - -| Bug | Manifestation | Root cause | Fix | -|-----|---------------|------------|-----| -| **1417** | Election stall, votes silently dropped | Vote reply index mapping: compressed local index → absolute peer index | `ha_repl_ack_abs_peer_idx` resolver (`d512275`) | -| **1418** | Term-desync + duplicate-node guard | Two factors: (1) ha->term/raft.term desync on standby, (2) start_node re-launch on already-running process | Multi-pass fix (`9b173c9`, `48b4dba`) | -| **1421** | AE-reject blindspot, term desync cycle | raft_on_higher_term_observed ignored stale-term AE when term was already advanced | Close blindspot (`327dbb9`) | -| **1422** | verify_no_loss false-positive after election | Race: job presence check ran before post-election convergence completed | Retry with backoff (`016c598`) | -| **1423** | vote-req failures silently dropped (production path) | Production vote-reply dispatch never called; only PreVote path ran | Route prod path through dispatch | -| **1424** | PreVote twin of 1423 | PreVote connect/monitor failure branches never prod | Same pattern, PreVote path | - -**Validation**: 40-round sequential PASS (111 jobs, 0 loss), 30-round chaotic PASS (93 jobs, 0 loss). - ---- - -## Jul 9-10: Chaotic Soak Debugging — Major Effort - -~6 chaotic soak rounds chasing persistent `committed_job_loss=5-8`: - -### Round 1: fdatasync investigation -- Initial hypothesis: bug-924 (overlayfs fdatasync EIO) -- **Disproved**: errors were `errno=22 EINVAL` on ext4, not overlayfs -- Stale attribution in buglog corrected - -### Round 2: Disk cleanup -- 11,062 stale `xce_waldump` tmp files accumulated across rounds -- Freed 156GB (86% → 68% disk), eliminated 2 false-positive losses - -### Round 3: persist_apply_pump_run_sync race -- CAS-coalescing pattern caused apply-pump to bail out before catching up -- **Fixed**: replaced with guaranteed loop-until-convergence (`last_applied_lsn >= committed_lsn`) -- Build: 920 tests PASS, 5/5 meson suites OK - -### Round 4: PreVote UAF (bug-1430) -- ASan caught heap-use-after-free in `ha_prevote_round_timeout_fn` -- **Fixed**: unconditional CAS before free in cancel path -- 19 PreVote unit tests pass, 0 regressions - -### Round 5: Force-flush (bug-1431) -- Added force-flush for commit durability path -- Reduced loss from 8→5 but didn't eliminate - -### Round 6: Residual loss (bug-1432) -- Remaining 8 losses identified as **pre-convergence false positives** -- New primary starts with `committed_lsn=0` during truncation, verify runs before convergence -- Not real data loss — timing artifact in soak harness - ---- - -## Jul 10: Controller Invariant Audit - -Created `xce-audit-invariants` skill. Ran full audit across 4 dimensions: - -| Dimension | Finding | Severity | -|-----------|---------|----------| -| Raft TLA+ checklist | bug-1434: `committed_lsn` CAS race in `persist_promotion_advance_committed` | CRITICAL | -| async-I/O thread role | bug-1433: `fdatasync` blocks on delaymgr thread (blocking-I/O violation) | CRITICAL | -| dispatch gate | Clean — all gates verified | — | -| join/startup race | Clean — no exploit path | — | - -### bug-1433: async-thread fdatasync block -- `flusher_flush_once` called `fdatasync()` directly, blocking the delaymgr worker -- Violation of async-I/O invariant (blocking ops must go through slowlane or dedicated thread) -- **Fix**: moved fdatasync to slowlane - -### bug-1434: committed_lsn CAS race -- `persist_promotion_advance_committed` used CAS without lock, racing with concurrent writers -- Violation of Raft TLA+ invariant (committed_lsn must be monotonic and consistent) -- **Fix**: replaced CAS with atomic store under existing lock - -**Validation**: 20-round chaotic + 20-round sequential both PASS (63 jobs, 0 loss, 0 divergence, all 3 nodes led). - -Committed: `40f510f`. - ---- - -## Jul 10-11: Spec Work - -### scheduler-framework (design discussion) -Four-level picker architecture designed: - -| Level | Picker | Plugin slot | Default (v1) | -|-------|--------|-------------|---------------| -| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` | -| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` | -| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` | -| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` | - -Key design decisions: -- Unified picker loop: always `project → queue → job`, identical code for v1+v2 -- Queue = scheduling mechanism (HOW), Project = resource ownership (WHO) -- Preemption: SIGSTP-based v1, LSF-style cross-queue victim selection -- New `XCE_DECISION_PREEMPT` decision type flowing scheduler→controller→worker -- `requirements.md` rewritten: 6→9 requirements, architecture diagrams, data structure changes -- `brief.md` updated: four-level architecture, queue vs project roles, preemption flow - -### admin-config-tool -Spec finalized: 10 EARS requirements, 3-skill harness (xce-build/deploy-ha/test-ha), Nix 7 packages, controller upgrade pipeline verified. Roadmap: rollback + entity-snapshot phases deferred. - -### ha-elect-ctx-lifecycle -Requirements generated (`58e047f`). New spec addressing election context lifecycle management. - ---- - -## Build & Test Infrastructure - -| Change | Detail | -|--------|--------| -| xce-build/xce-test-ha/xce-deploy-ha | Subagent dispatch fixed (`5e0579d`) | -| xce-test-ha skill | Updated docs: 5-50 round range (was stale 5-10) | -| xce-audit-invariants skill | New: audit+fix+regression loop workflow | - ---- - -## Bug Tally: Jul 6-11 - -### Fixed (17) -1416, 1417, 1418, 1419(×3 layers), 1420, 1421, 1422, 1423, 1424, 1426, 1427, 1428, 1430, 1431, 1433, 1434 - -### Known residual -- **bug-1432**: Pre-convergence false-positive committed_job_loss in soak harness (not real data loss) -- **bug-1425**: Intermittent client hang after failover in synchronous jobsub path (~2/9 attempts) - ---- - -## Commits (Jul 7-11) - -``` -40f510f fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race -58e047f spec(ha-elect-ctx-lifecycle): generate requirements -ea0ac81 save wolf state and document -c689351 chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail -5e0579d feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch -016c598 fix(xce-test-ha): bug-1422 verify_no_loss false-positive after election -327dbb9 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election -0941cf9 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss -11244dd fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard -f28ce57 fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap -1b6041a fix(xce-test-ha): bug-1420 harness jobid collision + bug-1419 regression coverage -0eb1054 fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor -fe4352f fix(raft-apply-decoupling): bug-1419 layer 1 — close resubmit-dedup gap left by apply-after-commit deferral -9b173c9 fix(xce-test-ha): guard start_node against duplicate launches (bug-1418) -d512275 fix(ha): correct real-vote reply index mapping for middle-priority nodes (bug-1417) -903b284 chore(raft-apply-decoupling): mark task group 7 complete, close out spec -40196be perf(raft-apply-decoupling): fix O(n^2) apply-pump WAL rescan, measure throughput (task 7.4) -``` - ---- - -## Current State - -| Area | Status | -|------|--------| -| HA sequential soak | 40-round PASS (zero loss) | -| HA chaotic soak | 20-round PASS after bug-1433/1434 fixes | -| raft-apply-decoupling | COMPLETE (all 7 task groups) | -| ha-term-safety | Phases 1-5 complete, tasks 1-5 done | -| ha-elect-ctx-lifecycle | Requirements generated, pending design | -| ha-commit-durability | Phases 6.1-6.4 ext4-validated | -| scheduler-framework | Requirements updated (design discussion), pending approval | -| admin-config-tool | Spec finalized, pending implementation | -| Overlayfs bug-924 | fdatasync wedge on overlayfs — known, workaround: use `~/tmp` (ext4) | diff --git a/reports/jul13-jul18-2026.md b/reports/jul13-jul18-2026.md deleted file mode 100644 index 7828e872..00000000 --- a/reports/jul13-jul18-2026.md +++ /dev/null @@ -1,175 +0,0 @@ -# Jul 13 → Jul 18 Progress Report - -## Overview - -Six days, 57 commits. Two threads dominated: **job-array-single-wal-record** landed as a complete feature (single-WAL-record array job submission, compact scheduler/API-server push, live apply fence, full regression GO — then reopened once for user-directed residual-gap closure and re-validated GO), and a **backlog of 9 HA/Raft specs that had 100%-complete tasks but had never been through a formal `/kiro-validate-impl` closure pass** got worked through systematically, surfacing one genuine production crash bug (bug-1447) that got root-caused and fixed. **ha-consistency-invariants** (the umbrella Raft+CP contract spec) closed out — all 4 sub-specs now complete. The controller test suite's Unity harness got a suite-wide correctness fix (83 files). Net: **9 bugs fixed** (1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449), **9 specs validated GO** in one day (Jul 18), **1 feature spec** (job-array-single-wal-record) shipped end-to-end. - ---- - -## Jul 13: Election liveness cleanup + ha-consistency-invariants closes out - -| Bug | Manifestation | Fix | -|-----|---------------|-----| -| **1435** | 40-round soak FAIL at round 36/40 — `verify_no_loss` got 0/111 jobs confirmed; freshly-elected primary answered `E_XCE_STANDBY` | New `ha_state_t.raft_lock` (recursive mutex) serializes every entry point into `raft_state_t` on the election/vote path (`510b566`) | -| **1436** | 40-round soak FAIL at round 36/40 — `xce_joblist`: connection reset by peer right after wait-primary confirmed the new primary serving in 46ms | `XCECLI_FAILOVER_CALL_COUNT` macro, a count-aware sibling of `XCECLI_FAILOVER_CALL` for `xce_get`-family calls (`40c2351`) | - -**ha-consistency-invariants** (the umbrella Raft+CP contract spec covering ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery): Criterion 2.9 (the Facts-Apply invariant, added 2026-06-25) marked **satisfied** — closed by raft-apply-decoupling's fact-apply-after-commit split plus decision D1 (block-but-durable quorum-loss policy). All 4 sub-specs now complete; the umbrella itself needs no separate task list by design (`895a204`). - -Also: `xce-test-ha` cross-run collision guard added to the harness (`b321963`). - ---- - -## Jul 14-17: job-array-single-wal-record — feature build-out - -The largest single thread this period — 24 commits. Single WAL record per array-job submission instead of N individual records, with N-way expansion on apply. - -**Build-out** (Jul 14-15): `WAL_JOB_ARRAY_SUBMIT` record type + compact capnp variant → `jobj.c` hardening → single-record append/commit → wire into `handle_jobsubmit` → `restore_job_array` primitive → wire array expansion into both live apply and `apply_wal_entry` → make the apply pump respect apply failures → **Live Apply Resource Fence** (self-healing, gates promotion on fence state) → compact scheduler-side and API-server-side (Go) local expansion of the push, so downstream consumers only see the compact form, not N expanded records on the wire. - -**Bugs surfaced during build-out**: -- **bug-1438**: array-submit test got a WAL type mismatch then hung all 3 threads on `futex_do_wait` — a WAL-header-offset helper copied from a zero-payload-only test assumed no payload; fixed by passing the entry's start offset explicitly instead of back-computing it (`jap_read_wal_hdr_at`/`jap_read_wal_entry_at`). -- **bug-1442**: AddressSanitizer double-free on `TestWireDispatch_OnePushProducesNRecordsAndLogsOnce` — Go's `go_push_handler` deferred `free_Request` while the datamsg-case free path double-freed the same buffer; fixed and verified. -- bug-1437, 1439, 1440, 1441 also surfaced (accounts-leak test finding, a controller.capnp.h build break, two Go cgo-in-test failures) but the user explicitly scoped those sessions to build/report only — fix paths were documented but **not applied**; these remain open, low-priority. - -**Full regression GO** recorded 2026-07-16 (`1cfdac4`/`2b0a37a`), backfilling bug-1444's fix into the record (`2a76940`). - -**Reopened** (`575f928`, user-directed) for three residual gaps left open by the first GO: -- **bug-1443**: `restore_job_array`'s bulk-insert pass could fail partway through, leaving a partial-visibility window — closed with new purely-additive `kvmgr_reserve()`/`xzarray_reserve()` primitives (`dcf9f39`). -- **bug-1444**: `replay_wal`'s job-count tally was blind to array submissions, causing an `APPLY_DIVERGENT` recovery mismatch — fixed (`16747aa`). -- Task 4.2's boot-replay fence gap: Live Apply Resource Fence extended to cover boot replay, not just live apply (`ea9d3d8`). -- A `jobArray` dataset leak on the scheduler receive path, found and fixed the next day (`53bd686`). - -Fresh `/kiro-validate-impl` GO recorded 2026-07-17 (`9cf61c9`), closing stale notes from the first pass. - ---- - -## Jul 17: bug-1445, first GO passes for ha-term-safety/controller-status, Unity harness fix - -- **nix**: repinned `librtti` to fix a `makeInstance` leak-on-init-failure (`8c7d26a`). -- **bug-1445**: a node's very first promotion-eligibility check in a fresh process ran *before* that node's own first `persist_recover()`/`replay_wal()` call, racing the Live Apply Resource Fence. Fixed by making `raft_cb_become_leader()`'s first-boot path re-check `persist_apply_resource_fenced()` immediately after `cluster_init_phase2()` completes (`ff01db3`). -- **ha-term-safety** and **controller-status** both got their first-ever `/kiro-validate-impl` pass — both **GO** (`84fde22`, `c08a348`). -- **Unity test harness**: nested `UNITY_BEGIN()`/`UNITY_END()` calls across 83 `controller/testsuite/*.c` files were masking `controller_unit_test`'s real exit code and mis-attributing failing tests to the wrong source file. Suite-wide fix (`UnitySetTestFile(__FILE__)` per group function) verified via a single consolidated 972-test, 0-failure summary block matching the ground-truth `RUN_TEST` count (`a81f2a2`). - ---- - -## Jul 18: Group 1 validate-impl batch — 9 specs closed, 1 real bug found and fixed - -A backlog audit found 9 HA/Raft specs with 100% of tasks checked but `spec.json` stuck at `phase: "tasks-generated"` — never run through a formal validation closure. Ran `/kiro-validate-impl` on all 9, each with a fresh ASAN build, full standalone test-suite run, and a live 10-round sequential HA soak: - -| Spec | Result | Notes | -|------|--------|-------| -| raft-sim-conformance | GO | TLA+ conformance harness; 293/293 lib tests, corrected a stale buglog entry (bug-1095) | -| libxce-failover | GO | `ready_for_implementation` flag was stale since 2026-06-18, not a real block | -| ha-entity-snapshots | GO | Traced and confirmed a 2026-07-02 task 7.1 blocker was genuinely resolved (real fix chain, not a checkbox) | -| ha-commit-durability | GO | Re-verified rather than trusting stale evidence given later churn; found bug-1446 (test-harness-only, BusyBox `date` lacks `%N`) | -| **raft-drive-async** | **NO-GO → GO** | Live soak crashed round 9 — see bug-1447 below; re-validated GO after the fix | -| ha-wal-replication | GO | 972/972 tests, 10/10 soak | -| ha-heartbeat-committed-propagation | GO | Needed 2 attempts — first agent run died mid-soak to a transient harness API-connection error, clean retry | -| ha-prevote | GO | Term climbed 2→13 cleanly across the soak, zero election-storm churn — exactly PreVote's purpose | -| ha-elect-ctx-lifecycle | GO | Found bug-1449 (harness-only, see below), closed via retry | - -**bug-1447** (the real one): raft-drive-async's soak crashed at round 9 — an ASAN SEGV deep inside OpenSSL, triggered during a reconnect-offload burst. Initial hypothesis (concurrent `SSL_do_handshake` from two `io_worker` threads) was **wrong and retracted** — `iomgr_start()` only ever creates one `io_worker` thread per `iomgr_t`. Root cause, confirmed via a targeted regression test: a **same-batch use-after-free** — `on_standby_connected()` evicts a stale duplicate peer channel mid-batch, but `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, not an event for that same endpoint already sitting in the *current* batch's array. Fixed by ref-counting every endpoint in a batch before dispatch and releasing only after the whole batch completes (mirrors the existing bug-1298 idiom). TDD RED/GREEN confirmed, 2 independent clean 10-round soaks, full suites clean (294/294 + 972/972). - -**bug-1449** (harness-only): `run-ha-soak.sh` had no wait for initial 3-of-3 config convergence before round 1's first kill — a slow cold-start election could let the kill land before config-push reached the standbys, producing a false "no primary elected" failure. Fixed by adding `wait_for_initial_config_convergence()`, mirroring the script's existing reconvergence-wait pattern. - ---- - -## Bug Tally: Jul 13-18 - -### Fixed (9) -1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449 - -### Known residual / open (low-priority, environment- or scope-limited) -- **1437**: accounts-leak test finding — user scoped session to build/report only, no fix applied -- **1439**: `controller.capnp.h` build break — build/report-only session, no fix applied -- **1440, 1441**: Go `cgo`-in-test failures in `apiserver/backend` — build/report-only sessions; fix paths documented (remove `import "C"` from the test file; add the missing `extern` declaration) but not applied - ---- - -## Commits (Jul 13-18) - -``` -8f614b6 chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449) -426728b fix(netio): bug-1447 same-batch endpoint use-after-free in io_worker dispatch -4d2c903 fix(test-harness): bug-1449 wait for initial config convergence before round-1 kill -7512dc8 docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs -781b0f7 chore: update OpenWolf session state + cerebrum learning (Unity test-harness fix) -a81f2a2 fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide -e6956ec chore: update OpenWolf session state -6592f9c chore: update OpenWolf session state + cerebrum learnings (meson parallel-timeout false-positive, /tmp/xce_* cleanup) -3f54ef3 chore: update OpenWolf session state (ha-term-safety + controller-status validation) -c08a348 docs(controller-status): record first /kiro-validate-impl pass -- GO -84fde22 docs(ha-term-safety): record first /kiro-validate-impl pass -- GO -1a0960b chore: update OpenWolf session state (bug-1445 fix) -ff01db3 fix: bug-1445 -- first-boot promotion check races its own first replay -5ddc20f docs: log bug-1445 for job-array-single-wal-record's first-boot promotion gap -fb701c2 chore: update OpenWolf session state (job-array-single-wal-record validation pass) -9cf61c9 docs(job-array-single-wal-record): record fresh /kiro-validate-impl GO, close stale notes -ccee18b chore: update OpenWolf session state -2cfb454 docs(job-array-single-wal-record): mark task 1.3's rtti leak note FIXED -8c7d26a chore(nix): repin librtti to fix makeInstance leak-on-init-failure -9176164 chore: update OpenWolf session state (anatomy, cerebrum, memory, token ledger) -b469b17 docs(job-array-single-wal-record): sync spec docs with phase-7 residual-gap closure -53bd686 fix(job-array-single-wal-record): free jobArray dataset on scheduler receive path -ea8d9d3 fix(job-array-single-wal-record): extend Live Apply Resource Fence to boot replay (task 4.2's gap) -5a81362 docs: mark bug-1443 FIXED (task 7.1, commit dcf9f39) -dcf9f39 fix(job-array-single-wal-record): close bug-1443's bulk-insert partial-visibility gap -575f928 plan(job-array-single-wal-record): reopen spec for residual gap closure -85ce5da docs: track recurrences of pre-existing bugs 1437/1438 during re-validation -2a76940 docs(job-array-single-wal-record): backfill bug-1444 for task 6.0a's replay_wal fix -2b0a37a docs(job-array-single-wal-record): record /kiro-validate-impl GO decision -1cfdac4 test(job-array-single-wal-record): full regression pass - GO -c613849 test(job-array-single-wal-record): chaotic-failover array retry scenario -0d8986a test(job-array-single-wal-record): large-array HA soak scenario -16747aa fix(job-array-single-wal-record): replay_wal's job-count tally was blind to array submissions -b7b90e9 test(job-array-single-wal-record): end-to-end integration test for the compact push -9e0d87f feat(job-array-single-wal-record): API-server-side (Go) local expansion of the compact push -1cbd7df fix(job-array-single-wal-record): wire dispatcher was silently dropping compact array pushes -d168471 feat(job-array-single-wal-record): scheduler-side local expansion of the compact push -e0cec75 feat(job-array-single-wal-record): build the compact API-server push -5e7a4aa feat(job-array-single-wal-record): build the compact scheduler push -1888498 test(job-array-single-wal-record): end-to-end fault-injection test for the live fence -8cb8c25 feat(job-array-single-wal-record): gate promotion on the live apply fence -b47cc66 feat(job-array-single-wal-record): add live, self-healing apply resource fence -cc7e99c feat(job-array-single-wal-record): make apply pump respect apply failures -107e383 test(job-array-single-wal-record): close remaining array expansion gaps -b0c7cfe feat(job-array-single-wal-record): wire array expansion into apply_wal_entry -0c4573b feat(job-array-single-wal-record): wire array expansion into live apply -7c755dc feat(job-array-single-wal-record): implement restore_job_array primitive -78eb983 test(job-array-single-wal-record): close remaining array append/commit gaps -d38ca72 feat(job-array-single-wal-record): wire array submissions into handle_jobsubmit -7c4afbb feat(job-array-single-wal-record): implement single-record array append/commit -76862bd feat(job-array-single-wal-record): harden jobj.c job construction primitives -bc4e70b feat(job-array-single-wal-record): add compact array push capnp variant -2e62e73 feat(job-array-single-wal-record): add WAL_JOB_ARRAY_SUBMIT record type -895a204 docs(spec): ha-consistency-invariants — Criterion 2.9 now satisfied -b321963 chore(wolf): session trail + xce-test-ha cross-run collision guard -40c2351 fix(client): bug-1436 count-aware failover macro for xce_get-family calls -510b566 fix(ha): bug-1435 raft_lock serializes concurrent raft_state_t access -``` - ---- - -## Current State - -| Area | Status | -|------|--------| -| job-array-single-wal-record | **COMPLETE** — fresh GO 2026-07-17, all residual gaps closed | -| ha-consistency-invariants | **COMPLETE** — umbrella + all 4 sub-specs (commit-engine, barrier-rollout, single-leader-role, convergence-recovery) | -| ha-term-safety | GO (first validate-impl pass, 2026-07-17) | -| controller-status | GO (first validate-impl pass, 2026-07-17) | -| raft-sim-conformance | GO (2026-07-18) | -| libxce-failover | GO (2026-07-18) | -| ha-entity-snapshots | GO (2026-07-18) | -| ha-commit-durability | GO (2026-07-18) | -| raft-drive-async | GO (2026-07-18, after bug-1447 fix — was NO-GO same day) | -| ha-wal-replication | GO (2026-07-18) | -| ha-heartbeat-committed-propagation | GO (2026-07-18) | -| ha-prevote | GO (2026-07-18) | -| ha-elect-ctx-lifecycle | GO (2026-07-18) | -| HA sequential soak (xce-test-ha) | Round-1 config-convergence race fixed (bug-1449); reliable across all 9 validation passes today | -| netio.c io_worker dispatch | Same-batch endpoint UAF fixed (bug-1447), regression test added | -| Controller test suite | 972/972 clean, Unity harness attribution fixed suite-wide | -| admin-config-tool | Unchanged — spec finalized, pending implementation | -| scheduler-framework | Unchanged — requirements-only, pending design | -| config-upgrade-snapshot | Unchanged — requirements-only | -| Known open (low-priority) | bug-1437 (test-only), bug-1439 (capnp build break), bug-1440/1441 (Go cgo-in-test) — all build/report-only sessions, fix paths documented not applied | diff --git a/reports/jul19-jul26-2026.md b/reports/jul19-jul26-2026.md deleted file mode 100644 index 250d4222..00000000 --- a/reports/jul19-jul26-2026.md +++ /dev/null @@ -1,225 +0,0 @@ -# Work Summary: Jul 19 – Jul 26, 2026 - -## Branch: `config` | Project: XCE Cluster Management System - ---- - -## Overview - -Week focused on three major workstreams: completing `clusterconfig-schema-split` (Jul 21), shipping `admin-config-tool` (Jul 22), and executing `config-upgrade-snapshot` tasks 8.0–8.9 (Jul 24–26). Two critical bugs discovered during soak testing (bug-1475 standby liveness stall, bug-1476 config-reload UAF) remain open. Two bugs fixed and committed (bug-1472 worker replication, bug-1474 repl slot clobber). - ---- - -## Day-by-Day Summary - -### Jul 19 (Fri) - -- **bug-1452** fixed: `commit_engine_fini`/`persist_stop` teardown race in controller -- **xce-audit-invariants** controller pass completed -- **bug-1453/1454/1455/1456**: scheduler + xcecli dead CA-cert plumbing removed, scheduler HA config bugs fixed - -### Jul 20 (Sat) - -- Continued scheduler/xcecli cleanups from Jul 19 -- Git: `94f50ce6` — fix(scheduler,cli): bug-1453–1456 remove dead CA-cert plumbing - -### Jul 21 (Mon) — `clusterconfig-schema-split` ship day - -- **All 12 tasks (1.1–6.2) completed**, spec marked implementation-complete -- Dead fields removed from `ClusterConfig` capnp schema and `cluster_config_t` -- Two-tier port fallback (`cluster_open_listener`), dead port defaulting removed -- Orphaned `HashZArrayConfig`/`ResmgrConfig` structs removed -- CLI flags for hash-table tunables (optional Req 6) -- 3-node HA live soak verified clean -- Commits: 17 commits from `0205ac3c` through `519e11c7` - -### Jul 22 (Tue) — `admin-config-tool` ship day - -- **All 18 tasks (1.1–5.6) completed**, spec marked implementation-complete -- **New binary**: `xceadmin` — Python-based admin config build/push/discovery tool -- Package: `xceadmin.build` (YAML→capnp), `xceadmin.push` (controller push), `xceadmin.discovery` (auto-detect), `xceadmin.cli` (CLI entry) -- Library: `xcecommon.schema`, `xcecommon.validation`, `xcecommon.mapping`, `xcecommon.wire` -- Schema fix: `Data` struct renamed to `XceData` (unshadow capnp builtin blob type) -- Wired into meson + nix build, HA deploy/test skills, `deploy-ha.sh --build-config` -- Tests: golden-file build determinism, mock-controller push/dry-run coverage -- Commits: 18 commits from `b30a856a` through `b6bf7cb8` - -### Jul 23 (Wed) — Stabilization + Spec expansion - -- **config-upgrade-snapshot** spec expanded: 6 requirements, 26 ACs, 9 tasks (8.0–8.9) -- `blocked_on` cleared to `[]` after bug-1418 re-verified FIXED (10-round chaotic soak PASS) -- **bug-1462** fixed: late-joiner config-push stall — `ha_promotion_config_tag_floor` clamps `committed_config_tag` to `max(safe_tag, self_tag)`. 20+20 round soaks PASS -- Limits.yaml wired through `cluster_config` into controller (real config path) -- xcecli: restored dangling CA-cert cleanup, enabled https support -- **resource-distribution-mapping** spec scaffolded, scope deferred (3 candidate sizes) -- **cc-sdd to Gitea sync** skill added -- **bug-1464** identified: `xce-build.sh` shared build-dir clobber (per-component `rm -rf`) -- Commits: 12 commits from `bf6a9b34` through `cae4ef0c` - -### Jul 24 (Thu) — config-upgrade-snapshot heavy lift - -Tasks 8.0–8.5 executed: - -- **Task 8.0**: HostAdminState/QueueAdminState capnp structs defined, snapshot format version bumped (`9e68335a`) -- **Task 8.1**: Host admin-state frames collected into checkpoint, queue frames migrated to capnp wire (`a71aa3f8`, `70ff3e5b`) -- **Task 8.2**: Embed active configuration bytes in every checkpoint — safety gate (`387ea35c`) -- **Task 8.3**: Restore host admin-state frame safely with create-if-missing (`5b1600a8`) -- **Task 8.4**: Configcompare tool + snapverify extended for byte-identical diff (`1d09631d`) -- **Task 8.5**: Add bug-857 regression test, correct design.md (`2bd44e8e`) -- **Materialization**: upgrade snapshot materialized on config-marker apply (`1da26b04`) -- **Activation fence**: hash-verification gate on materialized snapshot; primary serving/promotion gated on fence pass (`7b507156`, `89b85e16`) -- Removed restart/replay fallback for config activation (`41af0540`) - -Commits: 10 commits from `9e68335a` through `1da26b04` - -### Jul 25 (Fri) — Soak testing, bug discovery, task 8.6–8.9 - -- **Task 8.6**: Live config-upgrade soak infra built (`run-ha-soak.sh --config-upgrade`). Fixed bug-1468 (repl config-push duplicate-connect) and bug-1469 (config tag clamp). **Blocked on bug-1472** (out-of-scope HA replication defect for worker admin-status). -- **Task 8.7**: Fault-inject crash/restart + leadership change tests (`d4af605f`) -- **Task 8.8**: Overlapping config upgrades supersede cleanly (`b78efd03`) -- **Task 8.9**: Live WAL-trim-under-chaos soak infra, blocked on bug-1474 (`8b3e2ab0`) - -**Major bugs discovered (open):** - -| Bug | Severity | Description | -|-----|----------|-------------| -| **bug-1475** | HIGH | Standby catch-up liveness stall: WAL_CHECKPOINT at lsn=12 poisons applied-lsn advertisement. Root cause: `persist_write_checkpoint_marker` does NOT update term index → `on_get_entry_term(12)` returns 0 → raft library takes `ae_append` path not `ae_conflict_truncate`. Primary streams lsn=13 at raft index 12 → ctrl-1 applied_lsn never advances past 12 → infinite `ae_reject_logmismatch` loop. Full mechanism documented in claude-mem obs 16689–16706. | -| **bug-1476** | HIGH | `load_primary_config` concurrent UAF (heap-use-after-free). Two vCPU threads can concurrently free and decode `myconf->cluster_config` via `handle_repl_config_push` → ASAN crash in `decode_Resource_list`. No mutex protecting the read-free-decode sequence. | - -**Build issues:** -- Ninja/Meson parallel-build race (5th recurrence logged) -- ASAN rebuild: 2 full retries needed, all 1033 tests pass clean after retry2 - -Commits: 7 commits from `959b0cda` through `8b3e2ab0` - -### Jul 26 (Sat, early morning) — Bug fixes for 8.6/8.9 blockers - -- **bug-1474** FIXED: two-pass slot allocation (virgin→oldest-disconnect) in `repl.c`. WAL-retention guard clobber prevented standby slot reuse. 2 tests RED→GREEN. Commit: `c0b0c7d1` -- **bug-1472** FIXED: worker admin-status replication gap. `worker_from_capnp` used INSERT semantics (new worker) instead of UPDATE (existing worker). +3 tests. Commit: `7547a8f1` -- **1033 tests all pass** (C + Go + Python layers, ASAN clean) - -Commits: 2 commits - ---- - -## Spec Status - -| Spec | Phase | Status | -|------|-------|--------| -| **config-upgrade-snapshot** | Implementation | Tasks 8.0–8.9: 8.0–8.5 complete, 8.6 blocked on bug-1472 (now fixed), 8.7 done, 8.8 done, 8.9 blocked on bug-1474 (now fixed). Tasks 8.6 & 8.9 need soak re-verification. 6 reqs / 26 ACs. | -| **clusterconfig-schema-split** | **COMPLETE** | All 12 tasks done, live-soak verified | -| **admin-config-tool** | **COMPLETE** | All 18 tasks done, xceadmin binary shipped | -| **resource-distribution-mapping** | Scoping | Scaffolded, scope deferred | - ---- - -## Bug Register (Jul 19–26) - -| Bug | Date | Status | Description | -|-----|------|--------|-------------| -| bug-1452 | Jul 19 | **FIXED** | commit_engine_fini/persist_stop teardown race | -| bug-1453/4/5/6 | Jul 19–20 | **FIXED** | Dead CA-cert plumbing in scheduler/xcecli | -| bug-1418 | Jul 23 | **VERIFIED FIXED** | term desync + apply-divergent stall (re-verified with 10-round chaotic soak) | -| bug-1462 | Jul 23 | **FIXED** | Late-joiner config-push stall | -| bug-1464 | Jul 23 | **KNOWN** | xce-build.sh shared build-dir clobber (mitigation: use `/xce-build all`) | -| bug-1468 | Jul 25 | **FIXED** | Repl config-push duplicate-connect | -| bug-1469 | Jul 25 | **FIXED** | Config tag clamp on promotion | -| bug-1475 | Jul 25 | **OPEN** | Standby catch-up liveness stall (WAL_CHECKPOINT term-index gap) | -| bug-1476 | Jul 25 | **OPEN** | load_primary_config concurrent UAF | -| bug-1472 | Jul 25→26 | **FIXED** | Worker admin-status replication (INSERT→UPDATE semantics) | -| bug-1474 | Jul 25→26 | **FIXED** | Repl slot-reuse guard clobber (two-pass allocation) | - ---- - -## Commits: 63 total (Jul 19–26) - -``` -c0b0c7d1 fix(repl): harden standby slot allocation (bug-1474) -7547a8f1 fix(worker): replicate admin host-control status (bug-1472) -8b3e2ab0 feat(config-upgrade-snapshot): task 8.9 WAL-trim-under-chaos soak infra -b78efd03 feat(config-upgrade-snapshot): task 8.8 overlapping config upgrades -d4af605f test(config-upgrade-snapshot): fault-inject crash/restart + leadership (task 8.7) -959b0cda feat(config-upgrade-snapshot): live config-upgrade soak + bug-1468/1469 (task 8.6) -2bd44e8e feat(config-upgrade-snapshot): bug-857 regression test (task 8.5) -1d09631d feat(config-upgrade-snapshot): configcompare tool + snapverify (task 8.4) -4aeae6e2 test(config-upgrade-snapshot): WAL-trim safety + fault-injection seam -41af0540 feat(config-upgrade-snapshot): remove restart/replay fallback -89b85e16 feat(config-upgrade-snapshot): gate primary on activation fence -7b507156 feat(config-upgrade-snapshot): activation fence materialization+hash -1da26b04 feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply -00cd26a9 feat(config-upgrade-snapshot): decode host/queue admin-state on checkpoint load -5b1600a8 feat(config-upgrade-snapshot): restore host admin-state safely -38735ea5 feat(config-upgrade-snapshot): embed config bytes in checkpoint -70ff3e5b feat(config-upgrade-snapshot): migrate queue admin-state to capnp -a71aa3f8 feat(config-upgrade-snapshot): collect host admin-state frames -9e68335a feat(config-upgrade-snapshot): HostAdminState/QueueAdminState capnp structs -e0c5e8b7 fix(nix): pin capnpc source to immutable commit -cae4ef0c chore: save wolf state -c921e59b feat: resource distribution map project -1705495a fix(admin-config-tool): xceadmin config build warnings -4ec40167 fix(xcecli): restore CA-cert cleanup, enable https -ff1be480 merge -b87d5e68 chore: commit wolf status -60c25c39 chore(openwolf): update session memory/buglog/anatomy -8ee98bee docs(config-upgrade-snapshot): expand spec to 6 reqs/26 ACs -bf6a9b34 feat(tooling): cc-sdd to Gitea sync skill -18e7a0e9 feat(controller): wire limits.yaml through cluster_config -0649e17f fix(controller): clamp committed_config_tag on promotion (bug-1462) -ee8473bd feat(admin-config-tool): drop ha/data/, match real client TLS -db2b5f17 fix(admin-config-tool): pin xceadmin python3 to meson resolution -d5330fc9 feat(admin-config-tool): wire xceadmin into HA deploy/test skills -893a01db feat(admin-config-tool): install standalone xceadmin binary via meson -e4fd9e1d feat(admin-config-tool): wire xceadmin into meson + nix build -b6bf7cb8 chore(admin-config-tool): mark implementation-complete (18 tasks done) -a1478510 docs(admin-config-tool): commit spec docs and example configs -67cc3611 feat(admin-config-tool): task 5.6 mock-controller push/dry-run coverage -48a5a0b4 feat(admin-config-tool): task 5.4 golden-file build determinism tests -b5040e08 feat(admin-config-tool): tasks 5.1-5.3 mapping/wire/schema coverage -ad27147a feat(admin-config-tool): task 4.2 deploy-ha.sh --build-config -eeda32c6 feat(admin-config-tool): task 4.1 xceadmin.cli -c33ef7d4 feat(admin-config-tool): task 3.3 xceadmin.push -e53c086e fix(schema): rename Data -> XceData -7ed3a394 feat(admin-config-tool): task 3.2 xceadmin.discovery -72798d4a feat(admin-config-tool): task 3.1 xceadmin.build -77ee831e feat(admin-config-tool): task 2.4 xcecommon.wire -666a838f feat(admin-config-tool): task 2.3 xcecommon.mapping -3a2094a4 merge: job_peek_lchen -dbf5c6c5 feat(admin-config-tool): task 2.2 xcecommon.schema -12b13a54 feat(admin-config-tool): task 2.1 xcecommon.validation -798f19b8 feat(admin-config-tool): task 1.3 nix + xce-build.sh integration -2e4ef4e2 feat(admin-config-tool): task 1.2 shared exception taxonomy -b30a856a feat(admin-config-tool): task 1.1 xceadmin package skeleton -519e11c7 chore(controller): clusterconfig-schema-split mark complete -a74eb008 docs(controller): clusterconfig-schema-split close validation gaps -553da7c7 feat(controller): clusterconfig-schema-split task 6.2 CLI tunables -c3107b95 feat(controller): clusterconfig-schema-split task 6.1 CLI flags -edd79562 chore(controller): clusterconfig-schema-split task 5.3 single-node live-verified -3e01fb1b chore(controller): clusterconfig-schema-split task 5.2 3-node HA soak -f503a43d feat(controller): clusterconfig-schema-split task 5.1 regenerate cluster.bin -29f9b026 chore(controller): clusterconfig-schema-split task 4 build+test verified -ba6fdc57 feat(controller): clusterconfig-schema-split task 3 fix test fixtures -1fa78443 feat(controller): clusterconfig-schema-split task 2.3 remove dead port default -81fecb49 feat(controller): clusterconfig-schema-split task 2.2 remove candidates debug-dump -8f047d93 feat(controller): clusterconfig-schema-split task 1.3 remove orphaned structs -1229f9a9 feat(controller): clusterconfig-schema-split task 2.1 two-tier port fallback -0205ac3c feat(controller): clusterconfig-schema-split task 1.2 remove dead members -51524820 feat(controller): clusterconfig-schema-split task 1.1 remove dead fields -94f50ce6 fix(scheduler,cli): bug-1453/1454/1455/1456 dead CA-cert + HA config -619f41f7 chore: update OpenWolf session state (bug-1452 fix) -d67ccc22 fix(controller): bug-1452 commit_engine_fini/persist_stop teardown race -e1f4d676 chore: update OpenWolf session state (xce-audit-invariants pass) -``` - ---- - -## Next Steps - -1. **Soak re-verify tasks 8.6 and 8.9** with bug-1472 + bug-1474 fixes applied. Both blocking bugs now fixed, but soak must pass end-to-end. -2. **bug-1475** needs root-cause fix: `persist_write_checkpoint_marker` must update term index so `on_get_entry_term` returns correct term for WAL_CHECKPOINT slot. -3. **bug-1476** needs mutex guard on `load_primary_config`'s `myconf->cluster_config` read-free-decode sequence. -4. **Task 8.6** blocked_on cleared (bug-1472 fixed), can proceed with worker admin-status rotation verification. -5. **Task 8.9** blocked_on cleared (bug-1474 fixed), can proceed with WAL-trim-under-chaos verification. - ---- - -*Generated 2026-07-26 | Branch: `config` | 63 commits, 4 bugs fixed, 2 specs shipped, 2 bugs open* diff --git a/reports/jul27-aug02-2026.md b/reports/jul27-aug02-2026.md deleted file mode 100644 index b4888064..00000000 --- a/reports/jul27-aug02-2026.md +++ /dev/null @@ -1,85 +0,0 @@ -# XCE Project Status — Jul 27 – Aug 01, 2026 - -## TL;DR - -HA/Raft correctness work dominated the week: three permanent-wedge bugs closed (catch-up stall, quorum-loss deadlock, config-push race/UAF), `raft-adapter-conformance` spec landed complete, `admin-config-tool` and `config-upgrade-snapshot` both unblocked and near done. One exposed credential fixed. Scheduler work restarted: `scheduler-framework` decomposed into three child specs (`scheduler-core`, `scheduler-allocation`, `scheduler-preemption`), all currently at requirements-only (not yet approved for design). - -## Commits by day - -**Jul 27** -- `add report`, merge `working`→`config`, fix librtti package leak - -**Jul 28** — HA wedge fixes -- fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478) -- fix(ha): close permanent standby catch-up wedge after WAL_CHECKPOINT boundary (bug-1475) -- fix(config): serialize `load_primary_config` against concurrent invocations (bug-1476) -- fix(ha): retry `cluster_init_phase2` on failure instead of latching `phase2_done` unconditionally (bug-1477) -- docs: re-block config-upgrade-snapshot task 8.6 on bug-1477, then clear stale admin-config-tool blocker (bug-1468 already fixed) - -**Jul 29** — raft-adapter-conformance build-out -- feat: term-adoption wired into primary-side config-push reply handler -- feat: `term` field added to `ReplConfigPushReply` schema -- feat: explicit reply on every config-push outcome -- feat: route post-connect vote/PreVote failures to production tally -- fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479) -- test: PreVote post-connect failure routing + fast-quorum coverage; full config-push reply integrity; dormant AppendEntries sender non-regression; WAL catch-up reject reply-construction lock -- chore(wolf): logged bug-1480, bug-1481, bug-1482, bug-1483 (audit findings, not all fixed yet) - -**Jul 30** — conformance close-out + admin-config-tool E2E -- test(admin-config-tool): live single-node E2E test for config push -- fix(admin-config-tool): unwrap Request reply envelope before decoding as RequestReply -- docs(admin-config-tool): task 6.3 blocked pending flake reproduction (bug-1485) -- test(raft-adapter-conformance): batch payload persist-failure propagation regression lock -- docs: confirmed live soaks for tasks 7.1–7.4 (vote/PreVote post-connect failure, config-push rejection); extended controller conformance record; logged bug-1484 - -**Jul 31** -- fix(ha): retry chaotic-mode `verify_classify` before counting job loss (bug-1487) — false-positive job-loss fixed in soak harness, not controller -- fix(security): replace exposed `rshen/Letmein` admin credential in `auth.yaml` with placeholder; added `src/docs/deployment.md` full ops guide -- fix(waldump): decode `WAL_CONFIG_MARKER`, distinguish trimmed gaps from real loss -- docs(config-upgrade-snapshot): host admin-state frames, capnp migration, **bug-1418 blocker cleared** -- docs(raft-adapter-conformance): final design.md/research.md + spec.json state committed — **spec complete** -- **add scheduler related projects** — `scheduler-framework` decomposed into `scheduler-core` (1311-line requirements.md), `scheduler-allocation` (619-line), `scheduler-preemption` (109-line); all requirements-generated, none approved yet -- add wolf state (session memory checkpoint) - -**Aug 01** — no commits yet; working tree has only uncommitted `.wolf/` session-state churn (memory.md, token-ledger.json, hooks/_session.json). - -## Bugs opened/fixed this window - -| Bug | Status | Summary | -|---|---|---| -| bug-1475 | **Fixed** (2026-07-27/28, 2 findings) | Standby catch-up permanently wedged across a WAL_CHECKPOINT boundary — 3-part fix (config_repl.c prev-tracker, ha.c JIT resync, persist.c `TERM_INDEX_UNSET` sentinel) plus a second snapshot-install variant of the same class, closed with the same sentinel | -| bug-1476 | **Fixed** (2026-07-28, re-fixed same day) | Heap-use-after-free race in `load_primary_config` — first landed fix left the lock un-acquired (RED), re-fixed for real with `myconf->lock` + decode-into-local-then-swap | -| bug-1477 | **Fixed** | `cluster_init_phase2` latched `phase2_done` on failure, blocking HA bootstrap retry | -| bug-1478 | **Fixed** | Duplicated replication-target formula caused permanent catch-up wedge | -| bug-1479 | **Fixed** | Permanent quorum-loss deadlock on log-conflicted standby | -| bug-1480 | **Fixed** (2026-07-29 09:43, `b7828eed`) | Config-push reply missing explicit reject+term — fixed same day, buglog `fix:` field was stale (written 07:05, before landing) until corrected 2026-08-01 | -| bug-1481 | **Fixed** (2026-07-29 13:53, `fce73b5b`) | `vote_ep_err_handler` bypassed production tally on post-connect failure — fixed same day, buglog `fix:` field was stale until corrected 2026-08-01 | -| bug-1482 | **Fixed** | Not a real UAF — erroneous `free_packet` call in a test (pkt already owned/consumed by `xce_decode`); test + doc comment corrected | -| bug-1483 | Open | Pre-existing test-mode leak in `ha_arm_heartbeat_timeout_vtask` (c==NULL calloc never freed); workaround only (stub+reset hook in every LOST-path test), not fixed at root | -| bug-1484 | Open, explicit residual | No test drives real `repl_ack_decode_and_dispatch` term-adoption for heartbeat-reply/WAL-NACK vectors; Req 7.5 "already verified" claim rests on unverified ground; out of scope for raft-adapter-conformance, needs a follow-up task | -| bug-1485 | Open, flaky, 24 occurrences | `test_live_config_push_against_primary_reaches_success_path` — admin-config-tool task 6.3 blocked on this | -| bug-1486 | Documented (workflow, no code fix) | Concurrent `xce-build.sh` sessions clobber shared build dir (rm -rf race) — mitigation: check `ps auxf` before trusting a stale background build log | -| bug-1487 | **Fixed** | 30-round chaotic soak false-positive job-loss — actually a 6.2s Figure-8 commit-barrier window, not real loss; fix adds bounded retry to `verify_classify` in the soak harness | - -## Spec status changes - -- **raft-adapter-conformance** — requirements/design/tasks all approved, `ready_for_implementation`, `blocked_on: []`. Final design.md/research.md committed Jul 31. Effectively **done**. -- **admin-config-tool** — both historical blockers (bug-1468 dispatch-inversion, bug-1461 capnp `Data` shadowing) now marked RESOLVED; task 6.3 HA E2E test still blocked on a flaky-test repro (bug-1485). -- **config-upgrade-snapshot** — `blocked_on` cleared to `[]` (bug-1418 confirmed fixed, no regression); host admin-state frames and capnp migration work added. -- **scheduler-framework** — formally decomposed (phase: `decomposed`) into three child specs, all at `requirements-generated` / not approved: - - `scheduler-core` — 1311-line requirements.md - - `scheduler-allocation` — 619-line requirements.md - - `scheduler-preemption` — 109-line requirements.md - - Next step for this line of work is design approval on whichever child spec proceeds first. - -## Notable non-bug items - -- **Security**: a live, checked-in `cluster_admin` credential (`rshen/Letmein`) in `auth.yaml` was replaced with a placeholder — flag for credential rotation if that value was ever deployed anywhere real. -- **New doc**: `src/docs/deployment.md` — full ops guide for building/running all four daemons + xceadmin, written from direct source verification + a live single-node deploy; explicitly documents production gaps (no process supervision, plaintext non-controller traffic, exposed dev creds). -- **waldump**: now decodes `WAL_CONFIG_MARKER`, so trimmed-segment gaps are no longer misreported as real data loss during investigations. - -## Open threads heading into next session - -1. `scheduler-core`/`scheduler-allocation`/`scheduler-preemption` need requirements review + design phase — nothing is implementation-ready yet. -2. bug-1480/1481/1482/1483/1484/1485 remain open (raft-adapter-conformance audit residue + one ASan UAF in netio.c). -3. Uncommitted `.wolf/` session-state changes sitting in working tree (harmless, not code). diff --git a/reports/jun29-jul04-2026.md b/reports/jun29-jul04-2026.md deleted file mode 100644 index bb3fa780..00000000 --- a/reports/jun29-jul04-2026.md +++ /dev/null @@ -1,48 +0,0 @@ -# Work Summary: June 30 – July 4, 2026 - -## Monday June 30 — Raft Election + Bug Fixes -- **Phase 5 RED** shipped with heartbeat-freshness guard -- **3 concurrency bugs fixed**: bug-1282 (ABBA deadlock netio.c), bug-1286 (NULL ep→owner), bug-1298 (endpoint UAF refcount) -- **Tasks 5.5–5.8 complete**: election timeout resonance fix (per-campaign jitter [T,2T], bounded terms), ASAN soak 6/6 rounds green (182 elections, zero runaway) -- Task 5.9 backfill identified blocker for 7.2 acceptance soak - -## Tuesday July 1 — Spec Completion + Gap Audit -- **Raft_tla**: 12/13 tasks done (task 2 checkpoint refined via RED-reject-remediate, tasks 5-6 restore pipeline, task 7 ASAN regress shipped) -- **Ha-entity-snapshots**: 13/13 complete (checkpoint write, rotation guard, restore-before-advance) -- **Gap audit**: 3 blockers found blocking raft-ha-core-safety → raft-drive-async 7.2 acceptance gate (2 unscaffolded specs, 1 design-unapproved) - -## Wednesday July 2 — Soak Validation + More Bug Fixes -- **Raft-ha-core-safety phase 7.2** soak green; raft-drive-async task 3.3 unblocked -- **3 more bugs fixed**: bug-1386 (silent-send), bug-1389 (65KB chunk corruption), bug-1381 (endpoint UAF) -- **Diagnostic tools** added: snapverify/snapdiff to xce_test -- **Ha-commit-durability phases 6.1–6.4** ext4-validated (5/5 rolling+cold-restart, zero committed-job loss) -- Enabled `g_drive_async_configpush_offload` - -## Thursday July 3 — Raft-Apply-Decoupling Tasks 6.1–6.3 -- **Task 6.1** (ordered apply pump): committed; discovered **bug-1409** (apply_pump_drain unbounded busy-loop, missing return-code check) -- **Task 6.2** (fire continuation post-apply): rejected by adversarial review for **bug-1415** (multi-vcpu race: apply-pump orphans pending-apply registrations → hang); healed at `persist_register_pending_apply()` with self-heal check; committed -- **Task 6.3** (apply/notify timing integration): approved, committed - -## Friday July 4 — Tasks 6.4 + 7.1 + Bug Cleanup -- **Task 6.4** (quorum-loss catch-up + 4-vcpu burst tests): committed (2de410c), 30/30 consecutive runs pass -- **Task 7.1** (conformance harness extension): apply-after-commit + idempotent-reapply assertions in raft_sim.c; RED phase verified bidirectionally; 287/287 tests pass; adversarial review launched -- **Bug triage**: bug-1400 (submit.start_info leak), bug-1401 (xce_free_ctrlstatus macro freed inner fields not struct), bug-1396 (harness instrumentation) -- **buglog pruned**: 1406→721 entries (659 junk removed) -- **ha-consistency-invariants** spec closed - -## Key Metrics - -| Metric | Count | -|---|---| -| Commits | 7+ (across raft-apply-decoupling chain) | -| Bugs fixed | 9 (1282, 1286, 1298, 1381, 1386, 1389, 1400, 1401, 1415) | -| Bugs found+logged | 2 (1409, 1415) | -| Specs closed | 1 (ha-consistency-invariants) | -| Tests passing | 287/287 (xcelib_test) | -| ASAN soaks | 6-round election + 30-round apply-pump, all green | - -## Insights - -- **Adversarial review caught bug-1415**: a self-disclosed race condition the implementer flagged during task 6.2 — the review process worked as designed (RED→reject→remediate→GREEN) -- **buglog hygiene matters**: 659 junk entries (~47% of total) were removed — stale/duplicate observations degrade memory search quality and waste context tokens -- **The task chain dependency is tight**: 6.1→6.2→6.3→6.4 each depend on the prior's API surface (e.g., 6.2's `persist_take_pending_apply()` is 6.2's hand-off point to 6.3)