71 lines
9.7 KiB
Markdown
71 lines
9.7 KiB
Markdown
# Implementation Plan
|
|
|
|
> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
|
|
> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).
|
|
|
|
- [ ] 1. Foundation: status reply schema + codegen
|
|
- [x] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema
|
|
- Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
|
|
- Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
|
|
- Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
|
|
- _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_
|
|
|
|
- [ ] 2. Controller: status handler + any-role dispatch
|
|
- [x] 2.1 Implement the controller status handler
|
|
- Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
|
|
- Route the new opcode to this handler in the controller request dispatch switch.
|
|
- TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
|
|
- Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
|
|
- _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_
|
|
- _Boundary: handle_ctrlstatus_
|
|
- [x] 2.2 Allow the status query through the HA dispatch gate in every role
|
|
- Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
|
|
- TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
|
|
- Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
|
|
- _Requirements: 7.1, 7.2_
|
|
- _Boundary: ha_dispatch_gate_
|
|
- _Depends: 2.1_
|
|
|
|
- [ ] 3. libxce client API
|
|
- [ ] 3.1 Add the failover-aware any-node status call
|
|
- Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
|
|
- TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
|
|
- Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
|
|
- _Requirements: 1.1, 7.1, 7.2, 7.3_
|
|
- _Boundary: libxce client_
|
|
- _Depends: 1.1, 2.1, 2.2_
|
|
- [ ] 3.2 Add the authoritative reach-the-primary status call
|
|
- Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
|
|
- TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
|
|
- Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
|
|
- _Requirements: 2.3, 9.1, 9.2_
|
|
- _Boundary: libxce client_
|
|
- _Depends: 3.1_
|
|
|
|
- [ ] 4. Test client command + wait-primary integration
|
|
- [ ] 4.1 Add the `ctrlstatus` client command
|
|
- Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
|
|
- Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
|
|
- _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_
|
|
- _Boundary: cmd_ctrlstatus_
|
|
- _Depends: 3.1_
|
|
- [ ] 4.2 Switch wait-primary to the authoritative status probe
|
|
- Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
|
|
- Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
|
|
- _Requirements: 9.1, 9.2, 9.3, 9.4_
|
|
- _Boundary: cmd_wait_primary_
|
|
- _Depends: 3.2_
|
|
|
|
- [ ] 5. Validation (live cluster)
|
|
- [ ] 5.1 End-to-end status + authoritative wait-primary on a live cluster
|
|
- Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
|
|
- Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
|
|
- _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_
|
|
- _Depends: 4.1, 4.2_
|
|
|
|
## Implementation Notes
|
|
- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`.
|
|
- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
|
|
- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.
|
|
- 1.1 LEARNING (affects later tasks): the `$C.mapname(...)` capnp types are NOT auto-generated in this schema — the generated `controller.capnp.c` references HAND-AUTHORED C typedefs in `src/include/typemap.h` (same convention as `xce_clusterconf_t`). So `xce_ctrlstatus_t`/`xce_ctrlpeer_t` (and the `xce_reply_t.data.ctrlstatus` union member) live in `typemap.h`, verified field-for-field against the regenerated codec. `OPCode_ctrlstatus = 40`, `RequestReply_data_ctrlstatus = 12`. `configChecksum` is `xce_data_t*`. The handler/libxce tasks build/encode against these typemap.h types.
|
|
- 2.1 LEARNING + SCHEMA AMENDMENT: capnp `Data` in THIS schema maps to the project's STRUCTURED `xce_data_t` union (table-switched), NOT raw bytes — `encode_Data_ptr` can't serialize a SHA-256 blob. So `configChecksum @6` was changed `Data → Text` (a 64-char lowercase-hex string of the full 32-byte `config_sha256`); typemap member is `char*`; freed in `ctrlstatus_free_fields`. Ordinal @6 unchanged, no released decoder yet → safe. Handler lives in NEW `src/controller/ctrlstatus.c` (`ctrlstatus_fill`/`ctrlstatus_free_fields`/`handle_ctrlstatus`), proto in `src/controller/include/cluster.h`, routed at `cluster.c` opcode switch, test `src/controller/testsuite/test_ctrlstatus.c` (registered in `controller_test.c`, built via `controller/meson.build`). `primary_host` read under `ha->lock`; peer name/host strings are BORROWED from myconf (not freed). The libxce client (3.x) decodes `configChecksum` as Text.
|