Commit graph

56 commits

Author SHA1 Message Date
c3caab49a6 feat(election-iomgr-refactor): add on_connected fire-once callback to endpoint_t
Add void (*on_connected)(endpoint_t *ep) field to endpoint_t with
endpoint_set_on_connected() helper. io_handler fires it exactly once
immediately after ssl_handshake_done=1 then nulls the pointer.

Enables ha_start_election_async to send OP_requestVote as soon as
the TLS handshake with a peer completes — no blocking SSL_connect.

6 new unit tests pass; controller suite 1/1 OK.
2026-05-30 07:34:08 +00:00
4737b0f8a3 feat(election-iomgr-refactor): add delay_task_ms() for ms-granularity one-shot tasks
Add millisecond-granularity delay_task_ms() to delaymgr using
clock_gettime(CLOCK_MONOTONIC) for deadline computation. Extends
delay_task_t with nextrun_ts field; delay_worker distinguishes
ms-tasks from second-tasks via non-zero nextrun_ts. Backward
compatible with existing delay_task callers.

4 delay tests pass (2 new: tolerance and invalid-args).
Provides the timeout primitive for election and committing watchdogs.
2026-05-30 07:15:54 +00:00
0ec0ffa7e3 feat(ha-config-bootstrap): implement xce_retry_config_t and xce_request_with_retry (tasks 10.1, 10.2) 2026-05-29 03:00:34 +00:00
bb8e531a12 test(ha-config-bootstrap): add RED-phase tests for xce_request_with_retry (task 10.1) 2026-05-29 02:45:24 +00:00
a07a74a943 Merge branch 'persistent_ha' into scheduler 2026-05-12 10:52:46 +00:00
450a4d467c fix: sel_key_cmp_fn padding-byte mismatch in kvtab lookups
sel_key_cmp_fn compared keys with memcmp on the whole sel_node_key_t
struct, including the 4-byte padding between type and op1.  The stack-
allocated search key in gc_sweep_select_nodes can have different padding
bytes than the heap-allocated copy stored by key_add, causing key_del
to miss the entry and leave a dangling pointer in the CSE intern table.

Replace memcmp with field-by-field comparison on type, op1, op2.

Also fix GC sweep tests that asserted pointer inequality after re-intern:
calloc may lawfully reuse the freed address, so pointer inequality does
not reliably indicate a successful sweep.  Track the swept node's
node_id before freeing and assert the re-interned node gets a fresh id.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-12 09:53:59 +00:00
fd9cfce1c1 feat: P2.5-5/6 add --peers/--ca-cert parsing to worker+scheduler with validation
Worker (P2.5-5):
- Added parse_peers() static helper to src/worker/src/main.c
- Switched getopt to getopt_long to support --peers and --ca-cert long options
- When --peers is given: parses into xce_cluster_config_t, calls
  xce_cluster_config_validate(), logs INFO with candidate count/identity/ca_cert
- If validation fails: prints clear error and exits(1)
- WARN logged when candidates_count > 3
- Backward compatible: no --peers falls back to existing single-controller path

Scheduler (P2.5-6):
- Added parse_peers() static helper to src/scheduler/src/main.c
- Extended argp struct with --peers (-P) and --ca-cert (-C) options
- Same validation/logging pattern as worker
- Backward compatible: no --peers falls back to existing single-controller path

Tests (new):
- src/lib/testsuite/test_cluster_parse.c: 8 unit tests for parse_peers logic
  (single entry, multiple entries, no-name format, empty/null/missing-port errors,
  name-stripped correctly, MAX_HA_CANDIDATES cap respected)
- Registered in test.c and meson.build
- All 8 tests GREEN; 3 pre-existing failures in unrelated GC/CSE tests unchanged
2026-05-08 02:32:56 +00:00
0c11989da7 feat: P2.5-1/2/2a/3/8 xce_cluster_config_t, xce_connect_cluster, xce_reconnect_step
Add libxce cluster discovery and auto-reconnect state machine:

- src/include/xce/cluster.h: xce_candidate_t, xce_cluster_config_t,
  xce_reconnect_ctx_t types; full API with test-seam hook
  xce_cluster_set_connect_fn()
- src/lib/cluster.c: xce_cluster_config_validate() (rejects invalid
  candidate counts, empty hostname/port, missing identity/ca_cert),
  xce_connect_cluster() (candidate fallback with single sweep),
  xce_reconnect_step() state machine (IDLE→CONNECTING→CONNECTED,
  with exponential backoff; nanosleep skipped when mock is active)
- src/lib/meson.build: add cluster.c to libxce_src and
  test_cluster.c to libxce_test_src (P2.5-2a)
- src/lib/testsuite/test_cluster.c: 9 TDD tests covering validation,
  first-candidate success, fallback, all-fail NULL return, reconnect
  to CONNECTED, and backoff state transition (P2.5-8)

All 9 cluster tests pass; 3 pre-existing unrelated failures unchanged.
2026-05-08 02:16:08 +00:00
f155b1ef97 feat: xce_disable_hook() + _Atomic hook_fn_t + NULL-slot reuse (P1-2a)
- hook_item_t.fn changed to _Atomic hook_fn_t (C11 stdatomic)
- xce_disable_hook(): atomically stores NULL into the slot fn field;
  returns 0 on success, -ENOENT if fn not found, -EINVAL on bad args
- xce_add_hook(): scans for NULL (disabled) slots before allocating;
  uses atomic_store_explicit(release) when writing fn pointer
- xce_run_hooks(): atomic_load_explicit(acquire) + NULL guard so
  disabled slots are silently skipped without disrupting the loop
- Add src/lib/testsuite/test_hooks.c with 3 TDD tests:
  test_disable_hook_stops_invocation
  test_disable_hook_returns_enoent_if_not_found
  test_add_hook_reuses_null_slot
- Register test_hooks in meson.build and test.c runner

Required by ha_demote_to_standby() to atomically disable persistence
hooks when the controller steps down from primary, preventing unbounded
hook list growth across promote/demote cycles (resolution A6-015,
A7-004, A7-016).
2026-05-07 13:55:42 +00:00
813a5a3359 test: add KVMGR_KEY_INT coverage to kvmgr_capture_keys tests
Adds test_capture_keys_int_kvmgr() to verify integer-keyed kvmgr path.
Tests that kvmgr_capture_keys() correctly formats integer keys as decimal
strings via snprintf (PRId64), and validates key_len == strlen(key) + 1.

All 4 kvmgr_capture tests pass.
2026-05-07 11:12:11 +00:00
eaf1801d6d feat: implement kvmgr_capture_keys for lock-free checkpoint serialization (P0-14a)
Add kvtab_foreach walk primitive (P0-14) to kv.c/kv.h, then implement
kvmgr_capture_keys in kvmgr.c that deep-copies all key→index mappings
into a malloc'd slot_key_t[] array safe to read after the write lock is
released during checkpoint serialization (Part 3.8).

Key design points:
- Both kvtab_foreach and kvmgr_capture_keys take var (RTTI data pointer)
  and use RTTI_DATA_TO_OBJ internally — the RTTI framework's makeInstance
  returns the data region pointer, not the meta_obj_t header pointer
- String keys are copied via strncpy into slot_key_t.key[XCE_MAXNAME_LEN]
- Integer keys are formatted as decimal strings via snprintf
- Two-pass approach: count live entries first, allocate, then fill
- Tests cover string kvmgr (3 entries), empty kvmgr, and deep-copy safety
2026-05-07 11:07:41 +00:00
4c591a11d3 feat: implement xzarray COW freeze/thaw with pre-grown shadow array (P0-2)
- Add in_cow flag + shadow/shadow_caps fields to xzarray_private_t
- Convert xzarray_item_t.inuse from bitfield to _Atomic _Bool; update
  all read/write sites to use atomic_load/store_explicit
- Implement xzarray_freeze(): O(1) ptr swap, pre-grows active array
  1.5x+16 to avoid realloc during COW, sets pos past shadow range
- Implement xzarray_thaw(): merges shadow into active (shadow wins only
  for slots not updated during COW), supports optional retain_fn GC
- xzarray_get_vec(): falls back to shadow during COW so reads remain
  valid between freeze and thaw
- Add COW guard assertion in xzarray_clear()
- Wire vfreeze/vthaw into xzarray_type interface
- Tests: 5 new COW tests all green; 4 pre-existing failures unchanged
2026-05-07 09:46:28 +00:00
6d4d8eb81d feat: rename delaymgr precision→precision_ms, replace sleep() with nanosleep() (P0-2b)
- Rename `precision` field to `precision_ms` in delaymgr_t struct (delay.h)
- Replace `sleep(mgr->precision)` with `nanosleep()` for 10ms granularity (delay.c)
- Add `#include <time.h>` to delay.c (already in delay.h)
- Update delaymgr_init() to store precision_ms parameter
- Update all call sites: cluster.c (2→200ms), controller/testsuite/test.c (2→2000ms),
  lib/testsuite/test_delay.c (1→1000ms)
- Add TDD test test_delaymgr_precision_ms_field verifying field exists and stores correctly
- delay_task() delay parameter remains in seconds; only sleep granularity changes
2026-05-07 09:08:03 +00:00
c78f9cf91c feat: add vfreeze/vthaw slots to intf_varray_t; designated initializers (P0-0b)
rtti header already present at rtti-0.0.2 with vfreeze_fn_t, vthaw_fn_t,
retain_fn_t, and the two new slots in intf_varray_t. No header modification
needed.

Convert intf_varray_t compound-literal initializers in zarray.c and kvmgr.c
to designated style with explicit .vfreeze = NULL and .vthaw = NULL slots.
kv.c has no intf_varray_t initializer.

Add test_zarray_cow.c with test_freeze_thaw_slots_exist which verifies the
struct layout has the slots accessible and that they are NULL until P0-2
wires the real implementations. Wire test into meson.build and test.c.

Full clean rebuild confirmed: 27/28 targets linked, no new test failures.
Only the 4 pre-existing failures remain.
2026-05-07 08:54:18 +00:00
e0e2f56c81 feat: add persistence/HA system constants to config.h with tests (P0-A)
Adds 17 compile-time constants covering WAL format, HA topology,
snapshot sizing, and operational defaults. test_persistence_constants
in test_config.c validates every constant value; all 6 config tests
pass (4 pre-existing unrelated failures unchanged).
2026-05-07 08:41:19 +00:00
d7afd72e5b fix front end issues 2026-05-01 21:22:45 +08:00
177cf45af5 docs: mark Phase 6 complete in phase6-scheduler-integration-plan 2026-04-30 07:17:32 +00:00
1eb4eb1335 fix: use _Atomic flag for sel_alloc_thread_id registration, move reg before loop, add wrong-thread guard test 2026-04-30 06:15:01 +00:00
b722e5cb3d feat: add sel_alloc_thread_id ownership check in eval_select and gc_sweep (6-10) 2026-04-30 06:04:21 +00:00
4e177047c9 fix: correct SEL_NUM bit-cast in gc test; assert n1 swept; add recompact to protocol comment 2026-04-30 03:55:15 +00:00
9a19a8f932 feat: export gc_clear_marks + gc_mark_select_tree; remove clear from gc_sweep (phase 6)
- Add gc_clear_marks(): iterates intern table and zeroes all gc_mark bits
- Rename static gc_mark_tree → public gc_mark_select_tree (no attribute((unused)))
- Remove Phase 6 guard (live_jobs != NULL early return) from gc_sweep_select_nodes
- Remove the mark-clear loop from gc_sweep_select_nodes; add (void) casts for now-unused params
- Declare gc_clear_marks and gc_mark_select_tree in resreq.h with protocol comment
- Add TDD test test_gc_clear_marks_exported verifying the exported primitives
- Prefix all existing gc_sweep_select_nodes(NULL, 0) call sites in test_select.c with gc_clear_marks()
2026-04-30 03:40:24 +00:00
b38babb7d6 feat: implement Phase 4d const-fold transform for select AST
Adds sel_const_fold_pass() — a single-pass bottom-up const-fold that
runs after the Phase 4c flatten loop in parse_select():

  4d-1  EQ/NE with either operand SEL_ANY → SEL_TRUE / SEL_FALSE
        (checks both child slots — commutativity sort makes SEL_ANY
        position non-deterministic after makeSelectNode interning)
  4d-2  Identity laws: TRUE&&X→X, X&&TRUE→X, FALSE||X→X, X||FALSE→X
  4d-3  Annihilation laws: FALSE&&X→FALSE, TRUE||X→TRUE (symmetric)
  4d-4  Complement laws: !TRUE→FALSE, !FALSE→TRUE

Memory semantics:
  level==0 (API server): frees orphaned nodes via free_select_node /
    sel_free_tree; OOM from makeSelectNode returns original node safely
  level>0  (CSE/scheduler): never manually frees; annihilation folds
    return the already-interned left/right singleton directly (no
    redundant makeSelectNode call); OOM guard unconditional so CSE
    mode also degrades gracefully instead of returning NULL

9 new TDD tests added to test_select.c (RED→GREEN verified).
All 166 tests pass (157 prior + 9 Phase 4d).
2026-04-29 08:24:04 +00:00
dcb1b4fe25 feat: P0-2a C-side plumbing for host control
Add the full call chain for host control operations from the Go API
server through the C library to the controller:

- schema/controller.capnp: HostControl struct + hostctrlmsg @21
  in Request.body union
- include/typemap.h: xce_hostctrl_t* hostctrlmsg in body union
  (placed after @20 clrpreasonmsg, matching capnp ordinal)
- include/client.h, xceapi.h, request.h: declarations for
  xcecli_hostctrl, xce_hostctrl, xce_hostctrl_req
- lib/client.c: xcecli_hostctrl (validate + delegate to xce_hostctrl)
- lib/xceapi.c: xce_hostctrl (validate ctx state + send + wait reply)
- lib/request.c: xce_hostctrl_req (validate hostname/empty/nil,
  accept only HOST_OPEN/HOST_CLOSE, reject HOST_TAG/UNTAG)
- lib/helper.c: Request_body_hostctrlmsg cases in
  xce_request_body() and xce_request_msg_name()
- controller/cluster.c: OPCode_hostcontrol dispatch
- controller/include/cluster.h: do_hostcontrol_op declaration
- controller/worker.c: do_hostcontrol_op + fix pre-existing
  goto-clean bug in worker_ctrl that leaked write lock and
  reference on HOST_OPEN/HOST_CLOSE no-op paths

TDD: 8 xce_hostctrl_req validation tests (null ep, null ctrl,
null/empty hostname, HOST_DO_NONE no-op, HOST_TAG/UNTAG reject,
negative action reject, HOST_OPEN/CLOSE accept via xce_request).

116/116 xcelib tests pass. CGO_ENABLED=0 Go backend tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 07:46:41 +00:00
43dc12a36c save works 2026-03-06 18:20:30 +08:00
3f74f14687 save works 2026-03-04 18:18:53 +08:00
acfbec875b save works 2026-03-03 16:49:26 +08:00
f4b1ba4cd7 save works 2026-03-02 12:14:30 +08:00
b748e30d20 save works 2026-02-28 17:55:50 +08:00
598d381379 improve xkvmgr and testsuite for scheduler 2026-02-26 14:39:12 +08:00
2ee87d4ee6 use librtti for worker/queue/resource 2026-02-25 16:56:50 +08:00
532c68e6e1 - add implement job related objs based on librtti
- add lock()/unlock() to job object
2026-02-23 11:18:45 +08:00
f582742e77 add xzarray_t for replacement of zarray_t finally 2026-02-20 11:25:50 +08:00
ce69347581 - implement xstring by librtti 2026-02-14 14:38:52 +08:00
15fac3cda1 introduce kv 2026-02-13 17:05:41 +08:00
48ea4a0205 - introduce bitset
- add hash_block()
2026-02-13 11:01:33 +08:00
12dcc2a60d - save tmp works
- save works
2026-02-06 17:31:02 +08:00
cd4f3953b2 - use librtti
- Fix issue to xce_get() api cause client side crash
- add parse resreq section
- add rusage parser
- parse resreq with basic support alternative & compound, still miss select
2026-02-05 17:21:27 +08:00
924bc1517b using xvec related api in scheduler 2026-01-16 18:21:37 +08:00
ecf1916d78 support limit 2026-01-12 15:27:54 +08:00
780e9fb313 scheduling using real slots number 2025-12-30 18:13:15 +08:00
a27c8bfe6e enhance zarray library function 2025-12-26 16:23:04 +08:00
5560ae91da pass pend reason back to controller 2025-12-23 15:29:26 +08:00
c12fcf6905 improve ordq 2025-12-22 11:58:04 +08:00
f44ad07116 improve ordq 2025-12-17 15:28:27 +08:00
48ab2e60a2 enhance ordq 2025-12-17 12:14:52 +08:00
05abed1964 enhance ordq 2025-12-16 17:04:04 +08:00
cb0c7fe9e7 merge job rusage and pending reason support 2025-12-16 14:36:28 +08:00
de090dce35 use clang-format to format code 2025-11-25 14:58:09 +08:00
3488b62716 jobid format and parse 2025-11-19 14:44:49 +08:00
8f796b3109 security communication with openssl and simplify netio 2025-11-18 11:26:36 +08:00