Commit graph

116 commits

Author SHA1 Message Date
4903c83f28 feat: add PersistConfig, SnapshotHeader, ReplEnvelope Cap'n Proto schemas (P0-9)
Extends src/schema/controller.capnp with persistence structures needed
for WAL replication and snapshots:

- PersistConfig (8 fields, @0-@7): capnp-serialisable copy of the
  persistence configuration; mapped to capnp_persist_cfg_t to avoid
  collision with the plain-C persist_config_t in persist.h
- SnapshotHeader (9 fields, @0-@8): snapshot manifest with LSN, term,
  timestamp, record counts and checksums; mapped to capnp_snap_hdr_t
- ReplEnvelope (6 fields, @0-@5): replication wire envelope (NOT the
  on-disk format); named to avoid collision with wal_entry_hdr_t and
  the WALEntry_ internal symbol c-capnproto generates; mapped to
  capnp_repl_envelope_t

Also extends existing structs:
- Job.transientSince @11 (UInt64=0, mapname transient_since): set when
  job enters JOB_STAT_TRANSIENT; also adds uint64_t transient_since to
  xce_job_t in types.h
- Worker.deadSince @6 (UInt64=0, mapname dead_since): mirrors existing
  time_t dead_since field in xce_worker_t

OPCode enum extended with 7 replication opcodes @22-@28:
  replEntry, replAck, redirect, heartbeat, steppingDown,
  replSnapshot, replCatchUpDone

New C typedefs (capnp_persist_cfg_t, capnp_snap_hdr_t,
capnp_repl_envelope_t) added to typemap.h for the encode/decode shims.

TDD: test_persist_schema.c added (6 tests, all GREEN). Controller test
suite remains 1/1 PASS.
2026-05-07 10:19:33 +00:00
b0a5fbb4e9 feat: persist.c WAL writer with chained CRC32 and dedicated fsync pthread (P0-1)
- Expose persist_t struct concretely in persist.h (opaque pattern deferred)
- persist_set_workdir() derives wal/snap paths from workdir+name
- persist_init() creates WAL/snap directories, rejects NFS mounts
- persist_wal_async() writes WAL entries with chained CRC32 via zlib
- Dedicated fsync pthread fires every wal_sync_ms milliseconds
- persist_flush() forces fdatasync for tests and clean shutdown
- WAL_CHECKPOINT type rejected with -EINVAL (never written via async path)
- Stubs: persist_recover, persist_wal_sync_append, persist_checkpoint
- Add fsync_thread_running flag to guard persist_stop when thread not started
- Add zlib_dep to controller and unit test meson targets
- 3 new TDD tests: init_creates_directories, wal_async_writes_entry,
  checkpoint_type_rejected — all GREEN
2026-05-07 10:00:10 +00:00
fe69dcbd7a feat: add persist.h — WAL types, persist_config_t, slot_key_t (P0-B)
- Define wal_entry_hdr_t (40 bytes, _Static_assert enforced)
- Define wal_entry_type enum (WAL_JOB_SUBMIT=1 … WAL_CHECKPOINT=100)
- Define persist_config_t with persist_config_defaults() inline helper
- Define slot_key_t (index, key[XCE_MAXNAME_LEN], key_len)
- Forward-declare persist_t opaque struct and full persist_* API surface
- Bootstrap controller Unity test suite (controller_unit_test binary)
- Register test('controller test', …, suite: 'controller') in meson
- 4 new tests all PASS: wal_header_size, wal_entry_types,
  persist_config_defaults, slot_key_t_fields
2026-05-07 09:21:36 +00:00
b748e30d20 save works 2026-02-28 17:55:50 +08:00
ce69347581 - implement xstring by librtti 2026-02-14 14:38:52 +08:00
70970434ca initial work of api server 2026-01-20 18:20:24 +08:00
280c22671f support shared libxce(for api server) 2026-01-19 11:02:49 +08:00
8d78f0cbeb improve resmgr/jobmgr/qmgr/workermgr 2025-12-18 17:37:34 +08:00
1f1e4fdaba improve resmgr 2025-12-18 16:20:41 +08:00
89fc15b926 save works 2025-12-10 16:52:56 +08:00
5036d77f04 use struct2json and import autoscaler 2025-12-05 18:02:06 +08:00
e497dd1b69 job submit and support run in worker 2025-11-28 17:24:31 +08:00
cf425dfa56 support unit test cases of worker related 2025-11-25 16:52:03 +08:00
e3946383f4 no wrapp printf() and perror() for xce_test and lib tests 2025-11-18 11:59:51 +08:00
9ff857db9f reformat code and job array 2025-11-07 16:14:27 +08:00
Rongsong Shen
41f56fa6ed update controller code 2025-11-05 16:58:57 +08:00