lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/src/xceadmin/tests/test_config_build.py
lei.chen fc33a5da16 admin-plane breakpoint 2: YAML resourcemap wiring + queue payload deep-copy 
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.

1. YAML resourcemap wiring (admin-plane-gaps 1.10):
   - schema: Resource.distributions @8 / hostBased @9 uncommented --
     resourcemap had no wire destination at all before this.
   - resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
     (distributions + hbased_defined); resobj_destroy frees them. Deep
     copy is mandatory: the source is myconf->cluster_config, which
     keeps owning and later frees its tree.
   - controller: reconcile_config_resources + seed_sharedinst_from_config
     (cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
     distribution carries a value (the license-pool shape), from both
     config_activate_committed paths, ahead of the conf4sched push.
   - xceadmin: mapping.py maps resourcemap -> distributions (only the
     location:[all]+value shape sets SHARED|GLOBAL), build.py renders
     shared=true/distributions; golden expected.bin regenerated.

2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
   seeded push freed the resval entries AND the backing array via
   free_name_val_list, then xlist_fini freed the same array again.
   Detach xreslist before the FREEUP/xlist_fini block (same ownership
   class bug-1506 fixed in sched_worker_change). Latent since inception:
   every pre-1.10 push ran with nresvals==0 and never armed it.

3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
   queue_init (create) MOVED the definition's users/hosts/thresholds/
   window payloads out, and queue_refresh_def MOVED them out again --
   but reconcile_config_queues runs MORE THAN ONCE over the same
   cluster_config object (boot: pre-election activation AND phase2's
   build_queues), so the second pass refreshed live queues from the
   drained definition: thresholds became the poisoned (count>0,
   array=NULL) pair, the generated encode_Threshold_list has no NULL
   guard, and the next queue/worker push died on the NULL base.
   Both paths now deep-copy (queue_init and queue_refresh_def); the
   definition keeps its payloads, (NULL/0) genuinely means "no list in
   this config", and reconcile is idempotent. New dup_group_config
   helper in utils.c (the generated code has no GroupConfig dup).
   Mirrors resobj_init's distributions deep-copy and worker.c
   apply_join_state's dup pattern.

4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
   reconcile of a threshold-carrying queue and asserts definition
   intactness plus a real encode_QueueConf -> wire -> decode round
   trip; new test_config_resource_reconcile.c covers seed/deep-copy/
   non-seed classes and idempotent re-seed.

Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-23 14:29:29 +00:00

446 lines
16 KiB
Python
Raw Blame History

"""Golden-file-style build tests for ``xceadmin.build`` (design.md,
``xceadmin.build``; Requirements 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8,
1.9, 1.10, 1.11, 1.12, 8.3, 8.4, 9.1, 9.2, 9.3).
The ``tests/fixtures/minimal`` and ``tests/fixtures/full`` directories
(task 5.4) each pair a real, schema-valid ``configs/`` input with a
checked-in ``expected.bin`` -- unlike the dynamic build-then-compare
tests elsewhere in this file, these catch *unintended representation
drift*: a future change that is semantically correct (decodes fine,
right field values) but produces different serialized bytes than
before would still be a determinism regression (Requirement 9) that
purely structural assertions would never notice.
"""
from __future__ import annotations
import hashlib
from pathlib import Path
import pytest
import yaml
from xcecommon.errors import DuplicateEntryError, ReservedNameError, ValidationError
from xcecommon.schema import load_schema
from xceadmin import build
REPO_CONFIGS_DIR = Path(__file__).resolve().parents[2] / "configs"
def _write_yaml(path: Path, data: dict) -> None:
path.write_text(yaml.safe_dump(data))
# ---------------------------------------------------------------------------
# 1.1 / 1.5: closed-set scan, missing file -> empty section
# ---------------------------------------------------------------------------
def test_build_missing_all_files_still_succeeds(tmp_path):
"""Observable (task 2.1's bullet, reused here for build): a configs
directory with none of the 7 known files present still builds --
every section is empty, not an error.
"""
data = build.build_cluster_config(str(tmp_path))
mod = load_schema()
with mod.ClusterConfig.from_bytes(data) as reader:
assert reader.clusterName == ""
assert len(reader.queues) == 0
def test_build_ignores_unrecognized_extra_files(tmp_path):
(tmp_path / "notes.txt").write_text("not a config file")
(tmp_path / "schema").mkdir() # an operator-supplied schema/ dir -- ignored (1.6)
(tmp_path / "schema" / "queues.schema.json").write_text("{}")
data = build.build_cluster_config(str(tmp_path))
assert isinstance(data, bytes)
def test_build_missing_one_file_matches_present_empty_section(tmp_path):
_write_yaml(tmp_path / "xce.yaml", {"cluster_name": "c1", "heartbeat_interval": 5,
"inactive_heartbeat_count": 2})
without_resources = build.build_cluster_config(str(tmp_path))
_write_yaml(tmp_path / "resources.yaml", {"resources": []})
with_empty_resources = build.build_cluster_config(str(tmp_path))
assert without_resources == with_empty_resources
# ---------------------------------------------------------------------------
# 1.9 / determinism: same input twice -> identical bytes
# ---------------------------------------------------------------------------
def test_build_same_input_twice_is_byte_identical(tmp_path):
_write_yaml(tmp_path / "xce.yaml", {"cluster_name": "eda1", "heartbeat_interval": 30,
"inactive_heartbeat_count": 3})
_write_yaml(tmp_path / "queues.yaml", {"queues": [{"name": "normal", "priority": 30}]})
first = build.build_cluster_config(str(tmp_path))
second = build.build_cluster_config(str(tmp_path))
assert first == second
# ---------------------------------------------------------------------------
# worker-rejoin-resync (Requirements 3.7, 5.3): workerResyncWindowMs/
# workerResyncTimeoutMs are operator-settable through xce.yaml's scalar
# section, following the heartbeat_interval/inactive_heartbeat_count
# pattern exactly (task 1.5). These prove the VALUE round-trips
# (YAML -> binary -> decode) with a specific, checkable pair of numbers --
# not just that some golden blob happens to match.
# ---------------------------------------------------------------------------
def test_build_maps_worker_resync_window_and_timeout_fields(tmp_path):
_write_yaml(
tmp_path / "xce.yaml",
{
"cluster_name": "eda1",
"heartbeat_interval": 30,
"inactive_heartbeat_count": 3,
"worker_resync_window_ms": 900000,
"worker_resync_timeout_ms": 15000,
},
)
data = build.build_cluster_config(str(tmp_path))
mod = load_schema()
with mod.ClusterConfig.from_bytes(data) as reader:
assert reader.workerResyncWindowMs == 900000
assert reader.workerResyncTimeoutMs == 15000
def test_build_leaves_worker_resync_fields_at_default_when_absent(tmp_path):
_write_yaml(
tmp_path / "xce.yaml",
{"cluster_name": "eda1", "heartbeat_interval": 30, "inactive_heartbeat_count": 3},
)
data = build.build_cluster_config(str(tmp_path))
mod = load_schema()
with mod.ClusterConfig.from_bytes(data) as reader:
assert reader.workerResyncWindowMs == 0
assert reader.workerResyncTimeoutMs == 0
# ---------------------------------------------------------------------------
# Hard failures propagate (schema violation, reserved name, duplicates)
# ---------------------------------------------------------------------------
def test_build_raises_validation_error_on_schema_violation(tmp_path):
_write_yaml(tmp_path / "queues.yaml", {"queues": [{"priority": 30}]}) # missing "name"
with pytest.raises(ValidationError):
build.build_cluster_config(str(tmp_path))
def test_build_raises_reserved_name_error(tmp_path):
_write_yaml(tmp_path / "queues.yaml", {"queues": [{"name": "lost_and_found", "priority": 1}]})
with pytest.raises(ReservedNameError):
build.build_cluster_config(str(tmp_path))
def test_build_raises_duplicate_entry_error(tmp_path):
_write_yaml(
tmp_path / "queues.yaml",
{"queues": [{"name": "dup", "priority": 1}, {"name": "dup", "priority": 2}]},
)
with pytest.raises(DuplicateEntryError):
build.build_cluster_config(str(tmp_path))
# ---------------------------------------------------------------------------
# 1.7: unrecognized-but-schema-valid fields warn, never fail
# ---------------------------------------------------------------------------
def test_build_maps_resource_type_order_consumable_release(tmp_path, capsys):
"""type/order/consumable/release map to valtype/ordtype/flags (see
xcecommon.mapping's _RESOURCE_VALTYPE/_RESOURCE_ORDTYPE/flag constants)
-- no longer an "unmapped field" warning.
"""
_write_yaml(
tmp_path / "resources.yaml",
{
"resources": [
{
"name": "gpus",
"type": "numeric",
"order": "increase",
"consumable": True,
"release": True,
}
]
},
)
data = build.build_cluster_config(str(tmp_path)) # must not raise
mod = load_schema()
with mod.ClusterConfig.from_bytes(data) as reader:
resource = reader.resources[0]
assert resource.name == "gpus"
assert resource.valtype == 1 # XCE_VAL_NUMERIC
assert resource.ordtype == 0 # XCE_ORD_INCR
assert resource.flags == 0x80 | 0x20 # XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE
stderr = capsys.readouterr().err
assert "'type'" not in stderr
assert "'order'" not in stderr
assert "'consumable'" not in stderr
assert "'release'" not in stderr
def test_build_maps_resourcemap_to_distributions(tmp_path, capsys):
"""resourcemap maps to Resource.distributions (one entry per item,
order preserved; location -> hosts, value -> Value union member by
YAML type) -- no longer an "unmapped field" warning. Only the
location==["all"] + value shape (cluster-wide static shared pool)
additionally sets XCE_RESF_SHARED|XCE_RESF_GLOBAL (admin-plane-gaps
1.10); other shapes map definition-only.
"""
_write_yaml(
tmp_path / "resources.yaml",
{
"resources": [
{
"name": "licenses",
"type": "numeric",
"consumable": True,
"resourcemap": [{"location": ["all"], "value": 100}],
},
{
"name": "hostpool",
"type": "numeric",
"resourcemap": [
{"location": ["host1", "host2"], "value": 100},
{"location": ["default"]}, # dynamic, reported
],
},
{
"name": "flagres",
"type": "boolean",
"resourcemap": [{"location": ["host1"], "value": True}],
},
{
"name": "strres",
"type": "string",
"resourcemap": [{"location": ["host1"], "value": "v1"}],
},
]
},
)
data = build.build_cluster_config(str(tmp_path)) # must not raise
mod = load_schema()
with mod.ClusterConfig.from_bytes(data) as reader:
resources = {r.name: r for r in reader.resources}
licenses = resources["licenses"]
assert len(licenses.distributions) == 1
assert [str(h) for h in licenses.distributions[0].hosts] == ["all"]
assert licenses.distributions[0].value.u.which == "fvalue"
assert licenses.distributions[0].value.u.fvalue == 100.0
assert licenses.flags == 0x80 | 0x08 | 0x04 # CONSUMABLE|SHARED|GLOBAL
hostpool = resources["hostpool"]
assert len(hostpool.distributions) == 2
assert [str(h) for h in hostpool.distributions[0].hosts] == ["host1", "host2"]
assert hostpool.distributions[0].value.u.fvalue == 100.0
# dynamic shape: hosts mapped, value left unset (capnp union default)
assert [str(h) for h in hostpool.distributions[1].hosts] == ["default"]
assert hostpool.distributions[1].value.u.which == "ivalue32"
assert hostpool.flags == 0 # no SHARED|GLOBAL without location [all]
flag = resources["flagres"]
assert flag.distributions[0].value.u.which == "bvalue"
assert bool(flag.distributions[0].value.u.bvalue) is True
sres = resources["strres"]
assert sres.distributions[0].value.u.which == "svalue"
assert str(sres.distributions[0].value.u.svalue) == "v1"
stderr = capsys.readouterr().err
assert "'resourcemap'" not in stderr
def test_build_warns_on_schema_valid_unmapped_queue_fields(tmp_path, capsys):
_write_yaml(
tmp_path / "queues.yaml",
{"queues": [{"name": "night", "priority": 30, "nice": 10, "fairshare": [{"user": "u1", "share": 5}]}]},
)
build.build_cluster_config(str(tmp_path))
stderr = capsys.readouterr().err
assert "'nice'" in stderr
assert "'fairshare'" in stderr
assert "night" in stderr
def test_build_warns_on_schema_valid_unmapped_host_description(tmp_path, capsys):
_write_yaml(
tmp_path / "hosts.yaml",
{"hosts": [{"hostname": "h1", "description": "a worker node"}], "hostgroups": []},
)
build.build_cluster_config(str(tmp_path))
stderr = capsys.readouterr().err
assert "'description'" in stderr
assert "h1" in stderr
def test_build_no_warnings_when_only_mapped_fields_present(tmp_path, capsys):
_write_yaml(tmp_path / "scaler.yaml", {"scalers": [{"name": "s1", "type": "script", "script_path": "/x.sh"}]})
build.build_cluster_config(str(tmp_path))
assert capsys.readouterr().err == ""
# ---------------------------------------------------------------------------
# 4.4 undefined-ref warnings surface without blocking the build
# ---------------------------------------------------------------------------
def test_build_surfaces_undefined_ref_warnings_without_failing(tmp_path, capsys):
_write_yaml(
tmp_path / "queues.yaml",
{"queues": [{"name": "q1", "priority": 1, "hosts": {"members": ["nonexistent_hg"]}}]},
)
_write_yaml(tmp_path / "hosts.yaml", {"hosts": [], "hostgroups": []})
data = build.build_cluster_config(str(tmp_path)) # must not raise
assert isinstance(data, bytes)
stderr = capsys.readouterr().err
assert "nonexistent_hg" in stderr
# ---------------------------------------------------------------------------
# Real src/configs/*.yaml fixtures
# ---------------------------------------------------------------------------
@pytest.mark.skipif(not REPO_CONFIGS_DIR.is_dir(), reason="src/configs/ fixtures not present")
def test_build_real_repo_configs_succeeds_and_warns_on_known_gaps(capsys):
data = build.build_cluster_config(str(REPO_CONFIGS_DIR))
mod = load_schema()
with mod.ClusterConfig.from_bytes(data) as reader:
assert reader.clusterName == "eda1"
# queues.yaml/hosts.yaml still have several schema-valid-but-unmapped
# fields (description, nice, fairshare, ... -- see build.py's
# _UNMAPPED_BUT_VALID_ITEM_KEYS) and limits.yaml/queues.yaml reference
# bare usernames (e.g. "rshen") that the undefined-ref check can never
# resolve since users.yaml has no individual-user roster, only
# usergroups (Requirement 4.4 scopes that check to groups only) --
# both are expected warnings, not errors.
stderr = capsys.readouterr().err
assert "warning:" in stderr
@pytest.mark.skipif(not REPO_CONFIGS_DIR.is_dir(), reason="src/configs/ fixtures not present")
def test_write_output_real_repo_configs_hash_matches_manual_sha256(tmp_path):
data = build.build_cluster_config(str(REPO_CONFIGS_DIR))
out_path = tmp_path / "cluster.bin"
digest = build.write_output(data, str(out_path))
assert out_path.read_bytes() == data
assert digest == hashlib.sha256(data).hexdigest()
# ---------------------------------------------------------------------------
# write_output(): file path, stdout, and hash correctness
# ---------------------------------------------------------------------------
def test_write_output_writes_file_and_returns_correct_hash(tmp_path):
data = b"some cluster.bin bytes"
out_path = tmp_path / "out.bin"
digest = build.write_output(data, str(out_path))
assert out_path.read_bytes() == data
assert digest == hashlib.sha256(data).hexdigest()
def test_write_output_dash_writes_to_stdout_buffer(capsys):
data = b"stdout-bound bytes"
digest = build.write_output(data, "-")
captured = capsys.readouterr()
assert captured.out.encode() == data
assert digest == hashlib.sha256(data).hexdigest()
def test_write_output_does_not_print_the_hash_itself(tmp_path, capsys):
"""1.4's "print SHA-256 to stderr" is the CLI layer's job (task 4.1) --
write_output only returns the digest, it never prints anything.
"""
data = b"quiet"
build.write_output(data, str(tmp_path / "out.bin"))
captured = capsys.readouterr()
assert captured.err == ""
# ---------------------------------------------------------------------------
# Golden-file determinism (task 5.4)
# ---------------------------------------------------------------------------
FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures"
@pytest.mark.parametrize("fixture_name", ["minimal", "full"])
def test_golden_fixture_matches_checked_in_expected_bin(fixture_name):
fixture_dir = FIXTURES_DIR / fixture_name
data = build.build_cluster_config(str(fixture_dir / "configs"))
expected = (fixture_dir / "expected.bin").read_bytes()
assert data == expected
@pytest.mark.parametrize("fixture_name", ["minimal", "full"])
def test_golden_fixture_builds_twice_identically(fixture_name):
fixture_dir = FIXTURES_DIR / fixture_name / "configs"
first = build.build_cluster_config(str(fixture_dir))
second = build.build_cluster_config(str(fixture_dir))
assert first == second
assert hashlib.sha256(first).hexdigest() == hashlib.sha256(second).hexdigest()
def test_full_golden_fixture_covers_all_seven_file_types():
fixture_dir = FIXTURES_DIR / "full" / "configs"
present = {p.name for p in fixture_dir.glob("*.yaml")}
assert present == {
"hosts.yaml",
"queues.yaml",
"resources.yaml",
"users.yaml",
"limits.yaml",
"xce.yaml",
"scaler.yaml",
}
def test_minimal_golden_fixture_covers_hosts_and_queues_only():
fixture_dir = FIXTURES_DIR / "minimal" / "configs"
present = {p.name for p in fixture_dir.glob("*.yaml")}
assert present == {"hosts.yaml", "queues.yaml"}
Reference in a new issue View git blame Copy permalink