ykcheungab
bb15b750fb
复用现有 Docker 和 Nuitka 流程，隔离架构镜像并支持 aarch64 标签。已通过 47 项测试、代码及文档门禁，并完成 Python 3.10 ARM64 wheel 构建、隔离导入与内容审计。
503 lines
16 KiB
Python
503 lines
16 KiB
Python
"""Static content and metadata validation for built orisys wheels.
|
|
|
|
Everything here is pure standard library: the wheel is opened as a zip and
|
|
inspected in place. No installation, no network, no hardware.
|
|
|
|
The checks encode the guarantees a Nuitka-compiled release must satisfy:
|
|
|
|
* the filename, ``METADATA`` and ``RECORD`` agree on name/version/tags;
|
|
* PyPI-visible metadata (``Requires-Python``, classifiers, project URLs,
|
|
license) is present instead of silently dropped by the hand-written
|
|
``setup.py`` template in :mod:`wheel.package`;
|
|
* no Python sources ship inside ``orisys/`` (IP containment) while the compiled
|
|
extension is present;
|
|
* every file declared by ``[[tool.orisys.bundle.items]]`` is present and
|
|
byte-identical to the source tree;
|
|
* no caches, build leftovers or scratch directories leak in.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import base64
|
|
import hashlib
|
|
import re
|
|
import sys
|
|
import zipfile
|
|
from dataclasses import dataclass, field
|
|
from email.parser import Parser
|
|
from pathlib import Path
|
|
|
|
_BUILD_ROOT = Path(__file__).resolve().parents[1]
|
|
if str(_BUILD_ROOT) not in sys.path:
|
|
sys.path.insert(0, str(_BUILD_ROOT))
|
|
|
|
from wheel.layout import resolve_bundle_source_files # noqa: E402
|
|
from wheel.package import is_compliant_manylinux_wheel # noqa: E402
|
|
|
|
from common.paths import REPO_ROOT # noqa: E402
|
|
from common.pyproject_utils import ( # noqa: E402
|
|
bundle_items,
|
|
extras_require,
|
|
install_requires,
|
|
)
|
|
|
|
# PyPI's default per-project file limit is 100 MB; keep headroom.
|
|
MAX_WHEEL_BYTES = 90 * 1024 * 1024
|
|
|
|
_FILENAME_RE = re.compile(
|
|
r"^(?P<name>[^-]+)-(?P<version>[^-]+)-(?P<python>[^-]+)-(?P<abi>[^-]+)-(?P<platform>.+)\.whl$"
|
|
)
|
|
_COMPILED_EXT_RE = re.compile(r"^orisys/__init__\..+\.(?:pyd|so)$")
|
|
_FORBIDDEN_PREFIXES = (
|
|
"test/",
|
|
"tests/",
|
|
"data/",
|
|
"build/",
|
|
"dist/",
|
|
"sdk/",
|
|
".git",
|
|
)
|
|
_FORBIDDEN_SUFFIXES = (".pyc", ".pyo", ".c", ".pdb", ".map", ".spec")
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Check:
|
|
name: str
|
|
ok: bool
|
|
detail: str = ""
|
|
|
|
def render(self) -> str:
|
|
return f"[{'ok' if self.ok else 'FAIL'}] {self.name}: {self.detail}"
|
|
|
|
|
|
@dataclass
|
|
class WheelCheckResult:
|
|
wheel: Path
|
|
version: str | None = None
|
|
name: str | None = None
|
|
platform_tag: str | None = None
|
|
entry_count: int = 0
|
|
uncompressed_bytes: int = 0
|
|
file_bytes: int = 0
|
|
checks: list[Check] = field(default_factory=list)
|
|
warnings: list[str] = field(default_factory=list)
|
|
|
|
@property
|
|
def ok(self) -> bool:
|
|
return all(check.ok for check in self.checks)
|
|
|
|
@property
|
|
def failures(self) -> list[Check]:
|
|
return [check for check in self.checks if not check.ok]
|
|
|
|
def add(self, name: str, ok: bool, detail: str = "") -> bool:
|
|
self.checks.append(Check(name=name, ok=ok, detail=detail))
|
|
return ok
|
|
|
|
def warn(self, message: str) -> None:
|
|
self.warnings.append(message)
|
|
|
|
def render(self) -> str:
|
|
lines = [
|
|
f"wheel: {self.wheel.name}",
|
|
f" version={self.version} platform={self.platform_tag} "
|
|
f"entries={self.entry_count} file_size={self.file_bytes} "
|
|
f"uncompressed={self.uncompressed_bytes}",
|
|
]
|
|
lines.extend(f" {check.render()}" for check in self.checks)
|
|
lines.extend(f" [warn] {message}" for message in self.warnings)
|
|
lines.append(f" RESULT: {'PASS' if self.ok else 'FAIL'}")
|
|
return "\n".join(lines)
|
|
|
|
|
|
def _sha256(data: bytes) -> str:
|
|
return hashlib.sha256(data).hexdigest()
|
|
|
|
|
|
def _requirement_name(spec: str) -> str:
|
|
return re.split(r"[<>=!~\[; ]", spec, maxsplit=1)[0].strip().lower().replace("_", "-")
|
|
|
|
|
|
def _metadata_message(zf: zipfile.ZipFile, result: WheelCheckResult):
|
|
candidates = [name for name in zf.namelist() if name.endswith(".dist-info/METADATA")]
|
|
if not candidates:
|
|
result.add("metadata-present", False, "no *.dist-info/METADATA entry in the wheel")
|
|
return None
|
|
raw = zf.read(candidates[0]).decode("utf-8", errors="replace")
|
|
message = Parser().parsestr(raw)
|
|
result.add("metadata-present", True, candidates[0])
|
|
return message
|
|
|
|
|
|
def _check_filename(result: WheelCheckResult, expected_version: str | None) -> None:
|
|
match = _FILENAME_RE.match(result.wheel.name)
|
|
if match is None:
|
|
result.add("filename", False, f"unparsable wheel filename: {result.wheel.name}")
|
|
return
|
|
result.name = match.group("name")
|
|
result.version = match.group("version")
|
|
result.platform_tag = match.group("platform")
|
|
|
|
problems = []
|
|
if result.name != "orisys":
|
|
problems.append(f"distribution name is {result.name!r}, expected 'orisys'")
|
|
if expected_version is not None and result.version != expected_version:
|
|
problems.append(f"filename version {result.version!r} != expected {expected_version!r}")
|
|
result.add("filename", not problems, "; ".join(problems) or result.wheel.name)
|
|
|
|
|
|
def _check_metadata(
|
|
result: WheelCheckResult,
|
|
message,
|
|
expected_version: str | None,
|
|
repo_root: Path | None,
|
|
) -> None:
|
|
if message is None:
|
|
return
|
|
|
|
result.add(
|
|
"metadata-name",
|
|
message.get("Name") == "orisys",
|
|
f"Name={message.get('Name')!r}",
|
|
)
|
|
|
|
declared = message.get("Version")
|
|
expected = expected_version or result.version
|
|
if expected is not None and declared != expected:
|
|
result.add(
|
|
"metadata-version",
|
|
False,
|
|
f"METADATA Version={declared!r} != expected {expected!r}",
|
|
)
|
|
else:
|
|
result.add("metadata-version", True, f"Version={declared!r}")
|
|
|
|
requires_python = (message.get("Requires-Python") or "").strip()
|
|
result.add(
|
|
"metadata-requires-python",
|
|
bool(requires_python),
|
|
f"Requires-Python={requires_python!r}" if requires_python else "Requires-Python is missing",
|
|
)
|
|
|
|
classifiers = message.get_all("Classifier") or []
|
|
result.add(
|
|
"metadata-classifiers",
|
|
bool(classifiers),
|
|
f"{len(classifiers)} classifier(s)" if classifiers else "no Classifier fields",
|
|
)
|
|
|
|
project_urls = message.get_all("Project-URL") or []
|
|
result.add(
|
|
"metadata-project-urls",
|
|
bool(project_urls),
|
|
f"{len(project_urls)} Project-URL(s)" if project_urls else "no Project-URL fields",
|
|
)
|
|
|
|
license_values = [
|
|
(message.get("License") or "").strip(),
|
|
(message.get("License-Expression") or "").strip(),
|
|
]
|
|
has_license = any(license_values) or any(
|
|
value.startswith("License ::") for value in classifiers
|
|
)
|
|
result.add(
|
|
"metadata-license",
|
|
has_license,
|
|
f"License={license_values[0]!r} License-Expression={license_values[1]!r}"
|
|
if has_license
|
|
else "no License / License-Expression / 'License ::' classifier",
|
|
)
|
|
|
|
dynamic = message.get_all("Dynamic") or []
|
|
result.add(
|
|
"metadata-no-dynamic-version",
|
|
"version" not in dynamic,
|
|
f"Dynamic={dynamic!r}" if dynamic else "no Dynamic fields",
|
|
)
|
|
if dynamic:
|
|
result.warn(f"METADATA still declares Dynamic fields: {dynamic!r}")
|
|
|
|
if repo_root is not None and (repo_root / "pyproject.toml").is_file():
|
|
requires_dist = message.get_all("Requires-Dist") or []
|
|
provided = {_requirement_name(entry) for entry in requires_dist}
|
|
missing_core = [
|
|
spec for spec in install_requires(repo_root) if _requirement_name(spec) not in provided
|
|
]
|
|
result.add(
|
|
"metadata-requires-dist",
|
|
not missing_core,
|
|
f"missing core Requires-Dist entries: {missing_core!r}"
|
|
if missing_core
|
|
else f"{len(requires_dist)} Requires-Dist entries",
|
|
)
|
|
|
|
extras = message.get_all("Provides-Extra") or []
|
|
missing_extras = sorted(set(extras_require(repo_root)) - set(extras))
|
|
result.add(
|
|
"metadata-extras",
|
|
not missing_extras,
|
|
f"missing Provides-Extra: {missing_extras!r}"
|
|
if missing_extras
|
|
else f"Provides-Extra={sorted(extras)!r}",
|
|
)
|
|
|
|
|
|
def _check_compiled_only(result: WheelCheckResult, names: list[str]) -> None:
|
|
compiled = [name for name in names if _COMPILED_EXT_RE.match(name)]
|
|
result.add(
|
|
"compiled-extension",
|
|
bool(compiled),
|
|
f"{compiled[0]}" if compiled else "no orisys/__init__.<abi>.pyd|.so entry",
|
|
)
|
|
|
|
# orisys/examples/** is shipped as package data on purpose; every other
|
|
# orisys/**/*.py means source code leaked past the Nuitka compile step.
|
|
sources = [
|
|
name
|
|
for name in names
|
|
if name.startswith("orisys/")
|
|
and name.endswith(".py")
|
|
and not name.startswith("orisys/examples/")
|
|
]
|
|
result.add(
|
|
"no-python-sources",
|
|
not sources,
|
|
f"python sources leaked into the wheel: {sources[:5]!r}"
|
|
if sources
|
|
else "compiled package ships no python sources (examples excluded by design)",
|
|
)
|
|
|
|
blend = [name for name in names if name.startswith("orisys_blend/")]
|
|
result.add(
|
|
"orisys-blend-present",
|
|
"orisys_blend/__init__.py" in blend and "orisys_blend/kernel.py" in blend,
|
|
f"{sorted(blend)!r}",
|
|
)
|
|
|
|
|
|
def _check_bundle_fidelity(
|
|
result: WheelCheckResult,
|
|
zf: zipfile.ZipFile,
|
|
names: set[str],
|
|
repo_root: Path | None,
|
|
strict: bool,
|
|
) -> None:
|
|
if repo_root is None or not (repo_root / "pyproject.toml").is_file():
|
|
result.warn("pyproject.toml not found; bundle fidelity checks skipped")
|
|
return
|
|
|
|
missing: list[str] = []
|
|
mismatched: list[str] = []
|
|
checked = 0
|
|
|
|
for item in bundle_items(repo_root):
|
|
target = str(item["target"]).replace("\\", "/")
|
|
source_dir = repo_root / item["source"]
|
|
for source_file in resolve_bundle_source_files(repo_root, item):
|
|
if item.get("preserve_tree", False):
|
|
relative = source_file.relative_to(source_dir).as_posix()
|
|
entry = f"orisys/{target}/{relative}"
|
|
else:
|
|
entry = f"orisys/{target}/{source_file.name}"
|
|
|
|
if entry not in names:
|
|
missing.append(entry)
|
|
continue
|
|
checked += 1
|
|
if _sha256(zf.read(entry)) != _sha256(source_file.read_bytes()):
|
|
mismatched.append(entry)
|
|
|
|
# Both entries here mean "the wheel does not match this revision of the
|
|
# source tree". Against a fresh build that is fatal; against an already
|
|
# released wheel it is expected, so ``strict`` (set by preflight, unset by
|
|
# audit) decides whether it fails or warns.
|
|
if missing:
|
|
detail = (
|
|
f"{len(missing)} declared file(s) absent from the wheel (added to the "
|
|
f"source tree after this build?): {missing[:5]!r}"
|
|
)
|
|
result.add("bundle-completeness", not strict, detail)
|
|
if not strict:
|
|
result.warn(detail)
|
|
else:
|
|
result.add(
|
|
"bundle-completeness", True, f"{checked} declared bundled file(s) present"
|
|
)
|
|
|
|
if mismatched:
|
|
detail = (
|
|
f"{len(mismatched)} of {checked} bundled file(s) differ from the source "
|
|
f"tree (wheel predates or postdates it): {mismatched[:5]!r}"
|
|
)
|
|
result.add("bundle-fidelity", not strict, detail)
|
|
if not strict:
|
|
result.warn(detail)
|
|
else:
|
|
result.add(
|
|
"bundle-fidelity",
|
|
True,
|
|
f"{checked} bundled file(s) byte-identical to source",
|
|
)
|
|
|
|
|
|
def _check_forbidden_entries(result: WheelCheckResult, names: list[str]) -> None:
|
|
offenders = []
|
|
for name in names:
|
|
lowered = name.lower()
|
|
if lowered.startswith(_FORBIDDEN_PREFIXES) or lowered.endswith(_FORBIDDEN_SUFFIXES) or "__pycache__" in lowered or ".egg-info" in lowered:
|
|
offenders.append(name)
|
|
result.add(
|
|
"no-forbidden-entries",
|
|
not offenders,
|
|
f"unexpected entries: {offenders[:5]!r}" if offenders else "no caches/leftovers/scratch paths",
|
|
)
|
|
|
|
|
|
def _check_archive_paths(result: WheelCheckResult, names: list[str]) -> None:
|
|
unsafe = [
|
|
name
|
|
for name in names
|
|
if name.startswith(("/", "\\")) or ".." in Path(name.replace("\\", "/")).parts
|
|
]
|
|
result.add(
|
|
"archive-paths",
|
|
not unsafe,
|
|
f"absolute or traversing entries: {unsafe[:5]!r}" if unsafe else "all entries are relative",
|
|
)
|
|
|
|
|
|
def _check_record(result: WheelCheckResult, zf: zipfile.ZipFile, names: list[str]) -> None:
|
|
records = [name for name in names if name.endswith(".dist-info/RECORD")]
|
|
if not records:
|
|
result.add("record", False, "no *.dist-info/RECORD entry")
|
|
return
|
|
|
|
bad_paths: list[str] = []
|
|
bad_hashes: list[str] = []
|
|
bad_sizes: list[str] = []
|
|
present = set(names)
|
|
|
|
for line in zf.read(records[0]).decode("utf-8").splitlines():
|
|
if not line.strip():
|
|
continue
|
|
parts = line.split(",")
|
|
while len(parts) < 3:
|
|
parts.append("")
|
|
path, hash_field, size_field = parts[0], parts[1], parts[2]
|
|
|
|
if path not in present:
|
|
bad_paths.append(path)
|
|
continue
|
|
if not hash_field.startswith("sha256="):
|
|
continue
|
|
|
|
expected = hash_field[len("sha256=") :]
|
|
actual = base64.urlsafe_b64encode(
|
|
hashlib.sha256(zf.read(path)).digest()
|
|
).rstrip(b"=").decode("ascii")
|
|
if actual != expected:
|
|
bad_hashes.append(path)
|
|
if size_field and size_field.isdigit():
|
|
if int(size_field) != len(zf.read(path)):
|
|
bad_sizes.append(path)
|
|
|
|
problems = []
|
|
if bad_paths:
|
|
problems.append(f"paths not in archive: {bad_paths[:5]!r}")
|
|
if bad_hashes:
|
|
problems.append(f"sha256 mismatch: {bad_hashes[:5]!r}")
|
|
if bad_sizes:
|
|
problems.append(f"size mismatch: {bad_sizes[:5]!r}")
|
|
result.add(
|
|
"record",
|
|
not problems,
|
|
"; ".join(problems) if problems else f"{records[0]} verifies",
|
|
)
|
|
|
|
|
|
def _check_platform_tag(result: WheelCheckResult) -> None:
|
|
tag = result.platform_tag
|
|
if tag is None:
|
|
return
|
|
if "win" in tag:
|
|
ok = tag == "win_amd64"
|
|
result.add("platform-tag", ok, f"{tag!r} (expected 'win_amd64')")
|
|
return
|
|
ok = is_compliant_manylinux_wheel(result.wheel)
|
|
result.add(
|
|
"platform-tag",
|
|
ok,
|
|
f"{tag!r}" if ok else f"{tag!r} is not a supported manylinux x86_64 or aarch64 tag",
|
|
)
|
|
|
|
|
|
def _check_size(result: WheelCheckResult, max_bytes: int) -> None:
|
|
result.add(
|
|
"size-limit",
|
|
result.file_bytes <= max_bytes,
|
|
f"{result.file_bytes} bytes (limit {max_bytes})",
|
|
)
|
|
|
|
|
|
def inspect_wheel(
|
|
wheel_path: Path,
|
|
*,
|
|
expected_version: str | None = None,
|
|
repo_root: Path | None = None,
|
|
max_bytes: int = MAX_WHEEL_BYTES,
|
|
strict_bundle: bool = True,
|
|
) -> WheelCheckResult:
|
|
"""Validate one built wheel without installing it.
|
|
|
|
``strict_bundle`` controls how the wheel's bundled files are compared with the
|
|
current source tree: ``True`` (gate a fresh build, used by
|
|
``release preflight``) fails on any missing or changed declared file;
|
|
``False`` (audit an already-released wheel) reports the same findings as
|
|
warnings.
|
|
"""
|
|
wheel_path = Path(wheel_path).resolve()
|
|
if repo_root is None:
|
|
repo_root = REPO_ROOT if (REPO_ROOT / "pyproject.toml").is_file() else None
|
|
else:
|
|
repo_root = Path(repo_root).resolve()
|
|
|
|
result = WheelCheckResult(wheel=wheel_path)
|
|
if not wheel_path.is_file():
|
|
result.add("wheel-exists", False, f"not a file: {wheel_path}")
|
|
return result
|
|
|
|
result.file_bytes = wheel_path.stat().st_size
|
|
result.add("wheel-exists", True, str(wheel_path))
|
|
_check_filename(result, expected_version)
|
|
_check_platform_tag(result)
|
|
_check_size(result, max_bytes)
|
|
|
|
try:
|
|
zf = zipfile.ZipFile(wheel_path)
|
|
except zipfile.BadZipFile as exc:
|
|
result.add("zip-integrity", False, f"not a valid zip: {exc}")
|
|
return result
|
|
|
|
with zf:
|
|
corrupt = zf.testzip()
|
|
result.add(
|
|
"zip-integrity",
|
|
corrupt is None,
|
|
"CRC check passed" if corrupt is None else f"corrupt entry: {corrupt}",
|
|
)
|
|
|
|
names = zf.namelist()
|
|
name_set = set(names)
|
|
result.entry_count = len(names)
|
|
result.uncompressed_bytes = sum(info.file_size for info in zf.infolist())
|
|
|
|
_check_archive_paths(result, names)
|
|
_check_forbidden_entries(result, names)
|
|
message = _metadata_message(zf, result)
|
|
_check_metadata(result, message, expected_version, repo_root)
|
|
_check_compiled_only(result, names)
|
|
_check_bundle_fidelity(result, zf, name_set, repo_root, strict_bundle)
|
|
_check_record(result, zf, names)
|
|
|
|
return result
|