orisys-tech/orisys-app
3
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(studio-3d): 阶段 5 —— 发布保护（签名/可选加密/平台锁定）与性能长时报告

Browse Source 
- What: 新增 `scene3d/protect.py`（封套 v2：Ed25519 签名 + 可选 AES-256-GCM，nonce/salt 每包唯一、
  元数据作 AAD、明文摘要进签名、KeyProvider 只问 get_content_key）与 `record/visualization3d.py`
  （session.json 的 visualization3d 块：写入版本化、读取时缺块=不支持）；`package.py` 支持 v2
  （payload=plain 仅签名 / aes-gcm 签名+加密）并继续读 v1，新增解压比上限与"写入后自读回"校验；
  `contracts.py` 新增 PlatformLock/PlatformInfo/check_platform_lock、Sealing、platform_mismatch 诊断码；
  `Scene3DView` 在装配前查 OS/架构、拿到 capabilities 后查图形半区，违反即撤下场景并在面板显示原因；
  工具 `compile_scene.py` 新增 --sign-key/--write-verify-key/--encrypt-key-id/--key-file/--lock-from-probe，
  `validate_scene.py` 新增 --key-file/--key-id/--verify-key/--require-signature（发布门禁），
  新增 `bench_scene.py`（p50/p95 分阶段 + --cpu-only + --vertices N + --package）与
  `soak_scene.py`（RSS 四分位速率 + 队列上界 + 抽稀采样）；两份报告落到 docs/architecture/。
- Why: 规格 §5.2 的保护目标是"提高复制成本"而不是"防提取"，所以要的是签名（来源）+ 可选认证加密（机密性）
  + 密钥隔离，而不是自制加密或把 ZIP 当加密；§12 的"包保护/性能/长时"三行需要可复现的数字与判定标准，
  而不是一次性的观感。平台锁定还必须只降级三维：锁定不通过时二维照常。
- Verification: shared 全量通过（新增 test_scene3d_protect.py 35 项：封套往返/篡改/密钥/锁定/记录块/
  compile→validate 端到端发布流程；test_scene3d_package.py 20 项含 v1↔v2 兼容）；apps/studio 全量通过
  （test_scene3d_view.py 30 项含锁定拒绝与 session.json 块）；ui_check 566 项 0 失败；locale_check 0 缺失；
  真实资产端到端：1032 顶点编译出 signed+sealed+locked 包（56 927 B 密文），validate --require-signature 退出 0，
  改一个字节退出 2；性能报告（perf-report-windows-dev.json）：1032 顶点 compose p50 0.50 ms（30 fps 上限内），
  20022 顶点/39480 三角形 compose p50 6.15 ms（约 33 ms 帧的 18%），采集线程 submit p95 < 0.4 ms（合成器被人为放慢到 50 ms）；
  长时报告（soak-report-windows-dev.json）：无窗口 10 min/60 fps 尾段 0.354 MB/min，GUI 4 min/30 fps 尾段 0.809 MB/min，
  队列上界（mailbox=1、单飞行合成）两者均成立。过程中修掉：GUI 进程里的合成器计时被 GIL/vsync 污染（98.8 ms vs 真值 6.1 ms）、
  跨线程读 last_stats 撕裂、key id 无 provider 时会静默写明文包、soak 工具自身记时间列表会无界增长、RSS 读取失败伪装成零增长、
  平台锁定不是终态诊断。
- Not in this change: ARM 板卡的能力报告与 2 小时门禁运行（按用户要求暂缓；命令与判定标准已写入阶段 5 文档）；
  GPU 显存随时间的测量（无可移植 API，报告说明渲染器缓冲估计恒定且更新路径不新建 GPU 对象）；
  `cryptography` 在此环境无法联网安装，本机验收用的是本机已有的 cp310 构建（pyproject 里已声明为可选 extra
  `protected-assets`）。
This commit is contained in:
ykcheungab
2026-09-14 16:44:33 +08:00
parent ac4dcd643f
commit 5cd92254e1
24 changed files with 5448 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
shared/src/orisys_app_shared/record/__init__.py
+12
View File
@@ -21,12 +21,21 @@ from orisys_app_shared.record.playback import (
RecordingReader,
list_recordings,
)
from orisys_app_shared.record.visualization3d import (
BLOCK_VERSION as VISUALIZATION3D_BLOCK_VERSION,
VISUALIZATION3D_KEY,
Visualization3DRecord,
build_block as build_visualization3d_block,
read_block as read_visualization3d_block,
)
__all__ = [
"EVENTS_HEADER",
"FORCES_HEADER",
"FRAMES_HEADER",
"RECORD_FORMAT_VERSION",
"VISUALIZATION3D_BLOCK_VERSION",
"VISUALIZATION3D_KEY",
"ExperimentRecorder",
"RecordConfig",
"RecordReport",
@@ -34,8 +43,11 @@ __all__ = [
"RecordingFrame",
"RecordingInfo",
"RecordingReader",
"Visualization3DRecord",
"build_visualization3d_block",
"format_value",
"list_calibration_snapshots",
"list_recordings",
"read_visualization3d_block",
"store_calibration_snapshot",
]
shared/src/orisys_app_shared/record/visualization3d.py
+160
View File
@@ -0,0 +1,160 @@
"""The ``visualization3d`` block in a recording's ``session.json``.
Stage 5 in ``docs/architecture/studio-3d-stages.md``.
A recording that shows a three-dimensional surface has to say so, because the
picture is part of what was observed: an operator watching an exaggerated
deformation while a colleague reads the CSV is looking at a *different* claim than
the numbers, and six months later nobody can reconstruct which claim was on screen.
So the block records what the pane was, what it displayed it with, and how the
field was labelled — and nothing about the measurement, which the CSV already owns.
Two rules, both about records that outlive the build that wrote them:
* **the writer never fails a recording.** The block is assembled from a live widget,
so every read is guarded: a missing attribute becomes an explicit null rather than
an exception on the way into ``session.json``.
* **the reader treats a missing block as "not supported", not as an error.** A
recording made before this block existed, or by a build without 3D at all, is a
perfectly good recording. ``read_block`` says ``supported=False`` and why, so a
consumer can decide whether it cares.
The counters (frames shown, display frames dropped) are *not* in the block: they are
only known when the recording stops, and a block written at start cannot know them.
They travel as one ``visualization3d`` event instead, which the recorder already
timestamps and which keeps "what was configured" and "what happened" apart.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Mapping, Optional
#: The key under which the block lives in ``session.json``.
VISUALIZATION3D_KEY = "visualization3d"
#: Bumped when the block's *meaning* changes. A reader that does not know a version
#: reports ``supported=False`` rather than guessing at fields.
BLOCK_VERSION = 1
def build_block(
*,
enabled: bool,
available: bool,
reason: str = "",
scene: Optional[Mapping[str, Any]] = None,
gl: Optional[Mapping[str, Any]] = None,
display: Optional[Mapping[str, Any]] = None,
stream: Optional[Mapping[str, Any]] = None,
) -> dict[str, Any]:
"""One block for ``session.json``. Plain JSON types only, no failures."""
return {
"block_version": BLOCK_VERSION,
"enabled": bool(enabled),
"available": bool(available),
# Why not, in one line. Empty when the pane was up: a reason next to a working
# view only invites the reader to wonder which of the two is true.
"reason": "" if (enabled and available) else str(reason or ""),
"scene": _clean(scene),
"gl": _clean(gl),
"display": _clean(display),
"stream": _clean(stream),
}
def _clean(value: Optional[Mapping[str, Any]]) -> Optional[dict[str, Any]]:
"""Drop nothing silently: nulls stay nulls, and unknown types become text."""
if value is None:
return None
out: dict[str, Any] = {}
for key, item in value.items():
if item is None or isinstance(item, (bool, int, float, str)):
out[str(key)] = item
elif isinstance(item, (list, tuple)):
out[str(key)] = [str(v) if isinstance(v, (bytes, bytearray)) else v for v in item]
else:
out[str(key)] = str(item)
return out
@dataclass(frozen=True)
class Visualization3DRecord:
"""What a recording says about its three-dimensional view."""
supported: bool
enabled: bool = False
available: bool = False
reason: str = ""
scene: Mapping[str, Any] = field(default_factory=dict)
gl: Mapping[str, Any] = field(default_factory=dict)
display: Mapping[str, Any] = field(default_factory=dict)
stream: Mapping[str, Any] = field(default_factory=dict)
block_version: int = 0
def asset_hash(self) -> str:
return str(self.scene.get("asset_hash") or "")
def geometry_id(self) -> str:
return str(self.scene.get("geometry_id") or "")
def was_showing_measurements(self) -> bool:
"""Whether a 3D *measurement* was on screen, as opposed to a reference model.
``quality`` is the field's own claim: ``display_only`` means the pane was
showing an unmeasured surface, which is exactly the case a later reader must
not treat as observed data.
"""
return bool(self.enabled and self.available) and str(
self.stream.get("quality") or ""
) not in ("", "display_only")
def read_block(session: Optional[Mapping[str, Any]]) -> Visualization3DRecord:
"""Read the block out of a ``session.json`` mapping, tolerantly.
Never raises: a broken recording is still a recording, and a consumer asking
"was there a 3D view?" must get an answer for every file, including the ones
written before the question existed.
"""
if not isinstance(session, Mapping):
return Visualization3DRecord(supported=False, reason="no session metadata")
raw = session.get(VISUALIZATION3D_KEY)
if raw is None:
return Visualization3DRecord(
supported=False,
reason=(
"this recording has no visualization3d block: it was made before the "
"3D view was recorded, or by a build without one"
),
)
if not isinstance(raw, Mapping):
return Visualization3DRecord(
supported=False, reason="the visualization3d block is not an object"
)
try:
version = int(raw.get("block_version") or 0)
except (TypeError, ValueError):
return Visualization3DRecord(
supported=False, reason="the visualization3d block has no usable version"
)
if version != BLOCK_VERSION:
return Visualization3DRecord(
supported=False,
block_version=version,
reason=f"visualization3d block version {version} is not {BLOCK_VERSION}",
)
return Visualization3DRecord(
supported=True,
enabled=bool(raw.get("enabled")),
available=bool(raw.get("available")),
reason=str(raw.get("reason") or ""),
scene=_mapping(raw.get("scene")),
gl=_mapping(raw.get("gl")),
display=_mapping(raw.get("display")),
stream=_mapping(raw.get("stream")),
block_version=version,
)
def _mapping(value: Any) -> dict[str, Any]:
return dict(value) if isinstance(value, Mapping) else {}
shared/src/orisys_app_shared/scene3d/assets.py
+37 -6
View File
@@ -18,6 +18,7 @@ customer sees the wrong part move:
from __future__ import annotations
from dataclasses import replace
from pathlib import Path
from typing import Any, Optional
@@ -66,9 +67,12 @@ __all__ = [
class SceneCompiler:
"""Package a scene (and an optional housing GLB) for release. Offline only.
The signing and encryption that :class:`CompileOptions` describes are stage 5:
until then a request for an encrypted package is *refused*, rather than
quietly producing a plain one that a release script would ship as protected.
Signing and encryption are stage 5 and they are *optional*: with no
:class:`~orisys_app_shared.scene3d.contracts.Sealing` the output is the plain
container an engineering build has always produced. What this class will not do is
quietly substitute one for the other — a request for an encrypted package with no
key material, or with a crypto backend that is not installed, is refused rather
than satisfied with a plain file a release script would then ship as protected.
"""
def compile(
@@ -78,13 +82,35 @@ class SceneCompiler:
output: Path,
options: CompileOptions,
) -> CompileReport:
if options.encryption_key_id:
sealing = options.sealing
if options.encryption_key_id and not (sealing and sealing.encrypts()):
raise error(
"asset_auth_failed",
"encrypted packages are stage 5; refusing rather than writing a "
"plain package that claims to be protected",
"an encrypted package was requested but no content key is available "
"(the encryption key id is a name; the key has to come from somewhere)",
key_id=str(options.encryption_key_id),
)
if sealing is not None and (sealing.encrypts() or sealing.signs()):
from orisys_app_shared.scene3d.package import backend_available
if not backend_available():
raise error(
"asset_auth_failed",
"signing or encrypting a package needs the 'cryptography' package; "
"refusing rather than writing a plain package that claims to be "
"protected",
)
if options.encryption_key_id and sealing.key_id and (
str(options.encryption_key_id) != str(sealing.key_id)
):
# Two names for the same key is how a release ends up sealed for a key
# id nobody distributes.
raise error(
"asset_auth_failed",
"the compile options and the key material name different key ids",
options=str(options.encryption_key_id),
sealing=str(sealing.key_id),
)
scene = surface
if housing_glb is not None:
from orisys_app_shared.scene3d.glb import HousingLimits
@@ -106,12 +132,17 @@ class SceneCompiler:
f"the surface has {count} vertices (limit "
f"{int(options.max_surface_vertices)}); export a lower LOD",
)
if options.platform_lock is not None:
# The lock is part of the asset, so it travels into the manifest and hence
# into the signature.
scene = replace(scene, platform_lock=options.platform_lock)
validate_assets(scene)
report = write_package(
scene,
Path(output),
sensor_model=options.expected.sensor_model,
hardware_revisions=(options.expected.hardware_revision,),
sealing=options.sealing,
)
return report
shared/src/orisys_app_shared/scene3d/contracts.py
+254
View File
@@ -60,6 +60,9 @@ DIAGNOSTIC_SEVERITY: Mapping[str, Severity] = {
"gpu_budget_exceeded": "error",
"asset_rejected": "error",
"no_data": "info",
#: The package is not released for this machine. An error, but only for the 3D
#: view: the two-dimensional application carries on.
"platform_mismatch": "error",
#: An illustrative display deformation was applied. Informational by design:
#: the picture is legitimate, and the point is that it is *labelled*.
"display_illustrative": "info",
@@ -222,6 +225,211 @@ class ExpectedGeometry:
shape_hw: tuple[int, int]
@dataclass(frozen=True)
class PlatformLock:
"""Where a released package is allowed to be displayed.
The design document's stage 5 asks for a platform lock, and the reason is
commercial rather than technical: a package that ships a compiled surface should
not silently render on a machine nobody qualified. So the lock is *part of the
signed manifest* and the loader enforces it — but the enforcement is deliberately
one-sided: a locked-out machine loses the three-dimensional view and keeps
everything else (acquisition, recording, charts, calibration).
Every field is a *set of allowed values* and an empty set means "no restriction",
so an un-locked package (the default) is not the same as a package locked to
nothing. ``forbid_software_renderer`` defaults to ``True`` because a package that
was qualified on a GPU must not be signed off on llvmpipe.
"""
os_family: tuple[str, ...] = ()
machine: tuple[str, ...] = ()
gl_profile: Optional[Profile] = None
allowed_vendors: tuple[str, ...] = ()
allowed_renderers: tuple[str, ...] = ()
forbid_software_renderer: bool = True
min_texture_size: int = 0
#: Free text for the report ("qualified on the dev box, 2024-06").
note: str = ""
def is_unrestricted(self) -> bool:
return not (
self.os_family
or self.machine
or self.gl_profile
or self.allowed_vendors
or self.allowed_renderers
or self.min_texture_size
)
def to_dict(self) -> dict[str, Any]:
"""Canonical JSON form. Stored in the manifest, hence inside the signature."""
out: dict[str, Any] = {
"os_family": [str(v) for v in self.os_family],
"machine": [str(v) for v in self.machine],
"forbid_software_renderer": bool(self.forbid_software_renderer),
"min_texture_size": int(self.min_texture_size),
}
if self.gl_profile:
out["gl_profile"] = str(self.gl_profile)
if self.allowed_vendors:
out["allowed_vendors"] = [str(v) for v in self.allowed_vendors]
if self.allowed_renderers:
out["allowed_renderers"] = [str(v) for v in self.allowed_renderers]
if self.note:
out["note"] = str(self.note)
return out
@classmethod
def from_dict(cls, data: Optional[Mapping[str, Any]]) -> Optional["PlatformLock"]:
"""Read one back. A malformed lock is an error, not a silently open one."""
if data is None:
return None
if not isinstance(data, Mapping):
raise error("asset_rejected", "the platform lock is not an object")
profile = data.get("gl_profile")
if profile is not None and str(profile) not in ("desktop33", "gles30"):
raise error("asset_rejected", f"unknown gl_profile in the lock: {profile!r}")
def _strings(key: str) -> tuple[str, ...]:
value = data.get(key) or ()
if isinstance(value, str) or not isinstance(value, Iterable):
raise error("asset_rejected", f"the lock's {key} must be a list")
return tuple(str(item) for item in value)
try:
minimum = int(data.get("min_texture_size") or 0)
except (TypeError, ValueError) as exc:
raise error("asset_rejected", "the lock's min_texture_size is not a number") from exc
if minimum < 0:
raise error("asset_rejected", "the lock's min_texture_size is negative")
return cls(
os_family=_strings("os_family"),
machine=_strings("machine"),
gl_profile=str(profile) if profile else None, # type: ignore[arg-type]
allowed_vendors=_strings("allowed_vendors"),
allowed_renderers=_strings("allowed_renderers"),
forbid_software_renderer=bool(data.get("forbid_software_renderer", True)),
min_texture_size=minimum,
note=str(data.get("note") or ""),
)
@dataclass(frozen=True)
class PlatformInfo:
"""What this machine is, for the lock check.
Split in two on purpose: ``os_family``/``machine`` are known before any window
exists, while the graphics fields only exist once Qt has made a context. So a
package locked to another operating system is refused *before* a context is
created, and a package locked to a GPU is refused as soon as the capabilities are
known.
"""
os_family: str
machine: str
gl_profile: Optional[Profile] = None
vendor: str = ""
renderer: str = ""
software_renderer: bool = False
max_texture_size: int = 0
@classmethod
def here(cls, capabilities: Optional[GLCapabilities] = None) -> "PlatformInfo":
import platform as _platform
family = _platform.system().lower()
return cls(
os_family=family or "unknown",
machine=(_platform.machine() or "unknown").lower(),
gl_profile=(capabilities.profile if capabilities is not None else None),
vendor=(capabilities.vendor if capabilities is not None else ""),
renderer=(capabilities.renderer if capabilities is not None else ""),
software_renderer=bool(
capabilities.software_renderer if capabilities is not None else False
),
max_texture_size=int(
getattr(capabilities, "max_texture_size", 0) or 0
),
)
def check_platform_lock(
lock: Optional[PlatformLock], info: PlatformInfo
) -> tuple[Diagnostic, ...]:
"""Every way this machine violates the lock. Empty means "allowed".
The graphics half is only checked when ``info`` carries it: a pre-context check
(``PlatformInfo.here()`` with no capabilities) must not report a GPU problem it
cannot see yet, or every locked package would be refused before Qt starts.
"""
if lock is None or lock.is_unrestricted():
return ()
out: list[Diagnostic] = []
if lock.os_family and info.os_family not in {v.lower() for v in lock.os_family}:
out.append(
diagnostic(
"platform_mismatch",
"this package is not released for this operating system",
this=info.os_family,
allowed=",".join(lock.os_family),
)
)
if lock.machine and info.machine not in {v.lower() for v in lock.machine}:
out.append(
diagnostic(
"platform_mismatch",
"this package is not released for this processor architecture",
this=info.machine,
allowed=",".join(lock.machine),
)
)
graphics_known = info.gl_profile is not None or bool(info.vendor or info.renderer)
if not graphics_known:
return tuple(out)
if lock.gl_profile and info.gl_profile != lock.gl_profile:
out.append(
diagnostic(
"platform_mismatch",
"this package was compiled for another OpenGL profile",
this=str(info.gl_profile),
required=str(lock.gl_profile),
)
)
if lock.forbid_software_renderer and info.software_renderer:
out.append(
diagnostic(
"platform_mismatch",
"this package requires a GPU and this context is a software rasteriser",
renderer=info.renderer or "unknown",
)
)
haystack = f"{info.vendor} {info.renderer}".lower()
for label, allowed in (
("vendor", lock.allowed_vendors),
("renderer", lock.allowed_renderers),
):
if allowed and not any(item.lower() in haystack for item in allowed):
out.append(
diagnostic(
"platform_mismatch",
f"this package is not released for this graphics {label}",
this=haystack.strip() or "unknown",
allowed=",".join(allowed),
)
)
if lock.min_texture_size and info.max_texture_size < int(lock.min_texture_size):
out.append(
diagnostic(
"platform_mismatch",
"the GPU's maximum texture size is below what this package needs",
this=str(info.max_texture_size),
required=str(int(lock.min_texture_size)),
)
)
return tuple(out)
@dataclass(frozen=True)
class SceneAssets:
"""A validated scene: one sensing part, zero or more housing parts."""
@@ -242,6 +450,9 @@ class SceneAssets:
#: triangle is this?" is a question an operator will ask, and re-deriving it
#: from UVs alone is a guess.
source_vertex_of_render: Optional[I32] = None
#: Where this package may be displayed (stage 5). ``None`` means "not locked",
#: which is what every package written before the lock existed carries.
platform_lock: Optional[PlatformLock] = None
def part(self, part_id: str) -> Optional[MeshPart]:
for item in self.parts:
@@ -331,6 +542,34 @@ class RenderStats:
estimated_gpu_bytes: int = 0
@dataclass(frozen=True)
class Sealing:
"""How a package is protected while it is written.
The *material*, not the policy: the policy is ``CompileOptions.signing_key_reference``
and ``CompileOptions.encryption_key_id`` (names a person wrote down), and this is
the resolved key bytes and provider a build machine actually holds. Splitting the
two is what keeps an offline tool from needing the crypto backend while the
runtime, which only ever *reads* sealed packages, needs nothing but a public key.
"""
#: Which content key to use, and where the key comes from. ``None`` = plain.
key_id: str = ""
provider: Optional["KeyProvider"] = None
#: Build-time only. The private key never reaches the runtime path.
signing_key_pem: Optional[bytes] = None
signing_key_reference: str = ""
#: Optional: the writer's own public key, so a build can prove the package it just
#: wrote opens again instead of discovering the problem at the customer.
verify_key_pem: Optional[bytes] = None
def encrypts(self) -> bool:
return bool(self.key_id) and self.provider is not None
def signs(self) -> bool:
return self.signing_key_pem is not None
@dataclass(frozen=True)
class CompileOptions:
asset_id: str
@@ -341,6 +580,11 @@ class CompileOptions:
max_housing_triangles: int
signing_key_reference: str
encryption_key_id: Optional[str] = None
#: Where the released package may be displayed. ``None`` leaves it unlocked, which
#: is the right default for an engineering build and the wrong one for a release.
platform_lock: Optional[PlatformLock] = None
#: The resolved key material for signing and sealing (see :class:`Sealing`).
sealing: Optional[Sealing] = None
@dataclass(frozen=True)
@@ -570,6 +814,16 @@ def validate_assets(
raise error("geometry_mismatch", "a source vertex index is negative")
out: list[Diagnostic] = []
if assets.platform_lock is not None:
# The lock travels inside the signed manifest, so a shape that cannot round
# trip through JSON is a package nobody can check at the far end.
try:
PlatformLock.from_dict(assets.platform_lock.to_dict())
except ThreeDError as exc:
raise error(
"asset_rejected",
f"the platform lock is malformed: {exc.diagnostic.message}",
) from exc
if expected is not None:
if expected.geometry_id != assets.geometry_id:
raise error(
shared/src/orisys_app_shared/scene3d/digest.py
+16
View File
@@ -72,6 +72,22 @@ def content_digest(
return hasher.hexdigest()
def bytes_digest(data: bytes, *, domain: str = "bytes") -> str:
"""SHA-256 over raw bytes, labelled by what they are.
Used for a payload and for a ciphertext, where there is no array to name: the
label keeps a digest of one kind of object from ever colliding with another, and
the length is fed separately so two different lengths cannot hash the same way.
Still a *digest*: it detects corruption, it does not authenticate (see
``protect.py`` for that).
"""
hasher = hashlib.sha256()
_feed(hasher, "schema", DIGEST_SCHEMA.encode("ascii"))
_feed(hasher, f"bytes:{domain}:len", str(len(data)).encode("ascii"))
_feed(hasher, f"bytes:{domain}", bytes(data))
return hasher.hexdigest()
def geometry_id(parts: Iterable[MeshPart]) -> str:
"""The identity of the *shape*: part roles, positions and triangles.
shared/src/orisys_app_shared/scene3d/package.py
+427 -151
View File
@@ -1,28 +1,34 @@
"""The ``.oryscene`` runtime package: manifest, entries, verification.
"""The ``.oryscene`` runtime package: manifest, entries, verification, sealing.
Stage 2 in ``docs/architecture/studio-3d-stages.md``; signing and encryption are
stage 5 and are *not* here. What is here is the part that stage 5 wraps: a single
file whose every byte is accounted for in a manifest, and a reader that verifies
before it believes.
Stage 2 (the plain container) and stage 5 (signature, optional encryption, platform
lock) in ``docs/architecture/studio-3d-stages.md``.
The container is a zip archive (stdlib, no pickle, no code execution) holding:
The container is a zip archive (stdlib, no pickle, no code execution). Two envelope
versions exist and both are read:
``envelope.json``
magic, envelope version, payload state, entry count. Stage 5 adds the key id,
nonce and signature here.
``manifest.json``
canonical JSON: identity, coordinate frame, units, the UV domain, the parts,
and one entry record (dtype, shape, sha256) per array.
``sensing/*.npy`` (and ``housing/*.npy`` when present)
plain ``.npy`` arrays, written by :func:`numpy.save` into the archive.
``envelope_version`` 1 — plain
``envelope.json``, ``manifest.json``, ``sensing/*.npy``, ``housing/*.npy``.
Everything a v1 reader wrote before stage 5 still loads.
``envelope_version`` 2 — sealed
``envelope.json`` (public, signed metadata) and ``payload``: the AES-256-GCM
ciphertext of exactly the bytes a v1 package's *content* would have been, i.e. a
zip holding ``manifest.json`` and the arrays. Written only when a key provider and
a signing key are supplied; read only when the verification key (and, for an
encrypted one, the content key) is available.
The reader refuses, without trying to repair anything: a wrong magic, an unknown
envelope version, a missing manifest, an entry whose bytes do not match its
recorded hash, an entry the manifest does not describe (or describes and never
provides), a path that tries to escape, a dtype/shape that disagrees with the
record, non-finite positions, a binding that belongs to another mesh, and an asset
that does not match the device. An encrypted envelope is refused with its own
message rather than silently opened as if it were plain.
envelope version, a missing manifest, an entry whose bytes do not match its recorded
hash, an entry the manifest does not describe (or describes and never provides), a
path that tries to escape, a dtype/shape that disagrees with the record, non-finite
positions, a binding that belongs to another mesh, an asset that does not match the
device, a package larger than a geometry package has any business being, and — since
stage 5 — a bad signature, a ciphertext that was changed, and a platform the package
was not released for.
One property is worth stating because it is the point of the two versions: the
*payload* of a sealed package is built by the same code as the content of a plain
one, so a sealed package cannot drift into a different format from the one the tests
already cover.
"""
from __future__ import annotations
@@ -43,7 +49,9 @@ from orisys_app_shared.scene3d.contracts import (
ExpectedGeometry,
Material,
MeshPart,
PlatformLock,
SceneAssets,
Sealing,
SurfaceBinding,
error,
freeze,
@@ -56,9 +64,19 @@ from orisys_app_shared.scene3d.digest import (
content_digest,
geometry_id as compute_geometry_id,
)
from orisys_app_shared.scene3d import protect as _protect
#: Bumped when the container layout changes incompatibly.
ENVELOPE_VERSION = 1
#: Re-exported so a caller can ask "can this build seal a package at all?" without
#: importing the crypto module directly.
backend_available = _protect.backend_available
backend_version = _protect.backend_version
CryptoUnavailable = _protect.CryptoUnavailable
#: The version this build *writes*.
ENVELOPE_VERSION = 2
#: Every version this build *reads*. A v1 package is a release from before stage 5 and
#: has to keep working: refusing it would turn a format improvement into a recall.
SUPPORTED_ENVELOPE_VERSIONS = (1, 2)
SCHEMA_VERSION = "1.0"
MAGIC = "orisys-scene"
@@ -67,6 +85,12 @@ MAGIC = "orisys-scene"
MAX_ENTRIES = 64
MAX_ENTRY_BYTES = 256 << 20
MAX_TOTAL_BYTES = 512 << 20
#: And refuse a compression ratio that no geometry package has: a manifest is
#: repetitive JSON, not a megabyte of zeros, and an entry that expands 200-fold is a
#: decompression bomb aimed at this process's memory. Checked above a floor so a tiny
#: entry with an unusual ratio (an empty array compresses enormously) is not refused.
MAX_COMPRESSION_RATIO = 200
RATIO_FLOOR_BYTES = 4 << 20
#: The arrays a sensing part is made of, and the role each belongs to.
SENSING_ARRAYS = {
@@ -86,6 +110,9 @@ HOUSING_ARRAYS = {
_DTYPE_BY_KEY = {"f4": np.float32, "u4": np.uint32, "i4": np.int32}
_KEY_BY_DTYPE = {np.dtype(v): k for k, v in _DTYPE_BY_KEY.items()}
#: Inside a sealed package the ciphertext is one entry with this name.
PAYLOAD_ENTRY = "payload"
@dataclass(frozen=True)
class PackageContents:
@@ -167,21 +194,16 @@ def build_manifest(
],
"entries": {name: dict(entries[name]) for name in sorted(entries)},
"bounds_sensor_m": [[float(v) for v in row] for row in assets.bounds_sensor_m],
# ``None`` means "not locked". Stored as an explicit null rather than omitted,
# so a reader cannot mistake "the writer had no lock feature" for "unlocked".
"platform_lock": (
assets.platform_lock.to_dict() if assets.platform_lock is not None else None
),
}
def write_package(
assets: SceneAssets,
output: Path,
*,
sensor_model: str = "finger",
hardware_revisions: Iterable[str] = ("*",),
) -> CompileReport:
"""Write a runtime package atomically: a half-written package never appears."""
from orisys_app_shared.scene3d.contracts import validate_assets as _validate
_validate(assets)
output = Path(output)
def _arrays_of(assets: SceneAssets) -> dict[str, np.ndarray]:
"""Every array a package holds, in its canonical entry names."""
arrays: dict[str, np.ndarray] = {}
sensing = assets.sensing_part()
arrays["sensing/positions.npy"] = np.ascontiguousarray(sensing.positions_sensor_m)
@@ -198,7 +220,21 @@ def write_package(
arrays["housing/positions.npy"] = np.ascontiguousarray(part.positions_sensor_m)
arrays["housing/normals.npy"] = np.ascontiguousarray(part.normals_sensor)
arrays["housing/triangles.npy"] = np.ascontiguousarray(part.triangles)
return arrays
def _payload_bytes(
assets: SceneAssets,
*,
sensor_model: str,
hardware_revisions: Iterable[str],
) -> tuple[bytes, dict[str, Any], str]:
"""The *content* of a package as one zip in memory, plus its manifest and hash.
Shared by both envelope versions, which is what stops a sealed package from
becoming a second format with its own bugs: sealing encrypts exactly these bytes.
"""
arrays = _arrays_of(assets)
entries = {name: _record(array) for name, array in arrays.items()}
manifest = build_manifest(
assets, entries=entries, sensor_model=sensor_model,
@@ -209,30 +245,124 @@ def write_package(
)
manifest["asset_hash"] = digest
manifest["entry_digest"] = {name: record["sha256"] for name, record in entries.items()}
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, "w", compression=zipfile.ZIP_DEFLATED) as payload:
payload.writestr("manifest.json", json.dumps(manifest, indent=2, sort_keys=True))
for name in sorted(arrays):
payload.writestr(name, _npy_bytes(arrays[name]))
return buffer.getvalue(), manifest, digest
envelope = {
def write_package(
assets: SceneAssets,
output: Path,
*,
sensor_model: str = "finger",
hardware_revisions: Iterable[str] = ("*",),
sealing: Optional[Sealing] = None,
) -> CompileReport:
"""Write a runtime package atomically: a half-written package never appears.
Three cases, and the design document's order of protection decides them:
* **no key material** — ``envelope_version`` 1, plain and unsigned. This is what an
engineering build produces, and what every package written before stage 5 is. Its
manifest is still self-checked on read (hashes, shapes, identities), so a
corrupted package is refused; what it does not have is *origin*.
* **a signing key only** — ``envelope_version`` 2, still plain, but the manifest's
asset hash and the manifest itself are signed. This is the design's "完整性 and
来源" step: the client carries only a public key, and a customer who edits a mesh
or a sampling table is caught, because the hash that covers them is inside the
signature.
* **a content key too** — ``envelope_version`` 2 with the content encrypted by
AES-256-GCM (``protect.py``). Encryption is *in addition to* the signature, never
instead of it.
A sealed write also *reads itself back* when the caller supplied a verification key:
a build that cannot open its own release should not have produced it.
"""
from orisys_app_shared.scene3d.contracts import validate_assets as _validate
_validate(assets)
output = Path(output)
payload, manifest, digest = _payload_bytes(
assets, sensor_model=sensor_model, hardware_revisions=hardware_revisions
)
seal = sealing or Sealing()
if seal.key_id and seal.provider is None:
# A key id with no key is the dangerous half-state: the caller asked for a
# sealed package and would get a plain one that a release script then ships as
# protected. Refuse, rather than decide on the caller's behalf.
raise error(
"asset_auth_failed",
f"a content key id ({seal.key_id}) was given without a key provider: "
"refusing to write a plain package in its place",
)
members = _payload_members(payload)
common: dict[str, Any] = {
"magic": MAGIC,
"envelope_version": ENVELOPE_VERSION,
"payload": "plain",
"schema_version": SCHEMA_VERSION,
"asset_id": assets.asset_id,
"asset_version": assets.asset_version,
"entry_count": len(arrays),
"note": "unsigned and unencrypted: stage 5 adds the signature and the AEAD",
# The array entries, not the archive's members: this field has meant "how many
# arrays" since version 1, and a reader that counted members would disagree.
"entry_count": len(manifest.get("entries") or {}),
}
output.parent.mkdir(parents=True, exist_ok=True)
temporary = output.with_suffix(output.suffix + ".partial")
try:
with zipfile.ZipFile(temporary, "w", compression=zipfile.ZIP_DEFLATED) as archive:
archive.writestr("envelope.json", json.dumps(envelope, indent=2, sort_keys=True))
archive.writestr("manifest.json", json.dumps(manifest, indent=2, sort_keys=True))
for name in sorted(arrays):
buffer = _npy_bytes(arrays[name])
archive.writestr(name, buffer)
if seal.encrypts():
envelope, ciphertext = _protect.seal_payload(
payload,
key_id=seal.key_id,
provider=seal.provider,
signing_key_pem=seal.signing_key_pem,
signing_key_reference=seal.signing_key_reference,
extra=common,
)
with zipfile.ZipFile(temporary, "w", compression=zipfile.ZIP_STORED) as archive:
archive.writestr("envelope.json", json.dumps(envelope, indent=2, sort_keys=True))
# Stored, not deflated: the ciphertext is incompressible, and a second
# compression layer only hides the fact that it is ciphertext.
archive.writestr(PAYLOAD_ENTRY, ciphertext)
elif seal.signs():
envelope = dict(common)
envelope["envelope_version"] = 2
envelope["payload"] = "plain"
# For a plain payload the signed digest *is* the manifest's asset hash: the
# manifest already covers every entry's sha256, so signing it covers the
# geometry, the sampling table and the platform lock without inventing a
# second, differently-computed number for the same thing.
envelope["payload_sha256"] = digest
envelope = _protect.sign_envelope(
envelope,
private_key_pem=seal.signing_key_pem or b"",
reference=seal.signing_key_reference,
)
with zipfile.ZipFile(temporary, "w", compression=zipfile.ZIP_DEFLATED) as archive:
archive.writestr("envelope.json", json.dumps(envelope, indent=2, sort_keys=True))
for name in sorted(members):
archive.writestr(name, members[name])
else:
envelope = dict(common)
envelope["envelope_version"] = 1
envelope["payload"] = "plain"
envelope["note"] = (
"plain and unsigned: integrity only, no origin (see protect.py for what "
"a signature adds)"
)
with zipfile.ZipFile(temporary, "w", compression=zipfile.ZIP_DEFLATED) as archive:
archive.writestr("envelope.json", json.dumps(envelope, indent=2, sort_keys=True))
for name in sorted(members):
archive.writestr(name, members[name])
temporary.replace(output)
except Exception:
temporary.unlink(missing_ok=True)
raise
if seal.verify_key_pem is not None and (seal.encrypts() or seal.signs()):
# The build machine proves the release opens. Without this, the first machine to
# discover a broken envelope is the customer's.
read_package(output, keys=seal.provider, verify_key_pem=seal.verify_key_pem)
return CompileReport(
output=output,
asset_hash=digest,
@@ -241,6 +371,12 @@ def write_package(
)
def _payload_members(payload: bytes) -> dict[str, bytes]:
"""The members of the in-memory content zip, by name."""
with zipfile.ZipFile(io.BytesIO(payload)) as content:
return {name: content.read(name) for name in content.namelist()}
def _npy_bytes(array: np.ndarray) -> bytes:
import io
@@ -249,112 +385,220 @@ def _npy_bytes(array: np.ndarray) -> bytes:
return buffer.getvalue()
def read_package(path: Path, *, keys: Any = None) -> PackageContents:
"""Open, verify and hand back the arrays. Refuses rather than repairs."""
def _guard_sizes(archive: zipfile.ZipFile) -> None:
"""Refuse an archive that wants more memory than a geometry package needs.
Two limits, and the ratio is the one that matters: a 2 KB zip can *declare* a
256 MB entry, and the declared size is what this process would allocate while
decompressing it. So a ratio no geometry package reaches (a manifest is repetitive
JSON, not megabytes of zeros) is refused above a floor that keeps small,
genuinely-compressible arrays legitimate.
"""
names = archive.namelist()
if len(names) > MAX_ENTRIES:
raise error("asset_auth_failed", f"the package declares {len(names)} entries")
total = 0
for info in archive.infolist():
size = int(info.file_size)
total += size
if size > MAX_ENTRY_BYTES or total > MAX_TOTAL_BYTES:
raise error("asset_auth_failed", "the package is larger than allowed")
if size > RATIO_FLOOR_BYTES:
ratio = size / max(int(info.compress_size), 1)
if ratio > MAX_COMPRESSION_RATIO:
raise error(
"asset_auth_failed",
f"{info.filename} expands {ratio:.0f}x: refusing a decompression bomb",
)
def _read_envelope(archive: zipfile.ZipFile) -> dict[str, Any]:
try:
envelope = json.loads(archive.read("envelope.json"))
except KeyError as exc:
raise error("asset_auth_failed", "the envelope is missing") from exc
except Exception as exc: # noqa: BLE001
raise error("asset_auth_failed", f"the envelope is unreadable: {exc}") from exc
if not isinstance(envelope, dict) or envelope.get("magic") != MAGIC:
raise error("asset_auth_failed", "this is not an orisys scene package")
version = int(envelope.get("envelope_version", 0))
if version not in SUPPORTED_ENVELOPE_VERSIONS:
raise error(
"asset_auth_failed",
f"unsupported envelope version {envelope.get('envelope_version')!r} "
f"(this build reads {SUPPORTED_ENVELOPE_VERSIONS})",
)
envelope["envelope_version"] = version
return envelope
def _verify_members(
members: Mapping[str, bytes], *, expected_hash: Optional[str] = None
) -> PackageContents:
"""The stage-2 verification, over a name→bytes mapping from either envelope."""
try:
manifest = json.loads(members["manifest.json"])
except KeyError as exc:
raise error("asset_auth_failed", "the manifest is missing") from exc
except Exception as exc: # noqa: BLE001
raise error("asset_auth_failed", f"the manifest is unreadable: {exc}") from exc
if not isinstance(manifest, dict):
raise error("asset_auth_failed", "the manifest is not an object")
entries = manifest.get("entries")
if not isinstance(entries, dict) or not entries:
raise error("asset_auth_failed", "the manifest describes no entries")
recorded_digest = str(manifest.get("asset_hash") or "")
# Verify the asset hash over the entries and the manifest *without* the
# hash fields, exactly as the writer computed it.
stripped = {
key: value
for key, value in manifest.items()
if key not in ("asset_hash", "entry_digest")
}
recomputed = compute_asset_hash(
{name: str(record.get("sha256", "")) for name, record in entries.items()},
stripped,
)
if not recorded_digest or recomputed != recorded_digest:
raise error(
"asset_auth_failed",
"the manifest hash does not match its contents",
recorded=recorded_digest[:16],
recomputed=recomputed[:16],
)
if expected_hash and str(expected_hash) != recorded_digest:
# For a signed plain package this is where the signature meets the content: the
# signed digest and the manifest's own hash have to be the same number, or the
# signature covers something other than what was delivered.
raise error(
"asset_auth_failed",
"the signed payload digest does not match the manifest's asset hash",
signed=str(expected_hash)[:16],
manifest=recorded_digest[:16],
)
arrays: dict[str, np.ndarray] = {}
for name, record in entries.items():
_entry_name(name)
if name not in members:
raise error("asset_auth_failed", f"the manifest promises {name!r}, which is absent")
raw = members[name]
try:
# ``io.BytesIO``: numpy's reader wants a real file object (it seeks,
# tells and reads headers), and a hand-rolled stub fails obscurely.
array = np.load(io.BytesIO(raw), allow_pickle=False)
except Exception as exc: # noqa: BLE001
raise error(
"asset_auth_failed",
f"{name} is not a readable array: {type(exc).__name__}: {exc}",
) from exc
dtype_key = str(record.get("dtype"))
expected_dtype = _DTYPE_BY_KEY.get(dtype_key)
if expected_dtype is None or array.dtype != np.dtype(expected_dtype):
raise error(
"field_domain_mismatch",
f"{name}: dtype {array.dtype} does not match the manifest {dtype_key!r}",
)
expected_shape = tuple(int(v) for v in record.get("shape", ()))
if tuple(array.shape) != expected_shape:
raise error(
"field_domain_mismatch",
f"{name}: shape {tuple(array.shape)} does not match the manifest "
f"{expected_shape}",
)
if content_digest({"data": np.ascontiguousarray(array)}) != str(
record.get("sha256")
):
raise error("asset_auth_failed", f"{name} does not match its recorded hash")
arrays[name] = array
unknown = [
name
for name in members
if name not in entries and name not in ("envelope.json", "manifest.json")
]
if unknown:
raise error("asset_auth_failed", f"undeclared entries: {unknown}")
return PackageContents(manifest=manifest, arrays=arrays)
def read_package(
path: Path,
*,
keys: Any = None,
verify_key_pem: Optional[bytes] = None,
) -> PackageContents:
"""Open, verify and hand back the arrays. Refuses rather than repairs.
``keys`` is the :class:`~orisys_app_shared.scene3d.contracts.KeyProvider` for an
encrypted package; ``verify_key_pem`` the public key a signed one is checked
against. Both are optional, and each missing one is refused *specifically* — a
sealed package without a provider is not the same failure as a signed package
whose signature is wrong, and the message has to say which happened.
"""
path = Path(path)
if not path.is_file():
raise error("asset_rejected", f"package not found: {path}")
diagnostics: list[Diagnostic] = []
with zipfile.ZipFile(path) as archive:
_guard_sizes(archive)
envelope = _read_envelope(archive)
names = archive.namelist()
if len(names) > MAX_ENTRIES:
raise error("asset_auth_failed", f"the package declares {len(names)} entries")
total = 0
for info in archive.infolist():
total += int(info.file_size)
if int(info.file_size) > MAX_ENTRY_BYTES or total > MAX_TOTAL_BYTES:
raise error("asset_auth_failed", "the package is larger than allowed")
try:
envelope = json.loads(archive.read("envelope.json"))
except KeyError as exc:
raise error("asset_auth_failed", "the envelope is missing") from exc
except Exception as exc: # noqa: BLE001
raise error("asset_auth_failed", f"the envelope is unreadable: {exc}") from exc
if not isinstance(envelope, dict) or envelope.get("magic") != MAGIC:
raise error("asset_auth_failed", "this is not an orisys scene package")
if int(envelope.get("envelope_version", 0)) != ENVELOPE_VERSION:
raise error(
"asset_auth_failed",
f"unsupported envelope version {envelope.get('envelope_version')!r}",
)
if str(envelope.get("payload")) != "plain":
# Stage 5 territory. Saying so is better than pretending the bytes are
# data and failing later with a numpy error.
raise error(
"asset_auth_failed",
f"the payload is {envelope.get('payload')!r}; this build opens plain "
"packages only (signing and encryption are a later stage)",
)
try:
manifest = json.loads(archive.read("manifest.json"))
except KeyError as exc:
raise error("asset_auth_failed", "the manifest is missing") from exc
except Exception as exc: # noqa: BLE001
raise error("asset_auth_failed", f"the manifest is unreadable: {exc}") from exc
if not isinstance(manifest, dict):
raise error("asset_auth_failed", "the manifest is not an object")
entries = manifest.get("entries")
if not isinstance(entries, dict) or not entries:
raise error("asset_auth_failed", "the manifest describes no entries")
recorded_digest = str(manifest.get("asset_hash") or "")
# Verify the asset hash over the entries and the manifest *without* the
# hash fields, exactly as the writer computed it.
stripped = {
key: value
for key, value in manifest.items()
if key not in ("asset_hash", "entry_digest")
}
recomputed = compute_asset_hash(
{name: str(record.get("sha256", "")) for name, record in entries.items()},
stripped,
)
if not recorded_digest or recomputed != recorded_digest:
raise error(
"asset_auth_failed",
"the manifest hash does not match its contents",
recorded=recorded_digest[:16],
recomputed=recomputed[:16],
)
arrays: dict[str, np.ndarray] = {}
for name, record in entries.items():
entry = _entry_name(name)
if name not in names:
raise error("asset_auth_failed", f"the manifest promises {name!r}, which is absent")
raw = archive.read(name)
try:
# ``io.BytesIO``: numpy's reader wants a real file object (it seeks,
# tells and reads headers), and a hand-rolled stub fails obscurely.
array = np.load(io.BytesIO(raw), allow_pickle=False)
except Exception as exc: # noqa: BLE001
version = int(envelope["envelope_version"])
kind = str(envelope.get("payload"))
if version == 1:
if kind != "plain":
raise error(
"asset_auth_failed",
f"{name} is not a readable array: {type(exc).__name__}: {exc}",
f"envelope version 1 cannot carry a {kind!r} payload",
)
members = {
name: archive.read(name)
for name in names
if name != "envelope.json"
}
return _verify_members(members)
if kind == "aes-gcm":
if PAYLOAD_ENTRY not in names:
raise error("asset_auth_failed", "the encrypted payload entry is missing")
plaintext = _protect.open_payload(
envelope,
archive.read(PAYLOAD_ENTRY),
provider=keys,
public_key_pem=verify_key_pem,
)
try:
with zipfile.ZipFile(io.BytesIO(plaintext)) as content:
_guard_sizes(content)
members = {name: content.read(name) for name in content.namelist()}
except zipfile.BadZipFile as exc:
raise error(
"asset_auth_failed", "the decrypted payload is not a readable archive"
) from exc
dtype_key = str(record.get("dtype"))
expected_dtype = _DTYPE_BY_KEY.get(dtype_key)
if expected_dtype is None or array.dtype != np.dtype(expected_dtype):
raise error(
"field_domain_mismatch",
f"{name}: dtype {array.dtype} does not match the manifest {dtype_key!r}",
)
expected_shape = tuple(int(v) for v in record.get("shape", ()))
if tuple(array.shape) != expected_shape:
raise error(
"field_domain_mismatch",
f"{name}: shape {tuple(array.shape)} does not match the manifest "
f"{expected_shape}",
)
if content_digest({"data": np.ascontiguousarray(array)}) != str(
record.get("sha256")
):
raise error("asset_auth_failed", f"{name} does not match its recorded hash")
arrays[name] = array
unknown = [name for name in names if name not in entries and name not in (
"envelope.json", "manifest.json"
)]
if unknown:
raise error("asset_auth_failed", f"undeclared entries: {unknown}")
return PackageContents(manifest=manifest, arrays=arrays, diagnostics=tuple(diagnostics))
return _verify_members(members)
if kind == "plain":
# Signed, not encrypted: the arrays are in the clear, and the signature is
# what makes them the producer's. A reader that does not check it would be
# accepting an unsigned package while claiming otherwise.
if isinstance(envelope.get("signature"), Mapping):
if verify_key_pem is None:
raise error(
"asset_auth_failed",
"this package is signed; a verification key is required "
"(pass --verify-key or set one in the application)",
)
_protect.verify_envelope(envelope, public_key_pem=verify_key_pem)
members = {
name: archive.read(name)
for name in names
if name != "envelope.json"
}
return _verify_members(
members, expected_hash=str(envelope.get("payload_sha256") or "")
)
raise error(
"asset_auth_failed",
f"unsupported payload kind {kind!r} in envelope version {version}",
)
return PackageContents(manifest={}, arrays={}, diagnostics=tuple(diagnostics))
def scene_from_contents(
@@ -454,26 +698,26 @@ def scene_from_contents(
source_vertex_of_render=(
freeze(source, dtype=np.int32) if source is not None else None
),
platform_lock=PlatformLock.from_dict(manifest.get("platform_lock")),
)
validate_assets(assets, expected)
return assets
class PackagedAssetLoader:
"""The stage-2 loader: plain, verified packages (stage 5 adds the envelope)."""
"""The loader: plain, signed and encrypted packages through one entry point."""
def load(
self,
path: Path,
expected: ExpectedGeometry,
keys: Any = None,
*,
verify_key_pem: Optional[bytes] = None,
) -> SceneAssets:
if keys is not None:
raise error(
"asset_auth_failed",
"this build cannot use a key provider: packages are plain so far",
)
return scene_from_contents(read_package(path), expected)
return scene_from_contents(
read_package(path, keys=keys, verify_key_pem=verify_key_pem), expected
)
def validate(
self, assets: SceneAssets, expected: ExpectedGeometry
@@ -481,10 +725,16 @@ class PackagedAssetLoader:
return validate_assets(assets, expected)
def describe_package(path: Path) -> dict[str, Any]:
def describe_package(
path: Path,
*,
keys: Any = None,
verify_key_pem: Optional[bytes] = None,
) -> dict[str, Any]:
"""A report on a package without building a scene: for the validate tool."""
contents = read_package(path)
contents = read_package(path, keys=keys, verify_key_pem=verify_key_pem)
manifest = contents.manifest
lock = manifest.get("platform_lock")
return {
"asset_id": manifest.get("asset_id"),
"asset_version": manifest.get("asset_version"),
@@ -492,6 +742,7 @@ def describe_package(path: Path) -> dict[str, Any]:
"field_domain_id": (manifest.get("field_domain") or {}).get("id"),
"shape_hw": (manifest.get("field_domain") or {}).get("shape_hw"),
"asset_hash": manifest.get("asset_hash"),
"platform_lock": lock,
"parts": {part.get("id"): part.get("role") for part in manifest.get("parts", [])},
"entries": {
name: {"dtype": r.get("dtype"), "shape": r.get("shape")}
@@ -501,3 +752,28 @@ def describe_package(path: Path) -> dict[str, Any]:
"sensing": int(contents.arrays["sensing/positions.npy"].shape[0]),
},
}
def describe_envelope(path: Path) -> dict[str, Any]:
"""What the container says about itself, without opening the payload.
Used by the tools to report whether a package is plain, signed or encrypted —
and, importantly, to do so *without* a key: a support engineer can say "this
package is sealed for key id X and signed by Y" without being able to read it.
"""
path = Path(path)
if not path.is_file():
raise error("asset_rejected", f"package not found: {path}")
with zipfile.ZipFile(path) as archive:
_guard_sizes(archive)
envelope = _read_envelope(archive)
names = set(archive.namelist())
sealed_bytes = (
int(archive.getinfo(PAYLOAD_ENTRY).file_size)
if PAYLOAD_ENTRY in names
else 0
)
summary = _protect.envelope_summary(envelope)
summary["members"] = sorted(names)
summary["sealed_bytes"] = sealed_bytes
return summary
shared/src/orisys_app_shared/scene3d/protect.py
+532
View File
@@ -0,0 +1,532 @@
"""Sealed packages: Ed25519 signatures and optional AES-GCM, with no home-made crypto.
Stage 5 in ``docs/architecture/studio-3d-stages.md``; §5.2 of the design document is
the specification this module implements:
* **integrity and origin** (item 2): the build machine signs the package with an
Ed25519 private key and the client carries only the public key. A SHA-256 digest
says what the bytes are; it does not say who produced them, so the digest alone is
never described here as tamper protection.
* **optional encryption** (item 3): AES-256-GCM from a mature library, one fresh
nonce per sealed package, the envelope metadata bound as associated data, and the
plaintext digest inside the signed part. No XOR, no password hidden in the
extension, no encrypted-looking zip.
* **key isolation** (item 4): the signing private key never enters this module's
runtime path — :func:`sign_envelope` is a *build-time* function and only the public
key (``verify_envelope``) is needed to read a package. Content keys come from a
:class:`~orisys_app_shared.scene3d.contracts.KeyProvider`, so authorisation stays
outside the format.
* **decryption scope** (item 5): the payload is opened in memory and never written to
a plaintext temporary file; no function here logs a key, a nonce or a plaintext
digest of anything but a package the caller already holds.
Two things this module deliberately does **not** claim:
* it cannot stop somebody who controls the machine from reading the geometry after
decryption — the renderer needs the mesh, and a Python process cannot wipe its own
memory. The design document says the goal is to raise the cost of copying, not to
make it impossible, and that a structure which must not leak has to be replaced by
a visual proxy instead of by encryption;
* it does not protect anything if the same geometry ships in plaintext elsewhere.
Signing this package while the plain SDK mesh travels beside it protects nothing.
The format (``envelope_version`` 2, one ``payload`` entry):
``envelope.json``
public, signed metadata: magic, version, ``payload: "aes-gcm"``, ``key_id``,
``cipher``, ``nonce``, ``salt``, ``kdf``, ``payload_sha256`` (of the *plaintext*),
``ciphertext_sha256`` (of the stored bytes) and the ``signature`` block.
``payload``
the AES-256-GCM ciphertext of exactly the bytes a v1 package would have held: a
zip archive with ``manifest.json`` and the ``.npy`` entries.
The signed bytes are ``_DOMAIN + canonical_json(envelope without "signature") +
plaintext digest``, which is why the signature covers the manifest and every entry
(the manifest is inside the plaintext, whose digest is signed) as well as the key id,
the nonce, the salt and the ciphertext's own digest. The GCM associated data is the
same canonical envelope-without-signature bytes, so a package whose metadata was
edited fails decryption *and* verification, not just one of the two.
"""
from __future__ import annotations
import base64
import json
import secrets
from pathlib import Path
from typing import Any, Mapping, Optional
from orisys_app_shared.scene3d.contracts import (
Diagnostic,
KeyProvider,
diagnostic,
error,
)
from orisys_app_shared.scene3d.digest import bytes_digest
#: Separates our messages from any other use of the same key, so a signature made for
#: something else can never be replayed as a package signature.
_DOMAIN = b"orisys-scene/envelope/v2\x00"
#: The only cipher and KDF we write, and the only ones we accept on read. Names are
#: spelled out rather than negotiated: an unknown one is refused, not downgraded.
CIPHER = "AES-256-GCM"
KDF = "HKDF-SHA256"
SIGNATURE_ALGORITHM = "Ed25519"
NONCE_BYTES = 12
SALT_BYTES = 32
KEY_BYTES = 32
SIGNATURE_BYTES = 64
#: How the per-package content key is derived from the provider's key. The key id is
#: bound in, so a key handed out for one product cannot decrypt another product's
#: package even when the provider returns the same bytes.
_KDF_INFO = b"orisys-scene/content-key/v2\x00"
class CryptoUnavailable(RuntimeError):
"""The crypto backend is not installed. Its own type, so callers can say so."""
def backend_available() -> bool:
"""Whether a mature crypto backend is importable in this process.
Checked rather than assumed: the *runtime* needs the backend only to read a
sealed package, so a machine without it keeps working with plain packages, and
the compiler refuses to write a sealed one instead of writing a plain file that
claims to be protected.
"""
try:
import cryptography # noqa: F401
except Exception: # noqa: BLE001
return False
return True
def backend_version() -> str:
try:
import cryptography
return str(getattr(cryptography, "__version__", "unknown"))
except Exception: # noqa: BLE001
return "unavailable"
def _backend():
"""The AES-GCM/Ed25519/HKDF primitives, or a clear refusal."""
try:
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import (
Ed25519PrivateKey,
Ed25519PublicKey,
)
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
except Exception as exc: # noqa: BLE001
raise CryptoUnavailable(
"reading or writing a sealed scene package needs the 'cryptography' "
f"package, which is not installed here ({type(exc).__name__}: {exc})"
) from exc
return {
"hashes": hashes,
"serialization": serialization,
"Ed25519PrivateKey": Ed25519PrivateKey,
"Ed25519PublicKey": Ed25519PublicKey,
"AESGCM": AESGCM,
"HKDF": HKDF,
}
# --------------------------------------------------------------------- encoding
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _unb64(text: Any, *, field: str, length: Optional[int] = None) -> bytes:
if not isinstance(text, str) or not text:
raise error("asset_auth_failed", f"the envelope's {field} is missing")
try:
raw = base64.b64decode(text, validate=True)
except Exception as exc: # noqa: BLE001
raise error("asset_auth_failed", f"the envelope's {field} is not base64") from exc
if length is not None and len(raw) != length:
raise error(
"asset_auth_failed",
f"the envelope's {field} is {len(raw)} bytes, expected {length}",
)
return raw
def canonical_envelope(envelope: Mapping[str, Any]) -> bytes:
"""The bytes that are signed.
Everything except the signature itself, in a canonical JSON form. Sorted keys and
no whitespace, so the same content always signs the same bytes on every platform.
This *does* include the ciphertext's digest and length, which is why a package
whose ciphertext was swapped cannot keep a valid signature.
"""
body = {key: value for key, value in envelope.items() if key != "signature"}
return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode(
"utf-8"
)
#: Fields that cannot be part of the AEAD's associated data: they only exist *after*
#: the ciphertext does. They are still covered by the signature, so dropping them here
#: does not leave them unprotected — it just keeps the two mechanisms in their proper
#: order (encrypt, then describe the ciphertext, then sign everything).
_POST_ENCRYPTION_FIELDS = ("signature", "ciphertext_sha256", "ciphertext_bytes")
def gcm_aad(envelope: Mapping[str, Any]) -> bytes:
"""The associated data bound into the ciphertext: all pre-encryption metadata."""
body = {
key: value
for key, value in envelope.items()
if key not in _POST_ENCRYPTION_FIELDS
}
return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode(
"utf-8"
)
def signed_bytes(envelope: Mapping[str, Any], plaintext_digest: str) -> bytes:
"""Domain-separated message: the envelope bytes plus the plaintext's digest."""
return _DOMAIN + canonical_envelope(envelope) + b"\n" + str(plaintext_digest).encode("ascii")
# ------------------------------------------------------------------- key material
def generate_signing_key() -> bytes:
"""A fresh Ed25519 private key in PEM form, for a build machine."""
backend = _backend()
key = backend["Ed25519PrivateKey"].generate()
return key.private_bytes(
encoding=backend["serialization"].Encoding.PEM,
format=backend["serialization"].PrivateFormat.PKCS8,
encryption_algorithm=backend["serialization"].NoEncryption(),
)
def public_key_pem(private_key_pem: bytes) -> bytes:
"""The half that ships with the client."""
backend = _backend()
key = _load_private(private_key_pem, backend)
return key.public_key().public_bytes(
encoding=backend["serialization"].Encoding.PEM,
format=backend["serialization"].PublicFormat.SubjectPublicKeyInfo,
)
def _load_private(pem: bytes, backend: Mapping[str, Any]):
try:
return backend["serialization"].load_pem_private_key(bytes(pem), password=None)
except Exception as exc: # noqa: BLE001
raise error(
"asset_auth_failed",
f"the signing key is not a readable PEM private key: {type(exc).__name__}",
) from exc
def _load_public(pem: bytes, backend: Mapping[str, Any]):
try:
return backend["serialization"].load_pem_public_key(bytes(pem))
except Exception as exc: # noqa: BLE001
raise error(
"asset_auth_failed",
f"the verification key is not a readable PEM public key: {type(exc).__name__}",
) from exc
class MappingKeyProvider:
"""A :class:`KeyProvider` over an in-memory table.
For tests and for a build machine. A production client is expected to implement
the same one-method protocol over whatever it really has (a device key, a licence
server, an offline token) — which is why the format only ever asks for
``get_content_key(key_id)``.
"""
def __init__(self, keys: Optional[Mapping[str, Any]] = None) -> None:
self.keys: dict[str, bytes] = {
str(key_id): _key_bytes(value, key_id=str(key_id))
for key_id, value in (keys or {}).items()
}
def get_content_key(self, key_id: str) -> bytes:
key = self.keys.get(str(key_id))
if key is None:
# No key material in the message, ever.
raise error(
"asset_auth_failed",
f"no content key is available for key id {key_id!r}",
)
return key
def key_ids(self) -> tuple[str, ...]:
return tuple(sorted(self.keys))
def _key_bytes(value: Any, *, key_id: str) -> bytes:
if isinstance(value, bytes):
raw = value
elif isinstance(value, str):
text = value.strip()
try:
raw = bytes.fromhex(text)
except ValueError:
try:
raw = base64.b64decode(text, validate=True)
except Exception as exc: # noqa: BLE001
raise error(
"asset_rejected",
f"the key for {key_id!r} is neither hex nor base64",
) from exc
else:
raise error("asset_rejected", f"the key for {key_id!r} must be bytes or text")
if len(raw) != KEY_BYTES:
raise error(
"asset_rejected",
f"the key for {key_id!r} is {len(raw)} bytes; {KEY_BYTES} are required",
)
return raw
def load_key_file(path: Path, *, key_id: str) -> MappingKeyProvider:
"""A key provider with one key, read from a file (hex, base64 or raw)."""
path = Path(path)
if not path.is_file():
raise error("asset_rejected", f"the key file is missing: {path}")
raw = path.read_bytes()
text = raw.decode("utf-8", errors="ignore").strip()
try:
key = _key_bytes(text, key_id=key_id)
except Exception: # noqa: BLE001 - a binary key file is legitimate
key = _key_bytes(raw, key_id=key_id)
return MappingKeyProvider({key_id: key})
# ------------------------------------------------------------------- derive/open
def _content_key(provider: KeyProvider, key_id: str, salt: bytes) -> bytes:
backend = _backend()
master = provider.get_content_key(key_id)
if not isinstance(master, (bytes, bytearray)) or len(bytes(master)) != KEY_BYTES:
size = len(bytes(master)) if isinstance(master, (bytes, bytearray)) else "not bytes"
raise error(
"asset_auth_failed",
f"the provider returned {size} for a content key; {KEY_BYTES} bytes are required",
)
info = _KDF_INFO + str(key_id).encode("utf-8")
return backend["HKDF"](
algorithm=backend["hashes"].SHA256(),
length=KEY_BYTES,
salt=salt,
info=info,
).derive(bytes(master))
def seal_payload(
payload: bytes,
*,
key_id: str,
provider: KeyProvider,
signing_key_pem: Optional[bytes] = None,
signing_key_reference: str = "",
extra: Optional[Mapping[str, Any]] = None,
) -> tuple[dict[str, Any], bytes]:
"""Encrypt (and optionally sign) one plaintext payload.
Returns the public envelope and the ciphertext. A fresh random nonce and salt are
generated per call: the design demands that a nonce is never reused under one key,
and a 96-bit random nonce is the standard way to get that (with a test that the
same payload sealed twice shares neither nonce nor ciphertext). The salt keeps the
*derived* key unique per package as well, so a reused master key is never used
directly.
"""
backend = _backend()
plaintext = bytes(payload)
plaintext_digest = bytes_digest(plaintext, domain="payload")
salt = secrets.token_bytes(SALT_BYTES)
nonce = secrets.token_bytes(NONCE_BYTES)
key = _content_key(provider, str(key_id), salt)
envelope: dict[str, Any] = {
"envelope_version": 2,
"payload": "aes-gcm",
"cipher": CIPHER,
"kdf": KDF,
"key_id": str(key_id),
"salt": _b64(salt),
"nonce": _b64(nonce),
"payload_sha256": plaintext_digest,
"crypto_backend": f"cryptography/{backend_version()}",
}
if extra:
envelope.update({str(k): v for k, v in extra.items()})
aad = gcm_aad(envelope)
ciphertext = backend["AESGCM"](key).encrypt(nonce, plaintext, aad)
envelope["ciphertext_sha256"] = bytes_digest(ciphertext, domain="ciphertext")
envelope["ciphertext_bytes"] = len(ciphertext)
if signing_key_pem is not None:
envelope["signature"] = _sign(
envelope,
plaintext_digest=plaintext_digest,
private_key_pem=signing_key_pem,
reference=signing_key_reference,
)
return envelope, ciphertext
def _sign(
envelope: Mapping[str, Any],
*,
plaintext_digest: str,
private_key_pem: bytes,
reference: str,
) -> dict[str, Any]:
backend = _backend()
key = _load_private(private_key_pem, backend)
signature = key.sign(signed_bytes(envelope, plaintext_digest))
return {
"algorithm": SIGNATURE_ALGORITHM,
"key_id": str(reference or "unknown"),
"value": _b64(signature),
}
def sign_envelope(
envelope: Mapping[str, Any],
*,
private_key_pem: bytes,
reference: str = "",
) -> dict[str, Any]:
"""Sign an existing envelope (build-time). Returns a copy with ``signature``."""
if not envelope.get("payload_sha256"):
raise error("asset_auth_failed", "the envelope has no payload digest to sign")
out = dict(envelope)
out["signature"] = _sign(
out,
plaintext_digest=str(envelope["payload_sha256"]),
private_key_pem=private_key_pem,
reference=reference,
)
return out
def verify_envelope(
envelope: Mapping[str, Any], *, public_key_pem: bytes
) -> tuple[Diagnostic, ...]:
"""Verify the signature. Raises on a bad one; returns warnings otherwise."""
backend = _backend()
signature = envelope.get("signature")
if not isinstance(signature, Mapping):
raise error("asset_auth_failed", "the envelope is not signed")
if str(signature.get("algorithm")) != SIGNATURE_ALGORITHM:
raise error(
"asset_auth_failed",
f"unsupported signature algorithm {signature.get('algorithm')!r}",
)
raw = _unb64(signature.get("value"), field="signature", length=SIGNATURE_BYTES)
key = _load_public(public_key_pem, backend)
message = signed_bytes(envelope, str(envelope.get("payload_sha256") or ""))
try:
key.verify(raw, message)
except Exception as exc: # noqa: BLE001
raise error(
"asset_auth_failed",
"the package signature does not match its contents",
signing_key=str(signature.get("key_id") or ""),
) from exc
return ()
def open_payload(
envelope: Mapping[str, Any],
ciphertext: bytes,
*,
provider: Optional[KeyProvider],
public_key_pem: Optional[bytes] = None,
) -> bytes:
"""Verify, decrypt and hand back the plaintext payload.
Order matters and is not negotiable: the signature is checked *before* the key is
asked for and before anything is decrypted, so a tampered package cannot make the
client touch its key material or its cipher at all.
"""
backend = _backend()
if str(envelope.get("payload")) != "aes-gcm":
raise error(
"asset_auth_failed",
f"the payload is {envelope.get('payload')!r}, not {('aes-gcm')!r}",
)
if public_key_pem is not None:
verify_envelope(envelope, public_key_pem=public_key_pem)
elif "signature" in envelope:
raise error(
"asset_auth_failed",
"the package is signed but no verification key was supplied",
)
if str(envelope.get("cipher")) != CIPHER:
raise error("asset_auth_failed", f"unsupported cipher {envelope.get('cipher')!r}")
if str(envelope.get("kdf")) != KDF:
raise error("asset_auth_failed", f"unsupported key derivation {envelope.get('kdf')!r}")
if provider is None:
raise error(
"asset_auth_failed",
"this package is encrypted and no key provider was supplied",
)
recorded_cipher = str(envelope.get("ciphertext_sha256") or "")
if recorded_cipher and bytes_digest(ciphertext, domain="ciphertext") != recorded_cipher:
raise error("asset_auth_failed", "the ciphertext does not match its recorded hash")
declared = envelope.get("ciphertext_bytes")
if declared is not None and int(declared) != len(ciphertext):
raise error(
"asset_auth_failed",
f"the envelope declares {declared} ciphertext bytes, found {len(ciphertext)}",
)
salt = _unb64(envelope.get("salt"), field="salt", length=SALT_BYTES)
nonce = _unb64(envelope.get("nonce"), field="nonce", length=NONCE_BYTES)
key = _content_key(provider, str(envelope.get("key_id") or ""), salt)
aad = gcm_aad(envelope)
try:
plaintext = backend["AESGCM"](key).decrypt(nonce, bytes(ciphertext), aad)
except Exception as exc: # noqa: BLE001
raise error(
"asset_auth_failed",
"the package could not be decrypted: wrong key, or the bytes were changed",
key_id=str(envelope.get("key_id") or ""),
) from exc
recorded_plain = str(envelope.get("payload_sha256") or "")
if not recorded_plain or bytes_digest(plaintext, domain="payload") != recorded_plain:
raise error(
"asset_auth_failed",
"the decrypted payload does not match the digest that was signed",
)
return plaintext
def envelope_summary(envelope: Mapping[str, Any]) -> dict[str, Any]:
"""A log-safe view: no key material, no digests of anything secret.
Used by the validate tool and by diagnostics, so a support ticket can say what a
package is without printing anything that would help decrypt another one.
"""
signature = envelope.get("signature")
return {
"envelope_version": envelope.get("envelope_version"),
"payload": envelope.get("payload"),
"cipher": envelope.get("cipher"),
"key_id": envelope.get("key_id"),
"sealed": str(envelope.get("payload")) == "aes-gcm",
"ciphertext_bytes": envelope.get("ciphertext_bytes"),
"signed": isinstance(signature, Mapping),
"signing_key": (signature or {}).get("key_id") if isinstance(signature, Mapping) else None,
"crypto_backend": envelope.get("crypto_backend"),
}
def diagnose_unavailable(exc: BaseException) -> Diagnostic:
"""One place turns "no crypto backend" into the diagnostic the UI shows."""
return diagnostic(
"asset_auth_failed",
"sealed scene packages need the 'cryptography' package, which is not "
"installed on this machine; plain packages still work",
detail=str(exc),
)