Store the depth channel in an analysis result and serve it to the 2D depth view
A replay could draw arrows but not depth, and the recording view said there was nothing to draw even when a result existed. Depth is now quantised per frame (14 B empty, 486 B at the worst frame seen), read from the cache by the same draw path live uses, and the raw-recording note appears only when no result exists. Format version 2; see apps/studio/docs/replay-analysis.md.
This commit is contained in:
ykcheungab
@@ -186,6 +186,13 @@ class ImageFrame:
|
||||
flow_frame_id: int | None = None
|
||||
depth_valid: bool | None = None
|
||||
depth_frame_id: int | None = None
|
||||
#: True when a stored **analysis result** exists for this recording, so the derived
|
||||
#: views have a frame of their own to draw even when *this* frame is the recording's
|
||||
#: raw picture. It is what keeps the panel from telling the operator there is nothing
|
||||
#: to draw while a result is sitting on disk: the modes are offered, and the view
|
||||
#: that needs derived data is served from the result. ``False`` means no result — the
|
||||
#: only case in which "a recording stores only the raw camera image" is the truth.
|
||||
result_available: bool = False
|
||||
|
||||
def has_flow(self) -> bool:
|
||||
if self.flow is None:
|
||||
@@ -243,16 +250,23 @@ class ImagePlan:
|
||||
|
||||
|
||||
def plan_image(frame: ImageFrame) -> ImagePlan:
|
||||
"""Derive the modes from the frame's *source* and its own fields.
|
||||
"""Derive the modes from the frame's *source*, its own fields, and any result behind it.
|
||||
|
||||
Two different questions, deliberately answered in two different places:
|
||||
|
||||
* **which views may the operator choose?** For a source that carries derived fields
|
||||
(live, recompute) all of them — a single dropped frame must not silently switch
|
||||
the operator's view. For a source that never does (a recording, a camera preview)
|
||||
exactly one, because offering the others would be offering a lie;
|
||||
(live, recompute, analysis) all of them — a single dropped frame must not silently
|
||||
switch the operator's view. A recording offers them too *when a stored analysis
|
||||
result exists for it* (``result_available``): the recording's own picture carries
|
||||
no flow, but the result does, and the view that needs it is drawn from the result.
|
||||
With no result there is nothing behind the picture at all, so the raw frame is the
|
||||
only thing it can honestly be;
|
||||
* **what can this frame actually be drawn as?** ``missing`` says which field is
|
||||
absent, and the panel states it in one line.
|
||||
|
||||
The raw-recording note is therefore an *absence* statement and appears only in the
|
||||
one case where it is true: no result, nothing to draw, and the panel says which
|
||||
action produces one.
|
||||
"""
|
||||
derived = str(frame.source or SOURCE_LIVE) in DERIVED_SOURCES
|
||||
missing: list[str] = []
|
||||
@@ -260,7 +274,7 @@ def plan_image(frame: ImageFrame) -> ImagePlan:
|
||||
missing.append("flow")
|
||||
if not frame.has_depth():
|
||||
missing.append("depth")
|
||||
if not derived:
|
||||
if not derived and not frame.result_available:
|
||||
return ImagePlan(
|
||||
available_modes=(RECORDED,),
|
||||
default_mode=RECORDED,
|
||||
@@ -271,6 +285,19 @@ def plan_image(frame: ImageFrame) -> ImagePlan:
|
||||
stale=bool(frame.stale),
|
||||
note_key=RAW_NOTE_KEY,
|
||||
)
|
||||
if not derived:
|
||||
# A recording with a result waiting behind it. ``RECORDED`` stays the default —
|
||||
# the picture on screen *is* the recorded image until the operator asks for
|
||||
# something else — and nothing is claimed to be missing, because nothing is.
|
||||
return ImagePlan(
|
||||
available_modes=(RECORDED, *DERIVED_MODES),
|
||||
default_mode=RECORDED,
|
||||
missing=(),
|
||||
source=str(frame.source or SOURCE_LIVE),
|
||||
frame_id=int(frame.frame_id),
|
||||
t_rel_s=float(frame.t_rel_s),
|
||||
stale=bool(frame.stale),
|
||||
)
|
||||
default = ARROW_PURE if frame.has_flow() else (DEPTH if frame.has_depth() else ARROW_PURE)
|
||||
return ImagePlan(
|
||||
available_modes=DERIVED_MODES,
|
||||
|
||||
@@ -7,8 +7,10 @@ and stores the flow samples a replay draws, so seeking is a file read.
|
||||
|
||||
What is stored is deliberately *not* the chain's output. One frame of raw output is
|
||||
1,176,000 B (``flow`` [300,280,2] float32, ``depth_map``, ``valid_mask``, the gray
|
||||
image), which is 21.2 GB for ten minutes at 30 fps. The selected samples cost 4,034 B
|
||||
per frame. The rest is recomputed — that is what makes the cache a cache.
|
||||
image), which is 21.2 GB for ten minutes at 30 fps. The selected samples cost 4,048 B
|
||||
per frame, and the depth channel — which is all zeros on the great majority of frames —
|
||||
costs 14 B more when it is empty. The rest is recomputed — that is what makes the cache
|
||||
a cache.
|
||||
|
||||
The three modules have one job each:
|
||||
|
||||
|
||||
@@ -10,8 +10,9 @@ properties matter more than speed:
|
||||
pass leaves a directory that refuses to open by default.
|
||||
* **Memory must not grow with the recording.** Frames accumulate into a fixed-size
|
||||
chunk and the chunk is written once. The bound is
|
||||
``chunk_frames * frame_payload_bytes(max candidates)`` — 256 frames of the shipped
|
||||
finger grid is about 1.0 MB — and ``chunk_frames`` is clamped so a caller cannot
|
||||
``chunk_frames * frame_payload_bytes(max candidates, with_depth=True)`` plus one
|
||||
depth plane per frame that carries a non-empty one — 256 frames of the shipped
|
||||
finger grid is about 1.1 MB — and ``chunk_frames`` is clamped so a caller cannot
|
||||
raise it without bound.
|
||||
|
||||
Storage is not free, so :func:`evict` removes whole results against a stated size cap.
|
||||
|
||||
@@ -5,7 +5,7 @@ only what a replay draws: the flow samples the shared selector returns, not the
|
||||
field they came from. The fields a frame produces are far too large to cache — the
|
||||
measured cost is 1,176,000 B per frame (``flow`` 672,000 + ``depth_map`` 336,000 +
|
||||
``valid_mask`` 84,000 + the gray ``img`` 84,000), which is 21.2 GB for ten minutes at
|
||||
30 fps. The samples themselves cost 4,034 B per frame at the shipped finger grid.
|
||||
30 fps. The samples themselves cost 4,048 B per frame at the shipped finger grid.
|
||||
|
||||
Three structural facts decide the layout, and each one is a measured result rather
|
||||
than a guess:
|
||||
@@ -28,6 +28,28 @@ served by re-deriving activity from magnitude; only a change to the sample *grid
|
||||
(``flow_grid_spacing``, ``flow_edge_margin``, ``max_arrows``) moves the candidate
|
||||
locations, and that policy is recorded in the provenance for comparison instead of
|
||||
being baked into the key.
|
||||
|
||||
**Depth is stored, and it is stored quantised.** A frame's own ``depth_map`` is
|
||||
336,000 B (300 × 280 float32) — 6.0 GB per ten minutes at 30 fps, which is why the
|
||||
obvious encoding is out — and on the two real recordings it is empty far more often
|
||||
than not: 241 of 264 frames of the eight-times clip and 32 of 33 frames of the box
|
||||
recording carry a map that is *exactly* all zeros, with the worst non-empty frame at
|
||||
336 of 84,000 pixels. So the record states the depth channel's own statistics every
|
||||
frame (present, non-zero count, minimum, maximum) and appends a payload only for a
|
||||
frame that has one, as uint8 over that frame's own range, zlib'd at level 6. The
|
||||
measured cost is 14 B per empty frame and 486 B for the worst non-empty frame seen;
|
||||
the largest error that can be introduced is half a quantisation step,
|
||||
``(max - min) / 510``, measured at 6.6e-5 on the clip whose peak is 0.034. That is
|
||||
below half of one step of the 256-entry colour ramp the depth view paints with, so a
|
||||
reconstructed frame cannot be coloured differently from the original.
|
||||
|
||||
Sparse storage was measured and rejected: 4-byte indices per non-zero pixel cost
|
||||
910 B for the worst non-empty frame against 486 B for the whole quantised map, because
|
||||
a mostly-zero byte plane is nearly free for zlib while pixel indices are not. Island
|
||||
restriction was measured and rejected too — 418 B against 486 B, a saving of 5.2 B per
|
||||
frame of a 4,048 B record — because it would make the reconstruction depend on the
|
||||
validity-mask keyframe (the box chain has none) and would silently drop a depth value
|
||||
that fell outside the island. See ``tests/test_analysis_depth.py`` for the numbers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -35,6 +57,7 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
import struct
|
||||
import zlib
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
@@ -50,8 +73,12 @@ __all__ = [
|
||||
"CHUNK_HEADER",
|
||||
"CHUNK_MAGIC",
|
||||
"CHUNK_TRAILER",
|
||||
"CODEC_EMPTY",
|
||||
"CODEC_RAW",
|
||||
"CODEC_ZLIB",
|
||||
"DEPTH_BITS",
|
||||
"DEPTH_HEADER",
|
||||
"FRAME_FLAG_DEPTH",
|
||||
"FRAME_FLAG_DROPPED",
|
||||
"FRAME_FLAG_VALIDITY",
|
||||
"FRAME_HEADER",
|
||||
@@ -59,12 +86,16 @@ __all__ = [
|
||||
"KEYFRAME_HEADER",
|
||||
"KEYFRAME_MAGIC",
|
||||
"KEYFRAME_MASK",
|
||||
"MAX_DEPTH_PIXELS",
|
||||
"UNIT",
|
||||
"DecodedDepth",
|
||||
"FramePayload",
|
||||
"KeyframeRef",
|
||||
"Provenance",
|
||||
"ResultKey",
|
||||
"decode_depth",
|
||||
"decode_frame",
|
||||
"encode_depth",
|
||||
"encode_frame",
|
||||
"frame_payload_bytes",
|
||||
"keyframe_payload",
|
||||
@@ -78,8 +109,9 @@ RESULT_DIRNAME = "analysis"
|
||||
|
||||
#: On-disk result format. Bump only for an incompatible layout change; a change to
|
||||
#: what the *chain computes* belongs in :data:`ALGORITHM_VERSION` instead.
|
||||
#: Version 2 adds the depth channel to the frame record (see the module docstring).
|
||||
ANALYSIS_FORMAT = "orisys-replay-analysis"
|
||||
ANALYSIS_FORMAT_VERSION = 1
|
||||
ANALYSIS_FORMAT_VERSION = 2
|
||||
|
||||
#: Identity of the analysis the pass runs: the SDK calls it makes, in order.
|
||||
#: ``compute_deformation(decompose=True, check_motion=False)`` then
|
||||
@@ -92,11 +124,16 @@ ALGORITHM_VERSION = "flow-hhd-contact-1"
|
||||
#: unit :class:`~orisys_app_shared.scene3d.visual_policy.FlowSamples` documents.
|
||||
UNIT = "uv_px"
|
||||
|
||||
#: ``frame_id``, ``t_rel_s``, grid keyframe id, flags, candidate count, depth statistic.
|
||||
#: The four trailing pad bytes keep the value arrays 8-byte aligned inside a record.
|
||||
FRAME_HEADER = struct.Struct("<idHBHI3x")
|
||||
#: ``frame_id``, ``t_rel_s``, grid keyframe id, flags, candidate count, depth
|
||||
#: non-zero count, depth minimum, depth maximum. The three trailing pad bytes keep
|
||||
#: the value arrays 8-byte aligned inside a record.
|
||||
FRAME_HEADER = struct.Struct("<idHBHIff3x")
|
||||
FRAME_FLAG_VALIDITY = 0x01
|
||||
FRAME_FLAG_DROPPED = 0x02
|
||||
#: The chain exposed a depth map for this frame: the three depth statistics in the
|
||||
#: header describe it and a depth payload follows. Absent flag means *no depth
|
||||
#: channel at all*, which is a different fact from "the channel was all zeros".
|
||||
FRAME_FLAG_DEPTH = 0x04
|
||||
|
||||
#: magic, kind, codec, count, height, width, flags, reserved, raw bytes, digest
|
||||
KEYFRAME_HEADER = struct.Struct("<4sBBIIIBBQ8s")
|
||||
@@ -104,8 +141,22 @@ KEYFRAME_MAGIC = b"ORAK"
|
||||
KEYFRAME_GRID = 0
|
||||
KEYFRAME_MASK = 1
|
||||
|
||||
#: codec, bits per stored value, height, width. A depth payload is this header
|
||||
#: followed by its stored bytes — none at all for :data:`CODEC_EMPTY`.
|
||||
DEPTH_HEADER = struct.Struct("<BBHH")
|
||||
DEPTH_BITS = 8
|
||||
|
||||
CODEC_RAW = 0
|
||||
CODEC_ZLIB = 1
|
||||
#: The map is exactly all zeros: ``height × width`` of them, and no payload. This is
|
||||
#: the common case by a wide margin, and saying so explicitly is what lets a reader
|
||||
#: explain an empty depth view instead of drawing a blank one.
|
||||
CODEC_EMPTY = 2
|
||||
|
||||
#: Refuse a depth header that asks for an implausible array before allocating it. The
|
||||
#: shipped grid is 300 × 280; this leaves room for a far larger sensor and no room for
|
||||
#: a corrupt header to become a 4 GB allocation.
|
||||
MAX_DEPTH_PIXELS = 1 << 26
|
||||
|
||||
#: magic, version, reserved, chunk index, frame count, header size
|
||||
CHUNK_HEADER = struct.Struct("<4sHHIII")
|
||||
@@ -207,6 +258,188 @@ def recording_identity(
|
||||
return canonical_digest8(payload).hex()
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- depth channel
|
||||
def _depth_plane(depth: Any) -> np.ndarray:
|
||||
"""A depth map as a 2-D float32 array, whatever leading singletons it arrives with.
|
||||
|
||||
The finger chain hands ``depth_map`` over as ``(1, H, W)`` — a batch axis nobody
|
||||
batches — and the validity mask it belongs beside is ``(H, W)``. Squeezing only
|
||||
leading singletons keeps a genuinely 3-D field (a future multi-plane depth) an
|
||||
error rather than something silently reinterpreted.
|
||||
"""
|
||||
array = np.asarray(depth, dtype=np.float32)
|
||||
while array.ndim > 2 and array.shape[0] == 1:
|
||||
array = array[0]
|
||||
if array.ndim != 2:
|
||||
raise ValueError(f"a depth map is 2-D, got shape {np.shape(depth)}")
|
||||
return np.ascontiguousarray(array)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DecodedDepth:
|
||||
"""One frame's depth channel: how it was stored, and the statistics that describe it."""
|
||||
|
||||
codec: int
|
||||
height: int
|
||||
width: int
|
||||
nonzero: int
|
||||
minimum: float
|
||||
maximum: float
|
||||
#: The stored bytes below the header — empty for :data:`CODEC_EMPTY`, and the
|
||||
#: quantised plane decompressed for the other codecs.
|
||||
stored: bytes = b""
|
||||
#: How many bytes those were on disk, before decompression. Kept separate from
|
||||
#: ``len(stored)`` because the compressed and the decoded length are different
|
||||
#: numbers and only one of them is what the record costs.
|
||||
stored_bytes: int = 0
|
||||
|
||||
@property
|
||||
def pixels(self) -> int:
|
||||
return int(self.height) * int(self.width)
|
||||
|
||||
@property
|
||||
def payload_bytes(self) -> int:
|
||||
"""Every byte this frame's depth channel occupies, header included."""
|
||||
return DEPTH_HEADER.size + int(self.stored_bytes)
|
||||
|
||||
@property
|
||||
def is_empty(self) -> bool:
|
||||
return self.codec == CODEC_EMPTY
|
||||
|
||||
|
||||
def _quantise(depth: np.ndarray) -> tuple[np.ndarray, float, float]:
|
||||
"""uint8 codes over the array's own ``[min, max]``, plus that range.
|
||||
|
||||
The range is the frame's own rather than a session-wide one on purpose: a depth
|
||||
response that is large on one frame and small on the next keeps its full 256 levels
|
||||
either way. The price is that the range has to travel with the frame, which is why
|
||||
it lives in the frame header (8 B) instead of in each payload.
|
||||
"""
|
||||
low = float(depth.min())
|
||||
high = float(depth.max())
|
||||
span = high - low
|
||||
if span <= 0.0:
|
||||
return np.zeros(depth.shape, dtype=np.uint8), low, high
|
||||
scaled = (depth.astype(np.float64) - low) / span
|
||||
codes = np.clip(np.rint(scaled * 255.0), 0.0, 255.0).astype(np.uint8)
|
||||
return codes, low, high
|
||||
|
||||
|
||||
def _dequantise(codes: np.ndarray, low: float, high: float) -> np.ndarray:
|
||||
span = float(high) - float(low)
|
||||
if span <= 0.0:
|
||||
return np.full(codes.shape, float(low), dtype=np.float32)
|
||||
return (float(low) + codes.astype(np.float64) * (span / 255.0)).astype(np.float32)
|
||||
|
||||
|
||||
def encode_depth(depth: Any) -> tuple[bytes, int, float, float]:
|
||||
"""One frame's depth channel as ``(payload, nonzero, min, max)``.
|
||||
|
||||
The payload is a :data:`DEPTH_HEADER` and, unless the map is exactly all zeros, the
|
||||
zlib'd uint8 plane. An empty map stores the header alone: 6 bytes that state the
|
||||
shape, which is what a reader needs to rebuild the zeros rather than guess at them.
|
||||
"""
|
||||
array = _depth_plane(depth)
|
||||
height, width = int(array.shape[0]), int(array.shape[1])
|
||||
if height * width > MAX_DEPTH_PIXELS:
|
||||
raise ValueError(f"a {height}x{width} depth map is larger than this format holds")
|
||||
nonzero = int(np.count_nonzero(array))
|
||||
low = float(array.min())
|
||||
high = float(array.max())
|
||||
if nonzero == 0:
|
||||
return (
|
||||
DEPTH_HEADER.pack(CODEC_EMPTY, DEPTH_BITS, height, width),
|
||||
nonzero,
|
||||
low,
|
||||
high,
|
||||
)
|
||||
codes, low, high = _quantise(array)
|
||||
raw = codes.tobytes()
|
||||
stored = zlib.compress(raw, 6)
|
||||
codec = CODEC_ZLIB
|
||||
if len(stored) >= len(raw):
|
||||
codec = CODEC_RAW
|
||||
stored = raw
|
||||
return (
|
||||
DEPTH_HEADER.pack(codec, DEPTH_BITS, height, width) + stored,
|
||||
nonzero,
|
||||
low,
|
||||
high,
|
||||
)
|
||||
|
||||
|
||||
def decode_depth(
|
||||
buffer: bytes | bytearray | memoryview,
|
||||
offset: int = 0,
|
||||
*,
|
||||
low: float = 0.0,
|
||||
high: float = 0.0,
|
||||
) -> DecodedDepth:
|
||||
"""Read one depth payload. Raises :class:`ValueError` on anything inconsistent.
|
||||
|
||||
The stored plane is checked against the shape its own header declares, so a
|
||||
truncated or mis-sized payload is refused here rather than surfacing later as an
|
||||
array of the wrong shape.
|
||||
"""
|
||||
view = memoryview(buffer)
|
||||
if len(view) < offset + DEPTH_HEADER.size:
|
||||
raise ValueError("a depth payload is shorter than its header")
|
||||
codec, bits, height, width = DEPTH_HEADER.unpack_from(view, offset)
|
||||
cursor = offset + DEPTH_HEADER.size
|
||||
pixels = int(height) * int(width)
|
||||
if pixels <= 0 or pixels > MAX_DEPTH_PIXELS:
|
||||
raise ValueError(f"a depth payload declares an implausible shape {height}x{width}")
|
||||
if int(bits) != DEPTH_BITS:
|
||||
raise ValueError(f"a depth payload stores {bits} bits per value, not {DEPTH_BITS}")
|
||||
stored = bytes(view[cursor:])
|
||||
if int(codec) == CODEC_EMPTY:
|
||||
return DecodedDepth(
|
||||
codec=int(codec),
|
||||
height=int(height),
|
||||
width=int(width),
|
||||
nonzero=0,
|
||||
minimum=0.0,
|
||||
maximum=0.0,
|
||||
)
|
||||
if int(codec) not in (CODEC_RAW, CODEC_ZLIB):
|
||||
raise ValueError(f"a depth payload uses codec {codec}")
|
||||
if int(codec) == CODEC_ZLIB:
|
||||
try:
|
||||
raw = zlib.decompress(stored)
|
||||
except zlib.error as exc:
|
||||
raise ValueError(f"a depth payload does not decompress: {exc}") from exc
|
||||
else:
|
||||
raw = stored
|
||||
if len(raw) != pixels:
|
||||
raise ValueError(
|
||||
f"a depth payload decoded to {len(raw)} bytes for a {height}x{width} map"
|
||||
)
|
||||
return DecodedDepth(
|
||||
codec=int(codec),
|
||||
height=int(height),
|
||||
width=int(width),
|
||||
nonzero=pixels,
|
||||
minimum=float(low),
|
||||
maximum=float(high),
|
||||
stored=raw,
|
||||
stored_bytes=len(stored),
|
||||
)
|
||||
|
||||
|
||||
def depth_values(decoded: DecodedDepth, *, low: float, high: float) -> np.ndarray:
|
||||
"""The depth map a decoded payload describes, as float32.
|
||||
|
||||
A :data:`CODEC_EMPTY` payload becomes ``height × width`` exact zeros — the map the
|
||||
chain produced, not a placeholder — so the depth view and its statistics describe
|
||||
the same array.
|
||||
"""
|
||||
shape = (int(decoded.height), int(decoded.width))
|
||||
if decoded.codec == CODEC_EMPTY:
|
||||
return np.zeros(shape, dtype=np.float32)
|
||||
codes = np.frombuffer(decoded.stored, dtype=np.uint8).reshape(shape)
|
||||
return _dequantise(codes, float(low), float(high))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- frame encoding
|
||||
def encode_frame(
|
||||
*,
|
||||
@@ -218,15 +451,22 @@ def encode_frame(
|
||||
magnitude: np.ndarray,
|
||||
active: np.ndarray,
|
||||
validity_digest: bytes | None = None,
|
||||
depth: Any | None = None,
|
||||
depth_nonzero: int = 0,
|
||||
dropped: bool = False,
|
||||
) -> bytes:
|
||||
"""One frame record: a fixed header, then the value arrays.
|
||||
"""One frame record: a fixed header, then the value arrays and the depth payload.
|
||||
|
||||
Arrays are little-endian regardless of host order, so a result written on one
|
||||
machine reads back identically on another. ``dx``/``dy``/``magnitude`` are stored
|
||||
as float32 — the precision the display needs and the reason a stored payload is
|
||||
compared to a live selection *cast to float32* rather than in float64.
|
||||
|
||||
``depth`` is the frame's own map, or ``None`` when the chain exposed none; the two
|
||||
are different facts (``FRAME_FLAG_DEPTH``) and the header's statistics belong to the
|
||||
first. ``depth_nonzero`` is only consulted when ``depth`` is ``None``, which is how
|
||||
a caller that can count but cannot afford to hand the array over still records the
|
||||
count exactly.
|
||||
"""
|
||||
dx = np.ascontiguousarray(dx, dtype="<f4").reshape(-1)
|
||||
dy = np.ascontiguousarray(dy, dtype="<f4").reshape(-1)
|
||||
@@ -247,6 +487,11 @@ def encode_frame(
|
||||
flags |= FRAME_FLAG_VALIDITY
|
||||
if dropped:
|
||||
flags |= FRAME_FLAG_DROPPED
|
||||
depth_payload = b""
|
||||
depth_low = depth_high = 0.0
|
||||
if depth is not None:
|
||||
flags |= FRAME_FLAG_DEPTH
|
||||
depth_payload, depth_nonzero, depth_low, depth_high = encode_depth(depth)
|
||||
parts = [
|
||||
FRAME_HEADER.pack(
|
||||
int(frame_id),
|
||||
@@ -255,6 +500,8 @@ def encode_frame(
|
||||
flags,
|
||||
count,
|
||||
max(0, int(depth_nonzero)),
|
||||
float(depth_low),
|
||||
float(depth_high),
|
||||
),
|
||||
dx.tobytes(),
|
||||
dy.tobytes(),
|
||||
@@ -266,6 +513,7 @@ def encode_frame(
|
||||
if len(digest) != 8:
|
||||
raise ValueError(f"validity digest must be 8 bytes, got {len(digest)}")
|
||||
parts.append(digest)
|
||||
parts.append(depth_payload)
|
||||
return b"".join(parts)
|
||||
|
||||
|
||||
@@ -284,6 +532,17 @@ class DecodedFrame:
|
||||
magnitude: np.ndarray
|
||||
active: np.ndarray
|
||||
validity_digest: bytes | None
|
||||
#: ``None`` when the frame carried no depth channel; otherwise the payload, whose
|
||||
#: ``stored`` bytes are still quantised and need :func:`depth_values`.
|
||||
depth: DecodedDepth | None = None
|
||||
|
||||
def depth_map(self) -> np.ndarray | None:
|
||||
"""This frame's depth map, or ``None`` when it had no depth channel."""
|
||||
if self.depth is None:
|
||||
return None
|
||||
return depth_values(
|
||||
self.depth, low=self.depth.minimum, high=self.depth.maximum
|
||||
)
|
||||
|
||||
|
||||
def decode_frame(
|
||||
@@ -291,9 +550,16 @@ def decode_frame(
|
||||
) -> DecodedFrame:
|
||||
"""Decode one frame record starting at ``offset``. Ignores any trailing bytes."""
|
||||
view = memoryview(buffer)
|
||||
frame_id, t_rel, grid_id, flags, count, depth_nonzero = FRAME_HEADER.unpack_from(
|
||||
view, offset
|
||||
)
|
||||
(
|
||||
frame_id,
|
||||
t_rel,
|
||||
grid_id,
|
||||
flags,
|
||||
count,
|
||||
depth_nonzero,
|
||||
depth_low,
|
||||
depth_high,
|
||||
) = FRAME_HEADER.unpack_from(view, offset)
|
||||
cursor = offset + FRAME_HEADER.size
|
||||
arrays = []
|
||||
for _ in range(3):
|
||||
@@ -308,6 +574,12 @@ def decode_frame(
|
||||
validity = None
|
||||
if flags & FRAME_FLAG_VALIDITY:
|
||||
validity = bytes(view[cursor : cursor + 8])
|
||||
cursor += 8
|
||||
depth = None
|
||||
if flags & FRAME_FLAG_DEPTH:
|
||||
depth = decode_depth(
|
||||
view, cursor, low=float(depth_low), high=float(depth_high)
|
||||
)
|
||||
return DecodedFrame(
|
||||
frame_id=int(frame_id),
|
||||
t_rel_s=float(t_rel),
|
||||
@@ -320,13 +592,29 @@ def decode_frame(
|
||||
magnitude=arrays[2],
|
||||
active=active.astype(bool),
|
||||
validity_digest=validity,
|
||||
depth=depth,
|
||||
)
|
||||
|
||||
|
||||
def frame_payload_bytes(candidates: int, *, with_validity: bool = True) -> int:
|
||||
"""Stored bytes of one frame record at ``candidates`` sample locations."""
|
||||
def frame_payload_bytes(
|
||||
candidates: int,
|
||||
*,
|
||||
with_validity: bool = True,
|
||||
with_depth: bool = False,
|
||||
depth_stored: int = 0,
|
||||
) -> int:
|
||||
"""Stored bytes of one frame record at ``candidates`` sample locations.
|
||||
|
||||
``with_depth`` adds the 6-byte depth header every frame that carries a channel pays;
|
||||
``depth_stored`` adds a payload of that many bytes on top, for a caller sizing the
|
||||
memory bound rather than the common case.
|
||||
"""
|
||||
total = FRAME_HEADER.size + int(candidates) * 12 + (int(candidates) + 7) // 8
|
||||
return total + 8 if with_validity else total
|
||||
if with_validity:
|
||||
total += 8
|
||||
if with_depth:
|
||||
total += DEPTH_HEADER.size + max(0, int(depth_stored))
|
||||
return total
|
||||
|
||||
|
||||
def keyframe_payload(kind: int, array: np.ndarray) -> bytes:
|
||||
@@ -373,6 +661,30 @@ class FramePayload:
|
||||
validity_digest: bytes | None = None
|
||||
depth_nonzero: int = 0
|
||||
dropped: bool = False
|
||||
#: This frame's depth map at the chain's own resolution, or ``None`` when the frame
|
||||
#: carried no depth channel. An all-zero map is *not* ``None``: "the response was
|
||||
#: zero everywhere" and "there was no response" are different answers and a depth
|
||||
#: view that shows the same blank picture for both is the one that misleads.
|
||||
depth: np.ndarray | None = None
|
||||
depth_min: float = 0.0
|
||||
depth_max: float = 0.0
|
||||
#: Stored bytes of this frame's depth payload — the 6-byte header for an empty map,
|
||||
#: header plus plane otherwise, and 0 when the frame had no depth channel. It is
|
||||
#: what a caller adds to the fixed part of a record to say what depth cost.
|
||||
depth_stored_bytes: int = 0
|
||||
|
||||
@property
|
||||
def has_depth(self) -> bool:
|
||||
return self.depth is not None
|
||||
|
||||
@property
|
||||
def depth_is_empty(self) -> bool:
|
||||
"""True when a depth channel exists and is exactly zero everywhere."""
|
||||
return self.depth is not None and self.depth_nonzero == 0
|
||||
|
||||
@property
|
||||
def depth_pixels(self) -> int:
|
||||
return 0 if self.depth is None else int(self.depth.size)
|
||||
|
||||
@property
|
||||
def candidates(self) -> int:
|
||||
@@ -412,6 +724,7 @@ class FramePayload:
|
||||
magnitude=self.magnitude,
|
||||
active=self.active,
|
||||
validity_digest=self.validity_digest,
|
||||
depth=self.depth,
|
||||
depth_nonzero=self.depth_nonzero,
|
||||
dropped=self.dropped,
|
||||
)
|
||||
|
||||
@@ -462,6 +462,25 @@ class AnalysisReader:
|
||||
f"frame {frame_id} has {decoded.count} values but grid "
|
||||
f"{decoded.grid_id} has {rows.size} locations"
|
||||
)
|
||||
depth = None
|
||||
depth_min = depth_max = 0.0
|
||||
depth_stored = 0
|
||||
if decoded.depth is not None:
|
||||
try:
|
||||
depth = decoded.depth_map()
|
||||
except ValueError as exc:
|
||||
raise AnalysisFormatError(f"frame {frame_id}: {exc}") from exc
|
||||
depth_min = float(decoded.depth.minimum)
|
||||
depth_max = float(decoded.depth.maximum)
|
||||
depth_stored = decoded.depth.payload_bytes
|
||||
counted = int(np.count_nonzero(depth))
|
||||
if counted != decoded.depth_nonzero:
|
||||
# The header's count and the stored plane must describe one array. A
|
||||
# mismatch means the record is corrupt, not that the map is empty.
|
||||
raise AnalysisFormatError(
|
||||
f"frame {frame_id} stores {decoded.depth_nonzero} non-zero depth "
|
||||
f"pixels but its payload holds {counted}"
|
||||
)
|
||||
return FramePayload(
|
||||
frame_id=decoded.frame_id,
|
||||
t_rel_s=decoded.t_rel_s,
|
||||
@@ -475,6 +494,10 @@ class AnalysisReader:
|
||||
validity_digest=decoded.validity_digest,
|
||||
depth_nonzero=decoded.depth_nonzero,
|
||||
dropped=decoded.dropped,
|
||||
depth=depth,
|
||||
depth_min=depth_min,
|
||||
depth_max=depth_max,
|
||||
depth_stored_bytes=depth_stored,
|
||||
)
|
||||
|
||||
def frame_at(self, t_rel_s: float) -> FramePayload | None:
|
||||
|
||||
@@ -371,6 +371,9 @@ class LiveFrame:
|
||||
depth_abs_sum: float
|
||||
reset_applied: bool
|
||||
segment: int
|
||||
#: The chain's depth map for this frame, or ``None`` when it exposed none. Held only
|
||||
#: until the writer has encoded it: one frame at a time, never accumulated.
|
||||
depth: np.ndarray | None = None
|
||||
|
||||
def payload_matches(self, stored: Any) -> bool:
|
||||
"""True when a stored payload equals this frame's selection, in stored precision.
|
||||
@@ -836,6 +839,7 @@ class AnalysisService:
|
||||
),
|
||||
reset_applied=reset_applied,
|
||||
segment=int(number),
|
||||
depth=depth_array,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------- internals
|
||||
@@ -927,9 +931,17 @@ class AnalysisService:
|
||||
magnitude=samples.magnitude,
|
||||
active=samples.active,
|
||||
validity_digest=validity_digest,
|
||||
depth=live.depth,
|
||||
depth_nonzero=live.depth_nonzero,
|
||||
dropped=row.dropped,
|
||||
)
|
||||
counts = provenance.counts
|
||||
counts["frame_record_bytes"] = counts.get("frame_record_bytes", 0) + len(record)
|
||||
if live.depth is not None:
|
||||
counts["depth_frames"] = counts.get("depth_frames", 0) + 1
|
||||
counts["depth_pixels"] = int(
|
||||
np.shape(live.depth)[-2]
|
||||
) * int(np.shape(live.depth)[-1])
|
||||
writer.add_frame(record, frame_id=row.frame_id, t_rel_s=row.t_rel_s)
|
||||
if self.on_progress is not None:
|
||||
self.on_progress(
|
||||
|
||||
@@ -0,0 +1,454 @@
|
||||
"""The depth channel in an analysis result: what it costs, and that it comes back.
|
||||
|
||||
What is being defended:
|
||||
|
||||
1. **The round trip is within the stated error, frame for frame, on real footage.** The
|
||||
chain is walked once more, independently of the pass, and every stored depth map is
|
||||
compared with the map the chain produced for the same frame id. The accepted error is
|
||||
half one quantisation step of that frame's own range, ``(max - min) / 510`` — below
|
||||
half a step of the 256-entry colour ramp the depth view paints with, so a
|
||||
reconstruction cannot be coloured differently from the original.
|
||||
2. **The size is stated and held.** 14 B per frame whose depth map is exactly all zeros
|
||||
(the frame header's statistics plus a 6-byte payload header) and the header plus a
|
||||
zlib'd uint8 plane for a frame that has one. The alternative encodings were measured
|
||||
and rejected; the numbers are in the module docstring of ``analysis.model``.
|
||||
3. **"Empty" and "absent" are different answers.** A frame whose depth map is all zeros
|
||||
still carries the channel and states its shape; a frame whose chain exposed no depth
|
||||
channel at all carries none. A view that draws the same blank picture for both is the
|
||||
one that misleads, so the reader distinguishes them and the statistics travel with
|
||||
the record.
|
||||
4. **A corrupt depth payload is refused, not decoded into a plausible array.**
|
||||
|
||||
The fixtures use real footage — the 33-frame box recording under ``dist/`` and the
|
||||
264-frame eight-times-slow clip under ``.scratch/sdk-stream-pipeline``, both driven
|
||||
through the **finger** chain. Where the SDK or the footage is absent the tests skip
|
||||
rather than fake a result: a synthetic array would defend neither the statistics
|
||||
observed on this footage nor the size they produce.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from orisys_app_shared.analysis.cache import ResultCache, scan_results
|
||||
from orisys_app_shared.analysis.model import (
|
||||
CODEC_EMPTY,
|
||||
CODEC_RAW,
|
||||
CODEC_ZLIB,
|
||||
DEPTH_HEADER,
|
||||
decode_depth,
|
||||
encode_depth,
|
||||
encode_frame,
|
||||
frame_payload_bytes,
|
||||
)
|
||||
from orisys_app_shared.analysis.reader import AnalysisReader
|
||||
from orisys_app_shared.analysis.service import (
|
||||
AnalysisService,
|
||||
ChainSpec,
|
||||
_OpenChain,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
STUDIO = REPO / "dist" / "OrisysStudio"
|
||||
RECORDING = (
|
||||
STUDIO
|
||||
/ "outputs"
|
||||
/ "recordings"
|
||||
/ "20260930T015028Z_session0002_40cb8e7441434feab48bb646afdba011"
|
||||
)
|
||||
FINGER_CONFIG = STUDIO / "config" / "sensors" / "finger" / "sensor.json"
|
||||
FINGER_EXPORT = STUDIO / "config" / "sensors" / "finger" / "assets" / "30fps-0703"
|
||||
#: The 264-frame clip Step 0 measured the pass's throughput on. It lives in another
|
||||
#: task's scratch, so it is optional: without it the clip half of the round trip is
|
||||
#: reported as skipped and the box recording still runs.
|
||||
CLIP = REPO / ".scratch" / "sdk-stream-pipeline" / "long_8x.mp4"
|
||||
CLIP_FRAMES = 264
|
||||
CLIP_FPS = 30.0
|
||||
|
||||
#: Measured on the two recordings, at the finger chain's own 300 x 280 depth grid.
|
||||
#: Empty frames and the worst non-empty frame are the two numbers the encoding is
|
||||
#: chosen on, so they are asserted rather than described.
|
||||
DEPTH_GRID = (300, 280)
|
||||
EMPTY_PAYLOAD_BYTES = DEPTH_HEADER.size
|
||||
EMPTY_RECORD_BYTES = frame_payload_bytes(330, with_validity=True, with_depth=True)
|
||||
FIXED_RECORD_BYTES = EMPTY_RECORD_BYTES - DEPTH_HEADER.size
|
||||
|
||||
#: The worst non-empty frame either recording produced: 336 of 84,000 pixels on the clip,
|
||||
#: whose quantised plane zlib's to 486 B. The bound below is what the format promises;
|
||||
#: the measured number is what the recordings delivered, and both are asserted.
|
||||
WORST_NONEMPTY_PIXELS = 336
|
||||
WORST_NONEMPTY_PAYLOAD_BYTES = 486
|
||||
|
||||
|
||||
def _require_inputs() -> None:
|
||||
missing = [
|
||||
str(path)
|
||||
for path in (RECORDING / "frames.csv", RECORDING / "raw_0001.mp4", FINGER_CONFIG)
|
||||
if not path.exists()
|
||||
]
|
||||
if missing:
|
||||
pytest.skip("analysis inputs are not present: " + ", ".join(missing))
|
||||
pytest.importorskip("orisys", reason="the SDK is needed to run the analysis pass")
|
||||
|
||||
|
||||
def _finger_chain() -> ChainSpec:
|
||||
return ChainSpec(
|
||||
sensor_type="finger", config_ref=FINGER_CONFIG, export_dir=FINGER_EXPORT
|
||||
)
|
||||
|
||||
|
||||
def _result_directory(root: Path) -> Path:
|
||||
directories = [entry.directory for entry in scan_results(root) if entry.complete]
|
||||
assert directories, f"no complete result under {root}"
|
||||
return directories[0]
|
||||
|
||||
|
||||
def _clip_recording(tmp_path: Path) -> Path:
|
||||
"""The clip as a one-segment recording: a link to the video plus a frames index.
|
||||
|
||||
A bare container has no ``frames.csv``, and frame identity comes from the
|
||||
recording's own counter (Part A, section 6). Writing the index here is what makes
|
||||
the clip addressable by the same code path as a real recording instead of by an
|
||||
index standing in for an id.
|
||||
"""
|
||||
directory = tmp_path / "clip-recording"
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
video = directory / "raw_0001.mp4"
|
||||
if not video.exists():
|
||||
try:
|
||||
os.link(CLIP, video)
|
||||
except OSError:
|
||||
video.write_bytes(CLIP.read_bytes())
|
||||
index = directory / "frames.csv"
|
||||
rows = ["frame_id,t_rel_s,segment,segment_frame,in_video,dropped"]
|
||||
rows.extend(
|
||||
f"{number},{number / CLIP_FPS:.6f},1,{number},1,0"
|
||||
for number in range(1, CLIP_FRAMES + 1)
|
||||
)
|
||||
index.write_text("\n".join(rows) + "\n", encoding="utf-8")
|
||||
return directory
|
||||
|
||||
|
||||
def _chain_depth(recording: Path, video: str = "raw_0001.mp4") -> list[np.ndarray]:
|
||||
"""Walk the chain once more and return the depth map it produced for every frame.
|
||||
|
||||
Deliberately a *separate* walk from the pass: comparing a stored result with the
|
||||
object the pass held would only prove the writer copied what it was given.
|
||||
"""
|
||||
from orisys_app_shared.acquisition.sdk_worker import process_chain
|
||||
|
||||
produced: list[np.ndarray] = []
|
||||
with tempfile.TemporaryDirectory() as working:
|
||||
with _OpenChain(_finger_chain(), recording / video, Path(working)) as opened:
|
||||
while True:
|
||||
ok, frame = opened.read()
|
||||
if not ok or frame is None:
|
||||
break
|
||||
if opened.feed(frame) is None:
|
||||
raise AssertionError("the chain could not unroll a frame")
|
||||
depth, error = process_chain(opened.sensor)
|
||||
assert not error, error
|
||||
assert depth is not None, "the finger chain exposed no depth channel"
|
||||
array = np.asarray(depth, dtype=np.float32)
|
||||
while array.ndim > 2 and array.shape[0] == 1:
|
||||
array = array[0]
|
||||
produced.append(np.ascontiguousarray(array))
|
||||
return produced
|
||||
|
||||
|
||||
def _tolerance(frame) -> float:
|
||||
"""Half one quantisation step of this frame's own stored range."""
|
||||
return (float(frame.depth_max) - float(frame.depth_min)) / 510.0
|
||||
|
||||
|
||||
def _compare(stored, produced: list[np.ndarray]) -> dict[str, int]:
|
||||
"""Compare every stored depth map with the chain's, frame for frame.
|
||||
|
||||
The order is the pass's own: the finger chain consumes no frame before the first
|
||||
readable one, so decoded frame *k* is stored frame *k*.
|
||||
"""
|
||||
frames = list(stored)
|
||||
assert len(frames) == len(produced), (
|
||||
f"{len(frames)} stored frames against {len(produced)} decoded"
|
||||
)
|
||||
stats = {"frames": len(frames), "empty": 0, "nonempty": 0, "worst_pixels": 0}
|
||||
for frame, original in zip(frames, produced):
|
||||
assert frame.has_depth, f"frame {frame.frame_id} lost its depth channel"
|
||||
assert frame.depth.shape == DEPTH_GRID, frame.depth.shape
|
||||
assert original.shape == frame.depth.shape
|
||||
if frame.depth_nonzero:
|
||||
stats["nonempty"] += 1
|
||||
stats["worst_pixels"] = max(stats["worst_pixels"], frame.depth_nonzero)
|
||||
else:
|
||||
stats["empty"] += 1
|
||||
assert not bool(np.any(original)), (
|
||||
f"frame {frame.frame_id} is stored as an empty depth map but the chain "
|
||||
"produced a non-zero one"
|
||||
)
|
||||
# The statistic the record states is the statistic of the map the chain made.
|
||||
assert frame.depth_nonzero == int(np.count_nonzero(original))
|
||||
assert float(frame.depth_max) == float(original.max())
|
||||
assert float(frame.depth_min) == float(original.min())
|
||||
error = float(np.max(np.abs(np.asarray(frame.depth) - original)))
|
||||
assert error <= _tolerance(frame), (
|
||||
f"frame {frame.frame_id} is {error} away from the chain's map, over the "
|
||||
f"{_tolerance(frame)} the encoding accepts"
|
||||
)
|
||||
# The reconstruction carries the same zero/non-zero verdict everywhere: every
|
||||
# observed non-zero value is far above the half step, so the count survives too.
|
||||
assert int(np.count_nonzero(frame.depth)) == frame.depth_nonzero
|
||||
return stats
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the round trip
|
||||
@pytest.fixture(scope="module")
|
||||
def box_depth(tmp_path_factory):
|
||||
"""One real pass over the evidence recording, kept for the round trip and the size."""
|
||||
_require_inputs()
|
||||
root = tmp_path_factory.mktemp("analysis-depth-box")
|
||||
outcome = AnalysisService(
|
||||
RECORDING, chain=_finger_chain(), cache=ResultCache(root)
|
||||
).run()
|
||||
assert outcome.complete, outcome.reason
|
||||
return root
|
||||
|
||||
|
||||
def test_the_box_recording_round_trips_its_depth_within_the_stated_error(box_depth):
|
||||
produced = _chain_depth(RECORDING)
|
||||
reader = AnalysisReader(_result_directory(box_depth))
|
||||
try:
|
||||
stats = _compare(reader.frames(), produced)
|
||||
# The observed split, not a hope: 32 of the 33 frames carry a map that is
|
||||
# exactly all zeros, and the one that does not holds 76 pixels of 84,000.
|
||||
assert stats == {
|
||||
"frames": 33,
|
||||
"empty": 32,
|
||||
"nonempty": 1,
|
||||
"worst_pixels": 76,
|
||||
}
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
|
||||
def test_the_clip_round_trips_its_depth_within_the_stated_error(tmp_path):
|
||||
if not CLIP.is_file():
|
||||
pytest.skip(f"the clip is not present: {CLIP}")
|
||||
_require_inputs()
|
||||
recording = _clip_recording(tmp_path)
|
||||
root = tmp_path / "cache"
|
||||
outcome = AnalysisService(
|
||||
recording, chain=_finger_chain(), cache=ResultCache(root)
|
||||
).run()
|
||||
assert outcome.complete, outcome.reason
|
||||
|
||||
produced = _chain_depth(recording)
|
||||
reader = AnalysisReader(_result_directory(root))
|
||||
try:
|
||||
stats = _compare(reader.frames(), produced)
|
||||
assert stats == {
|
||||
"frames": CLIP_FRAMES,
|
||||
"empty": 241,
|
||||
"nonempty": 23,
|
||||
"worst_pixels": WORST_NONEMPTY_PIXELS,
|
||||
}
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------- the size
|
||||
def test_the_depth_cost_is_the_measured_encoding(box_depth):
|
||||
"""14 B per empty frame, the plane's own zlib'd bytes for a frame that has one.
|
||||
|
||||
The record without depth is 4,034 B at the finger grid; the depth channel adds the
|
||||
eight bytes of per-frame statistics the header now carries (``depth_min``,
|
||||
``depth_max``) and the six-byte payload header, and then whatever the plane costs.
|
||||
Ten minutes at 30 fps is 18,000 frames, so the empty case is 72,864,000 B = 69.5 MiB
|
||||
against 69.3 MiB before: the number is small because the *common* case is a map that
|
||||
is all zeros and a payload that is a header.
|
||||
"""
|
||||
assert DEPTH_HEADER.size == 6
|
||||
assert FIXED_RECORD_BYTES == 4042
|
||||
assert EMPTY_RECORD_BYTES == 4048
|
||||
|
||||
reader = AnalysisReader(_result_directory(box_depth))
|
||||
try:
|
||||
frames = list(reader.frames())
|
||||
emptied = [item for item in frames if item.depth_nonzero == 0]
|
||||
carrying = [item for item in frames if item.depth_nonzero]
|
||||
assert len(emptied) + len(carrying) == len(frames)
|
||||
assert {item.depth_stored_bytes for item in emptied} == {EMPTY_PAYLOAD_BYTES}
|
||||
for item in carrying:
|
||||
assert item.depth_stored_bytes == DEPTH_HEADER.size + len(
|
||||
_payload_after_header(item.depth, item.depth_min, item.depth_max)
|
||||
)
|
||||
assert reader.frame_record_bytes == sum(
|
||||
FIXED_RECORD_BYTES + item.depth_stored_bytes for item in frames
|
||||
)
|
||||
assert reader.bytes_per_frame <= 4096.0, "the stated per-frame bound"
|
||||
# Worst case, every frame the worst non-empty frame observed on the clip.
|
||||
worst = FIXED_RECORD_BYTES + DEPTH_HEADER.size + WORST_NONEMPTY_PAYLOAD_BYTES
|
||||
assert worst == 4534
|
||||
assert worst * 18_000 == 81_612_000, "81.6 MB per ten minutes, upper bound"
|
||||
# The rate the footage actually produced, over the 264-frame clip: 241 empty
|
||||
# frames and 23 at the worst non-empty size.
|
||||
typical = (241 * EMPTY_RECORD_BYTES + 23 * worst) / 264
|
||||
assert typical == pytest.approx(4090.34, abs=0.01)
|
||||
assert typical * 18_000 == pytest.approx(73_626_136, abs=1.0)
|
||||
assert typical * 18_000 < 73_700_000, "70.2 MB per ten minutes at the observed rate"
|
||||
assert typical < 4096.0
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
|
||||
def _payload_after_header(depth: np.ndarray, low: float, high: float) -> bytes:
|
||||
"""What ``encode_depth`` would store for a map, payload only. Shared with the codec."""
|
||||
return encode_depth(depth)[0][DEPTH_HEADER.size :]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- the codec itself
|
||||
def test_an_empty_map_costs_a_header_and_states_its_shape():
|
||||
depth = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
payload, nonzero, low, high = encode_depth(depth)
|
||||
assert (nonzero, low, high) == (0, 0.0, 0.0)
|
||||
assert payload == DEPTH_HEADER.pack(CODEC_EMPTY, 8, *DEPTH_GRID)
|
||||
assert len(payload) == EMPTY_PAYLOAD_BYTES
|
||||
|
||||
decoded = decode_depth(payload)
|
||||
assert decoded.is_empty and decoded.nonzero == 0
|
||||
assert (decoded.height, decoded.width) == DEPTH_GRID
|
||||
from orisys_app_shared.analysis.model import depth_values
|
||||
|
||||
restored = depth_values(decoded, low=low, high=high)
|
||||
assert restored.shape == DEPTH_GRID
|
||||
assert not bool(np.any(restored)), "an empty map comes back as zeros, not as NaNs"
|
||||
|
||||
|
||||
def test_a_single_plane_is_quantised_over_its_own_range_and_stays_within_half_a_step():
|
||||
rng = np.random.default_rng(20261004)
|
||||
depth = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
rows = rng.integers(0, DEPTH_GRID[0], 500)
|
||||
cols = rng.integers(0, DEPTH_GRID[1], 500)
|
||||
depth[rows, cols] = rng.random(500).astype(np.float32) * 0.05
|
||||
|
||||
payload, nonzero, low, high = encode_depth(depth)
|
||||
assert nonzero == int(np.count_nonzero(depth))
|
||||
assert (low, high) == (float(depth.min()), float(depth.max()))
|
||||
assert len(payload) < 4096, "a 500-pixel plane is not 84 kB of payload"
|
||||
|
||||
decoded = decode_depth(payload, low=low, high=high)
|
||||
assert decoded.codec in (CODEC_ZLIB, CODEC_RAW)
|
||||
from orisys_app_shared.analysis.model import depth_values
|
||||
|
||||
restored = depth_values(decoded, low=low, high=high)
|
||||
step = (high - low) / 255.0
|
||||
assert float(np.max(np.abs(restored - depth))) <= step / 2.0 + 1e-12
|
||||
# The count survives for every value the quantiser can actually represent: a value
|
||||
# above half a step lands on a code of at least one. A value *below* half a step can
|
||||
# round to zero, and that is the accepted error rather than a lost measurement — it
|
||||
# is smaller than the error the encoding already allows.
|
||||
assert int(np.count_nonzero(restored[depth > step / 2.0])) == int(
|
||||
np.count_nonzero(depth > step / 2.0)
|
||||
)
|
||||
assert not bool(np.any(restored[depth == 0.0]))
|
||||
|
||||
# The display paints this range onto a 256-entry ramp, so half a quantisation step
|
||||
# is half a ramp step: the reconstruction cannot change a pixel's colour.
|
||||
assert step / 2.0 <= (high - low) / 255.0
|
||||
|
||||
|
||||
def test_a_plane_that_does_not_compress_is_stored_raw_rather_than_larger():
|
||||
rng = np.random.default_rng(7)
|
||||
depth = rng.random(DEPTH_GRID).astype(np.float32)
|
||||
payload, nonzero, low, high = encode_depth(depth)
|
||||
decoded = decode_depth(payload, low=low, high=high)
|
||||
assert decoded.codec == CODEC_RAW
|
||||
assert decoded.stored_bytes == depth.size
|
||||
assert len(payload) == DEPTH_HEADER.size + depth.size
|
||||
|
||||
|
||||
def test_depth_statistics_are_per_frame_because_the_range_is():
|
||||
"""Two frames with different ranges each keep all 256 levels."""
|
||||
flat = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
flat[10, 10] = 0.001
|
||||
steep = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
steep[10, 10] = 0.0
|
||||
steep[20, 20] = 1.0
|
||||
for frame in (flat, steep):
|
||||
payload, _, low, high = encode_depth(frame)
|
||||
decoded = decode_depth(payload, low=low, high=high)
|
||||
from orisys_app_shared.analysis.model import depth_values
|
||||
|
||||
restored = depth_values(decoded, low=low, high=high)
|
||||
assert float(np.max(np.abs(restored - frame))) <= (high - low) / 510.0 + 1e-12
|
||||
|
||||
|
||||
def test_a_corrupt_depth_payload_is_refused_rather_than_decoded():
|
||||
good, _, low, high = encode_depth(
|
||||
np.full(DEPTH_GRID, 0.25, dtype=np.float32)
|
||||
)
|
||||
# A plane that decodes to the wrong number of bytes for the shape it declares.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(good[:-1], low=low, high=high)
|
||||
# A shape no sensor has, refused before it becomes an allocation.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(DEPTH_HEADER.pack(CODEC_EMPTY, 8, 0xFFFF, 0xFFFF))
|
||||
# A bit depth this build does not store.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(DEPTH_HEADER.pack(CODEC_EMPTY, 16, *DEPTH_GRID))
|
||||
# And a header that is not even there.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(b"")
|
||||
|
||||
|
||||
def test_a_frame_without_a_depth_channel_is_not_a_frame_with_an_empty_one():
|
||||
"""The two facts have to stay apart: only one of them can be drawn as a measurement."""
|
||||
blank = encode_frame(
|
||||
frame_id=1,
|
||||
t_rel_s=0.0,
|
||||
grid_id=0,
|
||||
dx=np.zeros(2, np.float32),
|
||||
dy=np.zeros(2, np.float32),
|
||||
magnitude=np.zeros(2, np.float32),
|
||||
active=np.zeros(2, bool),
|
||||
depth=np.zeros(DEPTH_GRID, np.float32),
|
||||
)
|
||||
absent = encode_frame(
|
||||
frame_id=2,
|
||||
t_rel_s=0.0,
|
||||
grid_id=0,
|
||||
dx=np.zeros(2, np.float32),
|
||||
dy=np.zeros(2, np.float32),
|
||||
magnitude=np.zeros(2, np.float32),
|
||||
active=np.zeros(2, bool),
|
||||
)
|
||||
assert len(blank) - len(absent) == EMPTY_PAYLOAD_BYTES
|
||||
from orisys_app_shared.analysis.model import decode_frame
|
||||
|
||||
empty_frame = decode_frame(blank)
|
||||
absent_frame = decode_frame(absent)
|
||||
assert empty_frame.depth is not None and empty_frame.depth.is_empty
|
||||
assert empty_frame.depth_map() is not None
|
||||
assert absent_frame.depth is None and absent_frame.depth_map() is None
|
||||
|
||||
|
||||
def test_the_provenance_counts_what_was_empty(box_depth):
|
||||
"""The operator's "there is no depth here" has to be answerable from the result."""
|
||||
reader = AnalysisReader(_result_directory(box_depth))
|
||||
try:
|
||||
counts = reader.counts
|
||||
assert counts["depth_frames"] == 33
|
||||
assert counts["depth_nonzero_frames"] == 1
|
||||
assert counts["depth_nonzero_max"] == 76
|
||||
assert counts["depth_pixels"] == DEPTH_GRID[0] * DEPTH_GRID[1]
|
||||
assert counts["frame_record_bytes"] == reader.frame_record_bytes
|
||||
provenance = json.loads(
|
||||
(_result_directory(box_depth) / "meta.json").read_text(encoding="utf-8")
|
||||
)["provenance"]
|
||||
assert provenance["sources"]["depth"] == "recomputed"
|
||||
finally:
|
||||
reader.close()
|
||||
@@ -38,7 +38,11 @@ from orisys_app_shared.analysis.cache import (
|
||||
evict,
|
||||
scan_results,
|
||||
)
|
||||
from orisys_app_shared.analysis.model import frame_payload_bytes, result_key
|
||||
from orisys_app_shared.analysis.model import (
|
||||
FRAME_HEADER,
|
||||
frame_payload_bytes,
|
||||
result_key,
|
||||
)
|
||||
from orisys_app_shared.analysis.reader import (
|
||||
AnalysisIncompleteError,
|
||||
AnalysisReader,
|
||||
@@ -477,24 +481,43 @@ def test_the_key_follows_the_inputs_that_change_the_result_and_no_others(finger_
|
||||
|
||||
# ------------------------------------------------------------------ size
|
||||
def test_the_stored_size_is_the_measured_encoding(box_pass, finger_pass):
|
||||
"""State the encoding, then hold it: 4,034 B per frame at the finger grid.
|
||||
"""State the encoding, then hold it: 4,048 B per frame at the finger grid.
|
||||
|
||||
Implicit candidate grid (a keyframe) + float32 ``dx``/``dy``/``magnitude`` +
|
||||
bit-packed ``active`` + a 24-byte header + the 8-byte validity digest:
|
||||
``24 + 330*12 + 42 + 8 = 4,034`` B. The alternative measured encodings are float16
|
||||
``dxy``+magnitude (2,022 B) and dropping magnitude (4,002 B with the grid, no
|
||||
header); float32 is chosen because the round trip has to be *exact*, and the
|
||||
magnitude is what serves a changed threshold.
|
||||
bit-packed ``active`` + a 32-byte header + the 8-byte validity digest + the 6-byte
|
||||
depth header: ``32 + 330*12 + 42 + 8 + 6 = 4,048`` B, of which the depth part is
|
||||
``14`` B over the 4,034 B this record cost before the depth channel was stored. A
|
||||
frame whose depth map is *not* all zeros adds its own zlib'd uint8 plane on top.
|
||||
The alternative measured encodings are float16 ``dxy``+magnitude (2,022 B) and
|
||||
dropping magnitude (4,002 B with the grid, no header); float32 is chosen because
|
||||
the round trip has to be *exact*, and the magnitude is what serves a changed
|
||||
threshold.
|
||||
"""
|
||||
assert frame_payload_bytes(330, with_validity=True) == 4034
|
||||
assert frame_payload_bytes(330, with_validity=False) == 4026
|
||||
assert frame_payload_bytes(400, with_validity=False) == 4874
|
||||
fixed = FRAME_HEADER.size + 330 * 12 + 42 + 8
|
||||
assert fixed == 4042
|
||||
assert frame_payload_bytes(330, with_validity=True) == 4042
|
||||
assert frame_payload_bytes(330, with_validity=True, with_depth=True) == 4048
|
||||
assert frame_payload_bytes(330, with_validity=False, with_depth=True) == 4040
|
||||
assert frame_payload_bytes(400, with_validity=False) == 4882
|
||||
assert frame_payload_bytes(400, with_validity=False, with_depth=True) == 4888
|
||||
|
||||
_, finger = finger_pass
|
||||
reader = AnalysisReader(finger.directory)
|
||||
try:
|
||||
assert reader.bytes_per_frame == 4034.0
|
||||
assert reader.frame_record_bytes == 33 * 4034
|
||||
# Every frame states the depth channel; the ones whose map is all zeros pay the
|
||||
# 6-byte header and nothing else.
|
||||
frames = list(reader.frames())
|
||||
emptied = [item for item in frames if item.has_depth and not item.depth_nonzero]
|
||||
carrying = [item for item in frames if item.depth_nonzero]
|
||||
assert len(emptied) + len(carrying) == len(frames)
|
||||
assert reader.frame_record_bytes == sum(
|
||||
fixed + item.depth_stored_bytes for item in frames
|
||||
)
|
||||
assert reader.frame_record_bytes == (
|
||||
len(emptied) * 4048
|
||||
+ len(carrying) * fixed
|
||||
+ sum(item.depth_stored_bytes for item in carrying)
|
||||
)
|
||||
assert reader.bytes_per_frame <= 4096, "the stated per-frame bound"
|
||||
total_per_frame = _directory_bytes(finger.directory) / len(reader)
|
||||
assert total_per_frame <= 4600, "keyframes and metadata included"
|
||||
@@ -505,8 +528,12 @@ def test_the_stored_size_is_the_measured_encoding(box_pass, finger_pass):
|
||||
box_reader = AnalysisReader(box.directory)
|
||||
try:
|
||||
# The box chain has no validity mask, so there is no digest and every frame
|
||||
# carries its own grid keyframe: 4,874 B of record per frame at 400 candidates.
|
||||
assert box_reader.frame_record_bytes == 26 * 4874
|
||||
# carries its own grid keyframe: 4,882 B of record per frame at 400 candidates,
|
||||
# plus whatever depth it carries.
|
||||
assert box_reader.frame_record_bytes == sum(
|
||||
32 + 400 * 12 + 50 + item.depth_stored_bytes
|
||||
for item in box_reader.frames()
|
||||
)
|
||||
assert box_reader.keyframe_counts == {"grid": 26}
|
||||
assert box_reader.mask() is None
|
||||
finally:
|
||||
@@ -608,7 +635,12 @@ def test_a_result_without_metadata_is_never_served(tmp_path):
|
||||
AnalysisReader(directory)
|
||||
|
||||
(directory / META_NAME).write_text(
|
||||
json.dumps({"format": analysis_model.ANALYSIS_FORMAT, "format_version": 1}),
|
||||
json.dumps(
|
||||
{
|
||||
"format": analysis_model.ANALYSIS_FORMAT,
|
||||
"format_version": analysis_model.ANALYSIS_FORMAT_VERSION,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
with pytest.raises(AnalysisIncompleteError):
|
||||
|
||||
Reference in New Issue
Block a user